What Happened:
A title launch increased Matchmaking's total traffic by ten times. The unexpected increase in load caused not only failures to clients but also had ripple effect on internal components.
Mitigation:
We continued to scale out our resources over the course of the week until we were able to safely handle the volume. We also spent time working with the customer to help significantly reduce their calling patterns to Matchmaking.
Next Steps:
To ensure we are able to scale to unexpected traffic like this in future, we have added items to our backlog. The overall requests are back to earlier load now. We will be scaling back down our services to reduce our COGs - this may result in some periodic issues, but they will be resolved shortly.