Status of PlayFab services and history of incidents

The incident began at 1000 PST in correlation with atypically high server allocation requests.

These high allocation rates uncovered a recently introduced bug that slowed down virtual machine deletion interactions between PlayFab and Azure. Because VMs were not being deleted, capacity safeguards throttled PlayFab's multiplayer server controller from creating new virtual machines. The issue could be mitigated by customers increasing the number of standing-by servers, but standing-by servers were rebuilt slowly throughout the afternoon in impacted regions.

The issue initially occurred in Brazil South but also impacted build time in US West and US South Central. A hotfix was introduced at 1600 PST and appears effective.

Additional analysis and remediation steps will be published next week.