Microsoft Azure PlayFab

System Status

Status of PlayFab services and history of incidents

Operational
Partial Outage
Major Outage
Multiplayer servers are not re-building in Brazil South
Incident Report for PlayFab
Postmortem

The incident began at 1000 PST in correlation with atypically high server allocation requests.

These high allocation rates uncovered a recently introduced bug that slowed down virtual machine deletion interactions between PlayFab and Azure. Because VMs were not being deleted, capacity safeguards throttled PlayFab's multiplayer server controller from creating new virtual machines. The issue could be mitigated by customers increasing the number of standing-by servers, but standing-by servers were rebuilt slowly throughout the afternoon in impacted regions.

The issue initially occurred in Brazil South but also impacted build time in US West and US South Central. A hotfix was introduced at 1600 PST and appears effective.

Additional analysis and remediation steps will be published next week.

Posted 3 months ago. Jul 02, 2019 - 18:58 PDT

Resolved
This incident has been resolved, and multiplayer servers re-building at normal speeds.
Posted 3 months ago. Jul 02, 2019 - 18:54 PDT
Identified
The building of new multiplayer servers in Brazil South is significantly slower than expected. We have identified the issue and are investigating a mitigation.
Posted 4 months ago. Jul 02, 2019 - 12:15 PDT
This incident affected: Multiplayer Game Servers 2.0 (Request Multiplayer Server API).