System Status
Status of PlayFab services and history of incidents
Operational
Partial Outage
Major Outage
PlayFab MultiPlayer API Errors
Incident
Report for PlayFab
Postmortem
Almost all HTTP requests sent to the PlayFab multiplayer server orchestrator (aka Control Plane) were dropped during the 90 minute outage. This occurred during an upgrade of the control plane, whereby normally the system kills existing connections to managed multiplayer servers in the current update domain and rebuild them. This is what occurred:
- VM heartbeat fails due to loss of connection
- VM heartbeat boosted to 30 times more frequent on failure
- Too many new connections caused CPU to spike to 100%
- The entire system fell into a negative feedback loop
Follow up:
- The most expensive requests (GetQosServers and VM heartbeat) have fixed to be dramatically less costly.
- We are improving our roll-out and testing processes to catch these issues before deployments.
Resolved
The incident has been resolved and postmortem will be posted next week.
Update
We're still investigating errors reported in the Multiplayer API for a subset of users. An update update will be posted in 20 minutes
Monitoring
The issue has been mitigated and the service is recovering. We are actively monitoring the service. An update will be posted in the next 20 minutes
Identified
The problem source has been identified and a service update is being deployed to fix a previously bad deployment. An update will be posted in the next 20 minutes.
Update
We are continuing to investigate this issue.
Investigating
We are experiencing API errors with PlayFab Multiplayer Servers. An update will be posted in the next 15 minutes
This incident affected: Multiplayer Game Servers 2.0 (Request Multiplayer Server API).