System Status
Status of PlayFab services and history of incidents
Operational
Partial Outage
Major Outage
Increased API Error Response Rate
Incident
Report for PlayFab
Resolved
API error rates and latency have returned to normal.
Our preliminary diagnosis of the cause of the incident is that a database cluster which houses player virtual currency balances went offline at approximately 20:25 UTC. A significant portion of API requests depend on this database cluster and thus began failing after it went offline. We manually restarted the database at approximately 20:56 UTC, and API error rates and response times began recovering shortly thereafter.
We operate this database in a multi-instance, mirrored configuration, but it did not fail-over automatically as it was expected to and as it has in the past. We will conduct a post-mortem of this incident, determine why the database did not automatically fail-over, and establish a plan to improve availability in this situation.
Our preliminary diagnosis of the cause of the incident is that a database cluster which houses player virtual currency balances went offline at approximately 20:25 UTC. A significant portion of API requests depend on this database cluster and thus began failing after it went offline. We manually restarted the database at approximately 20:56 UTC, and API error rates and response times began recovering shortly thereafter.
We operate this database in a multi-instance, mirrored configuration, but it did not fail-over automatically as it was expected to and as it has in the past. We will conduct a post-mortem of this incident, determine why the database did not automatically fail-over, and establish a plan to improve availability in this situation.
Update
We have restarted the affected database, and it is currently back online and recovering. We expect API error rates and latency to return to normal soon.
Monitoring
VC database issues have been resolved. We are monitoring recovery of various services.
Update
We are investigating an issue with the database that handles virtual currencies. This issue is impacting nearly all features for games that use virtual currencies.
Investigating
We're currently experiencing a higher than normal level of API errors and are investigating the issue.