API error rates and latency have returned to normal.
Our preliminary diagnosis of the cause of the incident is that a database cluster which houses player virtual currency balances went offline at approximately 20:25 UTC. A significant portion of API requests depend on this database cluster and thus began failing after it went offline. We manually restarted the database at approximately 20:56 UTC, and API error rates and response times began recovering shortly thereafter.
We operate this database in a multi-instance, mirrored configuration, but it did not fail-over automatically as it was expected to and as it has in the past. We will conduct a post-mortem of this incident, determine why the database did not automatically fail-over, and establish a plan to improve availability in this situation.
Posted 6 months ago. May 31, 2017 - 15:20 PDT
We have restarted the affected database, and it is currently back online and recovering. We expect API error rates and latency to return to normal soon.
Posted 6 months ago. May 31, 2017 - 14:12 PDT
VC database issues have been resolved. We are monitoring recovery of various services.
Posted 6 months ago. May 31, 2017 - 14:11 PDT
We are investigating an issue with the database that handles virtual currencies. This issue is impacting nearly all features for games that use virtual currencies.
Posted 6 months ago. May 31, 2017 - 13:46 PDT
We're currently experiencing a higher than normal level of API errors and are investigating the issue.