API error rates and latency have returned to normal.
Our preliminary diagnosis of the cause of the incident is that a database cluster which houses player virtual currency balances went offline at approximately 20:25 UTC. A significant portion of API requests depend on this database cluster and thus began failing after it went offline. We manually restarted the database at approximately 20:56 UTC, and API error rates and response times began recovering shortly thereafter.
We operate this database in a multi-instance, mirrored configuration, but it did not fail-over automatically as it was expected to and as it has in the past. We will conduct a post-mortem of this incident, determine why the database did not automatically fail-over, and establish a plan to improve availability in this situation.
Posted 22 days ago. May 31, 2017 - 15:20 PDT
We have restarted the affected database, and it is currently back online and recovering. We expect API error rates and latency to return to normal soon.
Posted 22 days ago. May 31, 2017 - 14:12 PDT
VC database issues have been resolved. We are monitoring recovery of various services.
Posted 22 days ago. May 31, 2017 - 14:11 PDT
We are investigating an issue with the database that handles virtual currencies. This issue is impacting nearly all features for games that use virtual currencies.
Posted 22 days ago. May 31, 2017 - 13:46 PDT
We're currently experiencing a higher than normal level of API errors and are investigating the issue.