Status of PlayFab services and history of incidents

On April 11th, 2024, between 12:20 PM and 1:28 PM PDT, some customers experienced intermittent errors and delays when accessing PlayFab's API. The incident was caused by a network configuration change that triggered a bug in one of our load balancers, resulting in traffic being routed incorrectly. We resolved the issue by rolling back the configuration change and applying a patch to the load balancer. 

Action Items 

• We improved our testing and validation procedures for network configuration changes to catch such bugs before they reach production. 

• We enhanced our monitoring and alerting systems to detect and report any anomalies in the load balancer's behavior and performance. 

• We updated our documentation and training materials to ensure that our engineers are aware of the potential impact of making network configuration changes.