Between 2025-03-18 19:00 UTC and 2025-03-19 17:00 UTC, some customers saw InternalServerErrors being returned from the Experimentation/GetExperiments API, or a blank page when loading the Experiments page in Game Manager. The incident was caused by the deployment of a bad configuration used by the experimentation service. We resolved the issue by fixing the configuration and redeploying the impacted service.

Impact

Any title that attempted to retrieve their experiment information via Game Manager or API saw InternalServerErrors or a blank Experiments page for the duration of the incident. There was no impact to any other Experimentation APIs or to the operation of the experiments themselves.

Root Cause Analysis

The root cause of the incident was the rollout of an incorrect configuration used by the Experimentation service.

Action Items

To prevent similar incidents from happening again, we have taken the following actions:

·       Enhanced monitoring and alerting systems to detect and report errors in service-to-service communication.