System Status

Status of PlayFab services and history of incidents

Operational

Partial Outage

Major Outage

PlayStream and Telemetry Event Delivery Delay

Incident Report for PlayFab

Postmortem

On August 1st, 2024, between 13:06 PDT and 14:25 PDT, some customers experienced delays in telemetry data processing. The incident was caused by a networking configuration change to a Key Vault, which resulted in parts of the service being unable to access the required resource. We resolved the issue by reverting the networking changes.

Impact

During the incident, all telemetry data was delayed by approximately 1 hour. This affected Data Explorer and data connections. However, reports were not impacted.

Action Items

To prevent similar incidents from happening again, we have taken the following actions:
• Created additional monitoring to reduce the time to root cause this kind of issue.
• Developed additional dashboards to allow us to more quickly identify the affected part of the pipeline.
• Authored a TSG (Technical Support Guide) describing how to use the new dashboards.

Posted Aug 21, 2024 - 13:00 PDT

Resolved

The issue has been resolved. PlayStream and Telemetry event delivery should be operating normally.

Posted Aug 01, 2024 - 14:50 PDT

Monitoring

We have deployed the fix for the identified issue, and event delivery times are returning to normal. Engineers are continuing to monitor to ensure there are no further issues.

Posted Aug 01, 2024 - 14:28 PDT

Identified

We have identified the cause for this issue and are in the process of updating to resolve it.

Telemetry events delivery is currently delayed by around 1 hour.

Posted Aug 01, 2024 - 14:07 PDT

This incident affected: PlayStream (Data Connections) and Analytics (Event History & Search).