System Status

Status of PlayFab services and history of incidents

Operational

Partial Outage

Major Outage

Small Percentage of Telemetry Event Delivery Delay and Event Loss

Incident Report for PlayFab

Postmortem

On August 22, 2024, between 19:04 PDT and 20:17 PDT, some customers experienced intermittent errors and delays when accessing PlayFab's services. The incident was caused by an event processor being persistently delayed. This led to a loss and delay in data processing and customer ingestion.

Impact

Overall, 3% of Entity profile change Playstream (V2) events were impacted, which is less than 0.1% of the overall playstream event volume. Actions, webhooks, and other subsequent playstream processing were delayed by at most one hour. For events created after August 22 until mitigation, customer ingestion (Data explorer, data connections, etc.) was delayed until August 29, or about 7 days. For events created before August 22 and after August 21, data was irrecoverably lost.

Posted Sep 10, 2024 - 10:55 PDT

Resolved

During the period starting 8/20/2024 at 2PM PDT and ending 8/21/2024 at 5PM PDT, telemetry related to a small percentage v2 entity creation and entity update events were not delivered to data connections nor data explorer. PlayStream events and Automation Rules related to these events were successfully processed, however the telemetry associated with these events has been lost.

Additionally, the delivery of a small percentage of telemetry related to v2 entity creation and entity update events that were emitted starting around 8/21/2024 at 5PM PDT and ending around 8/22/2024 at 7:30PM PDT was delayed. As of 8/29/2024 at 11:30 AM PDT, these events have been delivered to data explorer and data connections.

Posted Aug 29, 2024 - 11:11 PDT