System Status

Status of PlayFab services and history of incidents

Operational

Partial Outage

Major Outage

Throttling of FindLobbies and UpdateLobby calls (Single Title)

Incident Report for PlayFab

Postmortem

Incident Overview

Starting on February 12th, 2024, between 6:33 PM and 7:28 PM UTC, some customers experienced intermittent errors and delays when accessing PlayFab's Lobby/FindLobbies API. The incident was caused by a sudden increase in traffic from a popular game that exceeded the capacity of our search cluster. We resolved the issue by scaling out the cluster and adding more replicas to balance the load.

Action Items

To prevent similar incidents from happening again, we have taken the following actions:

· We added a new high-scale search cluster to handle the additional load.

Posted Mar 01, 2024 - 06:43 PST

Resolved

FindLobbies and UpdateLobby throttling has been resolved.

Posted Feb 17, 2024 - 16:01 PST

Update

UpdateLobby load spiked for a 23 minute period starting 11:45 PM PST, resulting in throttling, and then subsided. We continue to monitor as we approach our daily peak load.

Posted Feb 17, 2024 - 12:08 PST

Monitoring

FindLobby throttling mitigated at 11:36 AM PST. We are now monitoring until we officially resolve

Posted Feb 17, 2024 - 11:36 PST

Update

UpdateLobby throttling mitigated at 10:35 AM.

Posted Feb 17, 2024 - 10:35 PST

Identified

Increasing throttling limits.

Posted Feb 17, 2024 - 09:10 PST

Investigating

Increased inbound calls for a single title starting at 2:00 AM PST and continued upward until around 4:25 AM when throttling limits kicked in. We are currently investigating.

Posted Feb 17, 2024 - 04:25 PST

This incident affected: API (Lobby).