On May 21st, 2025, between 3:58 PM and 4:25 PM PDT, some customers experienced a partial outage impacting large API groups, including Catalog and Inventory services on PlayFab. The incident was caused by a configuration change during infrastructure migration, which resulted in a connectivity issue for backend databases. The issue was resolved by updating access settings to restore proper connectivity to the affected services.

Impact

During the incident, customers encountered degraded service with failed requests across multiple services reliant on Catalog—including Inventory and Redemption. Service availability dropped significantly below expected levels for approximately 27 minutes.

Root Cause Analysis

The outage was triggered when a migration enabled new access controls on backend databases. This inadvertently blocked necessary connections, leading to failures for dependent services. The infrastructure update caused public access to be disabled, preventing services from communicating as expected.

Action Items

To prevent similar incidents from happening again, we have taken the following actions:

·       We updated our migration procedures to ensure all required access settings are reviewed and applied before infrastructure changes.

·       We enhanced our monitoring to detect connectivity issues immediately after infrastructure updates.