Status of PlayFab services and history of incidents

On March 18th, 2024, between 4:45 PM and 10:40 PM UTC, some customers experienced errors and delays when using Segmentation, Reports and Trends, and Title Overview features on PlayFab's API. The incident was caused by a bug in the GDPR service that resulted in a high volume of delete operations on the Azure Data Explorer (ADX) cluster which hosts the data for these features. The delete operations increased the load on the cluster and triggered a throttling mechanism that aborted some queries. We resolved the issue by moving the queries to a secondary ADX cluster and fixing the bug in the GDPR service. 

Action Items 

• We fixed the bug in the GDPR service that caused the missed deletes and the high volume of re-run deletes. 
• We increased the batch size and reduced the frequency of the delete operations submitted to the cluster to reduce the load on the cluster. 
• We have stabilized the secondary cluster to continue to exclusively process queries from the Segmentation, Reports and Trends, and Title Overview features. The GDPR delete operations are now run only in the primary cluster without impacting the feature queries.