On April 17th, 2024, between 1:49 PM and 10:15 PM UTC, some customers experienced delays and errors when using PlayFab's Gaming Insights service, which provides reports, trends, and segmentation features for game developers. The incident was caused by a storage outage in the West US 2 region that throttled two of our Kusto clusters. We resolved the issue by scaling up the clusters while the storage outage was ongoing. 

Action Items 

To prevent similar incidents from happening again, we have taken the following actions: 

• We turned on the monitoring and alerting for the Kusto clusters, which was previously disabled. 
• We changed the autoscaling rules for both clusters to be more reactive to changes in CPU and cache utilization. 
• We improved our communication with the Kusto team and subscribed to their outage notifications.