Status of PlayFab services and history of incidents

On December 9th, 2024, between 3:00 PM and 11:00 AM PT the next day, some customers experienced issues with corrupted export files when accessing PlayFab's event data export to S3 buckets. The incident was caused by a bug introduced during an update to legacy code. We resolved the issue by deploying a hotfix and reprocessing the corrupted data. 

Impact 

There were 58 titles affected by this incident, specifically those configured for event exports to S3 buckets. The exports contained invalid characters, causing downstream parsing and decompression issues. The affected data was backfilled successfully by December 11th, 2024, at 7:00 PM PT. During the mitigation, exports to S3 were paused to prevent further impact. 

Root Cause Analysis 

The bug in the export process was introduced during an update to legacy code, which led to additional padding bytes being included in the export data. The codebase had not been actively maintained and lacked end-to-end tests, leaving the bug undetected during manual testing. 

Action Items 

To prevent similar incidents from happening again, we have taken the following actions: 

• Enhanced our monitoring and alerting systems to detect anomalies in export data. 

• Refactored the code for downloading and uploading blobs to S3. 

• Added end-to-end tests for exports to blob and S3. 

• Created tools for backfilling corrupted data.