Status of PlayFab services and history of incidents

*Root Cause: *Memory Leak

PlayStreamLiveSite had an existing memory leak which causes the VM to run out of memory after 6 days. Prior to this issue, there was period recycling of the IIS App Pool of 6 hours. When it was most recently deployed, a change which removed the recycling was also deployed which caused the VM to run out of memory and crash after 6 days.

Detection Details:

A pager alert was issued to the on call engineer. Other team members were able to validate the issue from the live site.

Mitigation Steps:

Memory was reset after restarting the service. IIS App Pool Recycling was enabled daily.

Fix:

Our engineers investigated the memory leak and added a fix to the SignalR subscriptions which we believe may be the cause of the leak.