*Root Cause: *Memory Leak
PlayStreamLiveSite had an existing memory leak which causes the VM to run out of memory after 6 days. Prior to this issue, there was period recycling of the IIS App Pool of 6 hours. When it was most recently deployed, a change which removed the recycling was also deployed which caused the VM to run out of memory and crash after 6 days.
Detection Details:
A pager alert was issued to the on call engineer. Other team members were able to validate the issue from the live site.
Mitigation Steps:
Memory was reset after restarting the service. IIS App Pool Recycling was enabled daily.
Fix:
Our engineers investigated the memory leak and added a fix to the SignalR subscriptions which we believe may be the cause of the leak.