No Data in Dashboard & PlayStream Debugger
Incident Report for PlayFab
Postmortem

*Root Cause: *Memory Leak

PlayStreamLiveSite had an existing memory leak which causes the VM to run out of memory after 6 days. Prior to this issue, there was period recycling of the IIS App Pool of 6 hours. When it was most recently deployed, a change which removed the recycling was also deployed which caused the VM to run out of memory and crash after 6 days.

Detection Details:

A pager alert was issued to the on call engineer. Other team members were able to validate the issue from the live site.

Mitigation Steps:

Memory was reset after restarting the service. IIS App Pool Recycling was enabled daily.

Fix:

Our engineers investigated the memory leak and added a fix to the SignalR subscriptions which we believe may be the cause of the leak.

Posted about 2 months ago. Oct 24, 2018 - 16:26 PDT

Resolved
We're seeing full recovery across all titles.
Posted 4 months ago. Aug 21, 2018 - 08:53 PDT
Update
We are continuing to investigate this issue.
Posted 4 months ago. Aug 21, 2018 - 08:37 PDT
Investigating
We are investigating what appears to be a global outage of the Dashboard & PlayStream Debugger feed in Game Manager. PlayStream appears to be unaffected as Event History & Search are still functioning.
Posted 4 months ago. Aug 21, 2018 - 08:37 PDT
This incident affected: Game Manager (PlayStream Debugger).