All Systems Operational
Uptime over the past 90 days. View historical uptime.
Dashboard and API ? Operational
90 days ago
99.87 % uptime
Today
Offline Job Processing ? Operational
90 days ago
99.96 % uptime
Today
Delivery ? Operational
90 days ago
99.94 % uptime
Today
Analytics & Update Processing ? Operational
90 days ago
99.97 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
had a major outage
had a partial outage

No incidents or maintenance related to this downtime.

System Metrics Month Week Day
Average Response Time ?
Fetching
Transactional Delivery Latency ?
Fetching
Past Incidents
May 28, 2020

No incidents reported today.

May 27, 2020

No incidents reported.

May 26, 2020
Increased HTTP 5xx error rate
Resolved - This incident has been resolved.
May 26, 11:43 PDT
Update - The backlog is reducing steadily and we expect to be clear in 10 minutes.
May 26, 11:28 PDT
Update - We expect delivery job backlog to be resolved in ~35 minutes. Player updates and delete will also be slow during this interval.
May 26, 10:53 PDT
Monitoring - A fix has been implemented and we are monitoring the results.
May 26, 10:45 PDT
Update - Deliveries are significantly delayed. We are processing the backlog of delivery tasks now.
May 26, 10:25 PDT
Update - We have identified the connection pooling issue and fixed it. Rollout will take 5 minutes
May 26, 10:23 PDT
Update - We are still having problems restoring connectivity to the databases for customers with apps starting with 20 through 3f and 80 through 8f.
May 26, 10:05 PDT
Update - We are experiencing connection pool related problems with processing delivery jobs. The dashboard and API are operational
May 26, 09:37 PDT
Update - We are continuing to work on a fix for this issue.
May 26, 09:26 PDT
Identified - We have a problem with the configuration of the database replica server promoted serving customers with app ids starting with 20 through 3f. We have identified the configuration problem, fixed it, and are restarting the webservers to refresh connection pools.
May 26, 09:25 PDT
Monitoring - We have promoted the database shards and restarted the site.
May 26, 09:13 PDT
Identified - We are promoting a replica for some database shards to which requires a rolling restart of all webservers. All apps may see some 500 errors for the dashboard while the restart proceeds.
May 26, 08:41 PDT
Investigating - We are currently investigating this issue.
May 26, 07:55 PDT
May 25, 2020

No incidents reported.

May 24, 2020

No incidents reported.

May 23, 2020

No incidents reported.

May 22, 2020

No incidents reported.

May 21, 2020

No incidents reported.

May 20, 2020
Connectivity Errors
Resolved - This incident has been resolved.
May 20, 16:20 PDT
Monitoring - A fix has been implemented and we are monitoring the results.
May 20, 16:18 PDT
Investigating - We are currently investigating this issue.
May 20, 16:16 PDT
Database upgrade
Completed - The scheduled maintenance has been completed.
May 20, 16:10 PDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 20, 16:05 PDT
Scheduled - We will be upgrading a database server.
May 15, 10:39 PDT
May 19, 2020

No incidents reported.

May 18, 2020

No incidents reported.

May 17, 2020

No incidents reported.

May 16, 2020
Site Outage
Resolved - This incident has been resolved.
May 16, 23:16 PDT
Update - We are continuing to monitor for any further issues.
May 16, 23:14 PDT
Monitoring - Connectivity has been fully restored and we are working through our delivery backlog. We expect delivery latency to return to normal in the next 10 minutes.
May 16, 23:00 PDT
Update - Correction: Connectivity has been mostly restored. API throughput is still partially degraded.
May 16, 22:52 PDT
Update - Connectivity has been restored; we are currently processing a large delivery backlog.
May 16, 22:50 PDT
Identified - The issue appears to be a major networking outage at our DC. We are coordinating with them on a solution.
May 16, 22:32 PDT
Update - The issue is more widespread than initially understood. We are continuing to investigate.
May 16, 22:28 PDT
Investigating - We are currently investigating this issue.
May 16, 22:26 PDT
May 15, 2020

No incidents reported.

May 14, 2020

No incidents reported.

← Incident History Powered by Statuspage