This incident has been resolved.
May 26, 11:43 PDT
The backlog is reducing steadily and we expect to be clear in 10 minutes.
May 26, 11:28 PDT
We expect delivery job backlog to be resolved in ~35 minutes. Player updates and delete will also be slow during this interval.
May 26, 10:53 PDT
A fix has been implemented and we are monitoring the results.
May 26, 10:45 PDT
Deliveries are significantly delayed. We are processing the backlog of delivery tasks now.
May 26, 10:25 PDT
We have identified the connection pooling issue and fixed it. Rollout will take 5 minutes
May 26, 10:23 PDT
We are still having problems restoring connectivity to the databases for customers with apps starting with 20 through 3f and 80 through 8f.
May 26, 10:05 PDT
We are experiencing connection pool related problems with processing delivery jobs. The dashboard and API are operational
May 26, 09:37 PDT
We are continuing to work on a fix for this issue.
May 26, 09:26 PDT
We have a problem with the configuration of the database replica server promoted serving customers with app ids starting with 20 through 3f. We have identified the configuration problem, fixed it, and are restarting the webservers to refresh connection pools.
May 26, 09:25 PDT
We have promoted the database shards and restarted the site.
May 26, 09:13 PDT
We are promoting a replica for some database shards to which requires a rolling restart of all webservers. All apps may see some 500 errors for the dashboard while the restart proceeds.
May 26, 08:41 PDT
We are currently investigating this issue.
May 26, 07:55 PDT