Backlog processing is now complete and all services have returned to normal.
Posted Sep 11, 2022 - 08:35 UTC
Update
The emergency maintenance on these systems continue. Based upon about 14 hours of processing, we now estimate that this should be completed by no later than 2022-09-11 1600 UTC. We will continue to update this page if there are any major changes in this estimate.
Posted Sep 09, 2022 - 21:21 UTC
Identified
We are currently performing emergency maintenance of some of our offline job processing around storing user notification event data. For the vast majority of our customers, the only impact is that some user event data (e.g., clicks, opens, and sends) on our Audience Activity dashboard page will be delayed until we can finish the emergency maintenance and clear up the backlog. However, this data will be available retroactively once the catchup is complete.
For Journeys customers, this will result in a similar delay, for click events, but could have some more noticeable effects. If you have a Journey that fires based upon the timing of recent click data, (e.g., "Send this notification six hours after they last clicked a message"), and that time (e.g., six hours) comes up during this maintenance, that Journey will travel down the "did not click" node of that Journey.
Similarly, for Journeys customers, if a Journey has a very long delay between click and message (e.g., one or more weeks) and that period finishes during the maintenance, the Journey will travel down the "did not click" node as well.
Other than these cases, the primary impact we expect for the vast majority of customers is that click processing will be delayed for approximately two days and then become retroactively correct.
We don't yet have a firm estimate as to when this catchup process will take, but we are optimistic that it will be completed no later than 2022-09-11 at 0500 UTC. We will update this notification when we have a better estimate of the progress.