Some customers experienced a failure of scheduled tasks configured to run on July 15, 2025 at 00:00 UTC. The incident was caused by a deployment overlapping with the scheduled task run time, leading to an interruption in the task processing service. The issue was resolved when tasks executed successfully during the following scheduled run (July 15, 2025 at 00:05 UTC or later).

Impact

Customers’ scheduled tasks did not execute at the expected time on July 15, 2025. This resulted in inconvenience and potential disruption for customers and their end users who relied on timely execution of these tasks. All affected tasks ran as expected during the next scheduled occurrence.

Root Cause Analysis

The root cause was a deployment that coincided with the scheduled task run, which caused the service to shut down and cancel the ongoing task processing. Additionally, code defects prevented the new service instance from queuing missed tasks when it started, and existing monitoring did not detect the issue because only a single execution was missed.

Action Items

To prevent similar incidents from happening again, we have implemented a mechanism to queue missed tasks when a new service instance starts, ensuring tasks are not skipped during deployments or restarts.