When a Dependency Melts Down: Our June 22–23 Incident Post-Mortem

Time (UTC)

What Happened Upstream (Full Report)

What Happened Internally / Our Experience

June 22, 19:00

The Trigger: AWS capacity shortage in us-east-1 forces ~10,000 runs to pile up in upstream queues.

Our background tasks quietly begin stalling. Queues start growing under the hood.

June 22, 20:09

The Meltdown: Upstream Kubernetes control plane hits 100% CPU, loses database quorum, and us-east-1 goes completely down.

Our production environment stops processing runs. Thousands of tasks platform-wide are backlog-locked.

June 22, 21:58

Upstream engineering is actively fighting a "reconnection storm" loop.

Incident Reported: Elevated failure rates are noted, and an internal incident room is initialized.

June 23, 00:59

Upstream provider recommends users manually switch workloads over to their eu-central-1 region.

Incident Lead Assigned: The incident response room is fully activated with an assigned lead to handle cross-region evaluation.

June 23, 01:01 - 02:10

Upstream provider deploys an emergency dashboard update to allow bulk-moving runs between regions.

Active Mitigation: At 01:10, the team manually cancels our queued US runs to clear our account's blocked execution capacity and routes all new traffic to the European region. By 02:10, messages begin processing again.

June 23, 02:02 - 02:33

Upstream provider is manually scaling up their database backing store.

Severity Upgraded to Critical: Over 30,000+ of our own tasks are now queued up and waiting. The team posts a critical status update to our public status page.

June 23, 04:35 - 05:05

Upstream infrastructure updates slow execution speeds back to a crawl.

The Webhook Bottleneck: Processing rates drop. By 05:05, we identify that our webhook-delivery queue is the primary bottleneck refusing to flush.

June 23, 08:38

The Second Cascade: Redirected global traffic overwhelms the provider’s European region, knocking it completely offline.

Total Standstill: Both upstream regions are now effectively dead. We have over 13,000+ tasks frozen in flight, entirely at the mercy of upstream recovery.

June 23, 10:34

Upstream engineering is struggling to bring their European region back online.

The Circuit Breaker Deployment: Realizing the primary background processing layer is completely stalled, our team shifts from infrastructure mitigation to an architectural bypass. We deploy an emergency hotfix across all core application and API environments to temporarily disable async task management.

June 23, 10:41

Upstream provider is still working on resizing their control plane nodes.

Immediate Relief: The architectural bypass works flawlessly. By routing around the frozen background queues, critical app processes fallback to direct execution paths. Engineering confirms: "Emails are flowing again."

June 23, 17:06

Upstream provider finally stabilizes both regions and restores processing speeds.

Queue Cleared: Engineering confirms our queues across both regions have completely emptied out. System performance returns to normal.

June 23, 19:31

Post-recovery stability holds.

Incident Resolved: Systems are fully nominal; the team enters the post-incident review flow.