A silent queue server left some GitHub Actions events too old to run

Text
iHOLYSHITVISUAL / TECHUnexpected technology eventsiHOLYSHIT / ILLUSTRATION
iHolyShit illustration

GitHub’s November 9 incident shows why clearing a backlog and completing the work behind it are different recovery milestones.

An automation can recover its queue without recovering every job that should have run. GitHub's report on a November 9 incident describes a slowed server that stopped providing the metrics used by its alerts. Some deliveries were delayed as much as 3.5 hours. Because Actions skips events older than 30 minutes, some workflows never started. The incident is resolved, not an assertion of a continuing outage.

GitHub Actions is the platform's system for automating builds, tests and deployments. A repository change can start work that tests a proposed modification or ships a release. Webhooks provide another connection, notifying external services when something happens. A single visible action can therefore launch a chain of less visible operations. A queue is the distribution depot between those operations, rather than proof that any downstream work has finished.

A useful everyday analogy is a courier clearing its distribution depot. That does not establish that every customer received a parcel on time. If a package contains a ticket for an event that has already ended, eventual delivery cannot recover its original purpose. Delivery, execution and completion are three different outcomes. Treating one as evidence of the others leaves room for work to disappear between dashboards.

For a development team, our assessment is that recovery checks should ask whether an expected test or deployment has a corresponding run, not simply whether the platform is reachable again. Emptying a backlog is an infrastructure milestone. Establishing that the business operations behind those messages occurred is a separate question. Faster normal operation cannot substitute for that reconciliation after a disruption.

GitHub's documentation describes another relevant boundary: failed webhook deliveries are not automatically resent. They can be redelivered manually or handled by code that checks delivery records. That is distinct from this incident's delayed events aging out of Actions. It does, however, demonstrate why “the platform will catch up” is not a universal recovery strategy. Resending a webhook should not be presented as a general repair for every missing Actions run.

There is no new programming product in this story. Its value is a concrete operational lesson: when a component stops sending its own health information, monitoring can lose sight of the very failure it is supposed to catch. For people relying on automated releases, evidence that the intended work happened matters more than a reachable server alone.

Incident date: November 9, 2026. Prepared November 14, 2026, 02:16 Beijing / November 13, 2026, 13:16 EST.

Share this article

What do you think of this story?

Anonymous feedback, up to 5 reactions per article.

Anonymous comments

Share your thoughts without signing in.

Discuss the technology and the story. Hate, politics and war, threats, abuse, personal accusations, sexual content, ads, links and personal information are not allowed.

0/500

No comments yet. Be the first to share.

Sources

A silent queue server left some GitHub Actions events too old to run|iHolyShit