When GitHub Actions Dies on Billing: a runbook for the silent failure mode
The symptom is almost comically uninformative: the job shows up red, lasts about 2 seconds, allocates no runner, executes not a single step, and leaves no logs at all. No message about quota, no link to billing, nothing. Just an X.
When a personal GitHub account runs out of hosted minutes, that’s what you get. And what happens next depends entirely on a question almost nobody asks in advance: what exactly stopped?
There are two possible answers to that question, and a single blown-quota episode produced them side by side, across two repositories. Both are instructive precisely because they’re opposites — and the difference between them is the difference between a scare and a silent outage.
Case A: the deploy stopped, so someone noticed
On Tamperlens, the deploy runs through GitHub Actions — push to main, CI green, runner on the VPS redeploys. When the quota ran out, the deploy stopped with it. It was noticed the same day, and the critical workflows moved to a self-hosted runner two days later.
This is the happy case. Coupling “checking” to “publishing” is normally a design defect, but here it worked as a detector: you cannot ignore a dead CI when it’s the same path the product takes to reach production.
Case B: the deploy did not stop, and nobody noticed
On haruo.dev it went the other way, and this is the more dangerous of the two.
The site publishes through the Cloudflare Pages git integration, not Actions — every push to main triggers a build and deploy straight on Cloudflare, outside GitHub. When the quota ran out, the site stayed up, normally, updating on every push, with lint, tests and link check switched off — and absolutely nothing indicating that.
Every commit of the following week went to production without passing a single gate. Not because someone decided to skip them, but because the gate died along an axis nobody was watching.
The rule that stands: every repository has a publishing path and a verification path. If both are the same system, one falls with the other and you find out. If they’re different systems — Pages publishes, Actions verifies — you have to monitor them separately, because one of them can die with no symptom. And the one that dies without a symptom is always the verification path, because it has no users.
The ugly case: the status page frozen for 4 days
The third effect is the most dangerous of the three, and it’s worth telling with numbers.
status.tamperlens.com was frozen from 2026-08-19 17:13 UTC. uptime.yml runs on cron, and from that moment every scheduled run died in the same ~3 seconds, no runner, no logs: 165 consecutive failures.
The frozen page is the visible half. The invisible half is that the probe was also the alarm. Since the 19th, nothing had been checking production at all. If the API had gone down, nothing would have said so.
Four days with monitoring switched off, and the only sign of it was a column of Xs in an Actions history nobody opens while everything appears fine.
And here’s the structural detail: when the other workflows moved to the self-hosted runner, uptime.yml could not follow. That runner is the box being watched. The page itself states, in its own copy, that it is probed from somewhere that is not the server. Moving the probe onto the box would not have fixed the page — it would have made the page lie.
This is the infrastructure version of the last post’s problem: a gate that cannot fail. A probe running on the machine it monitors cannot report that the machine went down. It reports silence, and silence looks like “no data”, which looks like “everything is fine”.
Exit 1: a self-hosted runner on the VPS
The obvious exit, and the one that solves most cases: register a self-hosted runner on the machine you already pay for.
It works, and it’s what the CI and deploy workflows use today. Three things that cost an attempt each, and are worth writing down somewhere:
- Tooling in a container, never installed on the host. The runner should not accumulate state. But if the workspace is mounted read-write and the container runs as root, you leave root-owned files in the runner’s workspace — which then break the next
npm ci. Running with--user $(id -u):$(id -g)fixes it. And$(id)does not expand inside a YAMLenv:block; that costs one run to learn. - npm cache inside the workspace, not in a named docker volume. A named volume is created owned by root; the container runs as a user; npm fails with
error writing to the directory. In the workspace the ownership is right and the cache still survives between runs. HOME=/tmp. npm requires a writableHOME, and the runner user’s home does not exist inside the container.
And the cost worth declaring rather than hiding: this became the third runner on the same box. Three services, three updates, three failure points — because organization runners don’t exist on a personal account, and GitHub won’t let three repositories point at the same runner outside an org. It’s a conscious cost, not an oversight. Noting it in the PR is cheaper than rediscovering in six months why the box has three near-identical systemd units.
Exit 2: get the watcher off the watched box
For the probe, a self-hosted runner is no good — for the reason above. The exit was different: a Cloudflare Worker with a cron that probes, folds the result into a history in KV, and serves the page. Cloudflare’s edge is neither the box nor GitHub, so the page’s premise survives.
Two decisions at that scale deserve a note, because they’re the kind of thing that ruins a monitoring migration:
The renderer was not reimplemented. The core was extracted whole out of the CLI script into a shared module, and both sides import the same file. The Worker cannot drift from the CLI on what the page says. The 32 existing tests still pass through the old path, which re-exports everything.
The five component ids stayed the same (api, site, checker, pricing, canonical). An id is not a label: it’s a storage key in the history. Renaming one silently orphans that component’s entire past — you gain a pretty page and lose the data that made it meaningful.
The reduction, stated rather than hidden
The Node probe did more than a Worker can. It executes the served scripts against a DOM and requires the demo report to render, plus a token-authenticated leg through POST /api/v1/inspect. None of that is portable to a Worker — no DOM, no eval. The Worker stops at the static layer.
That layer is not the consolation prize. It is exactly the one that would have caught the 2026-08-04 outage, when a syntax error in a served script left every checker page dead for two days while the HTML kept answering 200. The component description says what it actually checks now, and a test fails if anyone copies the more generous wording of the old probe over it.
workflow_dispatch stays on uptime.yml, so the deeper exercise is one click away before a release.
The load-bearing test
Workers have no node:vm, no eval and no new Function. So the function that parses the served bundle now takes the parser as an argument: Node binds node:vm, the Worker binds acorn.
If the two disagreed, the two vantage points would report different verdicts about the same served file — which is worse than either being wrong alone. So the test asserts that they agree across eight sources, including the exact 2026-08-05 breakage, modern syntax, template literals with braces, regex-vs-division ambiguity, and three deliberate syntax errors.
Total: 1,591 tests, 0 failures (1,577 previous + 14 new).
What is still not solved, and why
Being honest about the state: the Worker logs a failing probe with console.error, which reaches wrangler tail and nothing else.
It restored the page, not the alarm.
Which is a real improvement — the page is truthful again, and a human who opens it sees the current state — but it does not close the hole that caused the whole story. A four-day outage was invisible precisely because nobody was watching the watcher. Swapping one mute watcher for another mute watcher in a better location is half the way.
The runbook, short
If your jobs are dying in 2 seconds with no logs:
- Confirm it’s billing.
gh api repos/<owner>/<repo>/actions/runs --jq '.workflow_runs[0]'and look at the duration. Zero steps executed + seconds of duration + no logs = quota, not code. Don’t go hunting a bug in the repository; there is no bug in the repository. - List what stopped, not what turned red. Deploy, tests, lint, link check, probe cron, backup cron. For each one ask: does this have a visible symptom if it stops? The ones that don’t are your real exposure.
- Check what froze. Status page, coverage badge, dashboards fed by Actions. A number that stopped updating looks like a correct number. It’s the most polite form of lie a system can tell.
- Migrate by category, not by urgency. Build and test can go to a self-hosted runner. Probe and alarm cannot go onto the machine they observe — those need a third place (an edge runtime, an external uptime provider, anything that fails for different reasons than you do).
- Put a dead-man switch on everything periodic. The rule is simple: a system that speaks when things go wrong cannot detect its own death. A system that speaks when things go right can — the missing ping is the alert. It’s the only topology where “silence” means “problem” instead of “fine”.
And the sentence I’d write on the wall:
The watcher cannot live on the box it watches. And the watcher needs someone who notices when it stops talking.
The second half of that sentence is the piece this topology does not yet close. It’s recorded here as the next step, not as a conclusion.
Need a custom technical project?
Architecture, TypeScript, APIs and automation, from prototype to production. The person answering your email is the one writing the code, and the deadline I promise is the one I can meet.
Send me a message →