A Backup With a Dead-Man Switch — and the restore rehearsal that must know how to fail

There is a backup failure mode no error log ever records: the job that stops running. It doesn’t fail. Failing would be good, because failures show up in logs. It gets unloaded from the scheduler during a system update, or the destination disk goes missing, or a credential expires, and from then on nothing happens at all. No new line in the log. No alert, because the alert you configured fires when the job fails, and a job that doesn’t run doesn’t fail.

That is exactly how a nightly backup job in this infrastructure sat stopped for 14 days in August 2026, unloaded from launchd, the system scheduler. Nobody noticed. The log had no errors: it had an absence, and absence triggers nothing.

This post is the design that closes that hole, in three pieces. The first is the backup that pulls, not pushes. The second is the dead-man switch, the inverted signal that turns silence into an alarm. The third, the one almost everyone skips, is the restore rehearsal that actually runs. It carries a negative verification: a step that must fail for the rehearsal to pass.

---

Piece 1: the backup pulls, and the off-site copy is the one that matters

The base design: a VPS runs the services and produces local dumps every 6 hours (Postgres, SQLite with an online .backup(), configuration tarballs). A machine outside the datacenter, in this case a Mac, pulls those dumps every night via rsync, the utility that copies files between machines.

Pulling instead of pushing has a security consequence worth spelling out: the VPS has no credential whatsoever to write to the backup machine. An attacker who compromises the box can destroy its local dumps. The copies outside stay out of reach. In the same spirit, rsync runs with --ignore-existing and never with --delete: a snapshot that has arrived is neither overwritten nor removable by the origin. Pruning is local, by age (60 days), done by the pulling side.

The complete three layers, with what each one survives:

Layer	Where	Survives
Dumps on the VPS itself, every 6h	same disk as the database	logical corruption; not disk loss
Provider snapshots (7 rotating)	same provider	disk loss; not account loss
Nightly pull off-site	another machine, another place	loss of the entire provider

The third layer is the only copy outside the provider. That is why it is the only one carrying the dead-man switch. The other two have someone watching them: the provider itself and a cron on the box. The off-site one depends on a home machine and a scheduler that has already proven it knows how to die silently.

One detail that pays its rent: freshly arrived compressed files go through gunzip -t before counting as backup. A .gz truncated by a dropped connection is deleted on the spot. A corrupted backup discovered on restore day is not a backup: it is a prank with a scheduled date.

Piece 2: the dead-man switch — silence becomes the alarm

The fix for “the job that doesn’t run doesn’t fail” is inverting the signal’s direction. Instead of the job warning when something goes wrong, it warns when things go right. At the end of every successful run, a curl hits a healthcheck URL, the address that receives that sign of life. The service on the other side expects that ping on a known cadence (once a day, with 6 hours of grace). If the ping doesn’t arrive, it alerts.

The difference is structural: the alert no longer depends on the job being alive to be emitted. Job unloaded from the scheduler, machine off, network down, script hung before the end: every one of those failure modes collapses into the same observable symptom, silence. Including the ones nobody predicted. And silence now has an owner.

Two implementation cares:

• The check URL stays out of git. It is, in practice, a credential: anyone who knows it can silence the alarm by pinging in your place. It lives in an ignored file. If the file is missing, the script says so out loud (“dead-man switch inactive”) instead of proceeding quietly.
• The same pattern, inverted, runs inside the VPS. A guardian on a 10-minute cron checks disk (≥85%), unhealthy containers, stopped containers that had a restart policy, and a deep application healthcheck. If everything is fine, it pings OK; if something fails, it pings the failure variant with the reason in the body. And if the whole box dies? Silence. The healthcheck service converts that silence into an alert within at most an hour and a half. A dead VPS cannot warn that it died, and the design doesn’t need it to.

The general rule these two cases illustrate: the watcher cannot depend on the watched thing to emit the alarm. Everything else is implementation detail.

Piece 3: the restore rehearsal — quarterly, disposable, and with failure on purpose

A backup without a tested restore is a hypothesis with good marketing. The phrase is worn out; what is less worn out is the shape of a rehearsal that actually informs anything.

This infrastructure’s rehearsal runs in a single script, on the backup machine, against a disposable Postgres. It is an ephemeral container on a high port, torn down by a trap on exit, that never touches production. It takes the artifacts from the latest backup run, the same files a real restore would use, not a prepared copy. Then it restores the databases, runs the migrations, and checks.

The last full rehearsal: 37 verifications, 0 failures, data restored in under a minute, row counts checked table by table, 2,855 rows, 0 divergences.

The numbers that come out of the rehearsal are the ones no backup dashboard shows.

• RTO measured, not estimated, the time until you are back up: restoring the data takes seconds. The whole box, from scratch, is estimated at 35–60 minutes. That estimate is written next to the measured number, labeled as an estimate.
• Real RPO, how much data you lose between one copy and the next: the on-box dumps give 6 hours. The off-site copy, which depends on the nightly pull, can reach ~30 hours. It is the honest number, and it is bigger than intuition suggests.

The failure on purpose

The part of the design I most recommend copying: the rehearsal contains verifications that require a failure to pass.

The most important one involves encrypted credentials in the database. The rehearsal restores and decrypts a credential with the correct key, taken from the backup’s configuration tarball. That proves the key is in the backup too. Then it swaps in a wrong key and requires the decryption to raise. If the wrong key decrypts, the rehearsal aborts with an error: the credential was not encrypted at all, and the restore’s “success” would be hiding a leak.

It is the same principle as the test you watch fail before letting it pass, applied to infrastructure. A verification that has never been seen failing cannot distinguish “safe” from “unchecked”.

In the same vein, the rehearsal fails hard when an essential input is missing: it exits with an error, not a warning. The comment in the script itself sums up the philosophy. The rehearsal exists to say whether you can come back. Without the environment-variables tarball, the dump comes back with unreadable columns, and a rehearsal that “passes with caveats” in that condition would be certifying a restore that doesn’t work.

Other guards in the same category. An input older than 30 hours fails, because the promised RPO was broken and it is the rehearsal’s job to say so. A destination Postgres older than the origin’s major version fails. A restored schema with fewer tables than expected fails.

The rehearsal that lied

And the finding that justifies all of this: the first version of the rehearsal script itself lied. A docker exec -i inside a while read consumed the loop’s stdin, the standard input that fed it the table list. The list was swallowed on the first iteration. The script checked one table and reported “0 divergences” with all the confidence in the world.

The verification instrument has the same rights as any code: it breaks, it lies, it needs verifying. The fix came with an explicit count (“N tables checked”) in the report. “0 divergences across 54 tables” and “0 divergences” are very different sentences, and the second one hides exactly the bug that happened.

---

The checklist

1. The copy that matters is the one the origin cannot reach. Pull, not push; no --delete; age-based pruning on the pulling side.
2. Test integrity on arrival (gunzip -t or equivalent). A backup found corrupted at restore time is not a backup.
3. A dead-man switch on every scheduled job that matters. The job pings on success; the silence alerts. The ping URL is a credential: out of git.
4. The inside watcher pings failures with a reason; the box’s death becomes silence. And the silence already has a listener.
5. Rehearse the restore on a fixed cadence, against disposable infrastructure, with the real artifacts. Publish measured RTO and RPO, not wished-for ones.
6. Include negative verifications. The wrong key must fail. The missing input must abort. A rehearsal incapable of failing is not a rehearsal: it is a ceremony.
7. Distrust the instrument. The first bug a rehearsal finds is usually in the rehearsal itself, and that is not an embarrassment: it is the process working.