Hostname Sunset Without Poisoning Search Console

Retiring a product has one step that almost no sunset checklist includes: what to do with the hostname, the product’s public address. Skipping it doesn’t warn you at the time. It warns you weeks later, in the index report of your entire domain.

The case that produced this note: a product went offline on 2026-08-14. Containers down, final backup stored, documentation updated. An exemplary sunset, except for one detail.

The DNS record for keybelt.haruo.dev kept existing. And the hostname stayed configured in the ingress, the Cloudflare Tunnel’s list of routes, pointing at an origin (the destination server) that no longer existed. Result: every request to the subdomain became a 502, the server error. For a human, irrelevant: the product had no users left. For Googlebot, a different story.

Why one subdomain’s 502 contaminates the domain

Three mechanisms combine, and each is worth understanding:

1. An sc-domain property covers every subdomain. Search Console is Google’s dashboard about your site. Whoever verifies the domain there as a domain property (sc-domain:haruo.dev) gets aggregate reports for everything: the main site, the blog, and any subdomain, dead or alive. Four days after the sunset, the domain’s coverage report listed 7 pages under “Server error (5xx)”. All from the dead product, all polluting the dashboard of the site that matters.

2. 5xx is the worst possible terminal state. To Google, a server error is transient by definition: it keeps re-crawling and holds the URLs in the index for months, waiting for the server to come back. A 404 releases the URL over time; a 410 releases it faster; a 301 transfers it. A 502 never releases anything. The orphaned hostname becomes a wound that won’t heal, reopened on every recrawl.

3. The old robots.txt locks the exit. The dead product had Disallow: /api in its robots.txt, the file that tells the crawler where it may not go. With the site down, that robots file is no longer served. But Google remembers it, and the URLs under /api remain in “Blocked by robots.txt” (and one in “Indexed, though blocked by robots.txt”). The perverse detail: as long as robots doesn’t allow it, Google won’t recrawl those URLs. And therefore never finds out they’re dead. The block that protected the living API now prevents the burial of the dead one.

Summing up the state: 7 URLs in eternal 5xx, 2 trapped behind a ghost robots file, 1 indexed and blocked. None of them with a natural path out of the index.

The fix: a 56-line Worker on the dead hostname

The fix required touching neither DNS nor the tunnel ingress. That matters, because at the time of the incident the available token for wrangler, Cloudflare’s CLI, only had read permission on the zone. A Cloudflare Worker with a zone route (keybelt.haruo.dev/*), that is, a script Cloudflare hosts itself, intercepts requests at the edge, before they reach the tunnel. DNS keeps pointing at the tunnel; the route captures first.

The Worker does exactly three things:

// /robots.txt → 200, permissive plain text
"User-agent: *\nAllow: /"

// /api and /api/* → 410 Gone
"410 Gone — the API was discontinued on 2026-08-14."

// everything else → 301 to the main site
Response.redirect('https://haruo.dev/', 301);

Each line resolves one of the three mechanisms:

• The permissive robots actually being served (200, text/plain, 1h cache) unlocks the recrawl: Google can look again at the URLs the old robots hid. And only then can it see what happened to them.
• The 410 on the API routes says what the 502 never would: this died on purpose, stop coming back. Gone is the strongest removal signal the protocol has. It’s what gets a URL out of the index in days, not months.
• The 301 on everything else salvages what’s left. Any external link, bookmark, or old mention now points at the living site. That transfers what little signal the subdomain accumulated, instead of burning it on an error.

None of this requires the product back, a server, or a deploy in the site’s pipeline. The Worker is a separate artifact, deployed manually, outside the build. And the final teardown order was recorded alongside it: validate in the Search Console reports that the URLs are gone. Only then delete, in this order, the DNS record → the hostname in the tunnel ingress → the Worker itself. Tearing down before validating is recreating the problem to save three resources that cost nothing while parked.

The contrast: the sunset that created no orphan

Five days later, another of our products left the same VPS. That second sunset’s runbook shows what you learn from the first one: the hostname was never orphaned because it was never left without an origin.

The product (a translator with ~700 expressions in a database) became a static export. The data was exported to a versioned JSON. The same translation function that ran on the server now runs in the browser over that JSON, with parity verified case by case before the cutover. And the site moved to Cloudflare Pages.

The critical step in the runbook is the atomic ownership swap of the hostname. That means removing the public hostname from the tunnel and, in the same step, registering the custom domain on the Pages project. The subdomain went from “points at a container on the VPS” to “points at Pages” with no 5xx window. And the containers were only taken down after verification (curl -sI on the hostname, status 200, right headers, data being served).

Two details of that runbook worth stealing:

• The database volume survives 7 days past the cutover, with the rollback documented (bring the compose back up and recreate the hostname on the tunnel). A sunset with a cheap regret button is a sunset that gets executed without drama.
• The accepted loss was written down before the cut: social link previews lost their image, because a static export doesn’t generate dynamic OG, the preview image built on the fly. A known, accepted loss is different from a surprise. The difference is one line in the runbook.

The checklist that stuck

Every sunset of a product with its own hostname, in order:

1. Decide the hostname’s fate before tearing anything down. Either it gets a new origin (static, redirect, Pages), or it gets a tombstone Worker. “It gets nothing” is not an option: nothing means 502, and 502 is forever.
2. If tombstone: permissive robots + 410 on machine routes + 301 on the rest. The robots unlocks the recrawl; the 410 buries; the 301 salvages. A Worker with a zone route does all three without touching DNS.
3. If migration: swap the hostname’s owner atomically and verify the new origin responding before tearing down the old one.
4. Validate in Search Console before dismantling the tombstone. The URLs need to leave the reports; only then DNS → ingress → Worker, in that order.
5. Write the rollback and the accepted loss into the runbook, with an explicit regret window.

The cost of all this is an hour. The cost of skipping it was an index report polluted for weeks, on the domain that carries the main site. With a domain property in Search Console, there is no such thing as “a subdomain that doesn’t matter.” Every hostname that ever answered 200 is a promise made to the crawler, and a sunset is the act of unmaking that promise politely.