The Test That Derives the Published Number

There’s a class of bug no type checker catches, no ordinary unit test catches, and no code review catches reliably: the number that aged on the public page.

The page says “10 signal families”. The code has 18. Nobody lied — someone wrote 10 when it was 10, the product grew, and the string stayed. The code doesn’t reference the string; the string doesn’t reference the code. Nothing links them, so nothing keeps them together.

The house rule at Tamperlens is one sentence: every number printed on a public surface derives from the source of truth through a test. Not “is periodically reviewed”. Not “is documented where to update”. Derived, in CI, by a test that breaks.

Three PRs this week were essentially exercises in that rule. Two things became clear: when the pattern is applied it’s boring and invisible, and when it isn’t, the cost is always larger than the string change you’re imagining.

---

The case where it works: pricing

The simplest example is pricing, and it explains the format best.

src/services/plans.ts is the single source of truth for both the Stripe Price objects and the per-plan quota. The pricing page cannot have the number written by hand — if it drifts, it quotes an amount the customer will not be charged, which is the one failure a pricing page must not have.

The test does not compare two constants. It fetches the served HTML and asserts that the rendered value is the value derived from plans.ts:

for (const id of PLAN_IDS) {
  const quoted = MARK[url](displayPrice(id, "USD"));
  assert.ok(
    html.includes(`<span data-price="${id}">${quoted}</span>`),
    `${url} does not quote ${id} at ${quoted} (amount from src/services/plans.ts)`,
  );
}

That MARK is the interesting part, and it’s an i18n lesson that came from a real scare. The amount comes from plans.ts; the currency mark is a per-language convention. English writes $29 — the reader is already in a dollar context. Portuguese has to write US$ 29, because a bare $ reads as reais to a Brazilian, and the actual BRL charge is five times the number on the page.

/pt/precos was the one Portuguese surface still shipping the bare $, while /pt/, /pt/api and /pt/dashboard all wrote US$ . And the pricing script only repaints on CF-IPCountry: BR — so a Brazilian behind a VPN read the ambiguous form on the highest-intent page of the two.

There are two tests today: one deriving the amount from plans.ts, and one sweeping every Portuguese surface for a $ without US. The second is a sweep, not a list — because the next page somebody creates has to obey it too.

---

Case 1: the score of 82 that never existed

This is the most expensive of the three, because it wasn’t a number in text. It was a number that was computed.

The product publishes an invariant in three places: a signal family either fires once, with all its findings collected into a single evidence object, or it does not appear. The public page says it, its Portuguese twin says it, and ARCHITECTURE.md restates it. On top of that, the module that picks each family’s explanation maps one family to one explanation on the strength of it.

Three consumers built on the promise, before anything enforced it.

Measured by running the Office engine over the repository’s own fixtures:

fixture	family	signals
office-doctype.docx	office-structure-anomalies	[medium] [low] [info]
office-macro-template.docx	office-active-content	[high] [medium] [medium]
office-injection-markers.docx	office-hidden-content	[medium] [low]

Three families violating the invariant across three fixtures — and the signal registry could still append a fourth on top of any of them.

What it actually cost

Not just the report’s shape. The score.

The scoring function decays the second and later signals of a given severity because it assumes they are different families corroborating each other. A family firing three checks collected three independent contributions for one finding.

Result: office-macro-template.docx scored 82. On identical evidence, it now scores 70.

The 82 never existed. It wasn’t a score that ran high out of calibration generosity — it was a score adding the same evidence three times, and nobody could see that by looking at the number.

Alongside it, summary.signalCount — the number the report headline prints — counted findings on Office files and families everywhere else. Same field, two meanings, depending on the file type. And the report UI keys a signal’s brief text and its “what this signal means” link by family, so every duplicate card repeated both.

The test, and the fourth case the review had not found

The fix is a fold function. But what makes the fix stick is the test: it runs all three engines over every fixture in the repository, collects every violation before asserting (instead of stopping at the first), and separately pins that signalCount counts families rather than findings.

It turned up a fourth case: redaction-mixed-covers.pdf emits two signals in the same family — a filled box over text at high, a picture over text at info.

That one was not fixed, and the reason is the point. The split is deliberate, measured and fixture-backed: merging them was the earlier behaviour, and it was a bug — it announced “4 text runs are still readable underneath boxes drawn over them” at high when three of the four were a letterhead. The two findings carry the same evidence keys at different scopes, so the fold would silently drop one set of numbers — and the numbers are the whole reason for the split.

Resolving it means either nesting that family’s evidence (an API change to the product’s flagship family) or changing the published sentence from “one signal per family” to something like “one per finding kind”. Both are decisions about what the product promises, so the PR recorded the exception instead of taking it — on the public page, on its Portuguese twin, in ARCHITECTURE.md, and in the test, which exempts it by family: a second family appearing twice fails, and that same family appearing three times also fails.

The site stopped claiming something the engine disagrees with. Which is the minimum, and is different from having resolved it.

The bonus: the test that guarded itself with a regex

Along the way, a coverage test counted the Office families with a regex over the source of a module-private array. Its own comment called that a stand-in, “for the day someone exports it”.

That day arrived inside this same change — and the regex would have quietly started reading 7 the moment one of the families left the array. That number is published on two pages.

A test that derives the number from the wrong representation is not better than a hand-written constant. It’s worse, because it looks protected.

---

Case 2: the social card that said ten

This is the most visible and cheapest to fix — and the most educational about where a number can hide.

The brand-asset generator rendered “All ten signal families” onto a PNG that is the og:image of ten pages. The matching page opens with <h2>The eighteen families</h2>.

A wrong denominator on the most-shared image of a product whose entire differentiator is that its denominators are published and correct.

The existing sweep could not see it, twice over:

1. The string lives in scripts/, outside the directories that test sweeps.
2. What ships is a PNG. No text lint reads pixels.

Fixes: string corrected, card re-rendered with Playwright at 1200×630, and — a detail worth copying — only one PNG is in the diff. The command regenerates all ten; the other nine changed by render jitter alone and were reverted, so the binary change in history is exactly the card whose content moved.

Then the new test, which is what matters: it sweeps the generator for family counts and derives them from the registered modules. A second test closes the whole class the generator’s own header described in prose (“every page’s og:image must name the matching file — grep for the filename when adding a page here”): every card it names exists on disk and is referenced by at least one page. That grep now runs in CI instead of running in whoever’s memory.

Its sibling in the same PR

In the same batch, a blog post said a list of families suppressed by encryption “is five names long” and that an owner password “turns off a third of the engine”. The array in types.ts has held eight since version 1.15.0.

There’s a subtlety here I think is the most useful part of this whole post. The coverage tests exempt the blog, on a rule worth keeping: a dated post records what was measured the day it shipped, and moving a number without re-running the measurement claims a measurement nobody made.

But this particular number was never covered by that exemption. It is not a bench result. It is the length of an array in the current source, quoted in the present tense, checkable in one read — on the page whose entire claim is that its numbers are measured rather than asserted.

Corrected to eight, naming the three families the post never covered, and swapping “a third” for “eight of the eighteen”. With a dated callout explaining why this number could move while the post’s matrix could not. And the test now derives both figures from the constant.

The distinction between “a number that can be derived” and “a number that records a measurement from a date” is what makes this rule applicable without turning into tyranny.

---

Case 3: the bench that published its own failures as results

The third is my favourite, because the measuring instrument was lying, and the new test is about the instrument.

The post’s matrix — which signals survive which transformations — had to be re-run across all 18 families. It was. The first thing the run caught was the bench itself.

When an external tool exits non-zero, the function calling it returns no path, and the caller stores { error } — an object with no families key. The per-cell summary read that undefined exactly as it reads a family that stopped firing. The verdict became gone: “the signal disappeared”.

Nine of the 92 fixtures are legitimately refusable — one isn’t a PDF at all, another is already encrypted and cannot be re-encrypted, and one of the tools refuses six deliberately malformed files. Which means every column but one carried inflated “gone” counts.

Some before/after:

Cell	Published	Measured
structure-warnings × strip	gone on 3 of 7	gone on 0 — those three are files the tool refused
structure-warnings × encrypt	weaker, gone on 2 of 7	weaker, gone on 0
id-inconsistency × rewrite	gone on 5 of 8	gone on 3 of the 6 the tool processed
document-injection-markers × print	gone	unmeasured: n=1 and the tool refuses that fixture

And the detail that stings: the script already carried the rule it was breaking. Its header says a variant whose tool is missing is unmeasured, “absent, not zero”, because “a silently missing column is how a matrix starts lying”. It enforced that per column and not per file.

Fixed at the source: failures are counted separately, named with their filenames in the run header, and shown as unmeasured.

One important check before concluding anything: the whole bench was run on two engine versions over the same fixtures with the same tools. The outputs are byte-identical apart from the version line. The engine was not implicated; the instrument was.

The final shape: a record written by the tool itself

And here is the pattern that closes the post.

docs/evasion-bench-last-run.json is written by the bench (--record), never by hand. The test reads that record and fails when:

• the engine version moves past the version stored in the record;
• a new family has never been benched;
• any published n=, laundering figure or silenced-families bar disagrees with the record;
• and it re-derives the post’s conclusion (“nothing in the unconditional set survives all six”) rather than trusting the written sentence.

Every assertion was watched failing against a wrong value before it was allowed to pass. That’s the step separating a derivation test from a test that merely agrees with what’s already there.

Batch total: 1,602 tests, 0 failures.

---

The pattern, generalized

Three shapes, chosen by what kind of number it is:

1. A number that comes from a constant → the test imports the constant and asserts against the served HTML. Never compare two constants; compare the constant to what the user receives.
2. A number that comes from a sweep → the test does the sweep, it doesn’t maintain a list. A hand-written list excludes the page somebody creates tomorrow.
3. A number that comes from a measurement → the tool writes a JSON record, the test compares the page to the record, and the record knows which version was measured. That makes “update the number” and “redo the measurement” the same operation, which is the only honest arrangement.

And the test of the test, the part people skip: watch the assertion fail against the wrong value before you let it pass. A derivation test that has never been seen failing is indistinguishable from a test reading the same string on both sides.

What doesn’t scale is the alternative: trusting that someone will remember. This week: 82 → 70, ten → eighteen, five → eight, gone on 3 of 7 → gone on 0. None of those was found by careful reading. All of them were found by a new test — or, in the most revealing case, by the very tool producing them, the moment it was run while watching what it did with errors.