False-positive reduction benchmark on synthetic data
Synthetic data. Every alert on this page comes from a seeded generator: 1,200 alerts over 90 days, seed 20260915. No client, no real transaction and no real person is behind any number here.
By Douglas Haruo, KYC/AML engineering at Chipper Cash 2020–2025.
The numbers themselves prove nothing about your queue. What this page shows is the order of work: record a baseline, simulate a change on closed history, put the escalations it would have suppressed next to the hours it would have saved, and only then decide.
The queue
One row per alert, the shape a Sardine, Unit21, ComplyAdvantage or Alloy export takes once flattened: rule, vendor score from 0 to 100, the analyst’s disposition and the minutes the review took. Six rules, from structuring to PEP and adverse media.
A false positive here is an alert an analyst closed with no action. An escalation to SAR review is not one, whatever happens at level 2. Your institution may define it differently; the definition gets agreed in writing on day 1, because it moves the headline number.
Analyst time is priced at US$45 an hour, fully loaded. That is an assumption, not a measurement, and a client replaces it with their own figure.
Baseline, recorded before any change
| Measure | Value |
|---|---|
| Alert volume | 1,200 (13.3 a day, about 400 a month) |
| False-positive rate | 92.0% (1,104 closed with no action, 96 escalated) |
| Minutes per alert | median 8.5, mean 10.6, p90 18.2 (212.9 analyst hours in total) |
| Cost per alert | US$7.99 mean, US$6.38 median (US$9,582.02 in total) |
Two threshold changes, simulated on closed history
Each change suppresses alerts whose vendor score falls below the threshold, so they never reach the queue. Nothing here touched a live queue or a vendor setting.
Precision is the share of the remaining queue an analyst escalates. Recall is the share of all escalations that still reach an analyst. The interval is a 95% Wilson interval on recall.
| Threshold | Queue | Precision | Recall (95% CI) | Escalations suppressed | Analyst minutes saved | Cost saved | |
|---|---|---|---|---|---|---|---|
| Before | none | 1,200 | 8.0% | 100.0% | 0 of 96 | 0 | US$0 |
| Change A | score below 40 | 707 | 13.2% | 96.9% (91.2–98.9) | 3 of 96 | 4,515 (75.2 h) | US$3,386.17 |
| Change B | score below 50 | 442 | 19.7% | 90.6% (83.1–95.0) | 9 of 96 | 6,985 (116.4 h) | US$5,238.49 |
Change A removes 41% of the queue and costs 3 escalations. Change B removes 63% and costs 9. Choosing between them is the BSA Officer’s call. My part is to have the 3 and the 9 on paper, with the alert IDs behind them, before either change goes live.
The same two thresholds on a later period
A threshold that looks good on the history it was chosen against can look worse on the next month. So the same two thresholds were measured again on the last 30 days alone (alerts from 14 August 2026, 410 of them), without re-tuning.
| Threshold | Queue | Precision | Recall (95% CI) | Escalations suppressed | Analyst minutes saved | |
|---|---|---|---|---|---|---|
| Before | none | 410 | 9.0% | 100.0% | 0 of 37 | 0 |
| Change A | score below 40 | 234 | 15.0% | 94.6% (82.3–98.5) | 2 of 37 | 1,653 |
| Change B | score below 50 | 142 | 22.5% | 86.5% (72.0–94.1) | 5 of 37 | 2,581 |
Recall for change B falls from 90.6% to 86.5%, and its interval now reaches down to 72%. With 37 escalations in a month, one extra missed alert moves recall by almost 3 points. That width is the reason a threshold change gets a monthly re-measurement written into the retainer.
What this benchmark cannot tell you
- The generator does not drift. Every day of the synthetic history is drawn from the same distribution, so the difference between the two tables is sampling noise. A real queue also drifts: new products, new corridors, a vendor model update.
- The analyst is the reference, not the truth. Recall here means “still reaches the analyst”. It says nothing about escalations the analyst missed.
- After go-live, suppressed alerts are invisible. Once a threshold is live, nobody reviews what falls below it. Recall can only be measured again through below-the-line sampling: a random draw of suppressed alerts, reviewed by a human anyway, on a schedule.
- Why 1,200 alerts and not 50. A 50-alert set at this escalation rate holds about 4 escalations, and recall would move in 25-point steps. That is too coarse to compare two thresholds.
Reproducing it
The numbers come from alert-baseline, the kit I use for the two-week diagnostic: standard-library Python 3.9+, no network, no install. make benchmark regenerates both tables and make baseline the four numbers above; a unit test pins every figure on this page, so a change to the generator fails the build before it reaches a conversation. The repository is private. At a client, the same code runs inside their environment against their own export and emits only aggregates.
Want these numbers for your own alert queue?
Douglas Haruo, KYC/AML engineering at Chipper Cash from 2020 to 2025. The two-week diagnostic measures alert volume, false-positive rate, minutes per alert and cost per alert on the vendor tool you already run, inside your environment.
See the two-week diagnostic →