Alert triage: where the vendor agent ends and your tuning begins
A reference architecture for a 2026 AML alert stack, written for the BSA Officer who already runs Sardine, Unit21, Sumsub, ComplyAdvantage or Alloy in production. By Douglas Haruo, KYC/AML engineering at Chipper Cash 2020–2025.
In 2026 the triage agent is something you buy. Unit21 recommends dispositions and drafts SAR narratives. Bretton AI raised a US$75M Series B in February 2026 to do the same work for Mercury, Gusto and Ramp. If your team is thinking about building one, stop; that market has closed.
What the vendor does not ship is the part your examiner asks about. The diagram above splits every stage into what the vendor owns and what you own, and the second list is the longer one.
1. Detection: the vendor ships the engine, you own every number in it
The rule library, the scoring model and the name-matching algorithm are vendor code. The threshold on each rule, which customers a rule looks at, the fuzziness of a sanctions name match, the lookback window and the rules you switched off are your settings. If nobody has revisited them since onboarding, they describe a customer base you no longer have.
When a queue closes more than nine alerts in ten with no action, look at these settings first, because they are the part you can change this week without a vendor ticket. My synthetic benchmark queue closes 92% of its 1,200 alerts with no action. A single threshold change there cuts the queue by 41% and costs 3 of its 96 escalations. Whether those 3 are an acceptable price is your decision, and it has to be written down before the change goes live.
2. The agent: the vendor ships the model, you write its policy
The vendor owns the model, the prompt and the release schedule. You own the policy around it:
- which rules it may auto-close, and above what confidence;
- what must always reach a human (sanctions hits, PEP matches, anything touching an open SAR);
- how its agreement with your analysts is measured, and how often;
- the condition under which you switch it off.
Measure agreement on the raw 2×2 matrix. On my synthetic queue the recorded agent agrees with the analyst on 68.8% of alerts, Cohen’s kappa is 0.216 and PABAK is 0.375. Those three numbers look like a grade. The number that carries the risk is in one cell of the matrix: 8 alerts a human escalated and the agent would have closed. At an escalation rate of 8%, kappa drops for arithmetic reasons (the kappa paradox, Feinstein & Cicchetti 1990), so a kappa banding table tells you very little here.
The April 2026 model-risk guidance (SR 26-2 / OCC Bulletin 2026-13, which replaced SR 11-7) puts generative and agentic AI outside its scope. That leaves the evidence standard for the agent with your sponsor bank’s policy and, in New York, 3 NYCRR Part 504. The rules and scores feeding it are still models under the guidance, including the vendor’s, and it has a section on vendor and third-party products.
3. The human: close, escalate, file
The L1 analyst closes or escalates. L2 or the BSA Officer decides whether to file, against FinCEN’s 30-day clock from initial detection. The agent can shorten L1 time. It does not own the filing decision, and your policy should say so in a sentence an examiner can quote back to you.
4. The decision log: the one piece the vendor cannot write for you
The vendor can export its alerts and its agent’s recommendations. It cannot export your reasons. The log records:
- every setting change: who made it, when, why, the before/after simulation on closed history, and who approved it;
- every agent recommendation next to the human decision, with overrides in both directions;
- the vendor model version in force on the day of each decision;
- below-the-line samples: alerts a threshold suppressed, pulled at random and reviewed by a human anyway, because once a threshold is live nobody sees what it hides.
5. What the examiner asks for
Expect these questions. Why is this threshold set where it is, and what did you test before changing it? What does below-the-line testing show? Is the vendor’s model in your model inventory, and who validated it? Who is accountable for the agent’s auto-closures? The 2023 interagency guidance on third-party risk management is clear that using a vendor does not move that accountability off your institution.
One more boundary. Whoever tunes the rules cannot also be their independent validator. If I tune your thresholds, my evidence pack goes to your independent validator or internal audit; it does not replace them.
How I work with this
I run a two-week alert-baseline diagnostic inside your environment: volume, false-positive rate, minutes per alert and cost per alert, then the vendor-versus-tuning map above filled in for your stack. The kit is standard-library Python, needs no install and emits only aggregates. No alert, customer or transaction leaves your network.
Want these numbers for your own alert queue?
Douglas Haruo, KYC/AML engineering at Chipper Cash from 2020 to 2025. The two-week diagnostic measures alert volume, false-positive rate, minutes per alert and cost per alert on the vendor tool you already run, inside your environment.
See the two-week diagnostic →