Skip to content
amlcompliancefraudmodel riskai agents

Alert triage: where the vendor agent ends and your tuning begins

Douglas Haruo 6 min 10/6/2026

A reference architecture for a 2026 AML alert stack, written for the BSA Officer who already runs Sardine, Unit21, Sumsub, ComplyAdvantage or Alloy in production. By Douglas Haruo, KYC/AML engineering at Chipper Cash 2020–2025.

Alert triage stack, 2026: vendor, tuning, human, evidence Five stages left to right: your data, the vendor detection engine, the vendor AI agent, the human analyst and BSA Officer, and the examiner. Under each vendor stage, a dashed line separates what the vendor ships from the settings the institution owns. A decision log runs underneath every stage and is what the examiner and the independent validator read. YOUR DATA Inputs Transactions KYC profile Screening lists Case history VENDOR SHIPS Detection engine Rule library Scoring model Name-matching algorithm YOU TUNE Thresholds per rule Customer scope per rule Match tolerance, lookback Rules switched off VENDOR SHIPS Triage agent Recommends disposition Drafts SAR narrative Model and prompt YOU DECIDE What it may auto-close What must reach a human How it is measured When it is switched off HUMAN DECIDES Analyst L1 Close or escalate L2 / BSA Officer File the SAR or not 30-day filing clock READS IT Examiner Sponsor bank Independent validator YOU OWN, THE VENDOR CANNOT WRITE IT FOR YOU Decision log Every setting change: who, when, why, before/after simulation, approver Every agent recommendation next to the human decision, overrides in both directions Vendor model version in force on the day of each decision Below-the-line samples: alerts a threshold suppressed, reviewed by a human anyway The dashed line is where the vendor agent ends and your tuning begins.

In 2026 the triage agent is something you buy. Unit21 recommends dispositions and drafts SAR narratives. Bretton AI raised a US$75M Series B in February 2026 to do the same work for Mercury, Gusto and Ramp. If your team is thinking about building one, stop; that market has closed.

What the vendor does not ship is the part your examiner asks about. The diagram above splits every stage into what the vendor owns and what you own, and the second list is the longer one.

1. Detection: the vendor ships the engine, you own every number in it

The rule library, the scoring model and the name-matching algorithm are vendor code. The threshold on each rule, which customers a rule looks at, the fuzziness of a sanctions name match, the lookback window and the rules you switched off are your settings. If nobody has revisited them since onboarding, they describe a customer base you no longer have.

When a queue closes more than nine alerts in ten with no action, look at these settings first, because they are the part you can change this week without a vendor ticket. My synthetic benchmark queue closes 92% of its 1,200 alerts with no action. A single threshold change there cuts the queue by 41% and costs 3 of its 96 escalations. Whether those 3 are an acceptable price is your decision, and it has to be written down before the change goes live.

2. The agent: the vendor ships the model, you write its policy

The vendor owns the model, the prompt and the release schedule. You own the policy around it:

  • which rules it may auto-close, and above what confidence;
  • what must always reach a human (sanctions hits, PEP matches, anything touching an open SAR);
  • how its agreement with your analysts is measured, and how often;
  • the condition under which you switch it off.

Measure agreement on the raw 2×2 matrix. On my synthetic queue the recorded agent agrees with the analyst on 68.8% of alerts, Cohen’s kappa is 0.216 and PABAK is 0.375. Those three numbers look like a grade. The number that carries the risk is in one cell of the matrix: 8 alerts a human escalated and the agent would have closed. At an escalation rate of 8%, kappa drops for arithmetic reasons (the kappa paradox, Feinstein & Cicchetti 1990), so a kappa banding table tells you very little here.

The April 2026 model-risk guidance (SR 26-2 / OCC Bulletin 2026-13, which replaced SR 11-7) puts generative and agentic AI outside its scope. That leaves the evidence standard for the agent with your sponsor bank’s policy and, in New York, 3 NYCRR Part 504. The rules and scores feeding it are still models under the guidance, including the vendor’s, and it has a section on vendor and third-party products.

3. The human: close, escalate, file

The L1 analyst closes or escalates. L2 or the BSA Officer decides whether to file, against FinCEN’s 30-day clock from initial detection. The agent can shorten L1 time. It does not own the filing decision, and your policy should say so in a sentence an examiner can quote back to you.

4. The decision log: the one piece the vendor cannot write for you

The vendor can export its alerts and its agent’s recommendations. It cannot export your reasons. The log records:

  • every setting change: who made it, when, why, the before/after simulation on closed history, and who approved it;
  • every agent recommendation next to the human decision, with overrides in both directions;
  • the vendor model version in force on the day of each decision;
  • below-the-line samples: alerts a threshold suppressed, pulled at random and reviewed by a human anyway, because once a threshold is live nobody sees what it hides.

5. What the examiner asks for

Expect these questions. Why is this threshold set where it is, and what did you test before changing it? What does below-the-line testing show? Is the vendor’s model in your model inventory, and who validated it? Who is accountable for the agent’s auto-closures? The 2023 interagency guidance on third-party risk management is clear that using a vendor does not move that accountability off your institution.

One more boundary. Whoever tunes the rules cannot also be their independent validator. If I tune your thresholds, my evidence pack goes to your independent validator or internal audit; it does not replace them.

How I work with this

I run a two-week alert-baseline diagnostic inside your environment: volume, false-positive rate, minutes per alert and cost per alert, then the vendor-versus-tuning map above filled in for your stack. The kit is standard-library Python, needs no install and emits only aggregates. No alert, customer or transaction leaves your network.

Want these numbers for your own alert queue?

Douglas Haruo, KYC/AML engineering at Chipper Cash from 2020 to 2025. The two-week diagnostic measures alert volume, false-positive rate, minutes per alert and cost per alert on the vendor tool you already run, inside your environment.

See the two-week diagnostic →