Every model is sometimes wrong. The useful question is what happens when it is. With a chat assistant, the output is text that a person may copy into an email, a decision or a system without anyone checking. The control is training and good intentions. That is hard to evidence and harder to scale.
A decision service works the other way round. The model returns a typed answer and a probability. Code decides what that answer may trigger, and the set of possible actions is written down in advance.
Confidence is not permission
Jev's probability tells you how its answer is spread across the options. It does not grant authority. Policy should apply at least two tests before acting:
- Does the confidence clear the threshold set for this specific action? Thresholds are set on held-out labelled cases and the cost of a wrong action, not copied from a demo.
- Can the action be undone without harm? If not, a person decides, whatever the score.
Routing a ticket is reversible. Closing a complaint, releasing personal data or moving money is not.
Fail towards a person
Missing evidence, a response that doesn't validate, a timeout, an unexpected label: each should produce a review item with a correlation ID, never a silent approval. Instructions hidden inside the content being judged are treated as evidence, not as orders. Keep a circuit breaker and plan the capacity of the review queue for the day it opens.
Decide what never goes first
Some decisions should not be anyone's first pilot: autonomous payment approval, employment selection, diagnosis and irreversible account sanctions. Several of these sit in areas regulators already treat as high risk, such as employment and access to essential services under the EU AI Act. Starting elsewhere lets you build the evidence and the operating habits before anything consequential is at stake.
What a risk committee should ask
- What actions can the system take on its own? Show the list.
- What threshold applies to each, and what data set it?
- Which actions always go to a person?
- What happens on timeout, bad input or a new label?
- How do we switch it off, and who can?