Use cases

Check completed agent runs for possible failures

Proposed design Editorial draft

Completed agent traces provide a useful review opportunity because the checker need not perform the original action.

Agent traces Synthetic demo Proposed design
Queue · synthetic items
    RulesPolicy in code
    JevChoice · Score · Noul
    Frontier LLMDraft only
    HumanAccountable action
    Inputs

    Select an item.

    Jev + rules

    Judgments appear here.

    Outputs

    Tickets, drafts, queues.

    A proposed evaluator flags possible incomplete tasks, unsupported claims or tool misuse for inspection. Human reviewers establish whether the flags are useful.

    Do not treat a second AI judgment as independent ground truth merely because it comes from another model. The value is a better review queue and a faster route from recurring failure to a fix.

    Keep the prediction and final action as separate records. Log state, questions, answers, confidence and outcome so operations can reconstruct what happened.

    This pattern fits high-volume verification: every response or agent run can be checked without granting the verifier remediation authority.

    Next step

    Choose one trace failure with a verifiable definition.

    Pilot card

    What a first pilot of this decision needs. Proposed design, not a measured result.

    DecisionDid a completed agent run fail in a way nobody noticed?
    State Jev seesTask given, final output, tool calls and their results, stated success criteria.
    Questions
    • Noul Does the output fail the stated success criteria?
    • Noul Did the agent claim an action its tool results do not show?
    • Score How harmful would the failure be if left (1–5)?
    First action code may takeFile an issue against the agent with the run ID; retry where policy allows.
    Stays with a personChanging the agent, notifying affected users.
    Measure firstReal failures found per hundred runs; failures found by users before the check.
    Microsoft surfaceCopilot Studio or Foundry run logs to Fabric; adapter; Azure DevOps work item.
    Out of scopeSilently rewriting outputs.