Gauge.
how trustworthy is the failure rate you are acting on?
measuring the reliability of failure detection in multi-agent LLM systems · Hannah B. Pasandi, UC Berkeley Sky Computing Lab
in plain terms

Every team acts on a failure rate that comes from a person reading traces. Gauge measures how reproducible that reading is, without needing ground truth, and shows the honest answer is an interval, not a number.

the question

How trustworthy is the failure rate you are acting on?

Teams report agent failure rates by having a reviewer read traces, and everything downstream, routing, shipping decisions, safety claims, consumes that number as if it were ground truth. Gauge asks the instrument engineer's question instead: how reproducible is the reading, and where is it least trustworthy?

what the instrument shows

Same traces, three defensible answers

reviewer 1 reviewer 2 reviewer 3 three defensible labeling rules unanimity · majority · union miss rate: 0% · 5% · 12.5% report the interval, not the point dependence-aware reliability interval
The rule is the result. On the same traces, the single-reviewer miss rate is 0, 5, or 12.5 percent depending only on which defensible labeling rule you pick, and one in four flagged failures is contested. Coverage is not the gap. Capture-recapture puts the joint undercount near 1.0x: almost nothing escapes all reviewers, so the error is disagreement, not invisible failures. The split is structural. Per-mode reliability spans 0.76 to 1.00; locally checkable modes are reliable, whole-trajectory judgment modes are contested, and the contested modes are the frequent ones.
status

Gauge is in submission. Deliverables: a dependence-aware reliability interval, a fault-injection validity check, a reporting protocol, and a released measurement pipeline.

in the stack

Gauge produces what Confine consumes

In the AgenticStack agenda, Gauge measures the per-mode reliability of the verifier signal, and Confine is the contract that lets the router act on it. Measurement and mechanism, one loop.

Report the interval, not the point.

← back to AgenticStack