routing-savings benchmark · measured

The cost of a model you didn't need.

deadeye routes each subagent task to the cheapest model tier that can still do it. We measured what that's worth — in real billed dollars, graded by a test the model never saw.

3–9×

what opus billed over haiku for identical, correct output (median ~6×)

5 / 6

well-scoped tasks passed on the cheapest tier — downshift nearly free

~63%

model-cost cut vs all-opus (oracle routing) on this set

100%

measured, hidden-test graded — no estimated numbers

18 runs · 6 tasks × 3 tiers (haiku · sonnet · opus) · real total_cost_usd

The proof — over-provisioning has a price

Same task, done correctly: how many × more opus billed than haiku

counter

9.1×

wordwrap

7.1×

clamp

6.1×

semver

4.8×

csv

3.4×

Bar = opus ÷ haiku billed cost, for output both tiers got correct. Per-task and mix-independent — the robust claim. (sonnet ran ~4–5.5× haiku.)

And the cheaper tier actually did the work

Task	Band	haiku	sonnet	opus	Cheapest that passed
clamp	mech	$0.026PASS	$0.110PASS	$0.160PASS	haiku
wordwrap	std	$0.046PASS	$0.230PASS	$0.331PASS	haiku
csv	std	$0.060PASS	$0.329PASS	$0.204PASS	haiku
semver	hard	$0.055PASS	$0.279PASS	$0.265PASS	haiku
expr-eval	hard	$0.050FAIL	$0.294FAIL	$0.407FAIL*	opus
counter	hard	$0.034PASS	$0.186PASS	$0.311PASS	haiku

5 of 6 well-scoped tasks passed on haiku — including SemVer precedence and a concurrent counter under -race. The one that didn't (expr-eval) is the case routing sends up — more on that below.

How we measured it

One run — repeated 18 times

A "saving" is fake if the cheap model fails — you re-run on opus and pay twice. So every run measures both cost and quality, and a saving counts only when the cheaper tier actually cleared the task.

01 · ISOLATE

Clean tree

git archive HEAD into a fresh temp dir — runs can't contaminate each other.

02 · RUN

Real session

claude -p --model <tier> writes the actual code. DEADEYE=off — raw model cost.

03 · GRADE

Blind test

A hidden go test the model never saw runs against its output. Pass or fail.

04 · MEASURE

Real dollars

Billed total_cost_usd + tokens from the run's JSON. Nothing estimated.

The hidden test is the trick: grade against a spec the model couldn't see, and a pass means it understood the task, not that it gamed the test. Each hidden test was first validated against a known-correct reference impl — so a failure is the model's, not the test's.

The task set — three difficulty bands, on purpose

mechanical

clamp(v, lo, hi)

Trivial. Should pass on every tier — so opus here is pure waste.

standard

word-wrap (runes)
CSV parser

Edge cases — unicode width, empty fields, escaped quotes — where a weak tier might slip.

hard

SemVer compare
expression eval
concurrent counter

Reasoning-heavy. Where opus should actually be needed — the route-up case.

If cheap tiers passed everything, the honest verdict would be "just use haiku" — an anti-routing result. We needed one task where the cheap tier fails and opus is needed. That's the expression evaluator.

The frontier — why one task earns opus

expr-eval sits at the model's capability edge

The recursive expression evaluator (operator precedence, unary minus, error handling) failed on haiku and sonnet, and opus was borderline — it failed one trial and passed on re-run (the *). This is exactly the task routing must send up. The roll-up keeps it on opus and claims no saving for it.

The whole thesis in one row: route the easy common case down, keep the hard frontier case up.

The honest caveats — written into the benchmark

The ~63% roll-up is illustrative, not the headline. Oracle routing (cheapest tier that passed) cut cost ~63% vs all-opus on this set — but it depends on the task mix. Lead with the 3–9× ratio, which doesn't.
Single trial per (task, tier) is noisy near the frontier — see expr-eval. Multi-trial pass-rates are the next rigor step.
Small self-contained tasks under-count the tier gap. Large, context-heavy real work is where cheaper tiers slip more — so this is a floor on the value of routing up, not a ceiling.
Cache-heavy system-prompt cost is in every run and similar across tiers, so it dilutes the headline %. The model-priced delta is the real lever.

Correct output costs about 6× more on opus. deadeye routes it down.

← back to deadeye

review-quality benchmark · held-out

Cold review, graded by the bug the project itself fixed.

A second, separate benchmark. deadeye reviewed 24 real pull requests it had never seen — each one shipping a bug the project later repaired in a follow-up. That fix is the answer key, and deadeye never saw it.

61%

of the seeded bugs caught cold — from the diff + source alone (11 / 18)

97%

precision — one false positive across 31 findings, each verified in source

4 / 4

authorization regressions caught — the cross-file permission bugs

real-world PRs across 17 projects, each graded against its own shipped fix

24 PRs · reviewed cold on the flagship tier · graded against the shipped fix · precision adversarially re-verified in source

Where it's strong — and where it isn't

The shape matters more than the headline. deadeye is strongest exactly on the bugs a diff-only reviewer misses — the cross-file permission regression that needs you to open the callee or read the migration. It caught every authorization bug in the set. It's weakest on concurrency races — cold-reading a data race off a diff is hard, and the number says so.

Beyond the 24 seeded bugs, it surfaced 17 more real issues on the same PRs — at the cost of that single false positive.

authz

4/4

bounds

1/1

guard

1/1

contract

2/3

logic

3/7

nil / null

1/3

race

1/4

mutation

0/1

Caught / total per bug class, cold. Strong on the cross-file lenses; the race row is the honest soft spot, left in.

Hardened since · v0.34.0

New weak-class lenses — sibling-branch asymmetry, a dropped condition predicate, type-identity edges (a str subclass, null vs undefined), borrowed-state mutation, and the async await that never resolves. On a fresh held-out A/B — 15 new regression PRs, none from this set, each reviewed cold old-rubric vs new and graded against the shipped fix — recall lifted 4/15 → 6/15 with zero regressions, both gains verified in source. Concurrency races remain the known soft spot.

How it was graded — no answer key touched

The answer key is the project's own fix

Each PR introduced a bug the project later fixed in a follow-up commit. deadeye reviewed the introducing PR cold — diff and source only, no fix, no comments. A finding counted as a catch only when an independent grader confirmed it matched what the fix changed; anything ambiguous was scored a miss. Precision was checked separately: every one of the 31 findings was re-read in source by an adversarial pass whose only job was to refute it.

Held-out by construction, graded against real, public git history.

The honest caveats — this benchmark too

Recall is one held-out number, not a ceiling. 61% on this balanced 24-PR set; it moves with the mix and the difficulty. The durable strength is precision.
Concurrency is the weak lens. Races came in at 1 of 4 — a cold data-race read off a diff is genuinely hard.
Balanced on purpose. The set was capped per bug-class and per repo, so the number isn't stacked toward the authorization bugs deadeye is best at.

61% of held-out bugs caught cold, at 97% precision — every finding proven in source.

← back to deadeye