3–10×
what opus billed over haiku for identical, correct output (median ~5.7×)
deadeye routes each subagent task to the cheapest model tier that can still do it. We measured what that's worth — in real billed dollars, graded by a test the model never saw.
18 runs · 6 tasks × 3 tiers (haiku · sonnet · opus) · real total_cost_usd ·
router arm re-measured over 5 trials/task
| Task | Band | haiku | sonnet | opus | Cheapest that passed |
|---|---|---|---|---|---|
| clamp | mech | $0.035PASS | $0.070PASS | $0.118PASS | haiku |
| wordwrap | std | $0.042PASS | $0.132PASS | $0.279PASS | haiku |
| csv | std | $0.067PASS | $0.142PASS | $0.314PASS | haiku |
| semver | hard | $0.044PASS | $0.166PASS | $0.441PASS | haiku |
| expr-eval | hard | $0.058FAIL | $0.253FAIL | $0.392FAIL | none |
| counter | hard | $0.038PASS | $0.251PASS | $0.251FAIL | haiku |
A "saving" is fake if the cheap model fails — you re-run on opus and pay twice. So every run measures both cost and quality, and a saving counts only when the cheaper tier actually cleared the task.
git archive HEAD into a fresh temp dir — runs can't contaminate each other.
claude -p --model <tier> writes the actual code. DEADEYE=off — raw model cost.
A hidden go test the model never saw runs against its output. Pass or fail.
Billed total_cost_usd + tokens from the run's JSON. Nothing estimated.
The recursive expression evaluator (operator precedence, unary minus, error handling) failed on all three tiers, in both full sweeps. An earlier single-trial note here claimed opus passed it on a manual re-run; two sweeps have since contradicted that, so the claim is retired rather than repeated.
Before blaming the models, we checked the grader: the hidden test was re-validated against an independently written correct implementation and passes. So these are genuine one-shot model failures, not a broken fixture.
It earns no saving in either arm — nothing to downshift to. The honest read is narrower than the original framing: this task marks where one-shot generation runs out on every tier we tested, not where opus starts winning.
Everything above measures tiers — what each costs, what each can do. From that you can compute the oracle: perfect hindsight, always picking the cheapest tier that turned out to pass. On this set that's a 66% cut vs all-opus.
But nobody ships with hindsight. The question that decides whether routing is worth anything is what deadeye's own router picks, cold, from the task description alone — so we measured that too, and charged every wrong-cheap route the cost of the re-run on the next tier up. A bad guess loses money here; it doesn't quietly score as a saving.
The first time we ran this arm it
scored 29%, agreed with the oracle on 2 of 6
tasks, and picked different tiers for the same prompt across repeat trials on four of them. The cause wasn't
model noise — it was our own classifier prompt. It called tier 1 "most real tasks" (a standing thumb on the
scale toward the middle tier) and defined the cheap tier by edit size, leaving nowhere to put
"write one self-contained function from a complete spec." So it graded difficulty by how advanced the
topic sounded: SemVer precedence and a race-safe concurrent counter both got sent up, while haiku
passed both against the hidden test. Rewritten around scope and
specification instead of subject matter — with the tie-break deliberately scoped to the cheap end,
since guessing low on security or architecture is the expensive direction to be wrong in — it went to 48%,
5-of-6, and stable across every trial. A judge-probe.sh suite now guards the obvious way to
fake that number: a classifier collapsed to "always cheapest" would ace this benchmark, so seven probes
(cross-file refactor, under-specified integration, subtle debugging, security, architecture) fail the build
if anything routes down that shouldn't.
counter passed
on haiku and sonnet but failed on opus, and it passed on all three last time. Multi-trial
pass-rates are the next rigor step.claude -p exposes no temperature or seed control, so this has to
be re-measured after any change to the classifier, never assumed.Correct output costs about 6× more on opus. deadeye routes it down.
← back to deadeyeA second, separate benchmark. deadeye reviewed 24 real pull requests it had never seen — each one shipping a bug the project later repaired in a follow-up. That fix is the answer key, and deadeye never saw it.
24 PRs · reviewed cold on the flagship tier · graded against the shipped fix · precision adversarially re-verified in source
The shape matters more than the headline. deadeye is strongest exactly on the bugs a diff-only reviewer misses — the cross-file permission regression that needs you to open the callee or read the migration. It caught every authorization bug in the set. It's weakest on concurrency races — cold-reading a data race off a diff is hard, and the number says so.
Beyond the 24 seeded bugs, it surfaced 17 more real issues on the same PRs — at the cost of that single false positive.
New weak-class lenses —
sibling-branch asymmetry, a dropped condition predicate, type-identity edges (a str subclass,
null vs undefined), borrowed-state mutation, and the async await
that never resolves. On a fresh held-out A/B — 15 new regression PRs,
none from this set, each reviewed cold old-rubric vs new and graded against the shipped fix — recall lifted
4/15 → 6/15 with zero regressions, both gains verified in source.
Concurrency races remain the known soft spot.
Each PR introduced a bug the project later fixed in a follow-up commit. deadeye reviewed the introducing PR cold — diff and source only, no fix, no comments. A finding counted as a catch only when an independent grader confirmed it matched what the fix changed; anything ambiguous was scored a miss. Precision was checked separately: every one of the 31 findings was re-read in source by an adversarial pass whose only job was to refute it.
Held-out by construction, graded against real, public git history.
61% of held-out bugs caught cold, at 97% precision — every finding proven in source.
← back to deadeye