The data is three full sweeps, graded by the same panel of judges (Google Gemini and Claude Haiku) on test comprehensiveness and code quality, and pooled with the earlier Opus-4.7-era baselines onto one shared grading curve:

• Opus 4.8 — 7 tasks × 5 scripting languages × 4 effort levels (medium / high / xhigh / the new “ultra”) = 140 runs
• Sonnet 5 — 100 runs across three effort levels (low / medium / high)
• Fable 5 — 7 tasks × 4 languages × 2 effort levels (medium / high) = 56 runs

How to read the numbers: duration and cost figures here are geometric means — outlier-damped, so one unusually slow run can’t dominate a cell — and runs that hit the 30-minute timeout count against the duration statistics at their recorded wall clock (a ≥ marks a measurement the timeout capped). The tables also pool the benchmark’s two PowerShell variants (script-file vs. inline-tool) into a single pwsh column; in this harness they turned out to be replicates of each other.

Opus 4.8 tops the quality charts

Across the board, Opus 4.8 produces the strongest tests of any model in the benchmark — the newcomers included — and deliverable code at the top of the board (only Fable 5 at high effort matches it on workflow craft). At high, xhigh, and the new “ultra” effort it earns A‑/A grades for test quality in nearly every language — for example A (4.6) on the default language at ultra effort, and A (4.5) for PowerShell at high. The two judges, despite coming from different labs, agree on the ranking (Spearman rank correlations of +0.66 to +0.85 across the model and language-by-model rankings), so this isn’t one judge’s quirk.

…but you pay for it

The flip side is time and money. Relative to Opus 4.7, the premium is steepest at medium effort — roughly +54% on wall-clock time and +67% on cost — and, interestingly, compresses to about +30% at high effort (4.7’s “high” is comparatively expensive, so 4.8 closes part of the gap) before re-widening to about +75–80% at xhigh. At the top two efforts (xhigh and “ultra”), typical runs take 15–25 minutes and cost $4.30–$6.95 — with the slowest hitting the 30-minute timeout cap — landing in the D / D‑ bands on the speed and cost curves.

If you want most of Opus 4.8’s quality without the worst of the bill, medium effort is the value sweet spot: B+ on speed in three of the four languages, C+/C on cost, and still A‑ test quality everywhere except bash.

The new “ultra” effort

The Opus 4.8 sweep introduces a fourth effort level — “ultra” — which layers multi-agent orchestration on top of the highest reasoning setting. It tops the test-quality charts (it’s the single best column for tests) but is the most expensive option on the board, and it’s Opus‑4.8‑only, so there’s no older-model baseline to compare it against yet. Treat it as “spend more for the most thorough tests,” not as a free win.

It iterates a lot — but it isn’t getting stuck

Opus 4.8 writes more and denser tests than its predecessor, and it shows: it also trips GHA-bench’s “trap” detectors (heuristics that flag things like re-running the same test command many times) about twice as often as Opus 4.7 — 0.97 vs. 0.44 firings per run at matched effort levels. That sounds alarming, so I hand-reviewed all 201 occurrences the detectors originally flagged. The result:

• 99% show no looping at all.
• 86% are legitimate engineering — red-green TDD cycles, designing fixtures up front, fixing a real type error — that merely tripped a count-based heuristic.
• ~1% looked like genuine distress.

So read “~2× the traps” as “iterates ~2× more granularly,” not “fails ~2× as often.” The hand review also caught the detector itself misbehaving: 4.8 prefixes nearly every shell command with a long cd path, which was collapsing distinct commands into one de-duplication key and inflating the rerun count. That detector bug has since been fixed — the ~2× figure above is from the corrected detector, so the gap that remains is real iteration style, not measurement. One caveat still stands: the two runs used different Claude Code versions. The full investigation has the details.

Sonnet 5: pick your effort, pick your product

Sonnet 5 is really three different products depending on where you set the effort knob:

• Low effort is the budget pick. Typical runs land in 6–9 minutes for $0.60–$1.20 — the cheapest thing on the board after Haiku 4.5 — but the tests it writes are thin: C-range everywhere, down to D+ in TypeScript. Fine for scaffolding a workflow; don’t trust the test suite it leaves behind.
• Medium is the middle path — B-range speed and cost, B-range quality. Unremarkable in both directions.
• High effort turns Sonnet 5 into a genuine quality contender — A‑ tests in PowerShell (4.2) and TypeScript (4.3), in Opus 4.8 territory — but it pays in time: 13–25 minutes typical, and it is the most timeout-prone configuration in the benchmark. Six of its 35 high-effort runs hit the 30-minute wall (five of them in PowerShell), and those timeouts now count against its duration stats — that’s what the ≥30min “slowest run” figures in the table are.

Fable 5: near-top quality, premium bill

Fable 5 is the first of Anthropic’s Claude 5 family — a new tier above Opus, priced accordingly at double Opus rates ($10 / $50 per million tokens vs. $5 / $25). On this benchmark it mostly earns it:

• At medium effort it writes A-grade tests in three of the four languages — A (4.5) in PowerShell, A (4.4) in TypeScript, A‑ (4.1) in bash — in 8–15 minutes. That’s Opus-4.8-at-ultra-class test quality at roughly half the wall clock.
• High effort adds time and money ($5.47–$5.98 typical) without adding much: the grades barely move. Medium is clearly its value point.
• It’s the only new model with a clean sheet: zero failures and zero timeouts across all 56 runs.
• One quirk worth knowing: on the free-choice (“default”) language its test quality drops a full letter grade (B‑ 3.2, at both efforts) versus the pinned languages. Tell it what language to use and it shines; let it choose and it coasts.

The catch is the bill: D/D‑ on the cost curve in every language. If budget matters, Opus 4.8 at medium effort buys a similar test-quality profile for roughly half the price (with bash as its own weak spot).

Which should you use? Try it yourself

There’s no single answer — so weight it yourself. Pick a preset (Balanced, Max quality, Quality on a budget, Cheapest, Fastest) or drag the sliders to weight speed, cost, test quality, and code quality for your situation, and the table re-ranks all 76 model / effort / language combinations live. Hover or long-press a Duration cell to see that combination’s slowest run; † marks combos where a run hit the 30-minute timeout.

The complete data lives in the cross-run report on GitHub.

* The Gemini judge now runs via Google’s Antigravity (agy) CLI, which replaced the retired Gemini CLI in June 2026. It grades about 0.3 points stricter on a 1–5 scale than the prior harness (overall correlation r ≈ 0.90), so Opus 4.8’s quality grades are, if anything, very slightly conservative relative to the older models’.

** The runs being compared used different Claude Code versions (2.1.112–132 across the pooled Opus 4.7 rows; 2.1.193/195 for Opus 4.8; 2.1.197/198 for Sonnet 5; 2.1.198 for Fable 5). A clean, model-only comparison would re-run everything on one version; that’s on the to-do list.

Agentic safety. Although it shows improvements in some areas (such as refusing malicious requests), we found Opus 4.8 to be somewhat less robust than Opus 4.7 in several agentic contexts (such as vulnerability to prompt injection attacks). However, the application of our safeguards closes the gap between the models in practice. […]

- Anthropic, System Card: Claude Opus 4.8

How it works

Agents (currently a variety of Anthropic models set to various effort levels, driven by Claude Code) are given set of tasks they must automate using GitHub Actions, either using a particular scripting language or whichever they want.* They must use Test-Driven Development (TDD)– basically “write tests first, and don’t come back until they all pass”.**

A panel of judges (Google Gemini and Claude Haiku) then evaluates the comprehensiveness of the tests and the quality of the code.

Which model, effort level and scripting language should you use?

Adjust the sliders according to your priorities.

* When allowed to choose, the agents always choose Python.

** Agents run their tests locally in a container that leverages nektos act to emulate a GitHub-hosted runner.