The call
Yangze Liu and Zhongyi Han of Shandong University ran five models through Claude Code, OpenCode, and mini-SWE-agent on SWE-bench Verified. Claude Code and mini-SWE-agent, the heaviest and lightest harnesses, scored within five points of each other on 447 tasks, while cost per task varied by up to 3x with the same model, tasks, and prices (independent, arXiv 2610.04433). The preprint went up on 3 October; the coverage broke on 24 October (The Decoder, 09:21 UTC; AI Weekly; VentureBeat). It lands on a September result from UC Berkeley and Arena that found the same pattern on Claude and GPT models (independent, Arena).
What it is, and what it is not
What it is. A harness is everything around the model: the system prompt, the tool definitions, the loop that decides the next step. The Shandong team found that swapping harnesses flipped about 13% of outcomes on the 45 hardest tasks, the same rate as rerunning an unchanged setup, and that the system prompt and tool schemas resent at each step drove the cost gap (independent, arXiv). Claude Code sends 16,581 preamble tokens; mini-SWE-agent sends 829 (independent, via DEV Community).
What it is not. It is not a verdict on Claude models. The five tested models were Qwen3.6, Qwen3.8, DeepSeek-V4-Flash, GLM-5.3-Flash, and HY4-Preview; Claude Opus 5 ran only inside Claude Code, as a reference (independent, The Decoder). It is not your codebase: SWE-bench Verified is public open-source repositories, not Apex or Salesforce metadata. And it is a preprint, not peer-reviewed. Governance? The finding is about how you measure, not which tool you buy.
The number nobody is quoting
The headline is 3x. The number a budget owner should keep is the price of the extra solved task. On Arena’s own Claude result, the expensive harness bought 11 more solved tasks per 1,000 at $60 each.
| Measure | Figure | Label |
|---|---|---|
| Claude Fable 5, SWE-bench Lite, cost per task | Claude Code $1.33, Pi $0.67; 1.33 ÷ 0.67 = 1.99x | independent (Arena), our maths |
| Same pair, success rate | 97.8% against 96.7%, a 1.1-point gap | independent (Arena) |
| Per 1,000 tasks | $1,330 against $670, $660 more, for 978 against 967 solved, 11 more: $660 ÷ 11 = $60 per extra solved task | our maths |
| Cost per solved task | $1,330 ÷ 978 = $1.36 against $670 ÷ 967 = $0.69 | our maths |
| Rerun spread of an identical setup, 10 reruns, SWE-bench Verified | 2.2 to 6.0 points; Arena’s 1.1-point gap sits below the smallest spread | independent (Shandong, via AI Weekly), our maths |
| GLM-5.3-Flash, cost per task | Claude Code 1.81 yuan, mini-SWE-agent 0.97, OpenCode 0.56; 1.87x and 3.23x | independent (Shandong, via DEV Community), our maths |
| DeepSeek-V4-Flash, cost per task | Claude Code 0.98 yuan, mini-SWE-agent 1.02; the lighter harness costs 3.9% more | independent, our maths |
| DeepSeek-V4-Flash, model calls per task | mini-SWE-agent 141, Claude Code 104; 36% more steps | independent, our maths |
| Preamble tokens per task at those step counts | 16,581 × 104 = 1,724,424 against 829 × 141 = 116,889: 14.8x more preamble for about the same bill | our maths |
| Input tokens served from prompt cache | 96% to 99% in every arm | independent (via DEV Community) |
Two things follow. First, a 1.1-point accuracy edge is not something you can see in your own pilot, because an unchanged setup moves more than that between runs. Second, the size of the preamble is not the bill. Once it is cached, steps and the history each step resends decide the cost. A harness that is lean on paper and takes 36% more steps loses its advantage.
Missing
What the studies do not tell you, as of 25 October:
- Claude and GPT models inside third-party harnesses at Shandong scale; the Shandong runs used open-weight and Chinese API models.
- Dollar costs for the Shandong runs; every figure is in yuan.
- Results on private enterprise code: Apex, Salesforce metadata, or a legacy monolith.
- Wall-clock time per task and the reviewer time each output costs.
- How cache pricing differs by vendor, which decides whether a large cached preamble matters at all.
- A response from any harness vendor in the coverage we read.
Who owns what
The harness is their product. Cost per solved task is ours.
Put the meter where the work already lands. On a Salesforce team that is the CI run that validates a deployment and runs Apex tests against a sandbox or scratch org: an agent task counts as solved when that run passes, and its cost is the provider usage for that task. On an application team it is the same pull request check. Seat price tells you nothing here. Usage billing does. A survey reported by VentureBeat found 21% of enterprises have no automated way to stop agents’ token spending (independent, via AI Weekly).
Commercial decision: standardising on one harness saves training and support time. Standardising without a cost meter hides a 2x line item for a gap you cannot measure. Meter first, then standardise.
Monday morning
- Step 1. Pull real tasks. Take closed tickets with a passing CI run from your own repositories, enough to resolve a 5-point gap; the paper needed 447. Owner: engineering lead.
- Step 2. Run each candidate harness twice on the same model. If the gap between harnesses is smaller than the gap between two runs, it is noise. Owner: platform engineer.
- Step 3. Report cost per solved task, not cost per seat. Provider usage divided by tasks that pass CI. Owner: FinOps or the AI budget owner.
- Step 4. Set a hard spend stop on agent tokens per user and per project, before any wider rollout. Owner: AI budget owner.
- Step 5. Ask each harness vendor in writing for its preamble size, its cache behaviour, and its median steps per task, and date the reply. Owner: procurement.
We move this to WATCH if a replication on Claude and GPT models in third-party harnesses shows a harness accuracy gap that clears rerun noise.
Frequently asked questions
Does this mean Claude Code is a bad buy?
No. On the Arena tests Claude Code solved the most SWE-bench Lite tasks with Claude Fable 5, by 1.1 points, at about twice the cost. On the Shandong tests its accuracy matched the lightest harness. The finding is that harness choice moved cost far more than accuracy, so measure both on your own work before you standardise.
Why does the preamble size not decide the bill?
Because most of it is cached. The Shandong runs hit the prompt cache for 96% to 99% of input tokens, and on DeepSeek-V4-Flash the lightest harness took 141 model calls against 104 for Claude Code and ended up costing slightly more. Steps and growing history drive the bill once the fixed preamble is cached.
How big should our own harness bake-off be?
Bigger than most pilots. The paper shows 45 tasks detect a 13-point gap only about half the time, while 447 tasks resolve gaps of about 5 points, and reruns of an identical setup moved scores by 2.2 to 6.0 points. Run each harness at least twice and use real tasks from your own repositories.
Living page. Last updated 25 Oct 2026. What would change our call: a replication on Claude and GPT models showing a harness accuracy gap larger than rerun noise.