GoKawiil benchmark finds 64% of coding-agent inference spend went to failed runs
GoKawiil, working with Baseten, ran 27,280 coding agent runs across 22 harnesses, 5 models and 62 tasks using 83 billion tokens, expanding its earlier single-model FrontierHarness benchmark. The study found that 64% of total inference spend was consumed by runs that ultimately failed, and that the best-performing harness varied by model, with cost rankings shifting dramatically when models were swapped.
GoKawiil's interpretation of the reporting above, not reported fact.
The findings suggest that picking a harness and model independently could waste significant compute budget, since a harness tuned for one model may rank near the bottom with another, according to GoKawiil's data showing Codex CLI dropping from 1st to 16th place by cost when switching from GLM-5.3 to Kimi-K3. This implies teams building coding agents may need to re-evaluate their harness choice every time they change underlying models rather than treating the two as separate decisions.
- 64% of inference spend across 27,280 runs went to failed attempts
- Best-value harness differed per model, with cost rankings shifting by as many as 11 places out of 22
- Four harnesses—OpenCode v2, Pi, Codex CLI and ZCode—sat on the overall cost-vs-pass-rate frontier
Source: runta.com, 2026-11-17
Published there as: “In 27,280 coding agent runs, 64% of inference spend went to runs that failed”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.