Skip to content
Tech News
← Back to articles

GoKawiil benchmark finds 64% of coding-agent inference spend went to failed runs

read original more articles
GoKawiil Brief

GoKawiil, working with Baseten, ran 27,280 coding agent runs across 22 harnesses, 5 models and 62 tasks using 83 billion tokens, expanding its earlier single-model FrontierHarness benchmark. The study found that 64% of total inference spend was consumed by runs that ultimately failed, and that the best-performing harness varied by model, with cost rankings shifting dramatically when models were swapped.

Why It Matters

GoKawiil's interpretation of the reporting above, not reported fact.

The findings suggest that picking a harness and model independently could waste significant compute budget, since a harness tuned for one model may rank near the bottom with another, according to GoKawiil's data showing Codex CLI dropping from 1st to 16th place by cost when switching from GLM-5.3 to Kimi-K3. This implies teams building coding agents may need to re-evaluate their harness choice every time they change underlying models rather than treating the two as separate decisions.

Key Takeaways

Source: runta.com, 2026-11-17

Published there as: “In 27,280 coding agent runs, 64% of inference spend went to runs that failed”

Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.
Keep reading
glm-5.3 27 briefs → kimi k3 16 briefs → baseten 5 briefs → codex cli 4 briefs →