Tech News
← Home  ·  All topics

Frontierharness

2 GoKawiil briefs on this topic · RSS for frontierharness

GoKawiil benchmark finds 64% of coding-agent inference spend went to failed runs

GoKawiil, working with Baseten, ran 27,280 coding agent runs across 22 harnesses, 5 models and 62 tasks using 83 billion tokens, expanding its earlier single-model FrontierHarness benchmark. The study found that 64% of total inference spend was consumed by runs that ultimately failed, and that the best-performing harness varied by model, with cost rankings shifting dramatically when models were swapped.

Runta finds 17x cost gap across nine AI agent harnesses testing same model

Runta ran an evaluation called FrontierHarness Eval, testing nine different AI agent harnesses against the same underlying model to compare their performance and efficiency. The results showed the cost per successful task completion varied by as much as 17 times depending on which harness was used, despite the model being identical. Runta is now offering $100 in credits to developers who want to test their own harness on its platform.