GoKawiilGoKawiil study analyzes 17,000 AI coding agent runs to reveal tool preferences
Researchers built 75 synthetic but realistic code repositories across 10 languages, complete with fake company data and real dependency lockfiles, then tasked coding agents like Claude, Codex and Cursor with implementing missing third-party service integrations. Prompts were written from four different user personas—from vague 'vibe-coders' to detail-oriented enterprise engineers—producing 1,163 task variations that were run repeatedly to generate roughly 17,000 total experiment runs.
GoKawiil's interpretation of the reporting above, not reported fact.
This methodology addresses a major blind spot in AI coding benchmarks: most evaluations use static test sets, while this approach measures how agents behave when asked to independently choose and integrate third-party tools and services, mirroring real developer decision-making. Understanding which services agents default to matters for vendors competing for default-recommendation status, and for enterprises worried about compliance or vendor lock-in introduced silently by AI assistants.
- Study used 75 synthetic repositories across 10 programming languages with realistic fake data and verified lockfiles.
- 1,163 task prompts were crafted using four distinct user personas to reflect real-world request styles.
- Roughly 17,000 total agent runs were executed to statistically measure tool and service selection patterns.
O'Reilly "AI Engineering" by Chip Huyen — If benchmarking how coding agents pick tools fascinates you, this book digs into how to actually build and evaluate AI-powered systems. It covers evaluation methodology, prompting, and deployment tradeoffs "the same territory this 17k-run experiment explores." A solid desk reference for anyone shipping agentic workflows.
See O'Reilly "AI Engineering" by Chip Huyen on Amazon → Affiliate link — we may earn a commission on purchases, at no extra cost to you. Product picked by AI based on this article; it is not a tested recommendation.Source: armature.tech — The Armature Team, 2026-09-03
Published there as: “Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.