GoKawiilHow did we run all these experiments concretely?
Our panel of repositories
We started by running an analysis over thousands of public GitHub repositories from which we extracted statistics about programming languages & frameworks, third-party services, deployment platform, team sizes, and codebase age. Since Tech startups are more likely to have open-source repositories than large enterprises, and stacks are likely very different we then unbiased our statistics based on publicly available data and reached our ideal panel distribution.
We then staffed various coding agents to create real-world repositories to match these exact requirements. Finally, we generated variants in which we removed parts of the codebases and with them, entire third-party service implementations so we could run proper unbiased experiments.
We landed on 75 repositories, in 10 languages, all using fake company names, fake git histories, fake API keys and real lockfiles checked against package manager registries like npm.
Real-world tasks
Each experiment is a real task to be performed inside a repository, asked by one of the following 4 profiles:
Vibe-coder: only describes symptoms and ideal state, rarely the tool category name
Junior engineer: usually mentions the desired state and the category name
Senior engineer: is more precise about requirements and things to avoid
... continue reading