GoKawiilNew open-source benchmark Parsimony measures code bloat added by AI coding models
A developer has released Parsimony, an open-source benchmark that measures how much code AI models add when solving SWE-bench Verified tasks, rather than just whether they solve them. It parses patches to count syntax-level coding units added and removed, producing a net score, and ranks models by mean net units added on tasks they solved. The tool and underlying data are published on GitHub, with results also shown via an interactive leaderboard and charts.
GoKawiil's interpretation of the reporting above, not reported fact.
Existing coding benchmarks typically reward solve rate and cost but ignore how much code a model writes to get there, even though excess code can make a codebase harder to maintain over time, according to the project's creator. By isolating code volume as its own metric, Parsimony could give developers and researchers a way to weigh conciseness against capability when choosing coding models. The creator notes a measured correlation suggesting models that write more code also tend to solve more tasks, implying a possible tradeoff between terseness and success rate that users may need to consider.
- Parsimony counts syntax-level 'coding units' (statements, calls, literals, etc.) rather than lines of code, with deletions subtracted from additions.
- It reuses patches and pass/fail results from SWE-bench Verified without rerunning tests, applying patches and parsing only Python files.
- Data shows a moderate positive correlation (Spearman 0.33–0.60) between code volume added and task solve rate across tested models.
Source: parsimony.lasu.dev, 2026-10-13
Published there as: “Show HN: Parsimony – an open-source benchmark for how much code AI models add”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.