Tech News
← Home  ·  All topics

Swe Bench

2 GoKawiil briefs on this topic

New open-source benchmark Parsimony measures code bloat added by AI coding models

A developer has released Parsimony, an open-source benchmark that measures how much code AI models add when solving SWE-bench Verified tasks, rather than just whether they solve them. It parses patches to count syntax-level coding units added and removed, producing a net score, and ranks models by mean net units added on tasks they solved. The tool and underlying data are published on GitHub, with results also shown via an interactive leaderboard and charts.

Analysis flags flaws in popular napkin-math, SWE-Bench and winter tire benchmarks

A review examines three widely cited benchmark resources: the sirupsen/napkin-math GitHub project used for computer performance estimation interviews, a senior-level SWE-Bench coding evaluation, and standardized winter tire test ratings. The piece walks through each dataset's numbers and methodology, inviting readers to form their own estimates before revealing where the figures may be misleading or inconsistent.