GoKawiilAnalysis flags flaws in popular napkin-math, SWE-Bench and winter tire benchmarks
A review examines three widely cited benchmark resources: the sirupsen/napkin-math GitHub project used for computer performance estimation interviews, a senior-level SWE-Bench coding evaluation, and standardized winter tire test ratings. The piece walks through each dataset's numbers and methodology, inviting readers to form their own estimates before revealing where the figures may be misleading or inconsistent.
GoKawiil's interpretation of the reporting above, not reported fact.
Engineers and hiring managers often treat these benchmark tables as ground truth when making architecture decisions or evaluating candidates and AI coding models, so unexamined errors can propagate into real technical judgments. The comparison across unrelated domains—hardware latency, AI coding ability, and physical product safety—highlights a broader pattern of benchmarks being trusted more than their underlying rigor deserves.
- The popular napkin-math GitHub repo, used for performance interview prep, contains latency and throughput tables that may need closer scrutiny.
- SWE-Bench, a common AI coding model evaluation, is examined for the same kind of measurement reliability concerns.
- The piece extends its skepticism beyond software, applying similar critical benchmarking logic to winter tire performance testing.
Bridgestone Blizzak WS90 Winter Tires — The article's detour into tire benchmarks is a good reminder that real-world winter grip is measured on ice and snow, not marketing copy — and the Blizzak line is the classic benchmark-setter in dedicated winter rubber. A set swapped on for the cold season transforms braking and cornering confidence compared to all-seasons. Just check your vehicle's tire size before ordering.
See Bridgestone Blizzak WS90 Winter Tires on Amazon → Affiliate link — we may earn a commission on purchases, at no extra cost to you. Product picked by AI based on this article; it is not a tested recommendation.Source: danluu.com, 2026-09-11
Published there as: “Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.