GoKawiilBenchmarking Opus 5 on SlopCodeBench
we got better benchmarks
I've written before something along the lines of:
THERE ARE NO GOOD BENCHMARKS for a model's ability to maintain codebase quality
That wasn't entirely true. I love nothing more than burying a good lede.
Last Friday I dug into SlopCodeBench, a new-ish (March 2026) long-horizon coding benchmark from @GOrlanski's lab at UW Madison. It addresses the thing that bothers me most about coding benchmarks - that even "larger" more complex benchmarks still divulge the whole problem up front:
In contrast, each challenge in SlopCodeBench has multiple "checkpoints" - the model doesn't know the whole problem up front, it has to evolve the codebase over time as new requirements are divulged.
It's a good paper. It's not that long. You should read it.
What's cool about this benchmark is that it is unsaturated - at the time of running, the best models available, GPT-5.4 and Opus 4.6, got 11% and 17% strict pass rates, respectively.
benching opus 5 on slopcodebench
... continue reading