APL AI-Eval Flags Two MMLU Scores as Incomparable Despite Matching Benchmark Name
An analysis by Dmitrii Zatona examines two MMLU evaluation records for the same model family—one build scoring 0.781, another 0.79—that share identical provider, metric and benchmark labels but differ in dataset split, prompt format, grader and runner network access. Under the APL AI-Eval profile, which treats each evaluation as a content-addressed frame with its own hash, a query comparing the two scores returns 'incomparable' rather than a simple +0.009 delta.
GoKawiil's interpretation of the reporting above, not reported fact.
The piece argues that a shared benchmark name like MMLU is not enough to guarantee two scores measure the same thing, since undisclosed differences in implementation can swing results by full percentage points rather than fractions. This challenges the common practice of stacking leaderboard numbers side by side and suggests structural validity checks, not just benchmark labels, are needed before treating scores as comparable.
- Two MMLU scores (0.781 vs 0.79) for the same model family were ruled incomparable due to differing eval procedures.
- Benchmark names alone don't specify split, prompt format, grader, or network access, all of which can shift accuracy by points.
- APL AI-Eval's content-addressed framing checks structural comparability but does not verify which score is actually correct.
Source: zatona.dev, 2026-09-14
Published there as: “The Two MMLU Scores: What a Benchmark Name Does Not Fix”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.