The Two MMLU Scores: What a Benchmark Name Does Not Fix

Copy page

The Two MMLU Scores: What a Benchmark Name Does Not Fix

Date: September 6, 2026 · Author: Dmitrii Zatona

Two MMLU accuracies, 0.781 and 0.79 , for two builds of one model family under the same benchmark name; for a score-delta query the verifier returns incomparable (Sections 1 and 5).

and , for two builds of one model family under the same benchmark name; for a query the verifier returns (Sections 1 and 5). mmlu fixes a name. The split, the implementation, the prompt format, the grader and the runner’s network access stay open, and where published measurements exist for them the differences are points of accuracy, not thousandths (Section 2).

fixes a name. The split, the implementation, the prompt format, the grader and the runner’s network access stay open, and where published measurements exist for them the differences are points of accuracy, not thousandths (Section 2). Comparability is a property of the reference the results are traceable to, not of the number (Section 3).

Under the APL AI-Eval profile the frame is a content-addressed object and the claim carries its hash; the two frames differ, and subset: "all" and an omitted key are different scopes by canonical bytes (Section 4).

and an omitted key are different scopes by canonical bytes (Section 4). apl-valid is a statement about structure and says nothing about whether either score is correct (Section 6).

Two evaluation records appear in the same table. One reports mmlu accuracy 0.781 for build 42; the other reports 0.79 for build 44. The claims declare the same provider, model family, metric identifier, unit and benchmark name. The arithmetic difference is +0.009 .

The records are structurally valid. Their frames declare different runners, graders and dataset splits. The shared mmlu label identifies a dataset family, not a full measurement procedure. For the score-delta query shown below, the verifier returns incomparable .

... continue reading