GoKawiilAPL AI-Eval Flags Two MMLU Scores as Incomparable Despite Matching Benchmark Name
An analysis by Dmitrii Zatona examines two MMLU evaluation records for the same model family—one build scoring 0.781, another 0.79—that share identical provider, metric and benchmark labels but differ in dataset split, prompt format, grader and runner network access. Under the APL AI-Eval profile, which treats each evaluation as a content-addressed frame with its own hash, a query comparing the two scores returns 'incomparable' rather than a simple +0.009 delta.