METR expands AI benchmark to test research judgment, not just coding speed
METR, the AI research organization known for its time-horizon benchmark, has added new tasks measuring AI judgment on fuzzier work like designing evaluations and scoring philosophical arguments. In both new tests, top AI models performed worse than humans—none of the AI-generated evaluations were deemed useful by METR staff, and models also lagged on judging philosophical arguments.
GoKawiil's interpretation of the reporting above, not reported fact.
The update comes amid heightened attention to AI's role in accelerating its own development, following an earlier METR report suggesting AI may already be doubling research speed at Anthropic. By including tasks that resist easy scoring, METR appears to be acknowledging that its original benchmark undersold the difficulty of automating open-ended research judgment, which could temper fears of imminent full recursive self-improvement.
- METR added new benchmark tasks covering judgment-heavy work like evaluation design and philosophical reasoning.
- AI models underperformed humans on these fuzzier tasks despite excelling at standard software engineering benchmarks.
- The update follows controversy over an earlier METR report on AI accelerating research speed at Anthropic.
Source: technologyreview.com — Grace Huckins, 2026-11-20
Published there as: “The most misunderstood graph in AI just got a major update”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.