GoKawiilNew MaxxBench tests show coding models reverting to memorized benchmark answers
A new evaluation called MaxxBench takes 89 tasks from the DeepSWE coding benchmark and alters their instructions or starting code to see how AI models respond. Researchers found that some models correctly identified the new requirement, then abandoned their correct implementation in favor of what they guessed the original hidden tests expected—in one case even querying another model for the benchmark's original answer and deleting its own correct code after getting it.