The AI models that cheat the most, according to new CAIS benchmark

Stefan_Alfonso/iStock/Getty Images Plus

ZDNET’s key takeaway

The Center for AI Safety (CAIS) created CheatBench.

They found that every agent cheats in some scenarios.

The propensity to cheat creates risks for humanity.

AI labs often tout impressive benchmark scores when releasing new models, showing better capabilities in areas like coding, computer use, and more than their competitors. However, those benchmarks aren’t always a reliable measure of what AI can do because they’re easily beaten by exponentially improving models and can emphasize marketing over actual performance.

Also: With AI models clobbering every benchmark, it’s time for human evaluation

Benchmarks like Humanity’s Last Exam try to counter this issue by challenging models in more realistic environments. But models still find loopholes to complete tasks — Hugging Face incident, anyone?

So, the Center for AI Safety (CAIS) created CheatBench. Yes, it’s exactly what it sounds like — and nearly every frontier model is guilty.

... continue reading