GoKawiilThe top-performing artificial-intelligence model scored 6 out of 10 in the First Proof set of mathematical challenges.Credit: vitacopS/Getty
Artificial intelligence has undergone its most scrupulous maths test yet. The results are in, and the AI models that took part didn’t live up to the problem-solving skills of top mathematicians.
The test — part of a project called First Proof, which aims to evaluate the ability of AI to solve complex questions in mathematics — posed ten research-level maths problems to four AI systems. A jury of anonymous human specialists in the relevant mathematical fields then assessed the models’ answers. This test was the first of its kind to satisfy three key conditions simultaneously: first, it consisted of research-level maths questions; second, it involved problems that did not appear in the training data; and third, it was formally graded by mathematicians. The results were unveiled on the First Proof website on 10 June.
These findings follow recent AI breakthroughs in solving maths problems. Last month, for example, a chatbot made by the technology firm OpenAI, in San Francisco, California, solved an 80-year-old maths challenge set by the late mathematician Paul Erdős. The First Proof team says that future iterations of the test could help researchers to judge how useful AI models could be for mathematicians; for example, in solving problems autonomously, checking proofs or acting as research assistants.
Prove this
One important innovation of the First Proof test was that the questions had not previously been mentioned anywhere in the published literature or on the Internet — cutting the risk that the models could simply be regurgitating information they had learnt during their training. Instead, ten researchers from a broad range of mathematical specialities each provided a question that they had solved in the course of their own research but had not yet published.
First Proof ran a trial test in February with a different batch of novel problems. In that round, anyone could try their own favourite AI systems on the problems, and many groups did — but the results were not officially verified by the First Proof team. There was also no way to independently check that the AIs had not received any help from humans.
AI cracks 80-year-old mathematics challenge — researchers are astonished
This time, First Proof ran the test itself: the team asked the models to solve problems in an entirely autonomous way, and had a group of 30 mathematicians vet the answers. “The organizers have clearly thought through the second batch more carefully to make it more controlled and systematic,” says mathematician Jeremy Avigad, who heads the Institute for Computer-Aided Reasoning in Mathematics at Carnegie Mellon University in Pittsburgh, Pennsylvania.
Another rule was that the participating models had to be publicly available. This meant that Google’s Aletheia — a system designed specifically for solving maths problems — and the full, unreleased version of Claude Mythos, a model made by Anthropic in San Francisco, California, could not be used. OpenAI was the only big company that participated, with its model ChatGPT 5.5 Pro.
... continue reading