GoKawiilThe intersection of LLMs and mathematical research is a fascinating one. Mathematics is objective and verifiable, and today’s frontier models have been trained extensively on its concepts and methods. There is a particular sense of joy and wonder in watching LLMs solve previously unsolved (and genuinely difficult problems), while knowing that their capabilities emerge from mechanisms that are, at their core, surprisingly simple.
To me, this is an extraordinary testament to what humans can achieve collectively through ingenuity, hard work, and engineering. Like many others working in this space, I became interested in testing the limits of LLMs on open mathematical problems through my own lens. This post tells the story of that process.
Introduction
You have 4 cards from a standard deck of cards: Ace, King of Hearts and Spades (A♥, K♥, A♠, K♠). You can point to a card and ask one of two possible questions:
Is this card Red?
Is this card a King?
These 4 cards are now shuffled. It is easy to find which card is which by asking 8 questions (each question on each card). However, your goal is to come up with a strategy for asking questions such that you can always find which card is which in under 5 questions. Try your hand at the interactive media below. Drag a question from the right onto any card to learn one fact about it, then click “I know it!” once you know for sure the full arrangement.
24 arrangements left 0 queries used 5 information floor Cards: A♥ K♥ A♠ K♠ — two questions (red?, king?).
Goal: identify the full arrangement in under 5 queries. Shuffle & Play I know it! Questions · drag to a card Red suit? ( ♥♦ ) Is this card a King? Query log Pos Red? King?
This post is about a using an AI to solve an open question posed around a scaled version of this simple puzzle. The rest of the post is organized into these sections:
... continue reading