Analyst argues 'Minecraft in one prompt' demos have become gameable, not real benchmarks

Following the release of GPT Astra, viral demos such as recreating Minecraft, drawing a pelican on a bicycle as SVG, and simulating a bouncing ball in a rotating box swept social feeds. A commentary piece labels these 'demo-benchmarks': tasks that look impressive and are easy to grasp, but are narrow enough that labs can specifically optimize for them before each launch. It cites Thinking Machines' Inkling Small, a much smaller model that nearly matched or beat its larger sibling on tests like Humanity's Last Exam and GPQA Diamond, as evidence that public, static test sets get gamed through training choices.