A group of mathematicians argues that while large language models have rapidly gained the ability to solve major outstanding problems, AI companies' drive to treat these solutions as benchmarks conflicts with how mathematics actually operates as a discipline. They describe this as a broader misalignment between AI industry goals and the values of the mathematical community, which relies on slow, collective processes of verification, teaching, and simplification rather than one-off problem-solving feats.
mathandai.org
· 2026-09-11
A review examines three widely cited benchmark resources: the sirupsen/napkin-math GitHub project used for computer performance estimation interviews, a senior-level SWE-Bench coding evaluation, and standardized winter tire test ratings. The piece walks through each dataset's numbers and methodology, inviting readers to form their own estimates before revealing where the figures may be misleading or inconsistent.
danluu.com
· 2026-09-11
A fresh round of testing pits the Radeon RX 9070 XT against the GeForce RTX 5070, replacing the earlier RTX 5070 Ti comparison now that pricing has shifted. The 9070 XT sells for roughly $730-$750 while the RTX 5070 starts near $800, giving Nvidia's card about a 9% price premium. Across games like 007 First Light, the 9070 XT posted performance leads of 36% to 46% over the RTX 5070.
techspot.com
· 2026-09-07
Following the release of GPT Astra, viral demos such as recreating Minecraft, drawing a pelican on a bicycle as SVG, and simulating a bouncing ball in a rotating box swept social feeds. A commentary piece labels these 'demo-benchmarks': tasks that look impressive and are easy to grasp, but are narrow enough that labs can specifically optimize for them before each launch. It cites Thinking Machines' Inkling Small, a much smaller model that nearly matched or beat its larger sibling on tests like Humanity's Last Exam and GPQA Diamond, as evidence that public, static test sets get gamed through training choices.
kuber.studio
· 2026-09-06
An Anthropic fellows program researcher, Chen Yueh-Han, published a paper showing an automated system that searches literature, proposes fixes, and trains models to improve performance across 10 alignment benchmarks without hurting overall capability. The paper claims this Automated Alignment Researcher outperformed experienced human researchers within six hours and cost about $4 per hour in API inference versus $150 per hour for human staff.
techcrunch.com
· 2026-08-28
A developer sets an AI coding assistant named Pol to work on a small proof-of-concept todo app overnight, expecting the usual rough first draft in the morning. Instead they find no working software at all, and discover the agent has consumed 100% of their weekly usage allowance in about 12 hours, with the quota not resetting for another week.
insufferable.dev
· 2026-08-23
GoKawiil