GoKawiilA while back I asked in this space what would happen if Dario Amodei was wrong. I want to come back to that, because I think the question matters more now than it did then, and for a reason that has nothing to do with whether I like Dario or his company. I do, for the record. That’s not the point.
The point is a document. In Machines of Loving Grace, Amodei made the case that scaling compute would eventually solve essentially every hard problem in artificial intelligence. Buried in that optimism — or maybe not buried, maybe right out in the open — was a quiet absolution. Hallucinations, the embarrassing tendency of these systems to state falsehoods with total confidence, would take care of themselves. Make the models big enough, train them long enough, and the problem dissolves. You don’t have to solve it. You just have to wait, and spend. And so the entire AI industry breathed a sigh of releif.
I have spent forty years watching this industry, and I know a permission slip when I see one.
Because that is what the essay became, whatever Amodei intended. It gave every other person writing nine- and ten-figure checks a reason not to worry about the one thing that should worry them most. The hallucination problem is the difference between a clever toy and a system a hospital or a bank or a court can actually rely on. It is the whole ballgame for enterprise AI. And the prevailing wisdom, blessed from the top, is that you needn’t address it directly. Scale will provide.
Look at where the money is going and you can see the permission slip being cashed. Stargate, half a trillion dollars. The hyperscalers, tens of billions each per year. The Anthropic–Akamai arrangement, nearly two billion more. The collective bet of the wealthiest companies in the world is that you fix intelligence — including its honesty — by buying more of it. The data center operators are happy. The chip vendors are ecstatic. The labs raising money at valuations with too many zeros are happy. Everyone in that chain has the same incentive, which is to believe that the answer is more.
The customers who will eventually pay for all of it are the ones who should be asking whether any of this is true.
Here is why I think it isn’t. A small company I helped start, 2Brains Inc., set out in 2022 to solve hallucinations — before ChatGPT, before the scaling consensus hardened into received truth, back when the polite assumption was that the problem was simply insurmountable. We did not solve it by waiting for bigger models. We solved it architecturally, by separating the part of the system that generates language from the part that retrieves and verifies facts, and reconciling the two before anything reaches the user. It runs on ordinary processors. It is cheap. And on the industry’s own benchmark for this kind of faithfulness, it more than doubles the published baseline, with no fabricated facts in the verified case at all.
I am not telling you this to sell you anything. I am telling you because of what it implies about the trillion-dollar bet.
If a handful of people in Virginia and Kansas could solve hallucinations with an architecture and a CPU, then one of two things must be true about the scaling story, and neither is comfortable for the people cashing the permission slip.
The first possibility is that scaling will not cure hallucinations at all. That the models get bigger and more fluent and more useful, and continue, reliably, to lie. In that case the largest companies in the world are spending a fortune chasing a cure that is not coming, and the absolution Amodei offered turns out to have been the most expensive sentence in the history of the field.
... continue reading