Like many people, I got pretty excited when it was discovered that the transformer neural network architecture appeared to break through many years of stagnation in AI research. Chatbots suddenly had emergent capabilities, derived almost entirely from unstructured, unsupervised learning, far surpassing older technologies.
My first experiences were with unreleased models, pre-ChatGPT, and I was seriously impressed. Though these early, small, models would often mess up, even generating streams of garbage text, when they worked they worked. Spookily well. I completely understand why some people at the time thought they were sentient – this is a whole other discussion for another time.
People were saying that this meant that the AI winter was over, and a new era was beginning. I should explain for anyone who hasn't heard that term before, that way back in the day, when early AI research was seemingly yielding significant results, there was much hope, as there is now, but ultimately the technology stagnated. First time around, AI was largely symbolic – this basically means that attempts to model natural language understanding and reasoning were based essentially on hard-coded rules. This worked, up to a point, but it was soon clear that it was simply impractical to build a true AI that way. Human language is too messy for mechanised parsing to work in a general way. Reasoning required far too much world knowledge for it to be practical to write the code by hand, and nobody knew how to extract that knowledge without human intervention.
The other huge problem with traditional AI was that many of its algorithms were NP-complete, which meant that whilst a lot of the time you got a result, often you just didn't, with the algorithm taking an arbitrarily long time to terminate. I doubt anyone can prove this – I certainly wouldn't attempt it – but I strongly suspect that 'true AI', for useful definitions of that term, is at best NP-complete, possibly much worse. Though quantum computing in principle could give some leverage here, none of the technologies currently being built or that are being considered feasible are likely to be useful. Just not enough qubits to represent the kinds of data that would need to be processed – this is a way, way harder problem than trying to reverse encryption secured by the difficulty of prime factorization.
So then came transformers. Seemingly capable of true AI, or, at least, scaling to being good enough to be called true AI, with astonishing capabilities. For the uninitiated, a transformer is basically a big pile of linear algebra that takes a sequence of tokens and computes the likeliest next token. More specifically, they are fed one token at a time, which builds an internal state that ultimately guides the generation of the next token. This sounds bizarre and probably impossible, but the huge research breakthrough was figuring out that, by starting with essentially random coefficients (weights and biases) in the linear algebra, and during training back-propagating errors, these weights and biases could eventually converge on something that worked. Exactly why this works is still somewhat mysterious, though progress has been made.
Transformers aren't killed by the NP-completeness and scaling problems that caused the first AI winter. Technically, a single turn-of-the-handle, generating the next token from the previous token and some retained state, always takes the same amount of time. This inner loop isn't Turing-complete – a simple program with a while loop in it is computationally more powerful. If you allow a transformer to keep generating tokens indefinitely this is probably Turing-complete, though nobody actually does that because of the cost.
Transformers also solved scaling, because their training can be unsupervised (though, practically they do often need supervised training in order to create guardrails against dangerous behaviour). It is now standard practice to train new models on just about every book ever written and everything that can be scraped from the internet.
That's the good news. That was the good news. But we've gone past that point now, and we are now all up against the reality of widespread use of transformers.
All transformers have a fundamental limitation, which can not be eliminated by scaling to larger models, more training data or better fine-tuning. It is fundamental to the way that they operate. On each turn of the handle, transformers emit one new token (a token is analogous to a word, but in practice may represest word parts or even complete commonly used small phrases – this is why chatbots don't know how to spell!). In practice, the transformer actually generates a number for every possible output token, with the highest number being chosen in order to determine the token. This token is then fed back, so that the model generates the next token in the sequence. The problem with this approach is that the model will always generate a token, regardless of whether the context has anything to do with its training data. Putting it another way, the model generates tokens on the basis of what 'looks most plausible' as a next token. If this is a bad choice, and gets fed back, the next token will be generated to match that bad choice. And as the handle keeps turning, the model will generate text that looks plausible. Models are very good at this, because this is what they are trained to do. Indeed, it's all they can do. This is the root of the hallucination problem in transformers, and is unsolveable because hallucinating is all that transformers can do.
I would conjecture that this is another manifestation of the NP-completeness wall that slammed symbolic AI, causing the first AI winter. It's always possible to turn an NP-complete algorithm into one that runs quickly, if you don't mind that it fails to generate any output if you hit a timeout. The transformer equivalent of this is generating plausible, wrong, hallucinated output in cases where it can't pattern match a good result based on its training. The problem, though, is that with traditional AI algorithms you typically know if you've hit a timeout, or if none of your knowledge rules match. With transformers, generating wrong output looks exactly like generating correct output, and there is no way to know which is which.
... continue reading