Some thoughts on autoregressive models

Most generative AI models nowadays are autoregressive. That means they’re following the concept of next token prediction, and the transformer architecture is the current implementation that has been used for years now thanks to its computational efficiency. This is a rather simple concept that’s easy to understand - as long as you aren’t interested in the details - everything can be tokenized and fed into an autoregressive (AR) model. And by everything, I mean everything: text as you’d expect, but also images, videos, 3D models and whatnot. There is no limit to what can be represented and generated by an autoregressive model, and while pre-training is far from solved, I think it’s fair to say everyone more or less knows what to do. That’s why today’s autoregressive models, “multimodal reasoning general” large language models (LLMs), are statistical models so powerful that we may see traits of generalization. Update: since the article caught a bit of attention on HN, I modified some bi ... Read full article.