GoKawiilWe can learn hidden facts about how frontier models were trained by “probing” them with carefully curated requests.
By scoring them on niche facts we can approximate how many parameters models like GPT-5 and Opus have, using “Incompressible Knowledge Probes”
By measuring how the models break down tokens we can reveal facts about the datasets mixtures they used to train the model (or at least the tokenizer) using “Data Mixture Inference”
By scoring them on date or self-identification related questions you can also estimate training timelines (this post)
Everything here is an estimate. It’s possible that some speculation in this post is totally incorrect given there’s not a ton of publicly available ground truth to verify against.
How frontier models are trained
The 3 main stages of model training.
As a brief primer (see Alex Wa’s blog for more), how we train massive large language models has converaged into 3 stages:
Take a massive amount of general purpose data (aka scrape the internet) and “pre-train” a massive auto-complete model on that data. Use domain-specific “textbook quality” data to improve the base models and extend certain base capabilities like long-text understanding Turn the base model into the “assistant” persona, honing in on its personality, reasoning ability, and tool-calling.
While increasingly more compute is spent on post-training for boosting a model’s reasoning and problem solving, one of the most expensive and data-intensive steps is generating that pre-training checkpoint (by ‘checkpoint’ think of a massive claude-super-secret-2026-11-01-base.cpkt file).
... continue reading