GoKawiilResearch update on compute-efficient pretraining and scaling to trillion-parameter models.
Frontier pretraining is said to be a big-lab-only game. We don’t have 100k chips yet, so there’s only one way: algorithmic efficiency. After compounding for … a while …, our pretraining recipe is now >10x more compute-efficient than that of leading open-weight base models.
We match DeepSeek V4 Pro Base using ~50x fewer FLOPs – that’s around half of GPT3’s pretraining compute, or ~$0.5M on GB200. We continued scaling 10x (~$4M) and meaningfully outperformed all publicly available open base models on perplexity evals. By the scaling laws in Figure 1, training a model this capable would cost >$100M under DeepSeek V4 Pro’s recipe (and this is ignoring how much data exists). Of course, we won’t stop scaling there.
We believe pretraining, agentic RL, and long-context are sufficient to build superhuman coding agents and automate AI R&D. We started with long-context. Today’s blog post is about pretraining.
bits per byte (lower is better) Private Code Repos i Lower bits per byte is better. Training compute is in FLOPs on a logarithmic axis. The curve fits our current recipe; the dashed segment is a projection beyond the largest run. 0.185 0.204 0.223 0.242 0.261 10 21 10 22 10 23 10 24 10 25 DeepSeek V4 Flash: 0.206 bpb DeepSeek V4 Pro: 0.202 bpb Kimi K2: 0.205 bpb Nemotron 3 Ultra: 0.203 bpb V5 e21: 0.246 bpb V5 e22: 0.222 bpb V5 e23: 0.202 bpb V5 e24: 0.194 bpb 29x DSv4 Flash 48x DSv4 Pro 31x Kimi K2 47x Nemotron 3 Ultra Heldout Research Papers i Lower bits per byte is better. Training compute is in FLOPs on a logarithmic axis. The curve fits our current recipe; the dashed segment is a projection beyond the largest run. 0.36 0.41 0.46 0.52 0.57 10 21 10 22 10 23 10 24 10 25 DeepSeek V4 Flash: 0.415 bpb DeepSeek V4 Pro: 0.404 bpb Kimi K2: 0.421 bpb Nemotron 3 Ultra: 0.406 bpb V5 e21: 0.527 bpb V5 e22: 0.463 bpb V5 e23: 0.407 bpb V5 e24: 0.383 bpb 24x DSv4 Flash 45x DSv4 Pro 41x Kimi K2 35x Nemotron 3 Ultra Reasoning on heldout math problems i Lower bits per byte is better. Training compute is in FLOPs on a logarithmic axis. The curve fits our current recipe; the dashed segment is a projection beyond the largest run. 0.53 0.64 0.76 0.87 0.98 10 21 10 22 10 23 10 24 10 25 DeepSeek V4 Flash: 0.710 bpb DeepSeek V4 Pro: 0.678 bpb Kimi K2: 0.695 bpb Nemotron 3 Ultra: 0.619 bpb V5 e21: 0.890 bpb V5 e22: 0.770 bpb V5 e23: 0.647 bpb V5 e24: 0.587 bpb 72x DSv4 Flash 127x DSv4 Pro 58x Kimi K2 15x Nemotron 3 Ultra 6·N·D training FLOPs Figure 1 : Pretraining scaling laws against training compute, comparing to leading available open-weight base models . 1
We measured bits-per-byte loss (a metric that normalizes out differences in tokenizers) on heldout data and fit a scaling law to project how much compute is needed to reach a given level of capability. Better training compute efficiency means stronger models at all budgets.
We evaluated the latest available open-weight base models2 from DeepSeek, Moonshot (Kimi), and NVIDIA. Base models for Claude, Gemini, GPT-n, and many others aren’t openly available, but Kimi K3 and Meta’s Muse Spark indicate a 2.5x and 3.3x gain over Kimi K2, respectively. We evaluated logprobs for open models in both vLLM and SGLang on both GB200 and GB300 and found issues with some backends in the process. For further confirmation, we partnered with Fireworks to verify baseline logprobs in their in-house inference engine. Since models can learn their training parser’s characteristics, we built our eval sets using a different parser/OCR than the one our pretraining pipeline uses.
Evaluating generalization
To measure generalization, we evaluated loss on heldout data (Figure 1). Our code evals consist of our own codebase and private codebases we acquired from other startups. For reasoning evals, we generated CoT and step-by-step walkthroughs to heldout, private math problems using Kimi K3 and filtered for correct answers. For text and research, we used recent, low-citation research papers. We removed vendored OSS code and any document with a matching 96-character window of normalized text or Jaccard similarity above a sensitive threshold compared to our training data.3
Evaluating knowledge
... continue reading