GoKawiilOur new paper folds the systems stage into the scaling-law stage. Price every candidate architecture on what the cluster actually delivers, and the answer changes: the sparsity an MoE 'should' have depends on the cluster you train it on.
A scaling law converts a compute budget into a model: fix the FLOPs, minimize the predicted-loss curve, read off a parameter count and a token budget. It is the most useful instrument pretraining has, and it is denominated in a currency no cluster accepts. Clusters do not bill FLOPs. They bill GPU-hours.
I have been pulling at this thread on this blog for a while. The post on model size and training horizon ended by converting compute into machine time,
T machine = C train ρ MFU ⋅ ρ goodput ⋅ S , T_{\text{machine}} = \frac{C_{\text{train}}}{\rho_{\text{MFU}} \cdot \rho_{\text{goodput}} \cdot S}, T machine = ρ MFU ⋅ ρ goodput ⋅ S C train ,
and the pretraining research map filed MFU-aware comparison under where scaling laws get hard: two designs can tie on loss-per-FLOP and differ on loss-per-GPU-hour, and the second number is the one on the invoice. Our new paper with Soumajyoti Sarkar and Yuxin Tang takes that observation to its conclusion. The framework, MOSAIC, folds the systems stage into the scaling-law stage: one optimization picks the architecture, the token budget, and the distributed execution layout, for a specific cluster and a specific training window. The title gives away the finding: compute-optimal is not cluster-optimal.
Optimizing the numerator
Look at the denominator of that fraction. Goodput — how often the job runs, after checkpointing, node failures, and input stalls — is a property of the fleet, fixed before any architecture is chosen. Peak throughput S S S is a property of the hardware. MFU — how well the job runs while it runs — is a property of the architecture under its best feasible parallel layout, and it is not a small correction: two designs with identical predicted loss can differ several-fold in realized throughput.
The convention treats it as a constant anyway. A scaling-law stage picks the architecture and the token budget in FLOPs; a systems stage inherits the architecture and tunes the implementation around it. (The Kimi K2 report, for example, treats sparse-MoE scaling laws and systems tuning as separate stages.) That workflow optimizes the numerator while the cluster charges for the whole fraction.
Sparse MoEs are where it breaks
A mixture-of-experts model replaces the dense FFN with a set of experts and a router, which decouples active parameters — the per-token FLOPs — from total parameters — the memory footprint. That introduces axes a Chinchilla-style law cannot see: sparsity, and the expert split factor G G G, which measures how finely the FFN width is partitioned into experts. The paper fits a joint law over total parameters, sparsity, tokens, and G G G on roughly 150 from-scratch MoE pretraining runs, spanning 104M to 2.7B active parameters and total sizes up to 79B.
... continue reading