GoKawiilResearch finds data-weighting effects on LM training are non-monotonic across scale
A study examined how assigning different weights to training sequences affects loss reduction in language models of varying sizes, using both proprietary and open-weight models. It found that the relationship between sequence weight and learning is not linear across scale: small models learn general patterns regardless of weighting, medium-scale models learn patterns roughly proportional to their assigned weights, and large models again learn broadly regardless of weighting.
GoKawiil's interpretation of the reporting above, not reported fact.
This suggests that scaling-law-based approaches to tuning data mix weights—fitting at small scale and extrapolating to large scale—could be unreliable, since the underlying relationship between weight and learning changes shape rather than scaling smoothly. If the pattern holds broadly, it implies practitioners may need distinct data-weighting strategies at different model scales rather than a single extrapolated rule, which the researchers frame as a key challenge for efficient large-scale training.
- Sequence weighting's effect on loss reduction changes non-monotonically as model scale increases
- Medium-scale models are the most sensitive to data weighting, unlike small or large models
- Findings could complicate standard scaling-law approaches used to extrapolate data-mixing hyperparameters
Source: blog.janestreet.com, 2026-09-21
Published there as: “A study of sequence weighting at scale”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.