Researchers introduce 'Lattice of Latents' architecture to boost language model capacity
A new paper proposes the Lattice of Latents, a set of per-layer recurrent latent streams that persist across tokens alongside the standard residual stream running through model depth. Tested on a 406M-parameter Transformer trained on 20B FineWeb-Edu tokens, the method reduced validation perplexity from 15.41 to 14.40, with further gains when combined with hyper-connections and input residuals, down to 13.61. The approach also improved perplexity across Gated DeltaNet, Mamba-2, and hybrid architectures at multiple state sizes.