Researchers introduce 'Lattice of Latents' architecture to boost language model capacity
A new paper proposes the Lattice of Latents, a set of per-layer recurrent latent streams that persist across tokens alongside the standard residual stream running through model depth. Tested on a 406M-parameter Transformer trained on 20B FineWeb-Edu tokens, the method reduced validation perplexity from 15.41 to 14.40, with further gains when combined with hyper-connections and input residuals, down to 13.61. The approach also improved perplexity across Gated DeltaNet, Mamba-2, and hybrid architectures at multiple state sizes.
GoKawiil's interpretation of the reporting above, not reported fact.
The authors frame this as a new scaling axis distinct from deeper layers or looped computation: increasing the number of persistent computational states a model maintains, rather than just how much it computes per token. If the approach generalizes, it could offer model designers another lever for improving efficiency without simply adding depth or parameters, though these results are from research-scale models and have not been validated at production scale.
- Lattice of Latents adds persistent horizontal latent streams alongside standard depth-wise residual connections.
- On a 406M-parameter model, it cut validation perplexity from 15.41 to 14.40, and to 13.61 when combined with other residual enhancements.
- Gains were also observed across Gated DeltaNet, Mamba-2, and hybrid architectures at various state sizes.
Source: shivamduggal4.github.io, 2026-11-17
Published there as: “Latent Lattice Language Models”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.