Tech News
← Home  ·  All topics

Mamba 2

1 GoKawiil brief on this topic · RSS for mamba 2

Researchers introduce 'Lattice of Latents' architecture to boost language model capacity

A new paper proposes the Lattice of Latents, a set of per-layer recurrent latent streams that persist across tokens alongside the standard residual stream running through model depth. Tested on a 406M-parameter Transformer trained on 20B FineWeb-Edu tokens, the method reduced validation perplexity from 15.41 to 14.40, with further gains when combined with hyper-connections and input residuals, down to 13.61. The approach also improved perplexity across Gated DeltaNet, Mamba-2, and hybrid architectures at multiple state sizes.