GoKawiilTiny engine, immense model. Run GLM-5.2 (744B-parameter MoE) on a consumer machine with ~25 GB of RAM — in pure C, with zero dependencies, by streaming experts from disk.
$ ./coli chat 🐦 colibrì v1.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU ✓ pronto in 32s · residente 9.9 GB › ciao! ◆ Ciao! 😊 Come posso aiutarti oggi?
The idea
A 744B Mixture-of-Experts model activates only ~40B parameters per token — and only ~11 GB of those change from token to token (the routed experts). So:
the dense part (attention, shared experts, embeddings — ~17B params) stays resident in RAM at int4 (~9.9 GB);
(attention, shared experts, embeddings — ~17B params) stays (~9.9 GB); the 21,504 routed experts (75 MoE layers × 256 experts + the MTP head, ~19 MB each at int4) live on disk (~370 GB) and are streamed on demand, with a per-layer LRU cache, an optional pinned hot-store, and the OS page cache as a free L2.
The engine is a single C file ( c/glm.c , ~1,300 lines) plus small headers. No BLAS, no Python at runtime, no GPU.
What's implemented
Faithful GLM-5.2 ( glm_moe_dsa ) forward — validated token-exact against a transformers oracle (teacher-forcing 32/32, greedy 20/20 on a tiny-random model with the real architecture).
— validated token-exact against a oracle (teacher-forcing 32/32, greedy 20/20 on a tiny-random model with the real architecture). MLA attention (q/kv-LoRA, interleaved partial RoPE) with compressed KV-cache : 576 floats/token instead of 32,768 (57× smaller — GLM-5.2 has 64 heads and no GQA).
... continue reading