GoKawiilllama.cpp gets up to 42x faster prompt lookup drafting via optimized n-gram caches
Developer Hayder Tirmazi published a set of performance optimizations for llama.cpp's prompt lookup decoding (n-gram speculative drafting), claiming up to 42x faster drafting and up to 2.6x lower memory use. The work draws on techniques from Daniel Lemire and Martin Ankerl, and Lemire subsequently contributed a further patch that Tirmazi says boosts drafting speed by an additional 4.2x, for a combined speedup of up to 140x.
GoKawiil's interpretation of the reporting above, not reported fact.
Prompt lookup decoding is already used across engines like llama.cpp, vLLM, and Hugging Face transformers to speed up token generation without a separate draft model, so faster, leaner n-gram cache implementations could reduce latency and memory overhead broadly across local and server inference deployments. Because the improvements stem from low-level data structure engineering rather than model changes, they suggest meaningful efficiency gains are still available in mature open-source inference stacks through careful systems optimization.
- llama.cpp's prompt lookup (n-gram) drafting sped up by up to 42x through optimizations by Hayder Tirmazi.
- Memory usage for the n-gram caches dropped by up to 2.6x alongside the speed gains.
- A follow-up contribution from Daniel Lemire added a further 4.2x speedup, bringing the combined improvement to as much as 140x.
NVIDIA GeForce RTX 4070 GPU — If you're running llama.cpp with speculative and prompt lookup decoding, a capable GPU makes a huge difference in actual token throughput. The RTX 4070 offers strong VRAM and compute for local LLM inference experiments like the ones described in this article. It's a solid pick for hobbyists optimizing decoding speed on their own machines.
See NVIDIA GeForce RTX 4070 GPU on Amazon → Affiliate link — we may earn a commission on purchases, at no extra cost to you. Product picked by AI based on this article; it is not a tested recommendation.Source: jadidbourbaki.github.io — Hayder Tirmazi, 2026-09-26
Published there as: “Faster prompt lookup drafting in llama.cpp”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.