Tech News
← Home  ·  All topics

Speculative Decoding

4 GoKawiil briefs on this topic

Speculative decoding speeds LLMs by shifting GPU work, not reducing it

A technical explainer argues that speculative decoding does not make large language model inference faster by cutting the number of floating-point operations performed. In fact, total compute (FLOPs) for a given sequence increases under the technique, since a small draft model proposes tokens that a larger model must then verify in a single parallel forward pass.

llama.cpp gets up to 42x faster prompt lookup drafting via optimized n-gram caches

Developer Hayder Tirmazi published a set of performance optimizations for llama.cpp's prompt lookup decoding (n-gram speculative drafting), claiming up to 42x faster drafting and up to 2.6x lower memory use. The work draws on techniques from Daniel Lemire and Martin Ankerl, and Lemire subsequently contributed a further patch that Tirmazi says boosts drafting speed by an additional 4.2x, for a combined speedup of up to 140x.

Developer releases Jevgpt, an open-source small-vocabulary autoregressive chatbot on GitHub

A developer published Jevgpt, a hobby project posted on Show HN, describing it as an autoregressive chatbot built using classifier-based small token vocabularies. The creator says they addressed limited token exploration by adding a small speculative decoding pass, using the same model as both drafter and verifier to expand vocabulary options. The code is available on GitHub under the user giga-james.

AMD tests five speculative decoding methods for vLLM on Instinct MI300X and MI355X GPUs

AMD published results from experiments running speculative decoding in vLLM on its MI300X and MI355X Instinct GPUs using the ROCm software stack. The team compared five drafting approaches—native MTP, Gemma 4 MTP, EAGLE-3, DFlash, and DSpark—which differ in how draft tokens are generated and how they receive signals from the target model.