GoKawiilSpeculative decoding speeds LLMs by shifting GPU work, not reducing it
A technical explainer argues that speculative decoding does not make large language model inference faster by cutting the number of floating-point operations performed. In fact, total compute (FLOPs) for a given sequence increases under the technique, since a small draft model proposes tokens that a larger model must then verify in a single parallel forward pass.