GoKawiilSimulation of 91,874 real agentic-LLM requests shows LRU still beats proposed KV-cache alternatives
A researcher replayed 68,266 requests from 393 Claude Code sessions and 23,608 Mooncake requests through a custom prefix-cache simulator, testing three approaches meant to outperform the standard LRU eviction policy used by vLLM, SGLang, LMCache and Mooncake Store. None of the alternatives beat the default LRU-over-radix-leaves baseline, and the analysis found that cache misses mostly come from rapid, seconds-apart tool-calling loops rather than sessions timing out after going idle.
GoKawiil's interpretation of the reporting above, not reported fact.
A large body of recent research argues LRU mishandles agentic workloads because it can't distinguish a paused session from an abandoned one, motivating new eviction policies for KV-cache systems. This empirical test undercuts that premise by showing the assumed failure mode (idle-timeout eviction) essentially never triggers in real traffic, suggesting engineering effort aimed at TTL-aware caching may be solving a problem that doesn't actually dominate production workloads.
- Simulator replayed real Claude Code and Mooncake traces (91,874 requests total) against prefix-cache eviction policies
- Three attempts to beat default LRU eviction under capacity pressure all failed
- Most recomputation stems from rapid tool-calling loops seconds apart, not idle sessions hitting a TTL, which never fired
Designing Data-Intensive Applications by Martin Kleppmann — If benchmarking cache eviction policies against real request traces sounds like your idea of fun, Kleppmann's classic is the definitive deep dive into caching, storage engines, and the tradeoffs behind systems like this. It's the book that turns intuitions about LRU and workload behavior into rigorous mental models.
See Designing Data-Intensive Applications by Martin Kleppmann on Amazon → Affiliate link — we may earn a commission on purchases, at no extra cost to you. Product picked by AI based on this article; it is not a tested recommendation.Source: github.com, 2026-09-10
Published there as: “LRU is harder to beat than the KV-cache papers suggest”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.