GoKawiilNeuralwatt finds agent harness design drives most of AI inference's energy cost
Neuralwatt CEO Chad Gibson analyzed a month of cloud traffic and found that different agent harnesses produce wildly different prompt-cache hit rates, from 1.6% to 11.1% uncached input tokens. The study found that a cache miss roughly doubles a request's energy use, and that deep sessions past 128k tokens with major prompt re-reads can cost as much compute as roughly 600 normal turns at that depth.
GoKawiil's interpretation of the reporting above, not reported fact.
The findings suggest that efficiency gains in AI inference may depend less on model architecture or GPU hardware than on how client-side harnesses manage and edit prompts before sending them. Gibson attributes nearly all observed cache breaks to harnesses re-sending previously submitted content, implying that fixing harness design could meaningfully cut energy and compute costs without changing models or hardware at all.
- Uncached input tokens ranged from 1.6% to 11.1% depending on the harness used
- A cache miss roughly doubles energy use per request, and about 2.5x for GLM-5.3
- Deep sessions with major prompt re-reads past 128k tokens can cost as much as 600 normal turns
Source: neuralwatt.com — Neuralwatt Inc., 2026-10-16
Published there as: “Why harnesses matter for energy (and token) efficiency”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.