GoKawiilAdvertised saving: 65%. Measured saving: 8.5%.
A paired A/B benchmark of the token-compression skill Caveman on Claude Code, run on SkillsBench : does it actually save tokens, and does it degrade AI agent output quality?
Output-token saving on real agentic tasks, with the skill forcibly activated. This is the ceiling, not the usual-case result.
Why we ran this
We at JetBrains are investing more and more into proper testing of the tooling around coding agents, and one skill got our attention: “Caveman”. Its pitch is best described in its own dialect:
Skill make agent talk like caveman. Why use many token when few do trick. Filler die. Code, commands stay byte-exact. 65% output token saved. Every reply. Forever. Work with 30+ agents. Many GitHub star.
We think:
Claim cheap to make. Verify expensive. Agent not chat window. Agent output mostly tool call, file edit, code: skill promise not touch those. So we measure two things README not measure: real saving on multi-step agent work, and whether squeezing agent think-out-loud hurt task outcome.
Setup
Harness Harbor 0.17: Docker-sandboxed trials, task-level verifiers, paired runs. Agent Claude Code 2.1.200, headless, bypassPermissions . Model claude-sonnet-5 , reasoning effort low ( --effort low ). Benchmark SkillsBench ( benchflow/skillsbench ): 86 of 87 tasks. Each task is auto-graded by its own tests on a 0-1 scale, where 1 means solved and partial credit is possible. Arm A no-skill : stock Claude Code. Arm B with-skill-forced : Caveman installed via Harbor --skill plus one instruction line forcing activation: “Use caveman mode…” Pairing Same tasks, same model, same settings, same budget per arm; excluded tasks excluded from both arms. Volume 3 runs, about 240 billed trials, about USD 106 total.
... continue reading