GoKawiilJevBench publishes reproducible per-decision cost benchmark for typed decision models
A Show HN post introduces JevBench, a benchmark that measures typed decision models by cost per 1,000 decisions rather than per token, using actual token counts from 534 v1.2 test decisions. The methodology prices systems with public tariffs at their listed per-token rates, while unlisted open-weight models are priced using OpenRouter or DeepInfra hosting rates for the same or comparable weights, explicitly avoiding raw GPU rental costs. Reported figures include Jev 1.13.0 at $0.0399 per 1,000 decisions, SemIf at roughly $0.022, and Winnow-12B Q8 at roughly $0.037.
GoKawiil's interpretation of the reporting above, not reported fact.
By standardizing cost comparisons around decisions rather than tokens, the benchmark could make it easier for developers to compare real-world inference economics across differently sized and priced models. The choice to use commercial hosting rates instead of self-hosted GPU costs suggests an attempt to reflect costs realistic for production deployment rather than research experimentation, though this framing is the author's methodological choice rather than an industry standard.
- JevBench measures cost per 1,000 decisions using actual measured input/output tokens, not raw token pricing.
- Models without public pricing are estimated using OpenRouter or DeepInfra rates for identical or comparable weights.
- Reported costs range from about $0.022 to $0.0399 per 1,000 decisions across the compared systems.
Source: benchmarkheaven.com, 2026-09-22
Published there as: “Show HN: JevBench, a reproducible benchmark for typed decision models”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.