Tech News
← Home  ·  All topics

Jevbench

3 GoKawiil briefs on this topic · RSS for jevbench

AWS launches open-source Strands Decider 2B, a Jev-style decision model

Amazon Web Services released Strands Decider 2B, an open-source model built to choose between preset options and output a confidence score rather than generate text. It is small enough to run locally, launched the same week OpenAI unveiled a similar tool, and originated as a side project by AWS distinguished engineer Marc Brooker after he saw TypeSafe's Jev model.

Jeeves, a reasoning-based Jev-style classifier, outperforms Jev and Kev-9B benchmarks

Jeeves is a new 9B-parameter Jev-like decision model built on Qwen3.5-9B with LoRA and a pointer head, trained using SFT and CISPO with a block-4 diffusion drafter. It supports yes/no, multiple-choice, and rating questions through a Jev-compatible API, and reports test accuracy of 0.889 versus 0.822 for Kev-9B and 0.857 for Jev, plus a 0.935 score on JevBench's public tiers versus 0.866 for Jev.

JevBench publishes reproducible per-decision cost benchmark for typed decision models

A Show HN post introduces JevBench, a benchmark that measures typed decision models by cost per 1,000 decisions rather than per token, using actual token counts from 534 v1.2 test decisions. The methodology prices systems with public tariffs at their listed per-token rates, while unlisted open-weight models are priced using OpenRouter or DeepInfra hosting rates for the same or comparable weights, explicitly avoiding raw GPU rental costs. Reported figures include Jev 1.13.0 at $0.0399 per 1,000 decisions, SemIf at roughly $0.022, and Winnow-12B Q8 at roughly $0.037.