GoKawiilTypeSafe tests probability calibration of its Jev classifier model
A researcher evaluated Jev, TypeSafe's 'System One' classifier model built by attaching a classification head to a pretrained transformer, to see how well its output probability distributions match known physical answers. The test used questions with mathematically established solutions, such as the Maxwell-Boltzmann velocity distribution for a particle in thermodynamic equilibrium, comparing Jev's predicted probabilities against the known correct distributions. The author notes the entire experiment series cost less than $4.00 to run.
GoKawiil's interpretation of the reporting above, not reported fact.
Unlike generative LLMs, which produce freeform stochastic text poorly suited to classification, Jev returns typed, probability-weighted answers to multiple-choice queries, which could make it useful for tasks needing structured, calibrated outputs rather than plausible-sounding text. Whether its stated probabilities actually reflect real-world likelihoods determines if it can be trusted for applications like scientific estimation or decision support, so this kind of calibration testing matters for assessing practical reliability. The low cost suggests such models could offer an inexpensive alternative to large LLMs for narrow classification tasks, according to the author's informal testing.
- Jev is TypeSafe's new System One classifier model, contrasted with System Two LLMs like Claude or GPT.
- It attaches a classifier head to a pretrained transformer, returning typed, probability-weighted multiple-choice answers rather than freeform text.
- The author tested its calibration against known distributions like Maxwell-Boltzmann, with the full experiment costing under $4.00.
Source: maximumeffort.substack.com — Dylan Black, 2026-10-02
Published there as: “How accurately calibrated is Jev?”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.