GoKawiilBlog post argues LLM-generated confidence scores shouldn't be trusted by AI agents
A technical blog post examines the common practice of AI agent designers asking large language models to output a numeric confidence or probability score alongside their answers. The author argues there is no sound basis for trusting these numbers, contrasting them with how a human forecaster's confidence is grounded in historical data and a consistent updating process.
GoKawiil's interpretation of the reporting above, not reported fact.
If LLM confidence scores are essentially unfounded, agent systems that use them to decide when to defer to a human, retry a task, or flag uncertainty could be making decisions on a false signal. This could matter increasingly as more products embed autonomous agents that rely on self-reported certainty to gate actions. The post suggests developers may need alternative methods, such as calibration testing or ensemble checks, rather than simply trusting a model's stated confidence.
- The post critiques using LLMs to self-report confidence or probability scores
- It contrasts LLM outputs with human forecasters, whose confidence derives from consistent, data-grounded reasoning
- The argument implies agent designers should seek alternative, more rigorous ways to measure certainty
Source: rumple.github.io — Amrut Joshi, 2026-10-21
Published there as: “Using LLM for confidence scoring is a bad idea”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.