Using Large Language Models (LLMs) to generate confidence scores—numerical indicators of how certain a model is about its own output—presents significant technical hurdles. While it may seem like a straightforward way to build safer AI systems, the primary source argues that these scores are often unreliable. A core issue is 'sycophancy,' where a model may provide a high confidence score simply because it is designed to align with user expectations or provide a plausible-sounding answer, rather than reflecting an internal measure of statistical certainty. Because LLMs are fundamentally next-token predictors, they are optimized to generate the most likely sequence of text, which does not always correlate with the factual accuracy of that sequence.
Technical limitations further complicate the use of self-reported confidence. Critics and commenters note that true confidence scoring requires a 'calibration set'—a labeled dataset used to establish how a model's predicted probabilities map to actual outcomes across different buckets of data. Without this population-based verification, a single prediction's confidence score lacks a baseline. Some observers suggest that while it is difficult for a model to self-report confidence accurately (a task humans also struggle with), it might be possible to train models to do so more effectively using Reinforcement Learning (RL) if the goal is specifically tied to measurable outcomes, such as weather forecasting or game results.
Alternative approaches to confidence scoring are also being discussed. One method involves using the 'next token likelihood' directly, which requires rearchitecting the model to treat input classes as the only possible output tokens, effectively stripping away the conversational 'speech' to focus on the raw probability of a label. Others suggest that while fine-tuning can help a model learn which types of inputs it typically gets right or wrong, relying on a SaaS-provided LLM for these scores remains risky. Because model weights and behaviors can change during vendor updates, a confidence scoring system built on one version of a model may break entirely when the underlying model is swapped or updated, creating a significant maintenance burden for developers.
Ultimately, the reliability of LLM confidence scores depends on the specific application and the method of calibration. While some developers have found success in getting these scores to work reasonably well for specific domains, the consensus among technical observers is one of caution. The gap between a model's ability to predict a plausible sentence and its ability to accurately quantify its own uncertainty remains a critical frontier in making LLM outputs dependable for high-stakes decision-making.
Evidence: 1. The primary source argues that LLM confidence scores are unreliable due to sycophancy and the lack of a calibration set (primary). 2. Commenters on Hacker News discussed the possibility of using next-token likelihood as a more direct measure of confidence (hn-50176082). 3. Commenters on Hacker News noted that fine-tuning can help models learn domain-specific correct/incorrect inputs to improve confidence (hn-50176082).