4 ms·
When they say "can't hallucinate" they mean they produce a confidence value for every result, so you could see for example it has 0.1 confidence, and you can di
by dbbk 18d ago
When they say "can't hallucinate" they mean they produce a confidence value for every result, so you could see for example it has 0.1 confidence, and you can disregard the result - that'd be different from hallucinating where it believes it's correct
- bradly 17d agoWhat about the LLM calls though that are done midchain? In the Home Assistant video the multi-intent prompt gets split using what looks like a traditional llm model, which I'm assuming is vulnerable to classical hallucinations.
- CompleteSkeptic 18d agothat's right, but because these models are probabilistic, it's also possible to be confidently wrong (and all future models will be smarter still and still have that possibility)
- flockonus 17d agoCorrect. Not to say we're getting into the weeds of probability here as well. "What are the odds a thunder will strike in Paris at 1pm UTC of 2026-09-16" - that could be a 0.001 chance going from blind historical measurements; 0.01 if it's raining; or 1 or 1 after the date has passed.
- janalsncm 18d agoTechnically speaking when you send the prefix “The capital of France is “ into an LLM it will also produce probabilities across its whole vocabulary.
- jiggawatts 17d ago… which they could provide in their APIs but are vehemently opposed to because it makes distillation much easier, and faster.
- bigglebear 17d agoYeah. Yet another reason why open-weight models are better. If I want to use the logits, I can.
- sothatsit 17d agoThe probability values don’t really represent confidence in modern LLMs though, especially after RLHF and RLVR. System One says they use RLCD, Reinforcement Learning for Calibrated Decisions, which presumably has accurate probabilities as an explicit optimisation goal.
- nkozyra 17d agoHow is that different from RLVR?
- sothatsit 17d agoRLVR generally upweights tokens along the whole thinking trace that led to a correct answer, whether each token was "correct" or not. RLVR doesn't train a model to output an 80% likelihood, it just trains it to produce correct answers, and not to produce incorrect ones. System One hasn't said how RLCD works, but they do say it is explicitly training models to output "calibrated" probabilities, which makes it distinct from RLVR. This is how they describe it: > System One models are trained for calibrated decisions: their probabilities are optimized against outcomes to reflect uncertainty.
- porridgeraisin 16d agoIn RLCD (which is now an RL acronym that has 3 different unrelated expansions!), you basically massively negatively reward a distribution that is {yes: 0.9, no: 0.1} if the answer was no, and less negatively reward a {yes: 0.6, no: 0.4}. Many nuances when designing the details, but that is the rough idea. It is a known existing thing variously called "calibrated RL" or such. Implementing it on top of LLMs was difficult to get it to work, they seem to have done it up so its good enough for a polished product that works in a wide variety of usecases at the same time. I got accepted from the waitlist and it's really neat. Edit: it is now on vercel gateway. One thing to note, the out of distribution behaviour will be different from what we are used to with regular LLMs. Theoretically, it should be worse, but practically, it depends on their method.
- orbital-decay 17d agoYeah but what stops it from producing confidently incorrect outputs...
- zenlikethat 17d agoNothing, but imagine using LLMs for a classification task People out there are so resigned to the models being unreliable that they are really doing things like hallucinating deliberately, and then matching the hallucinations to embeddings - https://softwaredoug.com/blog/2026/08/10/hypothetical-classifications https://softwaredoug.com/blog/2026/08/10/hypothetical-classi... You could do that or you could just... use a model that will never produce unreliable outputs in the first place.
- threecheese 17d agoBut we're going from "Apple" to "Apple: 99% - trust me". It could still be an image of an orange :)
- zenlikethat 17d agoIt's pretty darn smart. If you did want to hack on it in earnest and find out for yourself, send me an email - nathan@typesafe.ai
- threecheese 16d agoOh I'm on the waiting list, I was just being a pedantic prick. I can't wait to try it.
- deleted 17d ago[deleted]
- nkozyra 17d agoI'm certainly not resigned to that, at least for classification. Even non-frontier models are absurdly good at this in a broad sense. Which would make it hard to judge "a model that will never produce unreliable outputs in the first place" against something that is already really, really good and exceptional in domain-specific areas with the tiniest amount of elbow grease. Speed and cost look good though (for now)!
- 8note 17d agoif it puts a high confidence value on a wrong answer, thats still hallucinating, no? llm hallucinations are high probability tokens that are incorrect vs the real world
- jubilanti 17d agoCorrect, they have not made a universal all-knowing omniscient oracle, which is what would be required for "can't hallucinate".
- spencerflem 17d agoNot to be tooo pedantic, but a bot that assigned 0 confidence to everything wouldn’t hallucinate. A calculator either gets the right answer or doesn’t answer. It wouldn’t have to be all knowing as long as it knew perfectly what it doesn’t know
- baq 17d agoA quantum calculator answers in distributions.
- stpedgwdgfhgdd 17d agoIn one universe that is true, in another one not.
- baq 17d agoCopenhagen is a beautiful city.
- eru 17d agoThat seems like a weird standard. I would be happy enough with: only produces what it can verify with sources. If you eg try to remember a court case (ie produce the reference via LLM token generation only), it's easy enough to check with your data whether it really exists. Similar for following links and other references. If your data or sources are wrong, obviously your report about them will be wrong. But I wouldn't call that a hallucination.
- darylteo 17d agoI read "hallucinations" as "generates novel output with no grounding/source". i.e. "it just made something completely up". I believe their "accuracy" metric (sonnet 5 level) is where "right/wrong" is measured.
- csomar 17d agoThat's really funny when you consider that generative models also don't hallucinate if you check up on them on every token generated?
- tahaazizi 13d ago[flagged]