2 ms·
The only story is here that some people think they can recognize a hallucination by how the text looks. You can't—the only difference between a hallucination an
by lolinder 3y ago
The only story is here that some people think they can recognize a hallucination by how the text looks. You can't—the only difference between a hallucination and a valid response is that one randomly sampled response happens to produce a factual statement and the other doesn't. There's no stylistic difference, there are no tells. The only way to recognize a hallucination is to fact check with an independent source.
I'm starting to agree with another commenter [0] that the word "hallucination" is a problem—it implies that there's some malfunction that sometimes happens to cause it to produce an inaccurate result, but this isn't a good model of what's happening. There is no malfunction, no psychoactive chemical getting in the way of normal processes. There is only sampling from the model's distribution.
[0] https://news.ycombinator.com/item?id=40033109 https://news.ycombinator.com/item?id=40033109
- fever_ 3y ago'hallucination' is nothing more than GIGO (garbage in garbage out) and it's the 2nd thing everyone learns right behind 'never trust user input'. For some reason everyone has thrown the basics out of the window so now we've got garbage in, garbage out called 'hallucinations' and 'prompt engineering' which is nothing more than being incapable of sanatizing input.
- cma 3y agoYou are saying all hallucinations originate in the training data? Remember as far as GIGO, there is literally a temperature parameter feeding in randomness to the output (though still sampling the probability mass).
- noman-land 3y agoThis is really well said. The malfunction is almost when it spews out something that happens to align with reality. It's actually us who are hallucinating its correct answers, not it who is hallucinating incorrect ones. It's like a stopped clock being right twice a day except if you expand the number of hours on the clock to trillions but it's still right the same percentage of time. The scale of right answers makes this seem impressive to us but it masks the reality of the scale of wrong answers we haven't seen yet.
- cjk2 3y agoWhat about "speaking in tongues"? You have no idea what you are saying but you follow through anyway.
- Ferret7446 3y agoI thought "hallucination" was hilarious when the term was first coined. We already have a word for it: "wrong". It gives off "blackmail is such an ugly word" vibes. It's WRONG. Maybe it's working as intended, maybe it isn't, but it's WRONG.