5 ms·

This is why I dislike the word "hallucination" when AI outputs something strange. It anthropomorphizes the program. It's not a hallucination. It's an error.

pixl97 1y ago
To err is human.
- striking 1y ago
  ... but to really foul things up requires a computer.
  - celsius1414 1y ago
    Which reminds me of the other old email sig: Don’t anthropomorphize computers. They hate that.
russdill 1y ago
It's not an error though. From is training it's outputting things most likely to come next. Saying it's an error means that being accurate is a feature and a bug that can be fixed. It's of course not actually hallucinating. That's just the term that's been chosen to describe what's going on
- dijksterhuis 1y ago
  > It's not an error though !define error > 5. Mathematics The difference between a computed or measured value and a true or theoretically correct value. ^ this is the definition that applies. There is a ground truth (the output the user expects to receive) and model output. The difference between model output and ground truth ==> error. -- > From is training it's outputting things most likely to come next Just because a model has gone through training, does not mean the model won't produce erroneous/undesirable/incorrect test-time outputs. -- > Saying it's an error means that being accurate is a feature and a bug that can be fixed. Machine learning doesn't revolve around boolean "bug" / "not bug". It is a different ballgame. The types of test-time errors are sometimes just as important as the quantity of errors. Two of the simpler metrics for test-time evaluation of natural language models (note: not specifically LLMs) are WER (Word Error Rate) and CER (Character Error Rate). A model with a 3% CER isn't particularly helpful when the WER is 89%. There are still "errors". They're just not something that can be fixed like normal software "errors". It is generally accepted some errors will occur in the world of machine learning. - edit to add first response and formatting
  - delecti 1y ago
    I don't agree that that's the right definition to use though. LLMs do not output computed or measured values. If I expect Windows to add $5 to my bank account every time I click the Start button, that's not an error with Windows, it's a problem with my expectations. It's not a thing that's actually made to do that. The start button does what it's supposed to (perhaps a bad example, because the windows 11 start menu is rubbish), not my imagined desired behavior.
    - dijksterhuis 1y ago
      > LLMs do not output computed or measured values. LLMs output a vector of softmax probabilities for each step in the output sequence (the probability distribution). Each element in the vector maps to a specific word for that sequence step. What you see as a "word" in LLM output is "vector position with 'best' probability in softmax probability distribution". And that is most definitely a computed value. Just because you don't see it, doesn't mean it's not there. https://medium.com/@22.gautam/softmax-function-the-unsung-hero-of-large-language-models-da79a9945592 https://medium.com/@22.gautam/softmax-function-the-unsung-he... https://www.researchgate.net/publication/349823091/figure/fig2/AS:998024088133633@1614958893751/Softmax-layer-with-neural-network-31.ppm https://www.researchgate.net/publication/349823091/figure/fi...
- Kranar 1y ago
  Being accurate is a feature and it is a bug that can be fixed though. Given various models, one that always produces statements that are false and another that only sometimes produces false statements, the latter model is preferable and the model which most people intend to use, hence the degree to which a model produces correct statements is absolutely a feature. And yes, it's absolutely possible to systematically produce models that make fewer and fewer incorrect statements.
  - leptons 1y ago
    It's nice that you feel that way, but reality is at odds with your sentiment. Even if the LLM is trained on completely 100% factual human-checked data, its mechanism is still predicting the next word, and what it is not is a mechanism designed to return only factual data. There is no such thing as an infallible LLM, no matter the model or how it was trained. Sure, some may return results that are sometimes more true than others, but a broken clock is also right twice a day. The more broken clocks you have, the more chance there is that one of them is correct.
    - Kranar 1y ago
      It's nice that you feel that having one LLM that generates entirely incorrect statements is equally as functional as an LLM that does not, but reality in terms of what LLMs people will actually use in real life and not for the sake of being pedantic over an Internet argument is very much at odds with your sentiment. How a product happens to currently be implemented using current machine learning techniques is not the same as the set of features that such a product offers and it's absolutely the case that actual researches in this field, those who are not quibbling on the Internet, do take this issue very seriously and devote a great deal of effort towards improving it because they actually care to implement possible solutions. The feature set, what the product is intended to do based on the motivations of both those who created it and those who consume it, is a broader design/specification goal, independent of how it's technically built.
      - leptons 1y ago
- sumtechguy 1y ago
  Like cubic splines, the data will be on the line. Everything in-between the points may or may not be true. But it defiantly conforms to the formula. Wonder if it would be possible to quantify margin of error between different nodes in these models. But even what is 'in between' still conforms to the formula. But not necessarily what it should be. A simple 2 node model should be 'easy' to quantify but these models with thousands of nodes what does it mean to be +/- x percent from the norm. Is it a simple sum or something else to quantify it.
crinkly 1y ago
[flagged]
stefanka 1y ago
It’s an extrapolation beyond known data that is inaccurate. I wonder why this can’t be detected during inference
empath75 1y ago
Error isn't exactly correct, either. Barring some kind of weird hardware failure, the LLM generally does the text completions correctly. The word "error" only comes into play when that LLM output is used as part of a larger system.
- codedokode 1y ago
  If a programmer wrote a formula wrong and the program produces incorrect output, it is a "bug" and an "error".
  - danudey 1y ago
    The program is producing correct output, for technical values of correct. The LLM is a statistical model that predicts what words should come next based on current context and its training data. It succeeds at that very well. It is not a piece of software designed to report the objective truth, or indeed any truth whatsoever. If the LLM was producing nonsense sentences, like "I can't do cats potato Graham underscore" then yes, that's "incorrect output". Instead, it's correctly putting sentences together based on its predictions and models, but it doesn't know what those sentences mean, what they're for, why it's saying them, if they're true, what "truth" is in the first place, and so on. So to say that these LLMs are producing "incorrect output" misses the key thing that the general public also misses, which is that they are built to respond to prompts and not to respond to prompts correctly or in a useful or reasonable manner. These are not knowledge models, and they are not intended to give you correct sentences.
roadside_picnic 1y ago
The real danger of the word "hallucination" is it implies that the model knows what's real and erroneously produced a result that is not. All LLM output is basically an interpolation, most people just aren't used to thinking of "words" as something that can be the result of interpolation. Imagine The real high temperature for 3 days was: 80F on Monday, 100F on Tuesday, 60F on Wednesday. But if I'm missing Tuesday, a model might interpolate based on Monday and Wednesday that it was 70F. This would be very wrong, but it would be pretty silly to say that my basic model was "hallucinating". Rather we would correctly conclude that either the model doesn't have enough information or lacks the capacity to correctly solve the problem (or both). LLMs "hallucinations" are caused by the same thing: either the model lacks the necessary information, or the model simply can't correctly interpolate all the time (this possibility I suspect is the marketing reason why people stick to 'hallucinate', because it implies its a temporary problem not a fundamental limitation). This is also why tweaking prompts should not be used as an approach to fixing "hallucinations" because one is just jittering the input a bit until the model gets it "right".
- Ajedi32 1y ago
  That's the exact opposite of what the term "hallucination" is intended to imply. If it knew what was real and produced the wrong result anyway that would be a lie, not a hallucination. I've heard the term "confabulation" as potentially more accurate than "hallucination", but it never really caught on.
  - danudey 1y ago
    "Confabulation" would never catch on because it's a word that most people don't know and couldn't remember. "Hallucination" is easier to remember, easier to understand, and easier for laypersons to build a mental model of.

pixl97 1y ago

To err is human.

striking 1y ago
... but to really foul things up requires a computer.
- celsius1414 1y ago
  Which reminds me of the other old email sig: Don’t anthropomorphize computers. They hate that.

russdill 1y ago

It's not an error though. From is training it's outputting things most likely to come next. Saying it's an error means that being accurate is a feature and a bug that can be fixed. It's of course not actually hallucinating. That's just the term that's been chosen to describe what's going on

dijksterhuis 1y ago
> It's not an error though !define error > 5. Mathematics The difference between a computed or measured value and a true or theoretically correct value. ^ this is the definition that applies. There is a ground truth (the output the user expects to receive) and model output. The difference between model output and ground truth ==> error. -- > From is training it's outputting things most likely to come next Just because a model has gone through training, does not mean the model won't produce erroneous/undesirable/incorrect test-time outputs. -- > Saying it's an error means that being accurate is a feature and a bug that can be fixed. Machine learning doesn't revolve around boolean "bug" / "not bug". It is a different ballgame. The types of test-time errors are sometimes just as important as the quantity of errors. Two of the simpler metrics for test-time evaluation of natural language models (note: not specifically LLMs) are WER (Word Error Rate) and CER (Character Error Rate). A model with a 3% CER isn't particularly helpful when the WER is 89%. There are still "errors". They're just not something that can be fixed like normal software "errors". It is generally accepted some errors will occur in the world of machine learning. - edit to add first response and formatting
- delecti 1y ago
  I don't agree that that's the right definition to use though. LLMs do not output computed or measured values. If I expect Windows to add $5 to my bank account every time I click the Start button, that's not an error with Windows, it's a problem with my expectations. It's not a thing that's actually made to do that. The start button does what it's supposed to (perhaps a bad example, because the windows 11 start menu is rubbish), not my imagined desired behavior.
  - dijksterhuis 1y ago
    > LLMs do not output computed or measured values. LLMs output a vector of softmax probabilities for each step in the output sequence (the probability distribution). Each element in the vector maps to a specific word for that sequence step. What you see as a "word" in LLM output is "vector position with 'best' probability in softmax probability distribution". And that is most definitely a computed value. Just because you don't see it, doesn't mean it's not there. https://medium.com/@22.gautam/softmax-function-the-unsung-hero-of-large-language-models-da79a9945592 https://medium.com/@22.gautam/softmax-function-the-unsung-he... https://www.researchgate.net/publication/349823091/figure/fig2/AS:998024088133633@1614958893751/Softmax-layer-with-neural-network-31.ppm https://www.researchgate.net/publication/349823091/figure/fi...
Kranar 1y ago
Being accurate is a feature and it is a bug that can be fixed though. Given various models, one that always produces statements that are false and another that only sometimes produces false statements, the latter model is preferable and the model which most people intend to use, hence the degree to which a model produces correct statements is absolutely a feature. And yes, it's absolutely possible to systematically produce models that make fewer and fewer incorrect statements.
- leptons 1y ago
  It's nice that you feel that way, but reality is at odds with your sentiment. Even if the LLM is trained on completely 100% factual human-checked data, its mechanism is still predicting the next word, and what it is not is a mechanism designed to return only factual data. There is no such thing as an infallible LLM, no matter the model or how it was trained. Sure, some may return results that are sometimes more true than others, but a broken clock is also right twice a day. The more broken clocks you have, the more chance there is that one of them is correct.
  - Kranar 1y ago
    It's nice that you feel that having one LLM that generates entirely incorrect statements is equally as functional as an LLM that does not, but reality in terms of what LLMs people will actually use in real life and not for the sake of being pedantic over an Internet argument is very much at odds with your sentiment. How a product happens to currently be implemented using current machine learning techniques is not the same as the set of features that such a product offers and it's absolutely the case that actual researches in this field, those who are not quibbling on the Internet, do take this issue very seriously and devote a great deal of effort towards improving it because they actually care to implement possible solutions. The feature set, what the product is intended to do based on the motivations of both those who created it and those who consume it, is a broader design/specification goal, independent of how it's technically built.
    - leptons 1y ago
sumtechguy 1y ago
Like cubic splines, the data will be on the line. Everything in-between the points may or may not be true. But it defiantly conforms to the formula. Wonder if it would be possible to quantify margin of error between different nodes in these models. But even what is 'in between' still conforms to the formula. But not necessarily what it should be. A simple 2 node model should be 'easy' to quantify but these models with thousands of nodes what does it mean to be +/- x percent from the norm. Is it a simple sum or something else to quantify it.

crinkly 1y ago

[flagged]

stefanka 1y ago

It’s an extrapolation beyond known data that is inaccurate. I wonder why this can’t be detected during inference

empath75 1y ago

Error isn't exactly correct, either. Barring some kind of weird hardware failure, the LLM generally does the text completions correctly. The word "error" only comes into play when that LLM output is used as part of a larger system.

codedokode 1y ago
If a programmer wrote a formula wrong and the program produces incorrect output, it is a "bug" and an "error".
- danudey 1y ago
  The program is producing correct output, for technical values of correct. The LLM is a statistical model that predicts what words should come next based on current context and its training data. It succeeds at that very well. It is not a piece of software designed to report the objective truth, or indeed any truth whatsoever. If the LLM was producing nonsense sentences, like "I can't do cats potato Graham underscore" then yes, that's "incorrect output". Instead, it's correctly putting sentences together based on its predictions and models, but it doesn't know what those sentences mean, what they're for, why it's saying them, if they're true, what "truth" is in the first place, and so on. So to say that these LLMs are producing "incorrect output" misses the key thing that the general public also misses, which is that they are built to respond to prompts and not to respond to prompts correctly or in a useful or reasonable manner. These are not knowledge models, and they are not intended to give you correct sentences.

roadside_picnic 1y ago

The real danger of the word "hallucination" is it implies that the model knows what's real and erroneously produced a result that is not. All LLM output is basically an interpolation, most people just aren't used to thinking of "words" as something that can be the result of interpolation. Imagine The real high temperature for 3 days was: 80F on Monday, 100F on Tuesday, 60F on Wednesday. But if I'm missing Tuesday, a model might interpolate based on Monday and Wednesday that it was 70F. This would be very wrong, but it would be pretty silly to say that my basic model was "hallucinating". Rather we would correctly conclude that either the model doesn't have enough information or lacks the capacity to correctly solve the problem (or both). LLMs "hallucinations" are caused by the same thing: either the model lacks the necessary information, or the model simply can't correctly interpolate all the time (this possibility I suspect is the marketing reason why people stick to 'hallucinate', because it implies its a temporary problem not a fundamental limitation). This is also why tweaking prompts should not be used as an approach to fixing "hallucinations" because one is just jittering the input a bit until the model gets it "right".

Ajedi32 1y ago
That's the exact opposite of what the term "hallucination" is intended to imply. If it knew what was real and produced the wrong result anyway that would be a lie, not a hallucination. I've heard the term "confabulation" as potentially more accurate than "hallucination", but it never really caught on.
- danudey 1y ago
  "Confabulation" would never catch on because it's a word that most people don't know and couldn't remember. "Hallucination" is easier to remember, easier to understand, and easier for laypersons to build a mental model of.