4 ms·

Andrej Karpathy on Hallucinations

20 points by sonabinu 3y ago

29athrowaway 3y ago
If LLMs produced music https://www.youtube.com/watch?v=ETTwP0mMmbk https://www.youtube.com/watch?v=ETTwP0mMmbk
alok-g 3y ago
I somehow do not think the 'hallucination' (or confabulation) problem, as defined by common people should be theoretically hard to solve. It may need a lot more computations though. We start with quasi-random initialization of network weights. The weights change under training based on actual data (assumed truthful), but some level of randomization is bound to remain in the network. There would be some weights which settle with low error margins, while some which would have wide error margins. Wider error margins are indicative or higher initial randomness remaining and lesser impact of the training data. Now when a query comes for which the network has not seen enough baking from the training data, the network will keep producing tokens based on weights that have a larger margin. And then, as others have noted, once it picks a wrong token, it will likely keep on the erroneous path. We could, in theory, however, maintain metadata around how each weight changed, and use that to foresee how likely would the network confabulate.
- numeri 3y ago
  It's currently a major area of research and an unsolved problem to find out what individual weights do – and the most recent research seems to suggest that there is not a one-to-one relationship between ideas and weights. In fact, one line of research is showing that each weight encodes multiple ideas with what is called polysemanticity [1], while another line of research seems to show that individual factoids are spread across thousands of weights and possibly even multiple layers [2]. [1]: https://transformer-circuits.pub/2022/toy_model/index.html https://transformer-circuits.pub/2022/toy_model/index.html [2]: https://rome.baulab.info/ https://rome.baulab.info/
  - alok-g 3y ago
    Sure ideas and facts would be spread out. The weights are cumulatively involved in generating the tokens. The error margins from them would also be accumulating for some paths while also decaying for some others. If the training and inference process could track, it should be possible to figure the error margins for the output tokens. However, such an approach would be too compute intensive to be practical.
  - kridsdale3 3y ago
    That really feels similar to genetics. Early on, people assumed a 1:1 relationship between genes and traits. Some are indeed, but many phenotypes are smeared across thousands of genes in a way that is fully mysterious to us.
ChatGTP 3y ago
TLDR I know I'm being super pedantic but the LLM has no "hallucination problem". Hallucination is not a bug, it is LLM's greatest feature. The LLM Assistant has a hallucination problem, and we should fix it. Sounds like a cop out.
- catchnear4321 3y ago
  it stops short of saying that humans should stop anthropomorphizing quite so hard, yes.
zer0c00ler 3y ago
Agreed mostly, although I'm less of a fan of the "dream" terminology. The key takeaway is that we can't trust LLM output and systems need to be designed accordingly. Eg Agents will never be able to securely take actions autonomously based on LLM responses.
65a 3y ago
I really prefer confabulation, because there are a lot of parallels with the human brain (e.g. https://people.uncw.edu/tothj/PSY510/Schnider-Confabulation-NRN-2003.pdf https://people.uncw.edu/tothj/PSY510/Schnider-Confabulation-...). Hallucination conceptually is often about sensing false inputs, whereas confabulation is about creating non-malicious false outputs.

20 points by sonabinu 3y ago

If LLMs produced music https://www.youtube.com/watch?v=ETTwP0mMmbk https://www.youtube.com/watch?v=ETTwP0mMmbk

I somehow do not think the 'hallucination' (or confabulation) problem, as defined by common people should be theoretically hard to solve. It may need a lot more computations though. We start with quasi-random initialization of network weights. The weights change under training based on actual data (assumed truthful), but some level of randomization is bound to remain in the network. There would be some weights which settle with low error margins, while some which would have wide error margins. Wider error margins are indicative or higher initial randomness remaining and lesser impact of the training data. Now when a query comes for which the network has not seen enough baking from the training data, the network will keep producing tokens based on weights that have a larger margin. And then, as others have noted, once it picks a wrong token, it will likely keep on the erroneous path. We could, in theory, however, maintain metadata around how each weight changed, and use that to foresee how likely would the network confabulate.

numeri 3y ago
It's currently a major area of research and an unsolved problem to find out what individual weights do – and the most recent research seems to suggest that there is not a one-to-one relationship between ideas and weights. In fact, one line of research is showing that each weight encodes multiple ideas with what is called polysemanticity [1], while another line of research seems to show that individual factoids are spread across thousands of weights and possibly even multiple layers [2]. [1]: https://transformer-circuits.pub/2022/toy_model/index.html https://transformer-circuits.pub/2022/toy_model/index.html [2]: https://rome.baulab.info/ https://rome.baulab.info/
- alok-g 3y ago
  Sure ideas and facts would be spread out. The weights are cumulatively involved in generating the tokens. The error margins from them would also be accumulating for some paths while also decaying for some others. If the training and inference process could track, it should be possible to figure the error margins for the output tokens. However, such an approach would be too compute intensive to be practical.
- kridsdale3 3y ago
  That really feels similar to genetics. Early on, people assumed a 1:1 relationship between genes and traits. Some are indeed, but many phenotypes are smeared across thousands of genes in a way that is fully mysterious to us.

ChatGTP 3y ago

TLDR I know I'm being super pedantic but the LLM has no "hallucination problem". Hallucination is not a bug, it is LLM's greatest feature. The LLM Assistant has a hallucination problem, and we should fix it. Sounds like a cop out.

catchnear4321 3y ago
it stops short of saying that humans should stop anthropomorphizing quite so hard, yes.

zer0c00ler 3y ago

Agreed mostly, although I'm less of a fan of the "dream" terminology. The key takeaway is that we can't trust LLM output and systems need to be designed accordingly. Eg Agents will never be able to securely take actions autonomously based on LLM responses.

65a 3y ago

I really prefer confabulation, because there are a lot of parallels with the human brain (e.g. https://people.uncw.edu/tothj/PSY510/Schnider-Confabulation-NRN-2003.pdf https://people.uncw.edu/tothj/PSY510/Schnider-Confabulation-...). Hallucination conceptually is often about sensing false inputs, whereas confabulation is about creating non-malicious false outputs.