3 ms·

> But as it started consuming more and more data that it had produced itself, its reliability became close to none. It's exactly the opposite with LLMs. See th

> But as it started consuming more and more data that it had produced itself, its reliability became close to none. It's exactly the opposite with LLMs. See the "model collapse" phenomenon (https://www.nature.com/articles/s41586-024-07566-y https://www.nature.com/articles/s41586-024-07566-y). > We show that, over time, models start losing information about the true distribution, which first starts with tails disappearing, and learned behaviours converge over the generations to a point estimate with very small variance. Furthermore, we show that this process is inevitable, even for cases with almost ideal conditions for long-term learning, that is, no function estimation error

eikenberry 2y ago
Aren't those saying the same thing? .. "its reliability became close to none" vs. "causes irreversible defects in the resulting models"
- renewiltord 2y ago
  There’s thing that happens in humans called “hallucination” where they just make up stuff. You can’t really just take it at face value. Sometimes, they’re just overfit so they generate the same tokens independent of input.
  - Guthur 2y ago
    That has never been the definition of hallucination, until LLMs. It's actually just called lying, dishonesty or falsehood. Hallucination is a distortion (McKenna might say liberation) of perception. If I hallucinate being covered in spiders, I don't necessarily go around saying, "I'm covered in spiders, if you cant see them you're blind" (disclaimer: some might, but that's not a prerequisite of an hallucination). The cynic in me thinks that use of the word hallucination is marketing to obscure functional inadequacy and reinforce the illusion that LLMs are some how analogous to human intelligence.
    - radlad 2y ago
      Lying, dishonesty, and falsehood all imply motive/intent, which is not likely the case when referring to LLM hallucinations. Another term is "making a mistake," but this also reinforces the similarities between humans and LLMs, and doesn't feel very accurate when talking about a technical machine. Sibling commenter correctly calls out the most similar human phenomenon: confabulation ("a memory error consisting of the production of fabricated, distorted, or misinterpreted memories about oneself or the world" per Wikipedia.)
      - usual_user 2y ago
        IMHO Lying means thinking one thing and saying another, hiding your true internal state. For it to be effective it also seems to require something like “theory of mind” (what does the other person know / think that I know).
    - jfim 2y ago
      I believe the term hallucination comes from vision models, where the model would "hallucinate" an object where none exists.
      - alcover 2y ago
        Wouldn't 'illusion' be the more precise term here ? When one thinks he recognizes something, whereas 'hallucination' is more of unreal appearing out of the blue ?
      - dijksterhuis 2y ago
        You might be thinking of deepdream... https://en.wikipedia.org/wiki/DeepDream https://en.wikipedia.org/wiki/DeepDream "Hallucinations" have only really been a term of art with regards to LLMs. my PhD in the security of machine learning started in 2019 and no-one ever used that term in any papers. the first i saw it was on HN when ChatGPT became a released product. Same with "jailbreaking". With reference to machine learning models, this mostly came about when people started fiddling with LLMs that had so-called guardrails implemented. "jailbreaking" is just another name for an adversarial example (test-time integrity evasion attack), with a slightly modified attacker goal.
  - goatlover 2y ago
    Confabulation is the accurate psychological term. Hallucination is a perceptual issue. The LLM term is misleading.
    - moffkalast 2y ago
      Confabulation would be actual false memories no? I suppose some of it are consistent false beliefs, but more often than not it's well.. lapsus. One token gets generated wrong, or the sampler picks something mindbogglingly dumb that doesn't make any sense because of high temperature, and the model can't help but try and continue as confidently as it can, pretending everything is fine without any option to correct itself. Some thinking models can figure these sort of mistakes out in the long run but it's still not all that reliable and requires training it that way from the base model. Confident bullshitting seems to be very ingrained in current instruct datasets.
  - ben_w 2y ago
    Getting high on their own supply. It's a problem when humans do this. That AI also do it… is interesting… but AI's failure is not absolved by human failure.
- qwertox 2y ago
  I think parent may have understood it as "second to none", as in "exceptional". That is at least the way I struggled with that sentence in the paper.
threeducks 2y ago
The dangers of "model collapse" are wildly overstated. Sure, if you feed the unfiltered output of an LLM into a new LLM, inbreeding will eventually collapse the model, but if you filter the data with some grounding in reality, the results will get better and better. The best example is probably AlphaGo Zero, which was grounded with the rules of the game and did not even receive any supervised data. For programming, grounding can happen by simply executing the code. But even if you do not have grounding, you can still just throw a lot of test time compute at the output to filter out garbage, which is probably good enough.
- wruza 2y ago
  Same for stable diffusion (being the same tech). I trained models purely on model outputs and didn’t see any horrific handwavy things everyone’s talked about. You can’t get more of what’s already there, obviously. But as a way to bake some prompt / control pic in, it works great. I actually find/feel this easier to achieve the same result, because I can vary the output with dynamic prompting and seeds, models, etc, while keeping the desired concept intact, with a little picking. This helps to better distill it into a model. Natural imagery is usually too diverse and requires lots of effort to dissect into the parts you want.

eikenberry 2y ago

Aren't those saying the same thing? .. "its reliability became close to none" vs. "causes irreversible defects in the resulting models"

renewiltord 2y ago
There’s thing that happens in humans called “hallucination” where they just make up stuff. You can’t really just take it at face value. Sometimes, they’re just overfit so they generate the same tokens independent of input.
- Guthur 2y ago
  That has never been the definition of hallucination, until LLMs. It's actually just called lying, dishonesty or falsehood. Hallucination is a distortion (McKenna might say liberation) of perception. If I hallucinate being covered in spiders, I don't necessarily go around saying, "I'm covered in spiders, if you cant see them you're blind" (disclaimer: some might, but that's not a prerequisite of an hallucination). The cynic in me thinks that use of the word hallucination is marketing to obscure functional inadequacy and reinforce the illusion that LLMs are some how analogous to human intelligence.
  - radlad 2y ago
    Lying, dishonesty, and falsehood all imply motive/intent, which is not likely the case when referring to LLM hallucinations. Another term is "making a mistake," but this also reinforces the similarities between humans and LLMs, and doesn't feel very accurate when talking about a technical machine. Sibling commenter correctly calls out the most similar human phenomenon: confabulation ("a memory error consisting of the production of fabricated, distorted, or misinterpreted memories about oneself or the world" per Wikipedia.)
    - usual_user 2y ago
      IMHO Lying means thinking one thing and saying another, hiding your true internal state. For it to be effective it also seems to require something like “theory of mind” (what does the other person know / think that I know).
  - jfim 2y ago
    I believe the term hallucination comes from vision models, where the model would "hallucinate" an object where none exists.
    - alcover 2y ago
      Wouldn't 'illusion' be the more precise term here ? When one thinks he recognizes something, whereas 'hallucination' is more of unreal appearing out of the blue ?
    - dijksterhuis 2y ago
      You might be thinking of deepdream... https://en.wikipedia.org/wiki/DeepDream https://en.wikipedia.org/wiki/DeepDream "Hallucinations" have only really been a term of art with regards to LLMs. my PhD in the security of machine learning started in 2019 and no-one ever used that term in any papers. the first i saw it was on HN when ChatGPT became a released product. Same with "jailbreaking". With reference to machine learning models, this mostly came about when people started fiddling with LLMs that had so-called guardrails implemented. "jailbreaking" is just another name for an adversarial example (test-time integrity evasion attack), with a slightly modified attacker goal.
- goatlover 2y ago
  Confabulation is the accurate psychological term. Hallucination is a perceptual issue. The LLM term is misleading.
  - moffkalast 2y ago
    Confabulation would be actual false memories no? I suppose some of it are consistent false beliefs, but more often than not it's well.. lapsus. One token gets generated wrong, or the sampler picks something mindbogglingly dumb that doesn't make any sense because of high temperature, and the model can't help but try and continue as confidently as it can, pretending everything is fine without any option to correct itself. Some thinking models can figure these sort of mistakes out in the long run but it's still not all that reliable and requires training it that way from the base model. Confident bullshitting seems to be very ingrained in current instruct datasets.
- ben_w 2y ago
  Getting high on their own supply. It's a problem when humans do this. That AI also do it… is interesting… but AI's failure is not absolved by human failure.
qwertox 2y ago
I think parent may have understood it as "second to none", as in "exceptional". That is at least the way I struggled with that sentence in the paper.

threeducks 2y ago

The dangers of "model collapse" are wildly overstated. Sure, if you feed the unfiltered output of an LLM into a new LLM, inbreeding will eventually collapse the model, but if you filter the data with some grounding in reality, the results will get better and better. The best example is probably AlphaGo Zero, which was grounded with the rules of the game and did not even receive any supervised data. For programming, grounding can happen by simply executing the code. But even if you do not have grounding, you can still just throw a lot of test time compute at the output to filter out garbage, which is probably good enough.

wruza 2y ago
Same for stable diffusion (being the same tech). I trained models purely on model outputs and didn’t see any horrific handwavy things everyone’s talked about. You can’t get more of what’s already there, obviously. But as a way to bake some prompt / control pic in, it works great. I actually find/feel this easier to achieve the same result, because I can vary the output with dynamic prompting and seeds, models, etc, while keeping the desired concept intact, with a little picking. This helps to better distill it into a model. Natural imagery is usually too diverse and requires lots of effort to dissect into the parts you want.