4 ms·

I wonder what is lost here. Surely there's a trade-off... I'm wondering if there's any effect of "creativity", or ability to interpolate between concepts. Hall

by patcon 2y ago

I wonder what is lost here. Surely there's a trade-off... I'm wondering if there's any effect of "creativity", or ability to interpolate between concepts. Hallucination and creativity feel very related to me. I understand hallucinating as simply being misaligned with the space humans feel appropriate to interpolate between

watsonmusic 2y ago
not all hallucinations are creativity Imaginate that for a RAG application, the model is supposed to follow the given documents
magicalhippo 2y ago
> Surely there's a trade-off... For one, speed and memory. They have twice as many Q and K weights in the attention blocks, leading to a ~10% reduction in throughput on their H100 (table 7 in appendix A).
- lennxa 2y ago
  they mention similar performance to vanilla transformer with significantly reduced param count though
- karmasimida 2y ago
  I mean it doesn’t necessarily needs 2x QK to match that performance, in terms of accuracy, of a regular transformer right?
dartos 2y ago
> Hallucination and creativity feel very related to me. Why? I see them as just sampling errors. Sure a mistake can spark inspiration sometimes, but creativity is much more than mistakes. > I understand hallucinating as simply being misaligned with the space humans feel appropriate to interpolate between These language models are next-token predictors. The way the next token is predicted is by sampling a probability space outputted by the model. That sampling process can be non deterministic. Hallucinations are when that sampling results in tokens that come together to create a false or otherwise unintended statement. You can just as well think of everything a model outputs as a hallucination, but we train the model to output a space what we want them to hallucinate is more likely. Otherwise it just outputs meaningless noise. “Hallucinate” is really an awful word for what it’s trying to describe.
- nextaccountic 2y ago
  > Sure a mistake can spark inspiration sometimes, but creativity is much more than mistakes. It looks like creativity has many steps but being able to come with novel, unprompted stuff is important, as long as you are able to discard the bullshit earlier. "Hallucination" is only a problem if later layers (or additional networks) can't detect and remove it
  - dartos 2y ago
    > "Hallucination" is only a problem if later layers (or additional networks) can't detect and remove it Yeah I mean sure. Anything is only a problem if it goes undetected. The issue is that if you rely on statistical model, you’ll always have hallucinations, so you can’t filter statistical output with another statistical model if you need real guarantees. Many products don’t need those guarantees though.
- thomastjeffery 2y ago
  Hallucinate is an awful word because of what it is trying to describe. Hallucination describes the same feature you just called "non deterministic sampling", but exclusively the cases that we don't like. It would be really convenient if we could actually draw that line, but we can't. If non-determinism is a core feature, then that feature will be present in every case; including the ones we find desirable, and the ones we find undesirable.
- skybrian 2y ago
  LLM’s are too unpredictable for many practical uses so I’d guess better predictability is better. Hopefully the change the paper proposes will help! But here’s a case for the other side: sure, most mistakes are just errors, but evolution happens via “mistakes.” Also, LLM’s often deliberately add add randomness at inference time.
  - dartos 2y ago
    > evolution happens via “mistakes.” That’s a nice slogan, but it’s a gross oversimplification. In the natural world, you can say that mistakes in DNA replication leads to evolution, but that’s discounting the entire process of natural selection. Same with creativity. Look at Picasso. His was a technically brilliant realistic painter at 15, but his work later in life evolved to be more abstract and weird. I don’t think that was the result of mistakes, but rather intentionally breaking patterns he learned in his youth.

by patcon 2y ago

watsonmusic 2y ago

not all hallucinations are creativity Imaginate that for a RAG application, the model is supposed to follow the given documents

magicalhippo 2y ago

> Surely there's a trade-off... For one, speed and memory. They have twice as many Q and K weights in the attention blocks, leading to a ~10% reduction in throughput on their H100 (table 7 in appendix A).

lennxa 2y ago
they mention similar performance to vanilla transformer with significantly reduced param count though
karmasimida 2y ago
I mean it doesn’t necessarily needs 2x QK to match that performance, in terms of accuracy, of a regular transformer right?

dartos 2y ago

> Hallucination and creativity feel very related to me. Why? I see them as just sampling errors. Sure a mistake can spark inspiration sometimes, but creativity is much more than mistakes. > I understand hallucinating as simply being misaligned with the space humans feel appropriate to interpolate between These language models are next-token predictors. The way the next token is predicted is by sampling a probability space outputted by the model. That sampling process can be non deterministic. Hallucinations are when that sampling results in tokens that come together to create a false or otherwise unintended statement. You can just as well think of everything a model outputs as a hallucination, but we train the model to output a space what we want them to hallucinate is more likely. Otherwise it just outputs meaningless noise. “Hallucinate” is really an awful word for what it’s trying to describe.

nextaccountic 2y ago
> Sure a mistake can spark inspiration sometimes, but creativity is much more than mistakes. It looks like creativity has many steps but being able to come with novel, unprompted stuff is important, as long as you are able to discard the bullshit earlier. "Hallucination" is only a problem if later layers (or additional networks) can't detect and remove it
- dartos 2y ago
  > "Hallucination" is only a problem if later layers (or additional networks) can't detect and remove it Yeah I mean sure. Anything is only a problem if it goes undetected. The issue is that if you rely on statistical model, you’ll always have hallucinations, so you can’t filter statistical output with another statistical model if you need real guarantees. Many products don’t need those guarantees though.
thomastjeffery 2y ago
Hallucinate is an awful word because of what it is trying to describe. Hallucination describes the same feature you just called "non deterministic sampling", but exclusively the cases that we don't like. It would be really convenient if we could actually draw that line, but we can't. If non-determinism is a core feature, then that feature will be present in every case; including the ones we find desirable, and the ones we find undesirable.
skybrian 2y ago
LLM’s are too unpredictable for many practical uses so I’d guess better predictability is better. Hopefully the change the paper proposes will help! But here’s a case for the other side: sure, most mistakes are just errors, but evolution happens via “mistakes.” Also, LLM’s often deliberately add add randomness at inference time.
- dartos 2y ago
  > evolution happens via “mistakes.” That’s a nice slogan, but it’s a gross oversimplification. In the natural world, you can say that mistakes in DNA replication leads to evolution, but that’s discounting the entire process of natural selection. Same with creativity. Look at Picasso. His was a technically brilliant realistic painter at 15, but his work later in life evolved to be more abstract and weird. I don’t think that was the result of mistakes, but rather intentionally breaking patterns he learned in his youth.