3 ms·

When people talk about stopping an LLM from "seeing hallucinations instead of the truth", that's like stopping an Ouija-board from "channeling the wrong spirits

by Terr_ 2y ago

When people talk about stopping an LLM from "seeing hallucinations instead of the truth", that's like stopping an Ouija-board from "channeling the wrong spirits instead of the right spirits." It suggests a qualitative difference between desirable and undesirable operation that isn't really there. They're all hallucinations, we just happen to like some of them more than others.

ml_more 2y ago
The problem is that LLMs are just convincing enough that people DO trust them which is sort of a problem since AI slop is creeping into everything. What can be done to solve it (while not perfect) is pretty powerful. You can force feed them the facts (RAG) and then verify the result. Which is way better than trusting LLMs while doing neither of those things (which is what a lot of people do today anyway). See the recent 5 cases of lawyers getting in trouble for ChatGPT hallucinating citations of case law. LLMs write better than most college students so if you do those two things (RAG + check) you can get college graduate level writing with accurate facts... and that unlocks a bit of value out in the world. Don't take my word for it look at the proposed valuations of AI companies. Clearly investors think there's something there. The good news is that it hasn't been solved yet so if someone wants to solve it there might be money on the table.
- wk_end 2y ago
  How do you check it? Take the example of case law. Would you need to formalize the entirety of case law? Would the AI then need to produce a formal proof of its argument, so that you can ascertain that its citations are valid? How do you know that the formal proof corresponds to whatever longform writing you ask the AI to generate? Is this really something that LLMs are suited for? That the law is suited for?
- latexr 2y ago
  > and that unlocks a bit of value out in the world. > Don't take my word for it look at the proposed valuations of AI companies. Clearly investors think there's something there. Investors back whatever they think will make them money. They couldn’t give less of a crap if something is valuable to the world, or works well, of is in any way positive to others. All they care is if they can profit from it and they’ll chase every idea in that pursuit. Source: all of modern history. https://www.sydney.edu.au/news-opinion/news/2024/05/02/how-corporations-harm-your-health-through-everyday-products---a-.html https://www.sydney.edu.au/news-opinion/news/2024/05/02/how-c... https://www.decof.com/documents/dangerous-products.pdf https://www.decof.com/documents/dangerous-products.pdf
  - Terr_ 2y ago
    > Investors back whatever they think will make them money. A not-flagrantly-illegal example of this might be casinos, where IMO it is basically impossible to argue the fleeting entertainment they offer offsets the financial ruin inflicted on certain vulnerable types of patron. > All they care is if they can profit from it Notably that isn't the same as the business itself being profitable: Some investors may be hoping they can dump their stake at a higher price onto a Greater Fool [0] and exit before the collapse. [0] https://en.wikipedia.org/wiki/Greater_fool_theory https://en.wikipedia.org/wiki/Greater_fool_theory
  - Gormo 2y ago
    > They couldn’t give less of a crap if something is valuable to the world "The world" is an abstraction: concretely, every bit of value that is generated within that abstraction accrues to someone in particular -- investors in AI projects, for example.
- threeseed 2y ago
  > Clearly investors think there's something there Of course. Because enterprise companies take a long time to evaluate new technologies. And so there is plenty of money to be made selling them tools over the next few years. As well as selling tools to those who are making tools. But from my experience in rolling out these technologies only a handful of these companies will exist in 5-10 years. Because LLMs are "garbage in, garbage out" and we've never figured out how to keep the "garbage in" to a minimum.
- Gormo 2y ago
  Sure, using RAG is great, but it limits the LLM to functioning as a natural-language search engine. That's a pretty useful thing in its own right, and will revolutionize a lot of activities, but it still falls far short of the expectations people have for generative AI.
Nehnehneh 2y ago
That's just not true. The training data is the underlying truth and that's not nothing but a lot. And hallucinations are pathes inside this space which are there for yet unknown reason. We like answers from LLMs which walk through this space reasonable.
- mrguyorama 2y ago
  >The training data is the underlying truth Correct. What is the training data? Language in the form of sentences and documents and words and "tokens". No human language has any normal or natural encoding of "fact" or "truthiness" which is the entire point. You can only rarely evaluate a string of text for truthiness without external context. An LLM "knows" the structure and look of valid text. That's why they rarely produce grammar mistakes, even when "hallucinating". A lie, a made up reference, a physical impossibility, contradictions, etc are all "valid sentences". That's why you can never prevent an LLM from producing falsehoods, lies, contradictions etc. Truthiness cannot be hacked in after the fact, and I currently believe that LLMs as an architecture are not powerful enough a statistical tool that you even COULD train an LLM that had "truthiness" of the entire corpus labeled somehow, especially since that's on it's own a fairly impossible task.
TZubiri 2y ago
I disagree with this take, Stallman has expressed it recently by linking some "scientific article". While I get that LLMs generate text in some way that does not guarantee correctness. There is a correlation between generated text and correctness, which is why millions of people use it... You can judge the correctness of a sentence generated by an LLM. In the same way you can judge the correctness of a human generated sentence. Now whether the truthness or correlation with reality of an LLM sentence can be judged on its own or whether it requires a human to interpret it is not very relevant, as sentences produced by the LLM are still correct most of the time. Just because it is not perfect doesn't make the correctness in the other cases useless, albeit perhaps less useful of course. This is nothing surprising of a statistical model, it tends to produce true results.
- Terr_ 2y ago
  > I disagree with this take, Stallman has expressed it recently by linking some "scientific article". I don't know how to parse this. What article did Stallman "link", and what are you saying Stallman "expressed" by linking/using it? > whether the truthness or correlation with reality of an LLM sentence can be judged on its own or whether it requires a human to interpret it is not very relevant It's incredibly relevant. We wouldn't even be having these debates if complex LLM judgements could always be verified without a human checking the logic. > sentences produced by the LLM are still correct most of the time At least half the problem here is that humans are accustomed to using certain cues as an indirect sign of time-investment, attentiveness, intelligence, truth, etc... and now those cues can be cheaply and quickly counterfeited. It breaks all those old correlations faster than we are adapting.
  - TZubiri 2y ago
    https://stallman.org/chatgpt.html https://stallman.org/chatgpt.html https://link.springer.com/article/10.1007/s10676-024-09775-5 https://link.springer.com/article/10.1007/s10676-024-09775-5
demaga 2y ago
> They're all hallucinations, we just happen to like some of them more than others. I love it! Puts things into perspective.
mdp2021 2y ago
> It suggests a qualitative difference And what is sought is, in a way, a jump to that qualitative difference. (And surely there are «desirable and undesirable operation[s]».) "Add something to the dices so that they can be well predictive".

by Terr_ 2y ago

ml_more 2y ago

The problem is that LLMs are just convincing enough that people DO trust them which is sort of a problem since AI slop is creeping into everything. What can be done to solve it (while not perfect) is pretty powerful. You can force feed them the facts (RAG) and then verify the result. Which is way better than trusting LLMs while doing neither of those things (which is what a lot of people do today anyway). See the recent 5 cases of lawyers getting in trouble for ChatGPT hallucinating citations of case law. LLMs write better than most college students so if you do those two things (RAG + check) you can get college graduate level writing with accurate facts... and that unlocks a bit of value out in the world. Don't take my word for it look at the proposed valuations of AI companies. Clearly investors think there's something there. The good news is that it hasn't been solved yet so if someone wants to solve it there might be money on the table.

wk_end 2y ago
How do you check it? Take the example of case law. Would you need to formalize the entirety of case law? Would the AI then need to produce a formal proof of its argument, so that you can ascertain that its citations are valid? How do you know that the formal proof corresponds to whatever longform writing you ask the AI to generate? Is this really something that LLMs are suited for? That the law is suited for?
latexr 2y ago
> and that unlocks a bit of value out in the world. > Don't take my word for it look at the proposed valuations of AI companies. Clearly investors think there's something there. Investors back whatever they think will make them money. They couldn’t give less of a crap if something is valuable to the world, or works well, of is in any way positive to others. All they care is if they can profit from it and they’ll chase every idea in that pursuit. Source: all of modern history. https://www.sydney.edu.au/news-opinion/news/2024/05/02/how-corporations-harm-your-health-through-everyday-products---a-.html https://www.sydney.edu.au/news-opinion/news/2024/05/02/how-c... https://www.decof.com/documents/dangerous-products.pdf https://www.decof.com/documents/dangerous-products.pdf
- Terr_ 2y ago
  > Investors back whatever they think will make them money. A not-flagrantly-illegal example of this might be casinos, where IMO it is basically impossible to argue the fleeting entertainment they offer offsets the financial ruin inflicted on certain vulnerable types of patron. > All they care is if they can profit from it Notably that isn't the same as the business itself being profitable: Some investors may be hoping they can dump their stake at a higher price onto a Greater Fool [0] and exit before the collapse. [0] https://en.wikipedia.org/wiki/Greater_fool_theory https://en.wikipedia.org/wiki/Greater_fool_theory
- Gormo 2y ago
  > They couldn’t give less of a crap if something is valuable to the world "The world" is an abstraction: concretely, every bit of value that is generated within that abstraction accrues to someone in particular -- investors in AI projects, for example.
threeseed 2y ago
> Clearly investors think there's something there Of course. Because enterprise companies take a long time to evaluate new technologies. And so there is plenty of money to be made selling them tools over the next few years. As well as selling tools to those who are making tools. But from my experience in rolling out these technologies only a handful of these companies will exist in 5-10 years. Because LLMs are "garbage in, garbage out" and we've never figured out how to keep the "garbage in" to a minimum.
Gormo 2y ago
Sure, using RAG is great, but it limits the LLM to functioning as a natural-language search engine. That's a pretty useful thing in its own right, and will revolutionize a lot of activities, but it still falls far short of the expectations people have for generative AI.

Nehnehneh 2y ago

That's just not true. The training data is the underlying truth and that's not nothing but a lot. And hallucinations are pathes inside this space which are there for yet unknown reason. We like answers from LLMs which walk through this space reasonable.

mrguyorama 2y ago
>The training data is the underlying truth Correct. What is the training data? Language in the form of sentences and documents and words and "tokens". No human language has any normal or natural encoding of "fact" or "truthiness" which is the entire point. You can only rarely evaluate a string of text for truthiness without external context. An LLM "knows" the structure and look of valid text. That's why they rarely produce grammar mistakes, even when "hallucinating". A lie, a made up reference, a physical impossibility, contradictions, etc are all "valid sentences". That's why you can never prevent an LLM from producing falsehoods, lies, contradictions etc. Truthiness cannot be hacked in after the fact, and I currently believe that LLMs as an architecture are not powerful enough a statistical tool that you even COULD train an LLM that had "truthiness" of the entire corpus labeled somehow, especially since that's on it's own a fairly impossible task.

TZubiri 2y ago

I disagree with this take, Stallman has expressed it recently by linking some "scientific article". While I get that LLMs generate text in some way that does not guarantee correctness. There is a correlation between generated text and correctness, which is why millions of people use it... You can judge the correctness of a sentence generated by an LLM. In the same way you can judge the correctness of a human generated sentence. Now whether the truthness or correlation with reality of an LLM sentence can be judged on its own or whether it requires a human to interpret it is not very relevant, as sentences produced by the LLM are still correct most of the time. Just because it is not perfect doesn't make the correctness in the other cases useless, albeit perhaps less useful of course. This is nothing surprising of a statistical model, it tends to produce true results.

Terr_ 2y ago
> I disagree with this take, Stallman has expressed it recently by linking some "scientific article". I don't know how to parse this. What article did Stallman "link", and what are you saying Stallman "expressed" by linking/using it? > whether the truthness or correlation with reality of an LLM sentence can be judged on its own or whether it requires a human to interpret it is not very relevant It's incredibly relevant. We wouldn't even be having these debates if complex LLM judgements could always be verified without a human checking the logic. > sentences produced by the LLM are still correct most of the time At least half the problem here is that humans are accustomed to using certain cues as an indirect sign of time-investment, attentiveness, intelligence, truth, etc... and now those cues can be cheaply and quickly counterfeited. It breaks all those old correlations faster than we are adapting.
- TZubiri 2y ago
  https://stallman.org/chatgpt.html https://stallman.org/chatgpt.html https://link.springer.com/article/10.1007/s10676-024-09775-5 https://link.springer.com/article/10.1007/s10676-024-09775-5

demaga 2y ago

> They're all hallucinations, we just happen to like some of them more than others. I love it! Puts things into perspective.

mdp2021 2y ago

> It suggests a qualitative difference And what is sought is, in a way, a jump to that qualitative difference. (And surely there are «desirable and undesirable operation[s]».) "Add something to the dices so that they can be well predictive".