8 ms·

One formulation is that these are hallucinations. Another is that these systems are "orthogonal to truth". They have nothing to do with truth or falsity. One

One formulation is that these are hallucinations. Another is that these systems are "orthogonal to truth". They have nothing to do with truth or falsity. One expression of that idea is in this paper: https://link.springer.com/article/10.1007/s10676-024-09775-5 https://link.springer.com/article/10.1007/s10676-024-09775-5

astrange 2y ago
That's unnecessarily negative. A better question is what the answer to a prompt is grounded in. And sometimes the answer is "nothing".
kreeben 2y ago
Your linked paper suffers from the same anthropomorphisation as does all papers who uses the word "hallucination".
- mordechai9000 2y ago
  It seems like a useful adaptation of the term to a new usage, but I can understand if your objection is that it promotes anthropomorphizing these types of models. What do you think we should call this kind output, instead of hallucination?
  - isidor3 2y ago
    An author at Ars Technica has been trying to push the term "confabulation" for this
    - jebarker 2y ago
      I think Geoff Hinton made this suggestion first.
- nerevarthelame 2y ago
  The criticism that people shouldn't anthropomorphize AI models that are deliberately and specifically replicating human behavior is already so tired. I think we need to accept that human traits will no longer be unique to humans (if they ever were, if you expand the analysis to non-human species), and that attributing these emergent traits to non-humans is justified. "Hallucination" may not be the optimal metaphor for LLM falsehoods, but some humans absolutely regularly spout bullshit in the same way that LLMs do - the same sort of inaccurate responses generated from the same loose past associations.
  - soloist11 2y ago
    People like that are often schizophrenic.
- fouc 2y ago
  > In this paper, we argue against the view that when ChatGPT and the like produce false claims they are lying or even hallucinating, and in favour of the position that the activity they are engaged in is bullshitting, in the Frankfurtian sense (Frankfurt, 2002, 2005). Because these programs cannot themselves be concerned with truth, and because they are designed to produce text that looks truth-apt without any actual concern for truth, it seems appropriate to call their outputs bullshit. > We think that this is worth paying attention to. Descriptions of new technology, including metaphorical ones, guide policymakers’ and the public’s understanding of new technology; they also inform applications of the new technology. They tell us what the technology is for and what it can be expected to do. Currently, false statements by ChatGPT and other large language models are described as “hallucinations”, which give policymakers and the public the idea that these systems are misrepresenting the world, and describing what they “see”. We argue that this is an inapt metaphor which will misinform the public, policymakers, and other interested parties.
- Karellen 2y ago
  Maybe another way of looking at it is - the paper is attempting to explain what LLMs are actually doing to people who have already anthropomorphised them. Sometimes, to lead people out of a wrong belief or worldview, you have to meet them where they currently are first.
soist 2y ago
It's like asking if a probability distribution is truthful or a liar. It's a category error to speak about algorithms as if they had personal characteristics.
- thwarted 2y ago
  The lie occurs when information which is known to be false or its truthfulness can not be assessed is presented as useful or truthful.
  - mistermann 2y ago
    This seems a bit ironic...you're claiming something needs to be true to be useful?
    - Y_Y 2y ago
      > Beauty is truth, truth beauty,—that is all > Ye know on earth, and all ye need to know. Keats - Ode on a Grecian Urn
  - deleted 2y ago
    [deleted]
  - soloist11 2y ago
    Lying is intentional, algorithms and computers do not have intentions. People can lie, computers can only execute their programmed instructions. Much of AI discourse is extremely confusing and confused because people keep attributing needs and intentions to computers and algorithms. The social media gurus don't help with these issues by claiming that non-intentional objects are going to cause humanity's demise when there are much more pertinent issues to be concerned about like global warming, corporate malfeasance, and the general plundering of the biosphere. Algorithms that lie are not even in the top 100 list of things that people should be concerned about.
    - skissane 2y ago
      > Lying is intentional, algorithms and computers do not have intentions. People can lie, computers can only execute their programmed instructions. Much of AI discourse is extremely confusing and confused because people keep attributing needs and intentions to computers and algorithms. How do you know whether something has “intentions”? How can you know that humans have them but computer programs (including LLMs) don’t or can’t? If one is a materialist/physicalist, one has to say that human intentions (assuming one agrees they exist, contra eliminativism) have to be reducible to or emergent from physical processes in the brain. If intentions can be reducible to/emergent from physical processes in the brain, why can’t they also be reducible to/emergent from a computer program, which is also ultimately a physical process (calculations on a CPU/GPU/etc)? What if one is a non-materialist/non-physicalist? I don’t think that makes the question any easier to answer. For example, a substance dualist will insist that intentionality is inherently immaterial, and hence requires an immaterial soul. And yet, if one believes that, one has to say those immaterial souls somehow get attached to material human brains - why couldn’t one then be attached to an LLM (or the physical hardware it executes on), hence giving it the same intentionality that humans have? I think this is one of those questions where if someone thinks the answer is obvious, that’s a sign they likely know far less about the topic than they think they do.
TheBlight 2y ago
My suspicion is shared reality will end up bending to accommodate LLMs not vice-versa. Whatever the computer says will be "truth."
- EForEndeavour 2y ago
  The botulinum that developed in this person's[1] garlic and olive oil mixture wouldn't particularly care to alter its toxicity to make Gemini's recommendation look better. [1] https://old.reddit.com/r/ChatGPT/comments/1diljf2/google_gemini_tried_to_kill_me/ https://old.reddit.com/r/ChatGPT/comments/1diljf2/google_gem...
  - TheBlight 2y ago
    Unfortunately there may be some unavoidable casualties.
skybrian 2y ago
The linked paper is about detecting when the LLM is choosing randomly versus consistently at the level of factoids. Procedurally-generated randomness can be great for some things like brainstorming, while consistency suggests that it's repeating something that also appeared fairly consistently in the training material. So it might be true or false, but it's more likely to have gotten it from somewhere. Knowing how random the information is seems like a small step forward.
- caseyy 2y ago
  I don’t know. It could be a misleading step. Take social media like Reddit for example. It has a filtering mechanism for content that elevates low-entropy thoughts people commonly express and agree with. And I don’t think that necessarily equates such popular ideas there to the truth.
  - skybrian 2y ago
    The conversations about people being misled by LLM's remind me of when the Internet was new (not safe!), when Wikipedia was new (not safe!) and social media was new (still not safe!) And they're right, it's not safe! Yes, people will certainly be misled. The Internet is not safe for gullible people, and LLM's are very gullible too. With some work, eventually they might get LLM's to be about as accurate as Wikipedia. People will likely trust it too much, but the same is true of Wikipedia. I think it's best to treat LLM's as a fairly accurate hint provider. A source of good hints can be a very useful component of a larger system, if there's something else doing the vetting. But if you want to know whether something is true, you need some other way of checking it. An LLM cannot check anything for you - that's up to you. If you have no way of checking its hints, you're in trouble.
bravura 2y ago
LLMs are trained with the objective: “no matter what, always have at least three paragraphs of response”. and that response is always preferred to silence or “unfriendly” responses like: “what are you talking about?” Then yes, it is being taught to bullshit. Similar to how an improv class teaches you to keep a conversation interesting and “never to say no” to your acting partner.
kouru225 2y ago
Yea IMO these LLMs seem more similar to a subconscious mind than a conscious mind. Jung would probably call it an "antinomy": it's goal is not to represent the truth, but to represent the totality of possible answers.

astrange 2y ago

That's unnecessarily negative. A better question is what the answer to a prompt is grounded in. And sometimes the answer is "nothing".

kreeben 2y ago

Your linked paper suffers from the same anthropomorphisation as does all papers who uses the word "hallucination".

mordechai9000 2y ago
It seems like a useful adaptation of the term to a new usage, but I can understand if your objection is that it promotes anthropomorphizing these types of models. What do you think we should call this kind output, instead of hallucination?
- isidor3 2y ago
  An author at Ars Technica has been trying to push the term "confabulation" for this
  - jebarker 2y ago
    I think Geoff Hinton made this suggestion first.
nerevarthelame 2y ago
The criticism that people shouldn't anthropomorphize AI models that are deliberately and specifically replicating human behavior is already so tired. I think we need to accept that human traits will no longer be unique to humans (if they ever were, if you expand the analysis to non-human species), and that attributing these emergent traits to non-humans is justified. "Hallucination" may not be the optimal metaphor for LLM falsehoods, but some humans absolutely regularly spout bullshit in the same way that LLMs do - the same sort of inaccurate responses generated from the same loose past associations.
- soloist11 2y ago
  People like that are often schizophrenic.
fouc 2y ago
> In this paper, we argue against the view that when ChatGPT and the like produce false claims they are lying or even hallucinating, and in favour of the position that the activity they are engaged in is bullshitting, in the Frankfurtian sense (Frankfurt, 2002, 2005). Because these programs cannot themselves be concerned with truth, and because they are designed to produce text that looks truth-apt without any actual concern for truth, it seems appropriate to call their outputs bullshit. > We think that this is worth paying attention to. Descriptions of new technology, including metaphorical ones, guide policymakers’ and the public’s understanding of new technology; they also inform applications of the new technology. They tell us what the technology is for and what it can be expected to do. Currently, false statements by ChatGPT and other large language models are described as “hallucinations”, which give policymakers and the public the idea that these systems are misrepresenting the world, and describing what they “see”. We argue that this is an inapt metaphor which will misinform the public, policymakers, and other interested parties.
Karellen 2y ago
Maybe another way of looking at it is - the paper is attempting to explain what LLMs are actually doing to people who have already anthropomorphised them. Sometimes, to lead people out of a wrong belief or worldview, you have to meet them where they currently are first.

soist 2y ago

It's like asking if a probability distribution is truthful or a liar. It's a category error to speak about algorithms as if they had personal characteristics.

thwarted 2y ago
The lie occurs when information which is known to be false or its truthfulness can not be assessed is presented as useful or truthful.
- mistermann 2y ago
  This seems a bit ironic...you're claiming something needs to be true to be useful?
  - Y_Y 2y ago
    > Beauty is truth, truth beauty,—that is all > Ye know on earth, and all ye need to know. Keats - Ode on a Grecian Urn
- deleted 2y ago
  [deleted]
- soloist11 2y ago
  Lying is intentional, algorithms and computers do not have intentions. People can lie, computers can only execute their programmed instructions. Much of AI discourse is extremely confusing and confused because people keep attributing needs and intentions to computers and algorithms. The social media gurus don't help with these issues by claiming that non-intentional objects are going to cause humanity's demise when there are much more pertinent issues to be concerned about like global warming, corporate malfeasance, and the general plundering of the biosphere. Algorithms that lie are not even in the top 100 list of things that people should be concerned about.
  - skissane 2y ago
    > Lying is intentional, algorithms and computers do not have intentions. People can lie, computers can only execute their programmed instructions. Much of AI discourse is extremely confusing and confused because people keep attributing needs and intentions to computers and algorithms. How do you know whether something has “intentions”? How can you know that humans have them but computer programs (including LLMs) don’t or can’t? If one is a materialist/physicalist, one has to say that human intentions (assuming one agrees they exist, contra eliminativism) have to be reducible to or emergent from physical processes in the brain. If intentions can be reducible to/emergent from physical processes in the brain, why can’t they also be reducible to/emergent from a computer program, which is also ultimately a physical process (calculations on a CPU/GPU/etc)? What if one is a non-materialist/non-physicalist? I don’t think that makes the question any easier to answer. For example, a substance dualist will insist that intentionality is inherently immaterial, and hence requires an immaterial soul. And yet, if one believes that, one has to say those immaterial souls somehow get attached to material human brains - why couldn’t one then be attached to an LLM (or the physical hardware it executes on), hence giving it the same intentionality that humans have? I think this is one of those questions where if someone thinks the answer is obvious, that’s a sign they likely know far less about the topic than they think they do.

TheBlight 2y ago

My suspicion is shared reality will end up bending to accommodate LLMs not vice-versa. Whatever the computer says will be "truth."

EForEndeavour 2y ago
The botulinum that developed in this person's[1] garlic and olive oil mixture wouldn't particularly care to alter its toxicity to make Gemini's recommendation look better. [1] https://old.reddit.com/r/ChatGPT/comments/1diljf2/google_gemini_tried_to_kill_me/ https://old.reddit.com/r/ChatGPT/comments/1diljf2/google_gem...
- TheBlight 2y ago
  Unfortunately there may be some unavoidable casualties.

skybrian 2y ago

The linked paper is about detecting when the LLM is choosing randomly versus consistently at the level of factoids. Procedurally-generated randomness can be great for some things like brainstorming, while consistency suggests that it's repeating something that also appeared fairly consistently in the training material. So it might be true or false, but it's more likely to have gotten it from somewhere. Knowing how random the information is seems like a small step forward.

caseyy 2y ago
I don’t know. It could be a misleading step. Take social media like Reddit for example. It has a filtering mechanism for content that elevates low-entropy thoughts people commonly express and agree with. And I don’t think that necessarily equates such popular ideas there to the truth.
- skybrian 2y ago
  The conversations about people being misled by LLM's remind me of when the Internet was new (not safe!), when Wikipedia was new (not safe!) and social media was new (still not safe!) And they're right, it's not safe! Yes, people will certainly be misled. The Internet is not safe for gullible people, and LLM's are very gullible too. With some work, eventually they might get LLM's to be about as accurate as Wikipedia. People will likely trust it too much, but the same is true of Wikipedia. I think it's best to treat LLM's as a fairly accurate hint provider. A source of good hints can be a very useful component of a larger system, if there's something else doing the vetting. But if you want to know whether something is true, you need some other way of checking it. An LLM cannot check anything for you - that's up to you. If you have no way of checking its hints, you're in trouble.

bravura 2y ago

LLMs are trained with the objective: “no matter what, always have at least three paragraphs of response”. and that response is always preferred to silence or “unfriendly” responses like: “what are you talking about?” Then yes, it is being taught to bullshit. Similar to how an improv class teaches you to keep a conversation interesting and “never to say no” to your acting partner.

kouru225 2y ago

Yea IMO these LLMs seem more similar to a subconscious mind than a conscious mind. Jung would probably call it an "antinomy": it's goal is not to represent the truth, but to represent the totality of possible answers.