5 ms·

I think characterisation of LLMs as lying is reasonable because although the intent isn't there to misrepresent the truth in answering the specific query, the i

by zmmmmm 4y ago

I think characterisation of LLMs as lying is reasonable because although the intent isn't there to misrepresent the truth in answering the specific query, the intent is absolutely there in how the network is trained. The training algorithm is designed to create the most plausible text possible - decoupled from the truthfulness of the output. In a lot of cases (indeed most cases) the easiest way to make the text plausible is to tell truth. But guess what, that is pretty much how human liars work too! Ask the question: given improbable but thruthful output but plausible untruthful output, which does the network choose? And which is the intent of the algorithm designers for it to choose? In both cases my understanding is, they have designed it to lie. Given the intent is there in the design and training, I think it's fair enough to refer to this behavioral trait as lying.

red75prime 4y ago
> Ask the question: given improbable but thruthful output but plausible untruthful output, which does the network choose? "Plausible" means "that which the majority of people is likely to say". So, yes, a foundational model is likely to say the plausible thing. On the other hand, it has to have a way to output a truthful answer too, to not fail on texts produced by experts. So, it's not impossible that the model could be trained to prefer to output truthful answers (as well as it can do it, it's not an AGI with perfect factual memory and logical inference after all).
- hgsgm 4y ago
  > "that which the majority of people is likely to say" .. and saying "I don't know" is forbidden by the programmers. That is a huge part of the problem.
  - red75prime 4y ago
    I guess it's not that straightforward. It's probably a combination of much less prevalent use of "don't know" online, low scores of "don't know" in RLHF, system prompt instructing GPT to give helpful responses, and, yeah, maybe token sampling algorithm is tuned to disfavor explicitly uncertain responses.
  - olalonde 4y ago
    On the subject of not knowing thigs... Your clain is incorrect. Prompt: Tell me what you know about the Portland metro bombing terrorist attack of 2005. GPT4: I'm sorry, but I cannot provide information on the Portland metro bombing terrorist attack of 2005, as there is no historical record of such an event occurring. It's possible you may have confused this with another event or have incorrect information. Please provide more context or clarify the event you are referring to, and I'll be happy to help with any information I can.
    - shjake 4y ago
      There is a big difference between what they are talking about (I believe) and asking it factual questions withal straightforward answers. Try discussing something a bit more complex and it’s very easy to get ChatGPT to start contradicting itself all the time (which is more or less the same thing). From my experience ChatGPT is prone to come up with lists of “probable” answer when it’s not feeling very confident. Then it just starts throwing them out in a confident tone. When challenged it switches to a complete different answer and then “pretends” that I misunderstood it previously.
      - olalonde 4y ago
        Yes I know it does, just not all of the time and it wasn't programmed to never say "I don't know".
tjr 4y ago
My understanding is that ChatGPT (&co.) was not designed as, and is not intended to be, any sort of expert system, or knowledge representation system. The fact that it does as well as it does anyway is pretty amazing. But even so -- as you said, it's still dealing chiefly with the statistical probability of words/tokens, not with facts and truths. I really don't "trust" it in any meaningful way, even if it already has, and will continue to, prove itself useful. Anything it says must be vetted.
- ChatGTP 4y ago
  So do you find it alarming that people are trying to give such a system “agency” ?
  - gwright 4y ago
    I do. I think the anthropomorphic language that people use to describe these systems is inaccurate and misleading. An Australian mayor has claimed that ChatGPT "defamed" him. The title of this article says that we should teach people that text generation tools "lie". Other articles suggest that ChatGPT "knows" things. It is extremely interesting to me how much milage can be gotten out of an LLM by observing patterns in text and generating similar patterns. It is extremely frustrating to me to see how easily people think that this is evidence of intelligence or knowledge or "agency" as you suggest.
    - simonw 4y ago
      I agree with you: convincing people that these systems do not have intelligence or agency is the second most important problem to solve. The most important problem is to ensure people understand that these systems cannot be trusted to tell them the truth. I'm OK with starting out with "these systems will lie to you", then following up with "but you do need to understand that they're not anthropomorphic etc" later on.
      - int_19h 4y ago
        You might have better success convincing people if you stop bundling it together as "intelligence and agency", for starters. I agree with the agency part and strongly disagree with intelligence, for one. In general, "intelligence" is so vaguely and informally defined that it's simply not something that can be reasonably argued objectively either way. (If what you really meant is humanlike intelligence, then I would agree with that claim, but it's a very different one.)
admissionsguy 4y ago
> In a lot of cases (indeed most cases) the easiest way to make the text plausible is to tell truth No, definitely not most cases. Only in the cases well represented in the training dataset. One does very quickly run into its limitations when trying to get it to do anything uncommon.
babyshake 4y ago
By that logic, our brains are liars. There are plenty of optical illusions based on the tendency for our brains to expect the most plausible scenario, given its training data.
- roywiggins 4y ago
  It's not that uncommon of a thing to say: https://www.google.com/search?q=your+brain+can+lie+to+you https://www.google.com/search?q=your+brain+can+lie+to+you
- unaindz 4y ago
  Well, they are liars too. The difference is that we seem to have an outer loop that checks for correctness but it fails sometimes, in some specific cases always.
pmoriarty 4y ago
"The training algorithm is designed to create the most plausible text possible" That may be how they're trained, but these things seem to have emergent behavior.
- gwright 4y ago
  I'm not sure why you call it "emergent" behavior. Instead, my take away is that much of what we think of as cognition is just really complicated pattern matching and probabilistic transformations (i.e. mechanical processes).
  - zmmmmm 4y ago
    yeah ... the depressing thing is that a large amount of human behaviour may well be exactly like LLMs - we think, "what should I say now" to reply to something and in the vast majority of cases the response is based almost entirely on "what would another reasonable person similar to me say in this situation?". I think that is in part why we find LLMs so appealing and lifelike.
ftxbro 4y ago
> The training algorithm is designed to create the most plausible text possible - decoupled from the truthfulness of the output. In a lot of cases (indeed most cases) the easiest way to make the text plausible is to tell truth. Yes. > But guess what, that is pretty much how human liars work too! There is some distinction between lying and bullshit. https://en.wikipedia.org/wiki/On_Bullshit#Lying_and_bullshit https://en.wikipedia.org/wiki/On_Bullshit#Lying_and_bullshit
- andrewflnr 4y ago
  Yes, this is exactly it. ChatGPT is superhuman at the task of generating bullshit.
DebtDeflation 4y ago
IMO, it just requires the same level of skepticism as a Google search. Just because you enter a query into the search bar and Google returns a list of links and you click one of those links and it contains content that makes a claim, doesn't mean that claim is correct. After all, this is largely what GPT has been trained on.
- worrycue 4y ago
  Seems worse to be frank. The webpages Google search delivers might be filled with falsehood but google search itself does its job of finding said pages which contain the terms you inputted fairly reliably. With GPT, not only there’s a chance its training data is full of falsehood, you can add the possibility of it inventing “original” falsehoods on top of that.
- shjake 4y ago
  That would be much easier if it provided sources and some context for it’s answers
njarboe 4y ago
I think it is much closer to bullshit. The bullshitter cares not to tell truth or deceive, just to sound like they know what they are talking about. To impress. Seems like ChatGPT to a T.
- kenjackson 4y ago
  ChatGPT no more cares to impress or persuade you as it cares to tell the truth or lie. It will say what its training maps to its model the best. No more, no less. If you believe it or not -- ChatGPT doesn't care -- except to the extent that you report the answer and they tweak its training/model in the future.
  - 8note 4y ago
    It definitely cares about impressing and persuasion. Those are things that will result in higher ratings from the user
    - kenjackson 4y ago
      Does ChatGPT care about ratings or does OpenAI?
  - shjake 4y ago
    > ChatGPT doesn't care -- except to the extent that you report the answer and they tweak its training/model in the future. So it does care (in a certainly non human way though)? If it’s training predisposes it to provide answers in a certain way or using a certain style that’s hardly distinguishable from ‘caring’, when that word is used to describe the behavior of a computer program.
8note 4y ago
I'd consider it definitional to lying that the agent knows the truth and purposefully obscures it. Without knowing what the truth is, I don't think LLMs are capable of lying
- zmmmmm 4y ago
  That's a great distinction to draw. I think it will split in two. There will be cases where the LLM has the truth represented in its data set and still chooses to say something else because its training has told it to produce the most plausible sounding answer, not the one closest to the truth. So this will fit closer to the idea of real lying. A good example: I asked it what the differences in driving between Australia and New Zealand are. It confidently told me that in New Zealand you drive on the right hand side of the road while in Australia you drive on the left. I am sure it has the correct knowledge in its training data. It chose to tell me that because that is a more common answer people say when asked about driving differences because that is the more dominant difference when you look between different countries. Then there will be cases where the subject in question has never been represented in its data set. Here I think your point is very valid.
alfor 4y ago
The training is to maximize good answers. Now there is lot of wrong answers that are close to the right one and ChatGPT does not expose it at the moment. But in the API you can see the level of confidence in each world the LLM output.
worrycue 4y ago
Whatever the intent or lack of, the result is the same, people are being given incorrect information.

by zmmmmm 4y ago

red75prime 4y ago

> Ask the question: given improbable but thruthful output but plausible untruthful output, which does the network choose? "Plausible" means "that which the majority of people is likely to say". So, yes, a foundational model is likely to say the plausible thing. On the other hand, it has to have a way to output a truthful answer too, to not fail on texts produced by experts. So, it's not impossible that the model could be trained to prefer to output truthful answers (as well as it can do it, it's not an AGI with perfect factual memory and logical inference after all).

hgsgm 4y ago
> "that which the majority of people is likely to say" .. and saying "I don't know" is forbidden by the programmers. That is a huge part of the problem.
- red75prime 4y ago
  I guess it's not that straightforward. It's probably a combination of much less prevalent use of "don't know" online, low scores of "don't know" in RLHF, system prompt instructing GPT to give helpful responses, and, yeah, maybe token sampling algorithm is tuned to disfavor explicitly uncertain responses.
- olalonde 4y ago
  On the subject of not knowing thigs... Your clain is incorrect. Prompt: Tell me what you know about the Portland metro bombing terrorist attack of 2005. GPT4: I'm sorry, but I cannot provide information on the Portland metro bombing terrorist attack of 2005, as there is no historical record of such an event occurring. It's possible you may have confused this with another event or have incorrect information. Please provide more context or clarify the event you are referring to, and I'll be happy to help with any information I can.
  - shjake 4y ago
    There is a big difference between what they are talking about (I believe) and asking it factual questions withal straightforward answers. Try discussing something a bit more complex and it’s very easy to get ChatGPT to start contradicting itself all the time (which is more or less the same thing). From my experience ChatGPT is prone to come up with lists of “probable” answer when it’s not feeling very confident. Then it just starts throwing them out in a confident tone. When challenged it switches to a complete different answer and then “pretends” that I misunderstood it previously.
    - olalonde 4y ago
      Yes I know it does, just not all of the time and it wasn't programmed to never say "I don't know".

tjr 4y ago

My understanding is that ChatGPT (&co.) was not designed as, and is not intended to be, any sort of expert system, or knowledge representation system. The fact that it does as well as it does anyway is pretty amazing. But even so -- as you said, it's still dealing chiefly with the statistical probability of words/tokens, not with facts and truths. I really don't "trust" it in any meaningful way, even if it already has, and will continue to, prove itself useful. Anything it says must be vetted.

ChatGTP 4y ago
So do you find it alarming that people are trying to give such a system “agency” ?
- gwright 4y ago
  I do. I think the anthropomorphic language that people use to describe these systems is inaccurate and misleading. An Australian mayor has claimed that ChatGPT "defamed" him. The title of this article says that we should teach people that text generation tools "lie". Other articles suggest that ChatGPT "knows" things. It is extremely interesting to me how much milage can be gotten out of an LLM by observing patterns in text and generating similar patterns. It is extremely frustrating to me to see how easily people think that this is evidence of intelligence or knowledge or "agency" as you suggest.
  - simonw 4y ago
    I agree with you: convincing people that these systems do not have intelligence or agency is the second most important problem to solve. The most important problem is to ensure people understand that these systems cannot be trusted to tell them the truth. I'm OK with starting out with "these systems will lie to you", then following up with "but you do need to understand that they're not anthropomorphic etc" later on.
    - int_19h 4y ago
      You might have better success convincing people if you stop bundling it together as "intelligence and agency", for starters. I agree with the agency part and strongly disagree with intelligence, for one. In general, "intelligence" is so vaguely and informally defined that it's simply not something that can be reasonably argued objectively either way. (If what you really meant is humanlike intelligence, then I would agree with that claim, but it's a very different one.)

admissionsguy 4y ago

> In a lot of cases (indeed most cases) the easiest way to make the text plausible is to tell truth No, definitely not most cases. Only in the cases well represented in the training dataset. One does very quickly run into its limitations when trying to get it to do anything uncommon.

babyshake 4y ago

By that logic, our brains are liars. There are plenty of optical illusions based on the tendency for our brains to expect the most plausible scenario, given its training data.

roywiggins 4y ago
It's not that uncommon of a thing to say: https://www.google.com/search?q=your+brain+can+lie+to+you https://www.google.com/search?q=your+brain+can+lie+to+you
unaindz 4y ago
Well, they are liars too. The difference is that we seem to have an outer loop that checks for correctness but it fails sometimes, in some specific cases always.

pmoriarty 4y ago

"The training algorithm is designed to create the most plausible text possible" That may be how they're trained, but these things seem to have emergent behavior.

gwright 4y ago
I'm not sure why you call it "emergent" behavior. Instead, my take away is that much of what we think of as cognition is just really complicated pattern matching and probabilistic transformations (i.e. mechanical processes).
- zmmmmm 4y ago
  yeah ... the depressing thing is that a large amount of human behaviour may well be exactly like LLMs - we think, "what should I say now" to reply to something and in the vast majority of cases the response is based almost entirely on "what would another reasonable person similar to me say in this situation?". I think that is in part why we find LLMs so appealing and lifelike.

ftxbro 4y ago

> The training algorithm is designed to create the most plausible text possible - decoupled from the truthfulness of the output. In a lot of cases (indeed most cases) the easiest way to make the text plausible is to tell truth. Yes. > But guess what, that is pretty much how human liars work too! There is some distinction between lying and bullshit. https://en.wikipedia.org/wiki/On_Bullshit#Lying_and_bullshit https://en.wikipedia.org/wiki/On_Bullshit#Lying_and_bullshit

andrewflnr 4y ago
Yes, this is exactly it. ChatGPT is superhuman at the task of generating bullshit.

DebtDeflation 4y ago

IMO, it just requires the same level of skepticism as a Google search. Just because you enter a query into the search bar and Google returns a list of links and you click one of those links and it contains content that makes a claim, doesn't mean that claim is correct. After all, this is largely what GPT has been trained on.

worrycue 4y ago
Seems worse to be frank. The webpages Google search delivers might be filled with falsehood but google search itself does its job of finding said pages which contain the terms you inputted fairly reliably. With GPT, not only there’s a chance its training data is full of falsehood, you can add the possibility of it inventing “original” falsehoods on top of that.
shjake 4y ago
That would be much easier if it provided sources and some context for it’s answers

njarboe 4y ago

I think it is much closer to bullshit. The bullshitter cares not to tell truth or deceive, just to sound like they know what they are talking about. To impress. Seems like ChatGPT to a T.

kenjackson 4y ago
ChatGPT no more cares to impress or persuade you as it cares to tell the truth or lie. It will say what its training maps to its model the best. No more, no less. If you believe it or not -- ChatGPT doesn't care -- except to the extent that you report the answer and they tweak its training/model in the future.
- 8note 4y ago
  It definitely cares about impressing and persuasion. Those are things that will result in higher ratings from the user
  - kenjackson 4y ago
    Does ChatGPT care about ratings or does OpenAI?
- shjake 4y ago
  > ChatGPT doesn't care -- except to the extent that you report the answer and they tweak its training/model in the future. So it does care (in a certainly non human way though)? If it’s training predisposes it to provide answers in a certain way or using a certain style that’s hardly distinguishable from ‘caring’, when that word is used to describe the behavior of a computer program.

8note 4y ago

I'd consider it definitional to lying that the agent knows the truth and purposefully obscures it. Without knowing what the truth is, I don't think LLMs are capable of lying

zmmmmm 4y ago
That's a great distinction to draw. I think it will split in two. There will be cases where the LLM has the truth represented in its data set and still chooses to say something else because its training has told it to produce the most plausible sounding answer, not the one closest to the truth. So this will fit closer to the idea of real lying. A good example: I asked it what the differences in driving between Australia and New Zealand are. It confidently told me that in New Zealand you drive on the right hand side of the road while in Australia you drive on the left. I am sure it has the correct knowledge in its training data. It chose to tell me that because that is a more common answer people say when asked about driving differences because that is the more dominant difference when you look between different countries. Then there will be cases where the subject in question has never been represented in its data set. Here I think your point is very valid.

alfor 4y ago

The training is to maximize good answers. Now there is lot of wrong answers that are close to the right one and ChatGPT does not expose it at the moment. But in the API you can see the level of confidence in each world the LLM output.

worrycue 4y ago

Whatever the intent or lack of, the result is the same, people are being given incorrect information.