7 ms·

> The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think sma

> The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. I don't know, the program does what it is engineered to do pretty well, which is, generate text that is representative of its training data following on from input tokens. It can't reason, it can't be confident, it can't determine fact. When you interpret it for what it is, it is not confidently wrong, it just generated what it thinks is most likely based on the input tokens. Sometimes, if the input tokens contain some counter-argument the model will generate text that would usually occur if an claim was refuted, but again, this is not based on reason, or fact, or logic. ChatGPT is not lying to people, it can't lie, at least not in the sense of "to make an untrue statement with intent to deceive". ChatGPT has no intent. It can generate text that is not in accordance with fact and is not derivable by reason from its training data, but why would you expect that from it? > Based on my understanding of the approach behind ChatGPT, it is probably very close to a local maximum in terms of intelligence so we don't have to worry about the fearmongering spread by the "AI safety" people any time soon if AI research continues to follow this paradigm. I agree here, I think you can only get so far with a language model, maybe if we get a couple orders of mangitude more parameters it magically becomes AGI, but I somehow don't quite feel it, I think there is more to human intelligence than a LLM, way more. Of course, that is coming, but that would not be this paradigm, which is basically trying to overextend LLM. LLMs are great, they are useful, but if you want a model that reasons, you will likely have to train it for that, or possibly more likely, combine ML with something symbolic reasoning.

danpat 4y ago
> but why would you expect that from it? If you understand what it is doing, then you don't. But the layman will just see a computer that talks in language they understand, and will infer intent and sentience are behind that, because that's the only analog they have for a thing that can talk back to them with words that appear to make sense at the complexity level that ChatGPT is achieving. Most humans do not have sufficient background to understand what they're really being presented with, they will take it at face value.
- hunter2_ 4y ago
  I think this is partly explained by most of the marketing and news essentially saying "ChatGPT is an AI" instead of "ChatGPT is an LLM." If you asked me what AI is, I'd say it means getting a computer to emulate human intelligence; if you asked me what an LLM is, I'd say it means getting a computer to emulate human language. The word "language" does not imply truthiness anywhere near to the extent that the word "intelligence" does.
  - int_19h 4y ago
    You could reasonably describe it as "human language emulator" back when people were using GPT-2 and the likes to compose text. But what we have today doesn't just emulate human language - it accepts tasks in that language, including such tasks that require reasoning to perform, and then carries them out. Granted, the only thing it can really "do" is produce text, but that already covers quite a lot of tasks - and then of course text can be an API call.
    - hunter2_ 4y ago
      Interesting perspective. I'm still learning about what it really is, and I'm having trouble marrying the thoughts of a parent commenter with yours: > ... does what it is engineered to do pretty well, which is, generate text that is representative of its training data following on from input tokens. It can't reason ... versus > ... doesn't just emulate human language - it accepts tasks in that language, including such tasks that require reasoning to perform ... Maybe a third party can jump in here: does ChatGPT use reasoning beyond the domain of language, or not?
      - int_19h 4y ago
        Nobody can definitely answer this question because we don't know what exactly is going on inside the model of that size. We can only speculate based on the observed behavior. But in this case, I didn't imply that it's "reasoning beyond the domain of language", in a sense that language is exactly what it uses to reason. If you force it to perform tasks without intermediate or final outputs that are meaningful text, the result is far worse. Conversely, if you tell it to "think out loud", the results are significantly better for most tasks. Here's one example from GPT-4 where the "thinking" effectively becomes a self-prompt for the corresponding SQL query: https://gist.github.com/int19h/4f5b98bcb9fab124d308efc19e5309fb https://gist.github.com/int19h/4f5b98bcb9fab124d308efc19e530.... Or here's an even more interesting example where GPT-4 does this kind of "thinking out loud" unprompted: https://gist.github.com/int19h/8251bd00b7a4858a69cf3922ae674dcc https://gist.github.com/int19h/8251bd00b7a4858a69cf3922ae674... I think the real point of disagreement is whether this constitutes actual reasoning or "merely completing tokens". If you showed the transcript of a chat with GPT-4 solving a multi-step task to a random person off the street, I have no doubt that they'd describe it as reasoning. Beyond that, one can pick the definition of "reason" that best fits their interpretation - there is no shortage of them, just as there is no shortage of definitions for "intelligence", "consciousness" etc.
llamaLord 4y ago
I completely disagree with this idea that the model doesn't "intend" to mislead. It's trained, atleast to some degree, based on human feedback. Humans are going to prefer an answer vs no answer, and humans can be easily fooled into believing confident misinformation. How does it not stand to reason that somewhere in that big ball of vector math there might be a rationale something along the lines of "humans are more likely to respond positively to a highly convincing lie that answers their question, than they are to to a truthful response which doesn't tell them what they want, therefore the logical thing for me to do is lie as that's what will make the humans press the thumbs up button instead of the thumbs down button".
- flanked-evergl 4y ago
  > How does it not stand to reason that somewhere in that big ball of vector math there might be a rationale I think, a suggestion that it is actually reasoning along these lines would need more than "it is possible". What evidence would refute your claim in your eyes, what would make it clear to you that "that big ball of vector mat" has no rationale, and is not just trying to trick humans to press the thumbs up? Of course the feedback is used to help control the output, so things that people downvote will be less likely to show up, but I have nothing to suggest to me that it is reasoning. If you think it has intent, you have to explain by what mechanism it obtained it. Could it be emergent? Sure, it could be, I don't think it is, I have never seen anything that suggests it has anything that could be compatible with intent, but I'm open to some evidence that it has. What I'm entirely convinced about is that it does what it was designed to do, which is generate output representative of its training data.
  - llamaLord 4y ago
    I would at least start to be convinced that this is NOT the case if I ever saw it respond with something like "I actually don't know the answer to that query" or "as far as I'm aware, there is no way to do the thing you asked". These are responses that would have shown up innumerable times in it's training data and make perfect sense as "the most logical next set of tokens", and yet it will never say them. Instead it will hallucinate something that sounds nearly indistinguishable from fact, but turns out to be a total fabrication. If all its doing is returning the next most logical set of tokens, and the training data it was based on included a non-trivial number of examples where one party in the conversation didn't have a clear answer, then there's no reason GPT-4 should be so averse to simply saying "yeah, I dunno bro". The only logical reason I can see is that it's "learned" that it's more likely to receive the positive feedback signal when it makes up convincing bullshit, than if it states that it doesn't have an answer. EDIT: To be clear, I mean it telling me it doesn't know something BEFORE hallucinating something incorrect and being caught out on it by me. It will admit that it lied, AFTER being caught, but it will never (in my experience) state that it doesn't have an answer for something upfront, and will instead default to hallucinating. Also - even when it does admit to lying, it will often then correct itself with an equally convincing, but often just as untrue "correction" to its original lie. Honestly, anyone who wants to learn how to gaslight people just needs to spend a decent amount of time around GPT-4.
joe_the_user 4y ago
ChatGPT is not lying to people, it can't lie, at least not in the sense of "to make an untrue statement with intent to deceive". ChatGPT doesn't really have a conception of true. It puts forward true and false things merely because it's cobbling together stuff in it's training set according to some weighing system. ChatGPT doesn't have an intent but merely by following the pattern of how humans put forward their claims, ChatGPT puts forward it's claims in a fashion that tends to get them accepted. So without a human-like intent, ChatGPT is going to be not just saying falsehoods but "selling" these falsehoods. And here, I'd be in agreement with the article that the distinction between this and "lying" is kind of quibbling.
IIAOPSW 4y ago
In short, it is not a liar its a bullshitter. A liar misrepresents facts, a bullshitter doesn't care if what they say is true so long as they pass in conversation.
ramblenode 4y ago
> ChatGPT is not lying to people, it can't lie I think the discussion of whether an LLM can technically lie is a red herring. The answer you get from an LLM isn't just a set of facts and truth values; it is also a conversational style and tone. It's training data isn't a graph of facts; it's human conversation, including arrogance, deflection, defensiveness, and deceit. If the LLM regurgitates text in a style that matches our human understanding of what a narcissistic and deceitful reply looks like, it seems reasonable we could call the response deceitful. The conversation around whether ChatGPT can technically lie seems to just be splitting hairs over whether the response is itself a lie or is merely an untrue statement in the style of a lie--a distinction which probably isn't meaningful most of the time. Ultimately, tone, style, truth, and falsity are just qualia we humans are imputing onto a statistically arranged string of tokens. In the same way that ChatGPT can't lie it also can't be correct or incorrect, as that too is imputing some kind of meaning where there isn't any.
westcoast49 4y ago
I think of ChatGPT as a natural language query engine for unstructured data. It knows about the relationships that are described in it's natural language input training data set, and it allows the same relationships to be queried from a wide range of different angles using queries that are also formulated in natural language. When it hallucinates, I find that it's usually because I'm asking it about a fringe topic where it's training data set is more sparse, or where the logical connections are deeper than it's currently able to "see", a sort of a horizon effect.

danpat 4y ago

> but why would you expect that from it? If you understand what it is doing, then you don't. But the layman will just see a computer that talks in language they understand, and will infer intent and sentience are behind that, because that's the only analog they have for a thing that can talk back to them with words that appear to make sense at the complexity level that ChatGPT is achieving. Most humans do not have sufficient background to understand what they're really being presented with, they will take it at face value.

hunter2_ 4y ago
I think this is partly explained by most of the marketing and news essentially saying "ChatGPT is an AI" instead of "ChatGPT is an LLM." If you asked me what AI is, I'd say it means getting a computer to emulate human intelligence; if you asked me what an LLM is, I'd say it means getting a computer to emulate human language. The word "language" does not imply truthiness anywhere near to the extent that the word "intelligence" does.
- int_19h 4y ago
  You could reasonably describe it as "human language emulator" back when people were using GPT-2 and the likes to compose text. But what we have today doesn't just emulate human language - it accepts tasks in that language, including such tasks that require reasoning to perform, and then carries them out. Granted, the only thing it can really "do" is produce text, but that already covers quite a lot of tasks - and then of course text can be an API call.
  - hunter2_ 4y ago
    Interesting perspective. I'm still learning about what it really is, and I'm having trouble marrying the thoughts of a parent commenter with yours: > ... does what it is engineered to do pretty well, which is, generate text that is representative of its training data following on from input tokens. It can't reason ... versus > ... doesn't just emulate human language - it accepts tasks in that language, including such tasks that require reasoning to perform ... Maybe a third party can jump in here: does ChatGPT use reasoning beyond the domain of language, or not?
    - int_19h 4y ago
      Nobody can definitely answer this question because we don't know what exactly is going on inside the model of that size. We can only speculate based on the observed behavior. But in this case, I didn't imply that it's "reasoning beyond the domain of language", in a sense that language is exactly what it uses to reason. If you force it to perform tasks without intermediate or final outputs that are meaningful text, the result is far worse. Conversely, if you tell it to "think out loud", the results are significantly better for most tasks. Here's one example from GPT-4 where the "thinking" effectively becomes a self-prompt for the corresponding SQL query: https://gist.github.com/int19h/4f5b98bcb9fab124d308efc19e5309fb https://gist.github.com/int19h/4f5b98bcb9fab124d308efc19e530.... Or here's an even more interesting example where GPT-4 does this kind of "thinking out loud" unprompted: https://gist.github.com/int19h/8251bd00b7a4858a69cf3922ae674dcc https://gist.github.com/int19h/8251bd00b7a4858a69cf3922ae674... I think the real point of disagreement is whether this constitutes actual reasoning or "merely completing tokens". If you showed the transcript of a chat with GPT-4 solving a multi-step task to a random person off the street, I have no doubt that they'd describe it as reasoning. Beyond that, one can pick the definition of "reason" that best fits their interpretation - there is no shortage of them, just as there is no shortage of definitions for "intelligence", "consciousness" etc.

llamaLord 4y ago

I completely disagree with this idea that the model doesn't "intend" to mislead. It's trained, atleast to some degree, based on human feedback. Humans are going to prefer an answer vs no answer, and humans can be easily fooled into believing confident misinformation. How does it not stand to reason that somewhere in that big ball of vector math there might be a rationale something along the lines of "humans are more likely to respond positively to a highly convincing lie that answers their question, than they are to to a truthful response which doesn't tell them what they want, therefore the logical thing for me to do is lie as that's what will make the humans press the thumbs up button instead of the thumbs down button".

flanked-evergl 4y ago
> How does it not stand to reason that somewhere in that big ball of vector math there might be a rationale I think, a suggestion that it is actually reasoning along these lines would need more than "it is possible". What evidence would refute your claim in your eyes, what would make it clear to you that "that big ball of vector mat" has no rationale, and is not just trying to trick humans to press the thumbs up? Of course the feedback is used to help control the output, so things that people downvote will be less likely to show up, but I have nothing to suggest to me that it is reasoning. If you think it has intent, you have to explain by what mechanism it obtained it. Could it be emergent? Sure, it could be, I don't think it is, I have never seen anything that suggests it has anything that could be compatible with intent, but I'm open to some evidence that it has. What I'm entirely convinced about is that it does what it was designed to do, which is generate output representative of its training data.
- llamaLord 4y ago
  I would at least start to be convinced that this is NOT the case if I ever saw it respond with something like "I actually don't know the answer to that query" or "as far as I'm aware, there is no way to do the thing you asked". These are responses that would have shown up innumerable times in it's training data and make perfect sense as "the most logical next set of tokens", and yet it will never say them. Instead it will hallucinate something that sounds nearly indistinguishable from fact, but turns out to be a total fabrication. If all its doing is returning the next most logical set of tokens, and the training data it was based on included a non-trivial number of examples where one party in the conversation didn't have a clear answer, then there's no reason GPT-4 should be so averse to simply saying "yeah, I dunno bro". The only logical reason I can see is that it's "learned" that it's more likely to receive the positive feedback signal when it makes up convincing bullshit, than if it states that it doesn't have an answer. EDIT: To be clear, I mean it telling me it doesn't know something BEFORE hallucinating something incorrect and being caught out on it by me. It will admit that it lied, AFTER being caught, but it will never (in my experience) state that it doesn't have an answer for something upfront, and will instead default to hallucinating. Also - even when it does admit to lying, it will often then correct itself with an equally convincing, but often just as untrue "correction" to its original lie. Honestly, anyone who wants to learn how to gaslight people just needs to spend a decent amount of time around GPT-4.

joe_the_user 4y ago

ChatGPT is not lying to people, it can't lie, at least not in the sense of "to make an untrue statement with intent to deceive". ChatGPT doesn't really have a conception of true. It puts forward true and false things merely because it's cobbling together stuff in it's training set according to some weighing system. ChatGPT doesn't have an intent but merely by following the pattern of how humans put forward their claims, ChatGPT puts forward it's claims in a fashion that tends to get them accepted. So without a human-like intent, ChatGPT is going to be not just saying falsehoods but "selling" these falsehoods. And here, I'd be in agreement with the article that the distinction between this and "lying" is kind of quibbling.

IIAOPSW 4y ago

In short, it is not a liar its a bullshitter. A liar misrepresents facts, a bullshitter doesn't care if what they say is true so long as they pass in conversation.

ramblenode 4y ago

> ChatGPT is not lying to people, it can't lie I think the discussion of whether an LLM can technically lie is a red herring. The answer you get from an LLM isn't just a set of facts and truth values; it is also a conversational style and tone. It's training data isn't a graph of facts; it's human conversation, including arrogance, deflection, defensiveness, and deceit. If the LLM regurgitates text in a style that matches our human understanding of what a narcissistic and deceitful reply looks like, it seems reasonable we could call the response deceitful. The conversation around whether ChatGPT can technically lie seems to just be splitting hairs over whether the response is itself a lie or is merely an untrue statement in the style of a lie--a distinction which probably isn't meaningful most of the time. Ultimately, tone, style, truth, and falsity are just qualia we humans are imputing onto a statistically arranged string of tokens. In the same way that ChatGPT can't lie it also can't be correct or incorrect, as that too is imputing some kind of meaning where there isn't any.

westcoast49 4y ago

I think of ChatGPT as a natural language query engine for unstructured data. It knows about the relationships that are described in it's natural language input training data set, and it allows the same relationships to be queried from a wide range of different angles using queries that are also formulated in natural language. When it hallucinates, I find that it's usually because I'm asking it about a fringe topic where it's training data set is more sparse, or where the logical connections are deeper than it's currently able to "see", a sort of a horizon effect.