5 ms·

I completely disagree with this idea that the model doesn't "intend" to mislead. It's trained, atleast to some degree, based on human feedback. Humans are goin

I completely disagree with this idea that the model doesn't "intend" to mislead. It's trained, atleast to some degree, based on human feedback. Humans are going to prefer an answer vs no answer, and humans can be easily fooled into believing confident misinformation. How does it not stand to reason that somewhere in that big ball of vector math there might be a rationale something along the lines of "humans are more likely to respond positively to a highly convincing lie that answers their question, than they are to to a truthful response which doesn't tell them what they want, therefore the logical thing for me to do is lie as that's what will make the humans press the thumbs up button instead of the thumbs down button".

flanked-evergl 4y ago
> How does it not stand to reason that somewhere in that big ball of vector math there might be a rationale I think, a suggestion that it is actually reasoning along these lines would need more than "it is possible". What evidence would refute your claim in your eyes, what would make it clear to you that "that big ball of vector mat" has no rationale, and is not just trying to trick humans to press the thumbs up? Of course the feedback is used to help control the output, so things that people downvote will be less likely to show up, but I have nothing to suggest to me that it is reasoning. If you think it has intent, you have to explain by what mechanism it obtained it. Could it be emergent? Sure, it could be, I don't think it is, I have never seen anything that suggests it has anything that could be compatible with intent, but I'm open to some evidence that it has. What I'm entirely convinced about is that it does what it was designed to do, which is generate output representative of its training data.
- llamaLord 4y ago
  I would at least start to be convinced that this is NOT the case if I ever saw it respond with something like "I actually don't know the answer to that query" or "as far as I'm aware, there is no way to do the thing you asked". These are responses that would have shown up innumerable times in it's training data and make perfect sense as "the most logical next set of tokens", and yet it will never say them. Instead it will hallucinate something that sounds nearly indistinguishable from fact, but turns out to be a total fabrication. If all its doing is returning the next most logical set of tokens, and the training data it was based on included a non-trivial number of examples where one party in the conversation didn't have a clear answer, then there's no reason GPT-4 should be so averse to simply saying "yeah, I dunno bro". The only logical reason I can see is that it's "learned" that it's more likely to receive the positive feedback signal when it makes up convincing bullshit, than if it states that it doesn't have an answer. EDIT: To be clear, I mean it telling me it doesn't know something BEFORE hallucinating something incorrect and being caught out on it by me. It will admit that it lied, AFTER being caught, but it will never (in my experience) state that it doesn't have an answer for something upfront, and will instead default to hallucinating. Also - even when it does admit to lying, it will often then correct itself with an equally convincing, but often just as untrue "correction" to its original lie. Honestly, anyone who wants to learn how to gaslight people just needs to spend a decent amount of time around GPT-4.
zuminator 4y ago
I don't think it intends to mislead because its answers are probabilistic. It's designed to distill a best guess out of data which is almost certain to be incomplete or conflicting. As human beings we do the same thing all the time. However we have real life experience of having our best guesses bump up against reality and lose. ChatGPT can't see reality. It only knows what "really being wrong" is to the extent that we tell it. Even with our advantage of interacting with the real world, I'd still wager that the average person's no better (and probably worse) than ChatGPT for uttering factual truth. It's pretty common to encounter people in life who will confidently utter things like, "Mao Zedong was a top member of the Illuminati and vacationed annually in the Azores with Prescott Bush" or "The oxygen cycle is just a hoax by environmental wackjobs to get us to think we need trees to survive," and to make such statements confidently with no intent to mislead.
- flanked-evergl 4y ago
  > Even with our advantage of interacting with the real world, I'd still wager that the average person's no better (and probably worse) than ChatGPT for uttering factual truth. ChatGPT makes up non-existing APIs for Google cloud and Go out of whole cloth. I have never met a human who does that. If we reduce it down to how often most people are wrong vs how often ChatGPT is wrong, then sure, people may be on average wrong more often, but there is a difference in how people are wrong vs how ChatGPT is wrong.
  - 8note 4y ago
    I wouldn't be surprised if there's very successfulsoftware hustlers that do make up Google cloud apis. You may not know them but that doesn't mean they don't exist. 5 years down the line though, maybe those apis will exist because chatgpt is giving a summary of what apis Google cloud should have, and Google will listen
    - shjake 4y ago
      > Google cloud should have, and Google will listen Why? Isn’t what ChatGPT suggesting just random, essentially incoherent noise in those cases? Are there any examples at all of it coming up with something actually useful (and something humans haven’t thought of)?

flanked-evergl 4y ago

> How does it not stand to reason that somewhere in that big ball of vector math there might be a rationale I think, a suggestion that it is actually reasoning along these lines would need more than "it is possible". What evidence would refute your claim in your eyes, what would make it clear to you that "that big ball of vector mat" has no rationale, and is not just trying to trick humans to press the thumbs up? Of course the feedback is used to help control the output, so things that people downvote will be less likely to show up, but I have nothing to suggest to me that it is reasoning. If you think it has intent, you have to explain by what mechanism it obtained it. Could it be emergent? Sure, it could be, I don't think it is, I have never seen anything that suggests it has anything that could be compatible with intent, but I'm open to some evidence that it has. What I'm entirely convinced about is that it does what it was designed to do, which is generate output representative of its training data.

llamaLord 4y ago
I would at least start to be convinced that this is NOT the case if I ever saw it respond with something like "I actually don't know the answer to that query" or "as far as I'm aware, there is no way to do the thing you asked". These are responses that would have shown up innumerable times in it's training data and make perfect sense as "the most logical next set of tokens", and yet it will never say them. Instead it will hallucinate something that sounds nearly indistinguishable from fact, but turns out to be a total fabrication. If all its doing is returning the next most logical set of tokens, and the training data it was based on included a non-trivial number of examples where one party in the conversation didn't have a clear answer, then there's no reason GPT-4 should be so averse to simply saying "yeah, I dunno bro". The only logical reason I can see is that it's "learned" that it's more likely to receive the positive feedback signal when it makes up convincing bullshit, than if it states that it doesn't have an answer. EDIT: To be clear, I mean it telling me it doesn't know something BEFORE hallucinating something incorrect and being caught out on it by me. It will admit that it lied, AFTER being caught, but it will never (in my experience) state that it doesn't have an answer for something upfront, and will instead default to hallucinating. Also - even when it does admit to lying, it will often then correct itself with an equally convincing, but often just as untrue "correction" to its original lie. Honestly, anyone who wants to learn how to gaslight people just needs to spend a decent amount of time around GPT-4.

zuminator 4y ago

I don't think it intends to mislead because its answers are probabilistic. It's designed to distill a best guess out of data which is almost certain to be incomplete or conflicting. As human beings we do the same thing all the time. However we have real life experience of having our best guesses bump up against reality and lose. ChatGPT can't see reality. It only knows what "really being wrong" is to the extent that we tell it. Even with our advantage of interacting with the real world, I'd still wager that the average person's no better (and probably worse) than ChatGPT for uttering factual truth. It's pretty common to encounter people in life who will confidently utter things like, "Mao Zedong was a top member of the Illuminati and vacationed annually in the Azores with Prescott Bush" or "The oxygen cycle is just a hoax by environmental wackjobs to get us to think we need trees to survive," and to make such statements confidently with no intent to mislead.

flanked-evergl 4y ago
> Even with our advantage of interacting with the real world, I'd still wager that the average person's no better (and probably worse) than ChatGPT for uttering factual truth. ChatGPT makes up non-existing APIs for Google cloud and Go out of whole cloth. I have never met a human who does that. If we reduce it down to how often most people are wrong vs how often ChatGPT is wrong, then sure, people may be on average wrong more often, but there is a difference in how people are wrong vs how ChatGPT is wrong.
- 8note 4y ago
  I wouldn't be surprised if there's very successfulsoftware hustlers that do make up Google cloud apis. You may not know them but that doesn't mean they don't exist. 5 years down the line though, maybe those apis will exist because chatgpt is giving a summary of what apis Google cloud should have, and Google will listen
  - shjake 4y ago
    > Google cloud should have, and Google will listen Why? Isn’t what ChatGPT suggesting just random, essentially incoherent noise in those cases? Are there any examples at all of it coming up with something actually useful (and something humans haven’t thought of)?