6 ms·

I wonder if the biggest shortcoming of GPT right now is not that it sometimes gets things wrong, or can't cite its sources, or whatever - maybe it needs to lear

I wonder if the biggest shortcoming of GPT right now is not that it sometimes gets things wrong, or can't cite its sources, or whatever - maybe it needs to learn when to say "I don't know the answer to that question". That's a pretty hard thing for most humans (and myself) to learn to say, and I suspect GPT's training data (tha internet) doesn't include a lot of "I'm not sure" language and probably does include a lot of "I'm definitely sure and definitely totally correct" language (maybe, I guess, no evidence to back up that suggestion, I'm not sure). Many of my favorite coworkers, friends, doctors, pundits are trustworthy exactly because they work hard to not profess knowledge they are unsure about. The reason (IMO) that Scott Alexander is a jewel of the internet is because of the way he quantifies uncertainty when working through a topic.

MagicMoonlight 4y ago
It can’t learn because it isn’t intelligent. It’s a markov chain. There is no way for it to make any decision at all. The fundamental system design is that it guesses the word that is most likely to be next. That’s it. There’s no concepts behind whatever it outputs, it has no idea what it means.
- skybrian 4y ago
  It's a next-word predictor like a Markov chain, but a Markov chain couldn't do all the things ChatGPT does. ChatGPT has learned lots of syntax-level patterns pretty well.
  - uh_uh 4y ago
    Is it actually a next-word predictor? I thought the training loss is against a set of words, not just one.
    - skybrian 4y ago
      I'm not sure what distinction you're getting at, but transformers do use "fill in the missing word" training and text generation chooses the next word (token actually) one at a time. Once it chooses a word, it doesn't go back.
- buzzerbetrayed 4y ago
  You’re just nitpicking. Everybody here knows that it isn’t intelligent. Words can mean more than one thing. And in this case the word “learn” is being used to describe what ChatGPT does to improve its answers over time. It’s perfectly reasonable to say that ChatGPT needs to lean to say “I don’t know”.
  - freejazz 4y ago
    No, it's just anthropomorphizing.
hooande 4y ago
You're flat out incorrect about this. GPT loves to say that it doesn't know things. Have you actually used it? Don't take my word for it. Ask it: "How many people are wearing tennis shoes in the United States right now?" The response I got: "I'm sorry, I don't have the current data on the number of people in the United States who are wearing tennis shoes right now. It's not possible to determine this number as it would require constantly monitoring the footwear of every person in the country, which is not feasible. The number would also vary greatly from moment to moment and would be influenced by a number of factors such as time of day, weather, and location." No hallucinations, no bullshitting. Just "I don't know" and an explanation of what it would take it find out. No idea why you think it has trouble expressing a lack of confidence in its answers.
- redox99 4y ago
  I've actually used it quite a bit and he's not "flat out incorrect". I'd say you're the one wrong. It might reply that when you ask it something that it can deduce it can't know, but ask it something reasonable and it will almost certainly hallucinate it instead of telling you it doesn't know about it. > What function in Unreal Engine makes an actor transparent? > The function to make an actor transparent in Unreal Engine is called "SetActorOpacity". (There is no function called SetActorOpacity)
- hesk 4y ago
  No, it's bullshit. Notice that the answer is not internally consistent. It first says that it does not have current data on that, which implies that it might have old data and that it would in principle be possible to generate that data. It then states that it is not possible to answer this question. Those two parts of the answer are not consistent with each other.
- camdenreslink 4y ago
  It isn’t saying “I don’t know,” it is saying “It can’t be known.” Which is actually an answer and something else entirely.
freejazz 4y ago
>"I wonder if the biggest shortcoming of GPT right now is not that it sometimes gets things wrong, or can't cite its sources, or whatever " It surprises me that people, here no less, seem to not realize that chatGPT has literally no idea what it's doing or saying. It's just trying to find the next word. It has no regard or understanding for what the word means. In response to this, people here just reflexively question what happens when a human is thinking, but I don't see that as a basis to assume that means chatGPT is thinking.

MagicMoonlight 4y ago

It can’t learn because it isn’t intelligent. It’s a markov chain. There is no way for it to make any decision at all. The fundamental system design is that it guesses the word that is most likely to be next. That’s it. There’s no concepts behind whatever it outputs, it has no idea what it means.

skybrian 4y ago
It's a next-word predictor like a Markov chain, but a Markov chain couldn't do all the things ChatGPT does. ChatGPT has learned lots of syntax-level patterns pretty well.
- uh_uh 4y ago
  Is it actually a next-word predictor? I thought the training loss is against a set of words, not just one.
  - skybrian 4y ago
    I'm not sure what distinction you're getting at, but transformers do use "fill in the missing word" training and text generation chooses the next word (token actually) one at a time. Once it chooses a word, it doesn't go back.
buzzerbetrayed 4y ago
You’re just nitpicking. Everybody here knows that it isn’t intelligent. Words can mean more than one thing. And in this case the word “learn” is being used to describe what ChatGPT does to improve its answers over time. It’s perfectly reasonable to say that ChatGPT needs to lean to say “I don’t know”.
- freejazz 4y ago
  No, it's just anthropomorphizing.

hooande 4y ago

You're flat out incorrect about this. GPT loves to say that it doesn't know things. Have you actually used it? Don't take my word for it. Ask it: "How many people are wearing tennis shoes in the United States right now?" The response I got: "I'm sorry, I don't have the current data on the number of people in the United States who are wearing tennis shoes right now. It's not possible to determine this number as it would require constantly monitoring the footwear of every person in the country, which is not feasible. The number would also vary greatly from moment to moment and would be influenced by a number of factors such as time of day, weather, and location." No hallucinations, no bullshitting. Just "I don't know" and an explanation of what it would take it find out. No idea why you think it has trouble expressing a lack of confidence in its answers.

redox99 4y ago
I've actually used it quite a bit and he's not "flat out incorrect". I'd say you're the one wrong. It might reply that when you ask it something that it can deduce it can't know, but ask it something reasonable and it will almost certainly hallucinate it instead of telling you it doesn't know about it. > What function in Unreal Engine makes an actor transparent? > The function to make an actor transparent in Unreal Engine is called "SetActorOpacity". (There is no function called SetActorOpacity)
hesk 4y ago
No, it's bullshit. Notice that the answer is not internally consistent. It first says that it does not have current data on that, which implies that it might have old data and that it would in principle be possible to generate that data. It then states that it is not possible to answer this question. Those two parts of the answer are not consistent with each other.
camdenreslink 4y ago
It isn’t saying “I don’t know,” it is saying “It can’t be known.” Which is actually an answer and something else entirely.

freejazz 4y ago

>"I wonder if the biggest shortcoming of GPT right now is not that it sometimes gets things wrong, or can't cite its sources, or whatever " It surprises me that people, here no less, seem to not realize that chatGPT has literally no idea what it's doing or saying. It's just trying to find the next word. It has no regard or understanding for what the word means. In response to this, people here just reflexively question what happens when a human is thinking, but I don't see that as a basis to assume that means chatGPT is thinking.