5 ms·

What struck me more about the last answer wasn't that GPT-3 didn't get the riddle, but that it couldn't interpret "no legs" as "zero legs" or similar, even in a

What struck me more about the last answer wasn't that GPT-3 didn't get the riddle, but that it couldn't interpret "no legs" as "zero legs" or similar, even in attempting to solve the riddle.

blixt 6y ago
Since GPT-3 transforms sentences into high-dimensional vectors with connections between all the words and of course its internal weights trained on both the English language and all languages in general, I find that it has a basically flawless ability to interpret normal language. I tested a prompt where the input was English and the output was Swedish, and it did a really good (but not always perfect) job at translating Swedish. I would assume the amount of Swedish in its training set is relatively tiny, so I find its ability to do that kind of translation in addition to all this other stuff really impressive. This also shows in its ability to translate really well between English and bash shell commands, or JSON data structures, or XML.
- aspenmayer 6y ago
  If this kind of thing interests you, look into the field of zero-shot learning. https://en.wikipedia.org/wiki/Zero-shot_learning https://en.wikipedia.org/wiki/Zero-shot_learning Zero-Shot Translation with Google’s Multilingual Neural Machine Translation System https://ai.googleblog.com/2016/11/zero-shot-translation-with-googles.html https://ai.googleblog.com/2016/11/zero-shot-translation-with... Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly https://ieeexplore.ieee.org/document/8413121 https://ieeexplore.ieee.org/document/8413121 also available at: https://arxiv.org/abs/1703.04394 https://arxiv.org/abs/1703.04394 mbsariyildiz / zsl_eval > Python implementation of zero-shot/generalized zero-shot evaluation framework proposed by Xian et al in Zero-Shot Learning - The Good, The Bad, The Ugly. https://github.com/mbsariyildiz/zsl_eval https://github.com/mbsariyildiz/zsl_eval
yellowstuff 6y ago
This is a wild guess- When GPT-3 got confused by "no legs", I don't think it's because it would have any trouble correctly interpreting "no legs" in other contexts. In this context it was searching in a semantic space set up by "a head, a tail, is brown", and the general form of a question looking for an animal. GPT-3 didn't find anything in the semantic space matching "no legs" so it assumed that was an error and corrected it. It's actually fairly close to how a human would solve the riddle, except when the human noticed nothing matched the explicit and implicit prompts, they would correct the implicit request for an animal, and then reinterpret the rest of the riddle.
- alexeldeib 6y ago
  A head, tail, is brown, no legs. For an animal that’s got me thinking of fish rather than typos :)
- LoSboccacc 6y ago
  > a head, a tail, is brown I wonder if brown would have been an issue, even without the animal context, as a search for object with these properties wouldn't have turned up the cent which is copper color it'd be interesting to ask gp3 what color is a cent and then redo the riddle with that
- red75prime 6y ago
  GPT-3 generates output word by word. And it doesn't make distinctions between prompt words and words it has generated. Generated words are selected somewhat randomly (if temperature >0). So "the train of GPT-3 thought" can be easily derailed by unfortunate random selection of a single word. When it generated "I don't know what you mean by", it had to continue with something relevant. In this case "no legs" was selected. In other words GPT-3 doesn't have machinery to stop and think "What I'm rambling about?".

What struck me more about the last answer wasn't that GPT-3 didn't get the riddle, but that it couldn't interpret "no legs" as "zero legs" or similar, even in attempting to solve the riddle.

blixt 6y ago

Since GPT-3 transforms sentences into high-dimensional vectors with connections between all the words and of course its internal weights trained on both the English language and all languages in general, I find that it has a basically flawless ability to interpret normal language. I tested a prompt where the input was English and the output was Swedish, and it did a really good (but not always perfect) job at translating Swedish. I would assume the amount of Swedish in its training set is relatively tiny, so I find its ability to do that kind of translation in addition to all this other stuff really impressive. This also shows in its ability to translate really well between English and bash shell commands, or JSON data structures, or XML.

aspenmayer 6y ago
If this kind of thing interests you, look into the field of zero-shot learning. https://en.wikipedia.org/wiki/Zero-shot_learning https://en.wikipedia.org/wiki/Zero-shot_learning Zero-Shot Translation with Google’s Multilingual Neural Machine Translation System https://ai.googleblog.com/2016/11/zero-shot-translation-with-googles.html https://ai.googleblog.com/2016/11/zero-shot-translation-with... Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly https://ieeexplore.ieee.org/document/8413121 https://ieeexplore.ieee.org/document/8413121 also available at: https://arxiv.org/abs/1703.04394 https://arxiv.org/abs/1703.04394 mbsariyildiz / zsl_eval > Python implementation of zero-shot/generalized zero-shot evaluation framework proposed by Xian et al in Zero-Shot Learning - The Good, The Bad, The Ugly. https://github.com/mbsariyildiz/zsl_eval https://github.com/mbsariyildiz/zsl_eval

yellowstuff 6y ago

This is a wild guess- When GPT-3 got confused by "no legs", I don't think it's because it would have any trouble correctly interpreting "no legs" in other contexts. In this context it was searching in a semantic space set up by "a head, a tail, is brown", and the general form of a question looking for an animal. GPT-3 didn't find anything in the semantic space matching "no legs" so it assumed that was an error and corrected it. It's actually fairly close to how a human would solve the riddle, except when the human noticed nothing matched the explicit and implicit prompts, they would correct the implicit request for an animal, and then reinterpret the rest of the riddle.

alexeldeib 6y ago
A head, tail, is brown, no legs. For an animal that’s got me thinking of fish rather than typos :)
LoSboccacc 6y ago
> a head, a tail, is brown I wonder if brown would have been an issue, even without the animal context, as a search for object with these properties wouldn't have turned up the cent which is copper color it'd be interesting to ask gp3 what color is a cent and then redo the riddle with that
red75prime 6y ago
GPT-3 generates output word by word. And it doesn't make distinctions between prompt words and words it has generated. Generated words are selected somewhat randomly (if temperature >0). So "the train of GPT-3 thought" can be easily derailed by unfortunate random selection of a single word. When it generated "I don't know what you mean by", it had to continue with something relevant. In this case "no legs" was selected. In other words GPT-3 doesn't have machinery to stop and think "What I'm rambling about?".