3 ms·

GPT-3 seems to refute these arguments. It is able to solve most understanding tasks reasonably well.

by namlem 5y ago

HarHarVeryFunny 5y ago
Perhaps, but "being able to do X" isn't the same as "being able to do X, using approach Y". After all, Deep Blue could play chess more than reasonably well, but no-one would suggest that it understood chess anymore than a calculator understands math. So, is GPT-3 more like a brute force calculator like Deep Blue (or AlphaGo for that matter), or is it more like a human in it's ability to "understand" language ? Given that GPT-3 is a language model, just trained to minimize errors when predicting next word in a sentence, and given the nature of it's transformer architecture, I'd say it's pretty clear it's really just another Deep Blue or AlphaGo. It's a "predict next word" brute force calculator, that - interestingly - is nonetheless able to perform many "NLU" tasks surprisingly well.
- namlem 5y ago
  AlphaGo is not merely a brute force calculator and neither is GPT-3. They have conceptual representations. That is the advantage of deep neural networks, the ability to form concepts, as biological neural networks do.
  - HarHarVeryFunny 5y ago
    They can form their own internal vector* representations, but they are still fixed architecture. Training AlphaGo can't change it from an RL architecture to something else, nor can training GPT-3 change it from a transformer into a more general cognitive architecture such as human brain. The datapath through a transformer is entirely prescribed. Once the weights are trained, feeding a test sample into in will indeed just result in a fixed series of calculations to produce an output. * It's a bit of a stretch to assert that deep neural nets are creating conceptual representations. For example if you look at the early layers of a CNN, what it's learned are just primitive (orientated edge, etc) feature detectors. At higher levels the more primitive features are combined into more complex ones, but they are just visual patterns, not concepts. It'll be the same for AlphaGo - it'll be creating it's own representations of complex board positions. Better than Deep Blue having to work with a human-design board representation, but at the end of the day it is nothing more than a board/position representation.
    - namlem 5y ago
      High level vector representations are concepts. That's what concepts are in the human brain too.
      - HarHarVeryFunny 5y ago
        What about a face embedding? The vector representation is the coordinates of the face (i.e. that combination of facial features) in face-space. I give up.

GPT-3 seems to refute these arguments. It is able to solve most understanding tasks reasonably well.

by namlem 5y ago

GPT-3 seems to refute these arguments. It is able to solve most understanding tasks reasonably well.

Perhaps, but "being able to do X" isn't the same as "being able to do X, using approach Y". After all, Deep Blue could play chess more than reasonably well, but no-one would suggest that it understood chess anymore than a calculator understands math. So, is GPT-3 more like a brute force calculator like Deep Blue (or AlphaGo for that matter), or is it more like a human in it's ability to "understand" language ? Given that GPT-3 is a language model, just trained to minimize errors when predicting next word in a sentence, and given the nature of it's transformer architecture, I'd say it's pretty clear it's really just another Deep Blue or AlphaGo. It's a "predict next word" brute force calculator, that - interestingly - is nonetheless able to perform many "NLU" tasks surprisingly well.

namlem 5y ago
AlphaGo is not merely a brute force calculator and neither is GPT-3. They have conceptual representations. That is the advantage of deep neural networks, the ability to form concepts, as biological neural networks do.
- HarHarVeryFunny 5y ago
  They can form their own internal vector* representations, but they are still fixed architecture. Training AlphaGo can't change it from an RL architecture to something else, nor can training GPT-3 change it from a transformer into a more general cognitive architecture such as human brain. The datapath through a transformer is entirely prescribed. Once the weights are trained, feeding a test sample into in will indeed just result in a fixed series of calculations to produce an output. * It's a bit of a stretch to assert that deep neural nets are creating conceptual representations. For example if you look at the early layers of a CNN, what it's learned are just primitive (orientated edge, etc) feature detectors. At higher levels the more primitive features are combined into more complex ones, but they are just visual patterns, not concepts. It'll be the same for AlphaGo - it'll be creating it's own representations of complex board positions. Better than Deep Blue having to work with a human-design board representation, but at the end of the day it is nothing more than a board/position representation.
  - namlem 5y ago
    High level vector representations are concepts. That's what concepts are in the human brain too.
    - HarHarVeryFunny 5y ago
      What about a face embedding? The vector representation is the coordinates of the face (i.e. that combination of facial features) in face-space. I give up.