5 ms·

The GPT approach must be very poor; with only a fraction of that information you can train a human to perform better than any GPT model shown thus far. When yo

The GPT approach must be very poor; with only a fraction of that information you can train a human to perform better than any GPT model shown thus far. When you look at power burned doing it, the gap is even wider. Obviously our wetware is optimized for it and we aren't true blank slates, but the enormous magnitude of the discrepancy makes me think we're on the wrong path.

monsieurbanana 6y ago
I'm sure it's much more probable than we're in the wrong path, rather than the good one, but I don't think the amount of information needed by a GPT model can give insights to that. Instead, as you pointed out, it's just proof of how efficient our wetware is.
gamblor956 6y ago
GPT and most modern "AI" approaches suffer from the same problem: they're just brute force algorithmic approaches that don't mimic "intelligence" in any way. Since there is no understanding, they can't learn in the same way humans or other living creatures do (in general terms, by extrapolation). The ironic thing is that ML became popular because we didn't have the technology back then to properly model actual neuronal activity. Now we do, but ML is so established that modeling "true" intelligence is no longer de jure.
- nialv7 6y ago
  That's a bold claim to make, given we ourselves don't even understand what understanding, or intelligence is. So how can you support your claim that what GPT-3 does is not intelligence.
  - gamblor956 6y ago
    I'm not the only one. It's industry consensus. https://www.technologyreview.com/2020/08/22/1007539/gpt3-openai-language-generator-artificial-intelligence-ai-opinion/ https://www.technologyreview.com/2020/08/22/1007539/gpt3-ope...
    - nialv7 6y ago
      That doesn't answer the question.
  - bigbubba 6y ago
    Personally I think our common definitions of intelligence are too narrowly defined, we have a bias towards recognizing intelligence similar to our own and a blindspot for intelligent systems unlike our own. I agree with Paul Stamets that mycelia networks may have a form of intelligence quite unlike our own. Large corporations likely have intelligence of their own ('slow AIs') and may in aggregate pursue goals that no individual participant in the organization agrees with (preference falsification demonstrates how such a thing might occur.) Closer to the traditional, I think reptiles may be far more intelligent that we usually give them credit for since we have a mammalian bias against their kind. Mammalians seem to have empathy circuits that fire more often for other mammals, and this inclines us to not consider the possible intelligence of non-mammals. So, GPT? I think that's probably a form of intelligence too, but not one that's particularly similar to our own. I also doubt that we'll have much luck getting it to ever perform at near-human levels, when you consider the data and power requirements. I think it's not simply a matter of silicon vs wetware; I think the GPT approach is substantially more different from our own than the hardware it runs on. It's a form of intelligence but that doesn't mean it is like our own.
nialv7 6y ago
> with only a fraction of that information you can train a human to perform better than any GPT model shown thus far. That's not a fair comparison. The human neural network is not trained _from scratch_. There is a very good fundamental structure we inherited through millions of years of evolution.
- bigbubba 6y ago
  Yes, I conceded that we aren't blank slates. I still think we're on the wring track.
- webmaven 6y ago
  > The human neural network is not trained _from scratch_. There is a very good fundamental structure we inherited through millions of years of evolution. It is worth noting that the training data used for training humans has been undergoing it's own optimization process for a while too (arguably for about 40ky, since the Upper Paleolithic Revolution, which is roughly when cultural evolution started taking over). In AI adjusting the training data is called Curriculum Learning. Right now most curriculum learning is done just by adjusting the order of training samples, rather than creating samples specifically optimized to facilitate learning (although GANs might be considered a Socratic approach to the latter, if you squint at it).
- jononor 6y ago
  Also, humans created all the text and the language and structures used within, for the purpose of communicating - so it is likely to be a good match for our cognitive systems.

monsieurbanana 6y ago

I'm sure it's much more probable than we're in the wrong path, rather than the good one, but I don't think the amount of information needed by a GPT model can give insights to that. Instead, as you pointed out, it's just proof of how efficient our wetware is.

gamblor956 6y ago

GPT and most modern "AI" approaches suffer from the same problem: they're just brute force algorithmic approaches that don't mimic "intelligence" in any way. Since there is no understanding, they can't learn in the same way humans or other living creatures do (in general terms, by extrapolation). The ironic thing is that ML became popular because we didn't have the technology back then to properly model actual neuronal activity. Now we do, but ML is so established that modeling "true" intelligence is no longer de jure.

nialv7 6y ago
That's a bold claim to make, given we ourselves don't even understand what understanding, or intelligence is. So how can you support your claim that what GPT-3 does is not intelligence.
- gamblor956 6y ago
  I'm not the only one. It's industry consensus. https://www.technologyreview.com/2020/08/22/1007539/gpt3-openai-language-generator-artificial-intelligence-ai-opinion/ https://www.technologyreview.com/2020/08/22/1007539/gpt3-ope...
  - nialv7 6y ago
    That doesn't answer the question.
- bigbubba 6y ago
  Personally I think our common definitions of intelligence are too narrowly defined, we have a bias towards recognizing intelligence similar to our own and a blindspot for intelligent systems unlike our own. I agree with Paul Stamets that mycelia networks may have a form of intelligence quite unlike our own. Large corporations likely have intelligence of their own ('slow AIs') and may in aggregate pursue goals that no individual participant in the organization agrees with (preference falsification demonstrates how such a thing might occur.) Closer to the traditional, I think reptiles may be far more intelligent that we usually give them credit for since we have a mammalian bias against their kind. Mammalians seem to have empathy circuits that fire more often for other mammals, and this inclines us to not consider the possible intelligence of non-mammals. So, GPT? I think that's probably a form of intelligence too, but not one that's particularly similar to our own. I also doubt that we'll have much luck getting it to ever perform at near-human levels, when you consider the data and power requirements. I think it's not simply a matter of silicon vs wetware; I think the GPT approach is substantially more different from our own than the hardware it runs on. It's a form of intelligence but that doesn't mean it is like our own.

nialv7 6y ago

> with only a fraction of that information you can train a human to perform better than any GPT model shown thus far. That's not a fair comparison. The human neural network is not trained _from scratch_. There is a very good fundamental structure we inherited through millions of years of evolution.

bigbubba 6y ago
Yes, I conceded that we aren't blank slates. I still think we're on the wring track.
webmaven 6y ago
> The human neural network is not trained _from scratch_. There is a very good fundamental structure we inherited through millions of years of evolution. It is worth noting that the training data used for training humans has been undergoing it's own optimization process for a while too (arguably for about 40ky, since the Upper Paleolithic Revolution, which is roughly when cultural evolution started taking over). In AI adjusting the training data is called Curriculum Learning. Right now most curriculum learning is done just by adjusting the order of training samples, rather than creating samples specifically optimized to facilitate learning (although GANs might be considered a Socratic approach to the latter, if you squint at it).
jononor 6y ago
Also, humans created all the text and the language and structures used within, for the purpose of communicating - so it is likely to be a good match for our cognitive systems.