4 ms·

I'm failing to connect the anecdotes with the conclusion. He's claiming the the ML is scaling well but then gives the data on how GPT-3 is expensively brute-for

I'm failing to connect the anecdotes with the conclusion. He's claiming the the ML is scaling well but then gives the data on how GPT-3 is expensively brute-forcing its way to "success". To me it just seems like what supercomputing is to normal computing: It makes the computationally expensive stuff do-able in a reasonable amount of time, or gives diminishing returns on existing algorithms. But it doesn't magic in any real advancements. The problem in AI/ML and the concept of "AI winter" to me was always the barrier of the fact that we're just doing predictions, with no deep meaning or comprehension of features. The layman thinks there's magic, but there's not, so when that truth is revealed there will be problems. There's nothing intelligent about our Artificial Intelligence; we're just doing statistics with big data and added sprinkles. OpenAI just proved they could do statistics with even bigger data and more expensive sprinkles. Has their work really shown we can get past that core problem? Personally, I don't see it.

LordDragonfang 6y ago
>we're just doing statistics with big data and added sprinkles. I mean, you could pretty much say that's how the human brain works, couldn't you?
- personjerry 6y ago
  That comparison is like comparing gardening my backyard to agricultural farming. Technically the core concept is similar, but I think we both know that the scale and depth makes them not at all the same.
  - H8crilA 6y ago
    There's a continuity of small changes between your backyard and industrial farming. Just like there's a continuity of small changes between you and your cat.
    - p1esk 6y ago
      There’s a continuity of small changes between any two things.
      - FeepingCreature 6y ago
        No. edit: For instance, there is no continuity of small changes in height that will bring you from climbing up incrementally larger trees to climbing to the moon. The question is whether gpt-3 vs human intelligence is more like climbing a tree vs climbing a mountain or more like climbing a tree vs building a rocket.
- qeternity 6y ago
  Well, we don’t really know. I think that’s one element missing from most DL analysis: our understanding of the brain is incredibly limited. And ANNs are incredibly crude imitations of actual NNs. So we’ve built massive, extremely crude approximations of a system we don’t really understand. As is often the case, the truth is somewhere in the middle. It’s almost certain that we won’t reach AGI without a fundamental breakthrough on soft/wet-ware but it’s also nearly certain that even with the best algorithms we will need to efficiently harness and coordinate massive compute power, as we’re learning to do now.
- tsimionescu 6y ago
  That's possibly true at the 'implementation' layer, but human learning is quite different, or at elast has some aspects that are vastly different. Human manuals don't contain millions of tagged examples, they contain explanations and a few dozen examples. So sure, when we learn how to walk or see or read, we may be in this mode of simply discerning patterns in large amounts of data. But when we learn maths or programming, we are using a completely different kind of learning.
- saberience 6y ago
  But thats the point, our brain is clearly a lot more than just a big table of probabilities. You only need to look at the absolutely insane volume of data and training time that these models need. How much time does it take a for human to understand a concept like "love" and what volume of training data is required? Computers would just regurgitate quotes from poetry or novels about love without any real understanding after billions of hours of training time and ingesting every document on the internet. A human can understand love in a tiny fraction of the time and with a tiny fraction of the volume of information processed and they also understand it in a more fundamental way and can articulate that in a way these models cannot. You might argue back, well the human brain has pre-trained neural networks with billions of hours of training time. Well, that isn't really the case. We don't start off with some pre-existing memory of what "love" means, or what "Physics" is, or trillions of bytes of data. All we have is a capacity to learn which is highly efficient, a conscious mind which is aware of itself, and certain fundamental drives driven by our bodies and instincts. If you have a human child and give it zero input information it will never learn a language or be capable at all in any sense of the term. So we become incredibly capable based on a tiny fraction of input data fed into us after birth. The way the human brain and mind works is deeply tied in to the experience of having a body, knowing we are mortal, and having fundamental drives such as a drive to survive, eat, drink, keep ourselves safe, and also a drive to be social, find a mate, and procreate. I would argue that we will never be able to have a computer/algorithm that thinks like we do unless it also has drives like we do and a body like we do, since so much of our process of thinking is tied in to having a body, our awareness of mortality, and our basic human drives and experience.
  - keenmaster 6y ago
    X = A+B A= C+D B = E+F Love = X = A+B = C+D+E+F Obviously the above is contrived and abstracted, but you get my point. If I took a little bit of time, I can schematically map every word to makes up the definition of love, and how they interact. Then I can associate 3D, real world graphical observations to each of those words and then love as a concept holistically (as humans do, we're not just confined to text data, we observe extremely rich 3D visual data, and audio data, and touch data, etc...). There's no reason to believe a massive "correlation machine" can't do the same thing with the right algorithms, enough compute power, and multimodal inputs. Furthermore, we can make the correlation machine even better by specializing parts of the hardware for certain tasks, just like the brain.
nl 6y ago
I'm becoming increasingly convinced that Artificial Intelligence will fail not because we can't do the "artificial" part but because there is so little we do that is "intelligent". > the fact that we're just doing predictions, with no deep meaning or comprehension of features What - specifically - do you mean by this?

LordDragonfang 6y ago

>we're just doing statistics with big data and added sprinkles. I mean, you could pretty much say that's how the human brain works, couldn't you?

personjerry 6y ago
That comparison is like comparing gardening my backyard to agricultural farming. Technically the core concept is similar, but I think we both know that the scale and depth makes them not at all the same.
- H8crilA 6y ago
  There's a continuity of small changes between your backyard and industrial farming. Just like there's a continuity of small changes between you and your cat.
  - p1esk 6y ago
    There’s a continuity of small changes between any two things.
    - FeepingCreature 6y ago
      No. edit: For instance, there is no continuity of small changes in height that will bring you from climbing up incrementally larger trees to climbing to the moon. The question is whether gpt-3 vs human intelligence is more like climbing a tree vs climbing a mountain or more like climbing a tree vs building a rocket.
qeternity 6y ago
Well, we don’t really know. I think that’s one element missing from most DL analysis: our understanding of the brain is incredibly limited. And ANNs are incredibly crude imitations of actual NNs. So we’ve built massive, extremely crude approximations of a system we don’t really understand. As is often the case, the truth is somewhere in the middle. It’s almost certain that we won’t reach AGI without a fundamental breakthrough on soft/wet-ware but it’s also nearly certain that even with the best algorithms we will need to efficiently harness and coordinate massive compute power, as we’re learning to do now.
tsimionescu 6y ago
That's possibly true at the 'implementation' layer, but human learning is quite different, or at elast has some aspects that are vastly different. Human manuals don't contain millions of tagged examples, they contain explanations and a few dozen examples. So sure, when we learn how to walk or see or read, we may be in this mode of simply discerning patterns in large amounts of data. But when we learn maths or programming, we are using a completely different kind of learning.
saberience 6y ago
But thats the point, our brain is clearly a lot more than just a big table of probabilities. You only need to look at the absolutely insane volume of data and training time that these models need. How much time does it take a for human to understand a concept like "love" and what volume of training data is required? Computers would just regurgitate quotes from poetry or novels about love without any real understanding after billions of hours of training time and ingesting every document on the internet. A human can understand love in a tiny fraction of the time and with a tiny fraction of the volume of information processed and they also understand it in a more fundamental way and can articulate that in a way these models cannot. You might argue back, well the human brain has pre-trained neural networks with billions of hours of training time. Well, that isn't really the case. We don't start off with some pre-existing memory of what "love" means, or what "Physics" is, or trillions of bytes of data. All we have is a capacity to learn which is highly efficient, a conscious mind which is aware of itself, and certain fundamental drives driven by our bodies and instincts. If you have a human child and give it zero input information it will never learn a language or be capable at all in any sense of the term. So we become incredibly capable based on a tiny fraction of input data fed into us after birth. The way the human brain and mind works is deeply tied in to the experience of having a body, knowing we are mortal, and having fundamental drives such as a drive to survive, eat, drink, keep ourselves safe, and also a drive to be social, find a mate, and procreate. I would argue that we will never be able to have a computer/algorithm that thinks like we do unless it also has drives like we do and a body like we do, since so much of our process of thinking is tied in to having a body, our awareness of mortality, and our basic human drives and experience.
- keenmaster 6y ago
  X = A+B A= C+D B = E+F Love = X = A+B = C+D+E+F Obviously the above is contrived and abstracted, but you get my point. If I took a little bit of time, I can schematically map every word to makes up the definition of love, and how they interact. Then I can associate 3D, real world graphical observations to each of those words and then love as a concept holistically (as humans do, we're not just confined to text data, we observe extremely rich 3D visual data, and audio data, and touch data, etc...). There's no reason to believe a massive "correlation machine" can't do the same thing with the right algorithms, enough compute power, and multimodal inputs. Furthermore, we can make the correlation machine even better by specializing parts of the hardware for certain tasks, just like the brain.

nl 6y ago

I'm becoming increasingly convinced that Artificial Intelligence will fail not because we can't do the "artificial" part but because there is so little we do that is "intelligent". > the fact that we're just doing predictions, with no deep meaning or comprehension of features What - specifically - do you mean by this?