14 ms·

I used to think that to achieve human-level intelligence with AI, we need to have human-sized neural network. It was a conservative guess, but seeing where thin

I used to think that to achieve human-level intelligence with AI, we need to have human-sized neural network. It was a conservative guess, but seeing where things are going, I now strongly believe we need a lot LESS parameters with AI, because AI is trained from literally nothing (from noise) on our current culture, from the Internet. It has no baggage, it's built from the ground up to be a modern 21st century human. We're not, we train decades to become ones. And struggle with it. Our genes are mostly tuned for our environment from thousands of years ago. A human brain carries a lot more baggage with it. Long story short, our biological brain hardware is much superior to current AI hardware, but our biological brain software (where we learn the culture that surrounds us) is much inferior to current AI machine learning. GPT-3 trained 175B on basically the whole internet in 15 days. That's insane. And now there are rumors that ChatGPT is executed on an optimized subset of this 175B model, which may be a bit more than 1/10 that size. I think going forward we'll see 20-30B parameter models perform much better than an average human in typical tasks. Especially after we evolve the topology of an AI to not be a straight directed acyclic graph, as it is now, but allow structures to loop internally, to invoke other structures with input (as subroutines) and so on. And also there's new hybrid analog/digital hardware in the pipeline, which makes AI execution 100 times more energy efficient and compact. So above-human level AI on your phone is on the horizon, I'd say. We're only getting started.

nwienert 4y ago
This is nonsense. These nets regurgitate the most average possible output given inputs. By definition. You’re conflating and hyping and mixing up so many weird things at once. You literally couldn’t get GPT to come up with a single novelty if you tried. It’s all remixing existing content, and again, doing so in a way to fit the average of the dataset. When you realize this you realize it has no intelligence as we most typically define it (novel solutions to novel problems).. its not AI. Call it what it is: a beautifully advanced way to regurgitate the exact most popular (mundane) reply you’d expect given a huge dataset. It’s sort of good for studying what already exists. It won’t even really ever show you the edges though so it’s actually almost dangerously deceptive as evidenced by this absurd rounding up people are doing. If you want to learn the gist of anything, ask GPT. If you want to know anything in depth, GPT in fact will only mislead you towards genericity, platitudinous mediocrity.
- BulgarianIdiot 4y ago
  Has GPT ever told you "let me think" and then you notice it actually pause for a few seconds? No. Its output is fixed cost. GPT can't think in depth because it's not allowed to. This is something I already covered above by noting that current NN models are a DAG. We're already experimenting with variations that can iterate internally, and therefore think deeper, using still their existing knowledge. And the results change dramatically. You're also not right that GPT can't produce anything novel. Here's a tiny, modest example: https://sumplete.com/about/ https://sumplete.com/about/ I won't waste my time, as I see you didn't read what you responded to the first time. But you need to start paying attention. You denial & empty confidence in what's happening will only get you so far.
  - nwienert 4y ago
    Considering my former co founder works at OpenAI and we spent years working on and talking about related ML with and around some of the best people in the world, considering I’ve implemented language models multiple times myself, please try and be more specific on where I’m wrong. I know many people who work in ML share this concern - people thinking it’s smart, it’s more than it is. It’s very useful no doubt, but your comment is an example of extrapolation from maladjusted priors. Your example proves my point, a very uninteresting looking game. I don’t even doubt GPT will find all sorts of interesting stuff by giving averaged out things. There’s plenty of gaps in the averages humans have missed. But nothing new, and until you can show that let’s just call it what it is.
    - deleted 4y ago
      [deleted]
    - BulgarianIdiot 4y ago
      Your goalpost moved from "nothing novel" to "it's novel, but it's not interesting to me". I'll let not the years ahead, but literally the DAYS ahead prove you wrong. Because things are changing quite rapidly lately.
    - ad404b8a372f2b9 4y ago
jerojero 4y ago
Here's a thought. Recently the whole connectome of the fruit fly has been mapped. 3000 neurons and 548k connections. The fruit flys brain is capable of piloting the body of the fruit fly on a precision and energy efficiency that's simply unmatched in any kind of aerial machine we have developed. And only with those many connections, half a million? Neurons are very complex computation machines that can module their response very carefully. The human brain does so much more than simply being able to process language. It's capable of one shot learning and to course correct incredibly easy. These features are very important for survivability and I would say are the "magic sauce" that we just haven't been able to replicate. So I'm actually not so sure this is simply a parameters game. There could be important structures and specialisation that we are missing. In the fruit fly there are 93 different types of neurons, in the human brain iirc it might be hundreds.
- BulgarianIdiot 4y ago
  A language model is also capable of one shot learning. Try it in the chat, you tell it something, once, and it incorporates it for the rest of the chat. :-) You may think I don't know what "one shot learning" means, but I do know, I'm just saying it's a problem of our perspective, not of the model's capability. Aside from it having no long term memory (by design) from the chat. Yet. Our high-level learning is absolutely nothing like how we learned as toddlers. It took us DECADES to understand complex concepts from "few shots". Only pretrained models can do that, and you're also one, in your adulthood. What you say about extra complexity from neuron types and neurotransmitters combining with one another to modulate even simple networks is true of course. But it's also true you don't need to replicate all those organic details in a NN, because they're equivalent to just a few more "regular" parameters. Think of an artificial neuron in a network as a Turing machine. A Turing machine can do everything with just some more tape, and a neural network can do everything with just some more parameters. And yet, our models show that we don't need AS many parameters as we thought. And that's exciting.
- kragen 4y ago
  half a million connections is about 500 times more connections than there are in a 6502, so structure clearly matters a lot
lelanthran 4y ago
> It has no baggage, it's built from the ground up to be a modern 21st century human. We're not, we train decades to become ones. The whole point of machine learning is to do all that "decades" of training in a few hours. In terms of learning material, it takes LLM a few orders of magnitude more material to learn than it does a human. After all, a human child trained with very few books (by age 7 a child would have seen only a few books) still outperforms CHatGPT trained with the same material, because ChatGPT trained on 4 books is basically useless.
- SanderNL 4y ago
  They train on years of real-time streaming video. Feedback loops all over the place. Higher order structures and rules all get learned way, way before they even are able to touch a book. I cannot phantom the amount of data and analyses that passes a typical human child in a day. So, sure, they can learn fast after all that. But so can GPT, which also one-shots a lot.
  - lelanthran 4y ago
    > They train on years of real-time streaming video. ... > I cannot phantom the amount of data and analyses that passes a typical human child in a day. You're counting every second of every image that a child sees as a separate "image" that is learned. I'm counting the number of objects that a child sees, because people aren't doing frame-capture and analysis on every image, they're doing lazy interpolation (which is why all your peripheral vision in in monochrome but you perceive it as full color). A 6 month old baby who has only ever seen close family (parents, siblings, pets) is quite able to discern the same features on strangers. Nothing we have in AI right now can take 3 example models of humans, and detect hair, nose and mouth in a brand new never before seen model of a human. Not really the same thing as training on millions of different images and videos. The number of distinct and different objects in a 7 yo child's training set is rarely the same as "every single youtube video ever uploaded". Going back to the original point: ChatGPT needs millions of books to learn to regurgitate information as well as a semi-intelligent young adult, while typical toddlers are quite able to regurgitate what they've just seen. Humans and animals get by with millions of times less training data than AI models need.

nwienert 4y ago

This is nonsense. These nets regurgitate the most average possible output given inputs. By definition. You’re conflating and hyping and mixing up so many weird things at once. You literally couldn’t get GPT to come up with a single novelty if you tried. It’s all remixing existing content, and again, doing so in a way to fit the average of the dataset. When you realize this you realize it has no intelligence as we most typically define it (novel solutions to novel problems).. its not AI. Call it what it is: a beautifully advanced way to regurgitate the exact most popular (mundane) reply you’d expect given a huge dataset. It’s sort of good for studying what already exists. It won’t even really ever show you the edges though so it’s actually almost dangerously deceptive as evidenced by this absurd rounding up people are doing. If you want to learn the gist of anything, ask GPT. If you want to know anything in depth, GPT in fact will only mislead you towards genericity, platitudinous mediocrity.

BulgarianIdiot 4y ago
Has GPT ever told you "let me think" and then you notice it actually pause for a few seconds? No. Its output is fixed cost. GPT can't think in depth because it's not allowed to. This is something I already covered above by noting that current NN models are a DAG. We're already experimenting with variations that can iterate internally, and therefore think deeper, using still their existing knowledge. And the results change dramatically. You're also not right that GPT can't produce anything novel. Here's a tiny, modest example: https://sumplete.com/about/ https://sumplete.com/about/ I won't waste my time, as I see you didn't read what you responded to the first time. But you need to start paying attention. You denial & empty confidence in what's happening will only get you so far.
- nwienert 4y ago
  Considering my former co founder works at OpenAI and we spent years working on and talking about related ML with and around some of the best people in the world, considering I’ve implemented language models multiple times myself, please try and be more specific on where I’m wrong. I know many people who work in ML share this concern - people thinking it’s smart, it’s more than it is. It’s very useful no doubt, but your comment is an example of extrapolation from maladjusted priors. Your example proves my point, a very uninteresting looking game. I don’t even doubt GPT will find all sorts of interesting stuff by giving averaged out things. There’s plenty of gaps in the averages humans have missed. But nothing new, and until you can show that let’s just call it what it is.
  - deleted 4y ago
    [deleted]
  - BulgarianIdiot 4y ago
    Your goalpost moved from "nothing novel" to "it's novel, but it's not interesting to me". I'll let not the years ahead, but literally the DAYS ahead prove you wrong. Because things are changing quite rapidly lately.
  - ad404b8a372f2b9 4y ago

jerojero 4y ago

Here's a thought. Recently the whole connectome of the fruit fly has been mapped. 3000 neurons and 548k connections. The fruit flys brain is capable of piloting the body of the fruit fly on a precision and energy efficiency that's simply unmatched in any kind of aerial machine we have developed. And only with those many connections, half a million? Neurons are very complex computation machines that can module their response very carefully. The human brain does so much more than simply being able to process language. It's capable of one shot learning and to course correct incredibly easy. These features are very important for survivability and I would say are the "magic sauce" that we just haven't been able to replicate. So I'm actually not so sure this is simply a parameters game. There could be important structures and specialisation that we are missing. In the fruit fly there are 93 different types of neurons, in the human brain iirc it might be hundreds.

BulgarianIdiot 4y ago
A language model is also capable of one shot learning. Try it in the chat, you tell it something, once, and it incorporates it for the rest of the chat. :-) You may think I don't know what "one shot learning" means, but I do know, I'm just saying it's a problem of our perspective, not of the model's capability. Aside from it having no long term memory (by design) from the chat. Yet. Our high-level learning is absolutely nothing like how we learned as toddlers. It took us DECADES to understand complex concepts from "few shots". Only pretrained models can do that, and you're also one, in your adulthood. What you say about extra complexity from neuron types and neurotransmitters combining with one another to modulate even simple networks is true of course. But it's also true you don't need to replicate all those organic details in a NN, because they're equivalent to just a few more "regular" parameters. Think of an artificial neuron in a network as a Turing machine. A Turing machine can do everything with just some more tape, and a neural network can do everything with just some more parameters. And yet, our models show that we don't need AS many parameters as we thought. And that's exciting.
kragen 4y ago
half a million connections is about 500 times more connections than there are in a 6502, so structure clearly matters a lot

lelanthran 4y ago

> It has no baggage, it's built from the ground up to be a modern 21st century human. We're not, we train decades to become ones. The whole point of machine learning is to do all that "decades" of training in a few hours. In terms of learning material, it takes LLM a few orders of magnitude more material to learn than it does a human. After all, a human child trained with very few books (by age 7 a child would have seen only a few books) still outperforms CHatGPT trained with the same material, because ChatGPT trained on 4 books is basically useless.

SanderNL 4y ago
They train on years of real-time streaming video. Feedback loops all over the place. Higher order structures and rules all get learned way, way before they even are able to touch a book. I cannot phantom the amount of data and analyses that passes a typical human child in a day. So, sure, they can learn fast after all that. But so can GPT, which also one-shots a lot.
- lelanthran 4y ago
  > They train on years of real-time streaming video. ... > I cannot phantom the amount of data and analyses that passes a typical human child in a day. You're counting every second of every image that a child sees as a separate "image" that is learned. I'm counting the number of objects that a child sees, because people aren't doing frame-capture and analysis on every image, they're doing lazy interpolation (which is why all your peripheral vision in in monochrome but you perceive it as full color). A 6 month old baby who has only ever seen close family (parents, siblings, pets) is quite able to discern the same features on strangers. Nothing we have in AI right now can take 3 example models of humans, and detect hair, nose and mouth in a brand new never before seen model of a human. Not really the same thing as training on millions of different images and videos. The number of distinct and different objects in a 7 yo child's training set is rarely the same as "every single youtube video ever uploaded". Going back to the original point: ChatGPT needs millions of books to learn to regurgitate information as well as a semi-intelligent young adult, while typical toddlers are quite able to regurgitate what they've just seen. Humans and animals get by with millions of times less training data than AI models need.