5 ms·

Funny that it does not need that much data to train your average 20th century human genius. I'd say that if we are dreaming of the future of ai, learning and re

Funny that it does not need that much data to train your average 20th century human genius. I'd say that if we are dreaming of the future of ai, learning and reasoning seems the greatest issue, not data. That said, the article title is about LLMs, so that's what will need changing I guess.

jstanley 2y ago
Humans aren't just text interfaces though. The majority of your input is not textual but is sights, sounds, feelings, etc., that LLMs don't (yet?) have access to. Humans receive an enormous amount of training data in forms not currently available to LLMs. If you locked baby Einstein in a room with the collected works of humanity and left him there for a lifetime, I doubt he'd have even learnt to read on his own.
- nope1000 2y ago
  We also inherit a lot of network topology already
- devoutsalsa 2y ago
  I remember listening to an AI researched in some interview over 20 years ago. He said that in his quest to create an AI, he realized at some point he could just have kids instead.
- mrweasel 2y ago
  We also have a funny way of applying solutions and lessons learn in one field to problems in completely unrelated areas. Given the statistical nature of LLMs I'm not convinced that they are able look across fields in the same way as a human brain, they lack creativity. The greatest advantage you can have in life is a creative mind and I don't believe that is something that can be taught. It can be stomped out of you as a child, but it's not learnable.
  - lassoiat 2y ago
    I have come to the point that it is not really fair to the LLM to statistically train it on human output and expect it to come up with something more than the average. There will be much value in automating the tedious and the routine. Of course, that doesn't make for a great science fiction story. We first have to placate all these science fiction fantasies and in the process we will automate the tedious and the routine as a side effect of trying to figure out how many AGI can dance on the head of a pin. Then human creativity will just be worth all the more.
- trott 2y ago
  The stream of data from vision does NOT explain why humans learn 1000x faster: Children who lost their sight early on, can grow up to be intelligent. They can learn English, for example. They don't need to hear 200B words, like GPT-3.
  - bhickey 2y ago
    The human brain isn't randomly initialized. It's undergone 500m years of pretraining.
    - LoganDark 2y ago
      This makes me wonder if human brains can be genetically predisposed to a particular dominant language. I'd imagine not since that isn't typically a factor in selection, but I still wonder.
      - bhickey 2y ago
        I doubt it. Language and human evolution operate on different time scales. We wouldn't be able to converse with someone from 13th century England. If anything I will expect selective pressure on languages—those that are easy to use are more likely to be adopted. Secondarily, I would expect this effect to be swamped by other factors (e.g. conquest).
      - Grimblewald 2y ago
        From what I have read and have come to understand, it is more that we are generally predisposed to human language in general, specific portions of our brain especially so.
        LoganDark 2y ago
        > it is more that we are generally predisposed to human language in general I understand, that's not what I was wondering.
    - trott 2y ago
- nkrisc 2y ago
  > I doubt he'd have even learnt to read on his own. Of course not, not any more than he’d learn to program a computer on his own. Reading and writing are not natural skills. They are a technology that was invented by humans and they must be taught in some capacity. As we learn to read and write, parts of our brain related to language and other skills (which we do possess naturally) are co-opted to enable reading and writing, an unnatural skill. Intelligence and language, however, are natural human abilities that we have evolved, likely over millions of years. Any parent will tell you this is obvious. It’s amazing how much children are able to infer and learn on their own.
- bluedevilzn 2y ago
  Helen Keller is the counter point to this. She’s obviously intelligent and had no sights or sounds as input but only texts.
verdverm 2y ago
This has become my thinking as well. There is plenty of data for humans, so this is really a hardware / algos problem, not a data one. Using lots of text and energy is just a step on the path. We are already moving on to text+image and not long until video and later robots (with sensors) too
bastien2 2y ago
That's because humans learn in stages of growing complexity and semantic depth, and LLMs don't. The chatbots do what infant humans do: mimic what it "sees" until it gets the pattern consistently matching what it saw without any capacity to understand what it is doing. Once humans have that part done, whole new layers of semantic learning kick in and create the critical analyses we perceive as "intelligence". LLMs, as a consequence of their design, lack those deeper layers. They are not artificially intelligent at all. Rather, they're the latest iteration of what centuries ago gave us steam-powered songbirds.

jstanley 2y ago

Humans aren't just text interfaces though. The majority of your input is not textual but is sights, sounds, feelings, etc., that LLMs don't (yet?) have access to. Humans receive an enormous amount of training data in forms not currently available to LLMs. If you locked baby Einstein in a room with the collected works of humanity and left him there for a lifetime, I doubt he'd have even learnt to read on his own.

nope1000 2y ago
We also inherit a lot of network topology already
devoutsalsa 2y ago
I remember listening to an AI researched in some interview over 20 years ago. He said that in his quest to create an AI, he realized at some point he could just have kids instead.
mrweasel 2y ago
We also have a funny way of applying solutions and lessons learn in one field to problems in completely unrelated areas. Given the statistical nature of LLMs I'm not convinced that they are able look across fields in the same way as a human brain, they lack creativity. The greatest advantage you can have in life is a creative mind and I don't believe that is something that can be taught. It can be stomped out of you as a child, but it's not learnable.
- lassoiat 2y ago
  I have come to the point that it is not really fair to the LLM to statistically train it on human output and expect it to come up with something more than the average. There will be much value in automating the tedious and the routine. Of course, that doesn't make for a great science fiction story. We first have to placate all these science fiction fantasies and in the process we will automate the tedious and the routine as a side effect of trying to figure out how many AGI can dance on the head of a pin. Then human creativity will just be worth all the more.
trott 2y ago
The stream of data from vision does NOT explain why humans learn 1000x faster: Children who lost their sight early on, can grow up to be intelligent. They can learn English, for example. They don't need to hear 200B words, like GPT-3.
- bhickey 2y ago
  The human brain isn't randomly initialized. It's undergone 500m years of pretraining.
  - LoganDark 2y ago
    This makes me wonder if human brains can be genetically predisposed to a particular dominant language. I'd imagine not since that isn't typically a factor in selection, but I still wonder.
    - bhickey 2y ago
      I doubt it. Language and human evolution operate on different time scales. We wouldn't be able to converse with someone from 13th century England. If anything I will expect selective pressure on languages—those that are easy to use are more likely to be adopted. Secondarily, I would expect this effect to be swamped by other factors (e.g. conquest).
    - Grimblewald 2y ago
      From what I have read and have come to understand, it is more that we are generally predisposed to human language in general, specific portions of our brain especially so.
      - LoganDark 2y ago
        > it is more that we are generally predisposed to human language in general I understand, that's not what I was wondering.
  - trott 2y ago
nkrisc 2y ago
> I doubt he'd have even learnt to read on his own. Of course not, not any more than he’d learn to program a computer on his own. Reading and writing are not natural skills. They are a technology that was invented by humans and they must be taught in some capacity. As we learn to read and write, parts of our brain related to language and other skills (which we do possess naturally) are co-opted to enable reading and writing, an unnatural skill. Intelligence and language, however, are natural human abilities that we have evolved, likely over millions of years. Any parent will tell you this is obvious. It’s amazing how much children are able to infer and learn on their own.
bluedevilzn 2y ago
Helen Keller is the counter point to this. She’s obviously intelligent and had no sights or sounds as input but only texts.

verdverm 2y ago

This has become my thinking as well. There is plenty of data for humans, so this is really a hardware / algos problem, not a data one. Using lots of text and energy is just a step on the path. We are already moving on to text+image and not long until video and later robots (with sensors) too

bastien2 2y ago

That's because humans learn in stages of growing complexity and semantic depth, and LLMs don't. The chatbots do what infant humans do: mimic what it "sees" until it gets the pattern consistently matching what it saw without any capacity to understand what it is doing. Once humans have that part done, whole new layers of semantic learning kick in and create the critical analyses we perceive as "intelligence". LLMs, as a consequence of their design, lack those deeper layers. They are not artificially intelligent at all. Rather, they're the latest iteration of what centuries ago gave us steam-powered songbirds.