4 ms·

A human trains on way less data then an AI. Chat GPT has processed over 500GB of text files from books, about 44 billion words. If you read a book a week you

by sjducb 4y ago

A human trains on way less data then an AI. Chat GPT has processed over 500GB of text files from books, about 44 billion words. If you read a book a week you might hit 70 million words by age 18

2OEH8eoCRo0 4y ago
I disagree. Starting from birth, humans train continuously on streamed audio, visual, and other data from 5 senses. An inconceivable amount.
- danenania 4y ago
  And prior to that was billions of years of training by evolution that got us to the point where we could 'fine tune' with our senses and brains. A little bit of data was involved in all that too.
  - wizofaus 4y ago
    I'd argue that is the fundamental difference though - brains that were able to make good guesses about what was going on in the environment with very limited information are the ones whose owners reproduced successfully etc. And it's not unreasonable to note that the information available to the brains of our forebears is therefore in a rather indirect but still significant way "encoded" into our brains (at birth). Do LLMs have an element of that at all in their programming? Do they need more, and if so, how could it be best created?
- Maursault 4y ago
  > and other data from 5 senses. It only makes your point stronger, but there are way more[1] than 5 human senses, not counting senses we don't have that, say, dolphins or other animals do. I can only name a few others, such as proprioception, direction, balance, and weight discrimination, but there are too many to keep track of them all. [1] https://www.nytimes.com/1964/03/15/archives/we-have-more-than-five-senses-most-people-take-the-faculties-of.html https://www.nytimes.com/1964/03/15/archives/we-have-more-tha...
- mcswell 4y ago
  You missed the point. ChatGPT trained on a gazillion words to "learn" a language. Children learn their language from a tiny fraction of that. Streamed visual, smell, touch etc. don't help learn the grammars of (spoken) languages.
  - sdiupIGPWEfh 4y ago
    That has me wondering now. It's absolutely true that children learn (and even generate) language grammar from a ridiculously small number of samples compared to LLMs. But could the availability of a world model, in the form of other sensory inputs, contribute to that capacity? Younger children who haven't fully mastered correct grammar are still able to communicate more sensibly than earlier LLMs, whereas the earlier LLMs tend toward more grammatically correct gibberish. What if the missing secret sauce to better LLM training is figuring out how to wire, say, image recognition into the training process?
    - sdiupIGPWEfh 4y ago
      It amuses me that this would be not unlike teaching an LLM with picture books.
  - hypertele-Xii 4y ago
    > visual, smell, touch etc. don't help learn the grammars of (spoken) languages. Of course they do! These are literally the things children learn to associate language with. "Ouch!" is what is said when you feel pain. An ML model can learn to associate the word "ouch" with the words "pain" and "feel", but it doesn't actually know what pain is, because it doesn't feel.
    - mcswell 4y ago
      No. First off, I said grammar, not word meaning. Second, saying "Ouch" is not even language. My cat says something when I step on her paw. That doesn't mean she understands language, nor that she speaks some language. Third, you're right about pain, but an ML model can associate the word "red" with the color, and "walk" with images of people walking, and "sailboat" with certain images or videos, and plenty of other concepts. If that was what learning a language was, then AIs would understand language in lots of areas, if not in the specific domain of pain. But they don't.
- VMG 4y ago
  You can run the numbers with HD video and audio with high words-per-minute and you'd probably still be orders of magnitude below the model sizes
- ASalazarMX 4y ago
  Last Christmas one of my nephews was gifted a noisy baby toy. I don't know what are his goals and constraints, but he's still training with it. Must have learned a lot by now.

by sjducb 4y ago

A human trains on way less data then an AI. Chat GPT has processed over 500GB of text files from books, about 44 billion words. If you read a book a week you might hit 70 million words by age 18

2OEH8eoCRo0 4y ago

I disagree. Starting from birth, humans train continuously on streamed audio, visual, and other data from 5 senses. An inconceivable amount.

danenania 4y ago
And prior to that was billions of years of training by evolution that got us to the point where we could 'fine tune' with our senses and brains. A little bit of data was involved in all that too.
- wizofaus 4y ago
  I'd argue that is the fundamental difference though - brains that were able to make good guesses about what was going on in the environment with very limited information are the ones whose owners reproduced successfully etc. And it's not unreasonable to note that the information available to the brains of our forebears is therefore in a rather indirect but still significant way "encoded" into our brains (at birth). Do LLMs have an element of that at all in their programming? Do they need more, and if so, how could it be best created?
Maursault 4y ago
> and other data from 5 senses. It only makes your point stronger, but there are way more[1] than 5 human senses, not counting senses we don't have that, say, dolphins or other animals do. I can only name a few others, such as proprioception, direction, balance, and weight discrimination, but there are too many to keep track of them all. [1] https://www.nytimes.com/1964/03/15/archives/we-have-more-than-five-senses-most-people-take-the-faculties-of.html https://www.nytimes.com/1964/03/15/archives/we-have-more-tha...
mcswell 4y ago
You missed the point. ChatGPT trained on a gazillion words to "learn" a language. Children learn their language from a tiny fraction of that. Streamed visual, smell, touch etc. don't help learn the grammars of (spoken) languages.
- sdiupIGPWEfh 4y ago
  That has me wondering now. It's absolutely true that children learn (and even generate) language grammar from a ridiculously small number of samples compared to LLMs. But could the availability of a world model, in the form of other sensory inputs, contribute to that capacity? Younger children who haven't fully mastered correct grammar are still able to communicate more sensibly than earlier LLMs, whereas the earlier LLMs tend toward more grammatically correct gibberish. What if the missing secret sauce to better LLM training is figuring out how to wire, say, image recognition into the training process?
  - sdiupIGPWEfh 4y ago
    It amuses me that this would be not unlike teaching an LLM with picture books.
- hypertele-Xii 4y ago
  > visual, smell, touch etc. don't help learn the grammars of (spoken) languages. Of course they do! These are literally the things children learn to associate language with. "Ouch!" is what is said when you feel pain. An ML model can learn to associate the word "ouch" with the words "pain" and "feel", but it doesn't actually know what pain is, because it doesn't feel.
  - mcswell 4y ago
    No. First off, I said grammar, not word meaning. Second, saying "Ouch" is not even language. My cat says something when I step on her paw. That doesn't mean she understands language, nor that she speaks some language. Third, you're right about pain, but an ML model can associate the word "red" with the color, and "walk" with images of people walking, and "sailboat" with certain images or videos, and plenty of other concepts. If that was what learning a language was, then AIs would understand language in lots of areas, if not in the specific domain of pain. But they don't.
VMG 4y ago
You can run the numbers with HD video and audio with high words-per-minute and you'd probably still be orders of magnitude below the model sizes
ASalazarMX 4y ago
Last Christmas one of my nephews was gifted a noisy baby toy. I don't know what are his goals and constraints, but he's still training with it. Must have learned a lot by now.