4 ms·

>I think ChatGPT is good evidence of this. What evidence do we have that this isn't how intelligence works? People don't need the gigantic amount of input data

by depr 3y ago

>I think ChatGPT is good evidence of this. What evidence do we have that this isn't how intelligence works? People don't need the gigantic amount of input data that ChatGPT needs to learn. However I'm not sure what exactly "this" is you and GP are referring to, and it may be possible to improve existing ideas so that it works with less input data.

streakfix 3y ago
``` USER: If a = 33223 and b = 22335, what is a + b? ASSISTANT: a + b = 55558. ``` The training data isn't the input. It's a part of the algorithm.
amitport 3y ago
"People don't need the gigantic amount of input data that ChatGPT needs to learn" Sure they do. Humans rely on tons of audio and video before they can even read (or walk).
- saurik 3y ago
  But I have learned to program and do complex math having read and analyzed ridiculously less source material on those subjects (and in fact in the case of programming having seen very little code as I mostly only had reference manuals at the time).
  - vidarh 3y ago
    We are trained in a way that involves far more back and forth, giving corrections to specific failure modes in our thinking, though. It'll be fascinating to see to what extent interleaving "independent study" (training on large chunks of data) with interaction fed back in as training data could reduce the amount of training needed.
  - xtreme 3y ago
    Before you learned how to code from a book, you had to learn how to read and write English. You also had to learn how to follow instructions, how to imbibe and compose information etc. How many books and hours of instruction did that take?
    - depr 3y ago
      Not as much as it took GPT to process all its input. >Let us consider the GPT-3 model with 𝑃 =175 billion parameters as an example. This model was trained on 𝑇 = 300 billion tokens. On 𝑛 = 1024 A100 GPUs using batch-size 1536, we achieve 𝑋 = 140 teraFLOP/s per GPU. As a result, the time required to train this model is 34 days. https://arxiv.org/pdf/2104.04473.pdf https://arxiv.org/pdf/2104.04473.pdf I'm not sure expressing brain capacity in FLOPs makes much sense, but I'm sure if it can be expressed in FLOPs, the amount of FLOPs going to learning for a normal human is less than that.
- depr 3y ago
  That is a highly questionable statement. For walking that is intuitively not true; most animals can walk almost immediately after birth. It would be very strange if human brains were so different that we had to learn walking from scratch. Indeed we do not; one of the most important reasons we can't walk from birth is because the brain has not grown enough yet. It grows by itself and then provides foundations needed for walking. Of course there is also a component to walking that is a learned skill that is refined. For reading, the same applies. Our brains are equipped with many of the foundational aspects required for reading, and we only _learn_ a part what is necessary for the skill of reading. Unlike computer models, brains are no tabula rasa. So we don't need the same input as computer models to learn.
SanderNL 3y ago
You have ingested truly astronomical amounts of data. Your body is covered with sensors of various kinds. Millions of datapoints streaming in day and night. You are not storing it all, but you are finding patterns and correlations in it from the day you were born. This all forms a base where after a decade+ you can learn things fast from books, but that’s comparable to an LLM’s in context learning. It’s fast, but it depends on a deep base of prior knowledge.
- depr 3y ago
  My body may have "ingested" it in some way but filtering starts even at the sense ("sensor") level. So there is no way I have learned from it. I didn't even have to learn how to filter, it is part of my genetic makeup. And even if I did, most of it would not be of the same category as what LLMs are learning; information about how a fabric feels or the sound of an ambulance doesn't teach me anything about programming or other things GPT can do. So when comparing the inputs, it makes no sense to count all the information that our senses are getting, to all the input for an LLM. >This all forms a base where after a decade+ you can learn things fast from books, but that’s comparable to an LLM’s in context learning Not really comparable
  - SanderNL 3y ago
    I think I implied a strong connection between LLMs and how biological intelligence works, but I certainly didn’t intend to. I think you discount general knowledge acquired non-verbally too easily. The sound of an ambulance and how it moves through space and how it correlates with visual information represents an astronomical amount of information that can be generalized from handsomely. All these data streams have to be connected, they correlate and intertwine in fantastically complex ways. I think the sound of an ambulance does teach you things that eventually help you “program”. You have witnessed similar (non-)events thousands if not millions of times. Each time it was accompanied with shitloads of sensory data from all modalities, both external and internal. The base of general patterns you start out from once you are ready for language is staggering. Again not saying LLMs work like that, because they do not. All I mean to do is put their information requirements in perspective. We ingest a lot more than a bunch of books.
godelski 3y ago
You have millions of years worth of training data. The model is just pruned, tuned, and specialized. More importantly, you have a lot of control over the tuning process. But it is naive to think that your life is all the training data you have and I'm not sure why this discussion is almost non-existent in threads like these.
esrauch 3y ago
This is actually a Noam Chomsky linguistics question: how much of language is innate/genetic and how much is learned. The common perception has been that children aren't exposed to enough data to arrive at their grammatical language skills, implying there's some proto language built in. Comparative analysis of languages has looked for what aspects are truly universal but there's actually not a lot of concrete true universals to ascribe to our genetic innate language. But if it is genetic that doesn't really mean it's fundamentally different than ChatGPT, it just took a different and drastically longer training period and then transfer learning when children learn their mother tongue.
- depr 3y ago
  >But if it is genetic that doesn't really mean it's fundamentally different than ChatGPT, it just took a different and drastically longer training period and then transfer learning when children learn their mother tongue. It doesn't necessarily mean it's fundamentally different, but it doesn't mean it is comparable either. Geoff Hinton doesn't think the brain does backpropagation. Training a neural net uses backpropagation. So if Hinton is correct then saying "it just took a longer training period" while brains doesn't learn like our current neural nets is glossing over a lot of things.

by depr 3y ago

streakfix 3y ago

``` USER: If a = 33223 and b = 22335, what is a + b? ASSISTANT: a + b = 55558. ``` The training data isn't the input. It's a part of the algorithm.

amitport 3y ago

"People don't need the gigantic amount of input data that ChatGPT needs to learn" Sure they do. Humans rely on tons of audio and video before they can even read (or walk).

saurik 3y ago
But I have learned to program and do complex math having read and analyzed ridiculously less source material on those subjects (and in fact in the case of programming having seen very little code as I mostly only had reference manuals at the time).
- vidarh 3y ago
  We are trained in a way that involves far more back and forth, giving corrections to specific failure modes in our thinking, though. It'll be fascinating to see to what extent interleaving "independent study" (training on large chunks of data) with interaction fed back in as training data could reduce the amount of training needed.
- xtreme 3y ago
  Before you learned how to code from a book, you had to learn how to read and write English. You also had to learn how to follow instructions, how to imbibe and compose information etc. How many books and hours of instruction did that take?
  - depr 3y ago
    Not as much as it took GPT to process all its input. >Let us consider the GPT-3 model with 𝑃 =175 billion parameters as an example. This model was trained on 𝑇 = 300 billion tokens. On 𝑛 = 1024 A100 GPUs using batch-size 1536, we achieve 𝑋 = 140 teraFLOP/s per GPU. As a result, the time required to train this model is 34 days. https://arxiv.org/pdf/2104.04473.pdf https://arxiv.org/pdf/2104.04473.pdf I'm not sure expressing brain capacity in FLOPs makes much sense, but I'm sure if it can be expressed in FLOPs, the amount of FLOPs going to learning for a normal human is less than that.
depr 3y ago
That is a highly questionable statement. For walking that is intuitively not true; most animals can walk almost immediately after birth. It would be very strange if human brains were so different that we had to learn walking from scratch. Indeed we do not; one of the most important reasons we can't walk from birth is because the brain has not grown enough yet. It grows by itself and then provides foundations needed for walking. Of course there is also a component to walking that is a learned skill that is refined. For reading, the same applies. Our brains are equipped with many of the foundational aspects required for reading, and we only _learn_ a part what is necessary for the skill of reading. Unlike computer models, brains are no tabula rasa. So we don't need the same input as computer models to learn.

SanderNL 3y ago

You have ingested truly astronomical amounts of data. Your body is covered with sensors of various kinds. Millions of datapoints streaming in day and night. You are not storing it all, but you are finding patterns and correlations in it from the day you were born. This all forms a base where after a decade+ you can learn things fast from books, but that’s comparable to an LLM’s in context learning. It’s fast, but it depends on a deep base of prior knowledge.

depr 3y ago
My body may have "ingested" it in some way but filtering starts even at the sense ("sensor") level. So there is no way I have learned from it. I didn't even have to learn how to filter, it is part of my genetic makeup. And even if I did, most of it would not be of the same category as what LLMs are learning; information about how a fabric feels or the sound of an ambulance doesn't teach me anything about programming or other things GPT can do. So when comparing the inputs, it makes no sense to count all the information that our senses are getting, to all the input for an LLM. >This all forms a base where after a decade+ you can learn things fast from books, but that’s comparable to an LLM’s in context learning Not really comparable
- SanderNL 3y ago
  I think I implied a strong connection between LLMs and how biological intelligence works, but I certainly didn’t intend to. I think you discount general knowledge acquired non-verbally too easily. The sound of an ambulance and how it moves through space and how it correlates with visual information represents an astronomical amount of information that can be generalized from handsomely. All these data streams have to be connected, they correlate and intertwine in fantastically complex ways. I think the sound of an ambulance does teach you things that eventually help you “program”. You have witnessed similar (non-)events thousands if not millions of times. Each time it was accompanied with shitloads of sensory data from all modalities, both external and internal. The base of general patterns you start out from once you are ready for language is staggering. Again not saying LLMs work like that, because they do not. All I mean to do is put their information requirements in perspective. We ingest a lot more than a bunch of books.

godelski 3y ago

You have millions of years worth of training data. The model is just pruned, tuned, and specialized. More importantly, you have a lot of control over the tuning process. But it is naive to think that your life is all the training data you have and I'm not sure why this discussion is almost non-existent in threads like these.

esrauch 3y ago

This is actually a Noam Chomsky linguistics question: how much of language is innate/genetic and how much is learned. The common perception has been that children aren't exposed to enough data to arrive at their grammatical language skills, implying there's some proto language built in. Comparative analysis of languages has looked for what aspects are truly universal but there's actually not a lot of concrete true universals to ascribe to our genetic innate language. But if it is genetic that doesn't really mean it's fundamentally different than ChatGPT, it just took a different and drastically longer training period and then transfer learning when children learn their mother tongue.

depr 3y ago
>But if it is genetic that doesn't really mean it's fundamentally different than ChatGPT, it just took a different and drastically longer training period and then transfer learning when children learn their mother tongue. It doesn't necessarily mean it's fundamentally different, but it doesn't mean it is comparable either. Geoff Hinton doesn't think the brain does backpropagation. Training a neural net uses backpropagation. So if Hinton is correct then saying "it just took a longer training period" while brains doesn't learn like our current neural nets is glossing over a lot of things.