4 ms·

"People don't need the gigantic amount of input data that ChatGPT needs to learn" Sure they do. Humans rely on tons of audio and video before they can even rea

"People don't need the gigantic amount of input data that ChatGPT needs to learn" Sure they do. Humans rely on tons of audio and video before they can even read (or walk).

saurik 3y ago
But I have learned to program and do complex math having read and analyzed ridiculously less source material on those subjects (and in fact in the case of programming having seen very little code as I mostly only had reference manuals at the time).
- vidarh 3y ago
  We are trained in a way that involves far more back and forth, giving corrections to specific failure modes in our thinking, though. It'll be fascinating to see to what extent interleaving "independent study" (training on large chunks of data) with interaction fed back in as training data could reduce the amount of training needed.
- xtreme 3y ago
  Before you learned how to code from a book, you had to learn how to read and write English. You also had to learn how to follow instructions, how to imbibe and compose information etc. How many books and hours of instruction did that take?
  - depr 3y ago
    Not as much as it took GPT to process all its input. >Let us consider the GPT-3 model with 𝑃 =175 billion parameters as an example. This model was trained on 𝑇 = 300 billion tokens. On 𝑛 = 1024 A100 GPUs using batch-size 1536, we achieve 𝑋 = 140 teraFLOP/s per GPU. As a result, the time required to train this model is 34 days. https://arxiv.org/pdf/2104.04473.pdf https://arxiv.org/pdf/2104.04473.pdf I'm not sure expressing brain capacity in FLOPs makes much sense, but I'm sure if it can be expressed in FLOPs, the amount of FLOPs going to learning for a normal human is less than that.
depr 3y ago
That is a highly questionable statement. For walking that is intuitively not true; most animals can walk almost immediately after birth. It would be very strange if human brains were so different that we had to learn walking from scratch. Indeed we do not; one of the most important reasons we can't walk from birth is because the brain has not grown enough yet. It grows by itself and then provides foundations needed for walking. Of course there is also a component to walking that is a learned skill that is refined. For reading, the same applies. Our brains are equipped with many of the foundational aspects required for reading, and we only _learn_ a part what is necessary for the skill of reading. Unlike computer models, brains are no tabula rasa. So we don't need the same input as computer models to learn.

"People don't need the gigantic amount of input data that ChatGPT needs to learn" Sure they do. Humans rely on tons of audio and video before they can even read (or walk).

saurik 3y ago

But I have learned to program and do complex math having read and analyzed ridiculously less source material on those subjects (and in fact in the case of programming having seen very little code as I mostly only had reference manuals at the time).

vidarh 3y ago
We are trained in a way that involves far more back and forth, giving corrections to specific failure modes in our thinking, though. It'll be fascinating to see to what extent interleaving "independent study" (training on large chunks of data) with interaction fed back in as training data could reduce the amount of training needed.
xtreme 3y ago
Before you learned how to code from a book, you had to learn how to read and write English. You also had to learn how to follow instructions, how to imbibe and compose information etc. How many books and hours of instruction did that take?
- depr 3y ago
  Not as much as it took GPT to process all its input. >Let us consider the GPT-3 model with 𝑃 =175 billion parameters as an example. This model was trained on 𝑇 = 300 billion tokens. On 𝑛 = 1024 A100 GPUs using batch-size 1536, we achieve 𝑋 = 140 teraFLOP/s per GPU. As a result, the time required to train this model is 34 days. https://arxiv.org/pdf/2104.04473.pdf https://arxiv.org/pdf/2104.04473.pdf I'm not sure expressing brain capacity in FLOPs makes much sense, but I'm sure if it can be expressed in FLOPs, the amount of FLOPs going to learning for a normal human is less than that.

depr 3y ago

That is a highly questionable statement. For walking that is intuitively not true; most animals can walk almost immediately after birth. It would be very strange if human brains were so different that we had to learn walking from scratch. Indeed we do not; one of the most important reasons we can't walk from birth is because the brain has not grown enough yet. It grows by itself and then provides foundations needed for walking. Of course there is also a component to walking that is a learned skill that is refined. For reading, the same applies. Our brains are equipped with many of the foundational aspects required for reading, and we only _learn_ a part what is necessary for the skill of reading. Unlike computer models, brains are no tabula rasa. So we don't need the same input as computer models to learn.