4 ms·

> Hasn't the typical human taken in orders of magnitude more data than this example? Have you read GPT-3s 175 billion parameters (words, sentences, papers, I d

> Hasn't the typical human taken in orders of magnitude more data than this example? Have you read GPT-3s 175 billion parameters (words, sentences, papers, I don't care) of anything? Do you know all the words used in that corpus? Nobody has or does. A child of a small age can listen to a very small set of things and not just come up with words to communicate with mama and papa what they learned, but they can reuse it. And this I think is key, because the language part of that is at least partially secondarily. The little kid understands what they're talking about even if they have a hard time communicating it to an adult. The fact they take creative leaps to use their extremely limited vocabulary to communicate their knowledge is amazing.

ipnon 6y ago
I recall a researcher filming their child from the day they were born until they began to speak. They wanted to find how many times a child had to hear a word in order to be able to repeat it back to the parent. The result, I think, was that if the child heard the word 2,000 times at all, they would be able to repeat it. But, if they heard the word 600 times at the same place, for instance the end of the couch, that would be enough to repeat it. The human brain requires less training, but to some extent it is pretrained by our genetic code. The human brain will take on a predictable structure with any sort of training. This post was generated using GPT-3. [;)]
- sah2ed 6y ago
  > I recall a researcher filming their child from the day they were born until they began to speak. Can’t tell if you are kidding or not, but if you aren’t, mind sharing links about the researcher for the curious?
  - ipnon 6y ago
    Let me take a look, I was not kidding. I recall some mainstream media coverage in the last decade. edit: Can't seem to find it which is a shame. I think it may have been included in a TED talk.
abernard1 6y ago
> This post was generated using GPT-3. [;)] Your post was generated using GPT-3 and 175 billion parameters of pre-existing human writing, contextualized, distilled, and cross-referenced with terminology we've agreed on for centuries. It's a parrot, and I remain unimpressed. Take the learned knowledge of GPT-3 (because it must be so smart right?) and have it actually do something. Buy stocks, make chemical formulas, build planes. If you are not broke or dead by the end of that exercise, I'll be impressed and believe GPT-3 knows things.
- kevmo314 6y ago
  > That post was generated using GPT-3 and 175 billion parameters of pre-existing human writing, contextualized, distilled, and cross-referenced with terminology we've agreed on for centuries. So basically like DNA?
  - saberience 6y ago
    DNA is nothing like the training of GPT. DNA does not encord a massive amount of statistics of words and language and how concepts, words, etc, relate to one another. All DNA does it encode for how to grow, build, and mantain a human body. That human body has the potential to learn a language and communicate, but if you put a baby human inside an empty room and drop in food, it will never learn language and never communicate. DNA isn't magic and comparing "millions of years of evolution" of DNA is nothing like the Petabytes of data that GPT-3 needs to operate. Again DNA has no knowledge embedded in it, it has no words or data embedded. Data in the sense that we imagine Wikipedia stored in JSON files on a hard disk. DNA stores an algorithm for growth of a human, that's it. The GPT-3 model is probably > 700GB in size. That is, for GPT to be able to generate text it needs an absolutely massive "memory" of existing text which it can recite verbatim. In contrast, young human children can generate more novel insights with many orders of magnitude less data in "memory" and less training time.
  - varjag 6y ago
    Since literacy or human knowledge isn't encoded in DNA, it's nothing like it.
- 6y ago

ipnon 6y ago

I recall a researcher filming their child from the day they were born until they began to speak. They wanted to find how many times a child had to hear a word in order to be able to repeat it back to the parent. The result, I think, was that if the child heard the word 2,000 times at all, they would be able to repeat it. But, if they heard the word 600 times at the same place, for instance the end of the couch, that would be enough to repeat it. The human brain requires less training, but to some extent it is pretrained by our genetic code. The human brain will take on a predictable structure with any sort of training. This post was generated using GPT-3. [;)]

sah2ed 6y ago
> I recall a researcher filming their child from the day they were born until they began to speak. Can’t tell if you are kidding or not, but if you aren’t, mind sharing links about the researcher for the curious?
- ipnon 6y ago
  Let me take a look, I was not kidding. I recall some mainstream media coverage in the last decade. edit: Can't seem to find it which is a shame. I think it may have been included in a TED talk.

abernard1 6y ago

> This post was generated using GPT-3. [;)] Your post was generated using GPT-3 and 175 billion parameters of pre-existing human writing, contextualized, distilled, and cross-referenced with terminology we've agreed on for centuries. It's a parrot, and I remain unimpressed. Take the learned knowledge of GPT-3 (because it must be so smart right?) and have it actually do something. Buy stocks, make chemical formulas, build planes. If you are not broke or dead by the end of that exercise, I'll be impressed and believe GPT-3 knows things.

kevmo314 6y ago
> That post was generated using GPT-3 and 175 billion parameters of pre-existing human writing, contextualized, distilled, and cross-referenced with terminology we've agreed on for centuries. So basically like DNA?
- saberience 6y ago
  DNA is nothing like the training of GPT. DNA does not encord a massive amount of statistics of words and language and how concepts, words, etc, relate to one another. All DNA does it encode for how to grow, build, and mantain a human body. That human body has the potential to learn a language and communicate, but if you put a baby human inside an empty room and drop in food, it will never learn language and never communicate. DNA isn't magic and comparing "millions of years of evolution" of DNA is nothing like the Petabytes of data that GPT-3 needs to operate. Again DNA has no knowledge embedded in it, it has no words or data embedded. Data in the sense that we imagine Wikipedia stored in JSON files on a hard disk. DNA stores an algorithm for growth of a human, that's it. The GPT-3 model is probably > 700GB in size. That is, for GPT to be able to generate text it needs an absolutely massive "memory" of existing text which it can recite verbatim. In contrast, young human children can generate more novel insights with many orders of magnitude less data in "memory" and less training time.
- varjag 6y ago
  Since literacy or human knowledge isn't encoded in DNA, it's nothing like it.
6y ago