4 ms·

> We have no evidence that human cognition and GPT-3 cognition are distinct in some fundamental way. We do know that the human brain can generalize from much l

by 0-_-0 5y ago

> We have no evidence that human cognition and GPT-3 cognition are distinct in some fundamental way. We do know that the human brain can generalize from much less data. > We do not know if the discrepancy is a matter of degree, or a matter of category. That's not even a useful distinction. E.g. you could argue the difference between human intelligence and a random classifier is also a matter of degree. I would say the difference is a matter of degree, but the difference right now is so enormous that it seems like a different category.

lostmsu 5y ago
> We do know that the human brain can generalize from much less data. Adult human brains receive magnitudes more data than GPT-3 by age 18. Probably even by age one. Take vision, for example, which is believed to be approx. 8000x8000 24Hz stream with HDR (so more than 3 bytes per pixel). This alone generates (uncompressed) 252TB per year. Slightly over half of GPT-3 training data is Common Crawl, which only recently grew to 320TB. Where have you seen a 2yo baby, who can solve tasks GPT-3 can?
- 0-_-0 5y ago
  Where have you seen a 2yo baby, who read 320TB of text? If we fed a few years worth of video an audio to a neural network, could it write War and Peace? Could it play chess based on a text description of the game?
  - lostmsu 5y ago
    > who read 320TB of text When you said "less data", you did not specify the type of data. > Could it play chess based on a text description of the game? We don't know - nobody tried it yet. We tried with 320TB text only. > If we fed a few years worth of video an audio to a neural network, could it write War and Peace? As Sonny from "I, robot" said, "can you?"
    - 0-_-0 5y ago
      >you did not specify the type of data. Obviously, it has to be the same data to make a comparison. > We don't know - nobody tried it yet. How would you even train a network unsupervised so that it can play chess from a text description? Nobody knows. That's the point. >"can you?" That doesn't matter, Tolstoy could, therefore it's within human capability.
      - lostmsu 5y ago
        > Obviously, it has to be the same data to make a comparison. No, it is very much not obvious. There's quite a bit of research showing models, that receive X samples of type A and Y samples on type B might be better in tasks on A, than models that are trained just on X.

by 0-_-0 5y ago

lostmsu 5y ago

> We do know that the human brain can generalize from much less data. Adult human brains receive magnitudes more data than GPT-3 by age 18. Probably even by age one. Take vision, for example, which is believed to be approx. 8000x8000 24Hz stream with HDR (so more than 3 bytes per pixel). This alone generates (uncompressed) 252TB per year. Slightly over half of GPT-3 training data is Common Crawl, which only recently grew to 320TB. Where have you seen a 2yo baby, who can solve tasks GPT-3 can?

0-_-0 5y ago
Where have you seen a 2yo baby, who read 320TB of text? If we fed a few years worth of video an audio to a neural network, could it write War and Peace? Could it play chess based on a text description of the game?
- lostmsu 5y ago
  > who read 320TB of text When you said "less data", you did not specify the type of data. > Could it play chess based on a text description of the game? We don't know - nobody tried it yet. We tried with 320TB text only. > If we fed a few years worth of video an audio to a neural network, could it write War and Peace? As Sonny from "I, robot" said, "can you?"
  - 0-_-0 5y ago
    >you did not specify the type of data. Obviously, it has to be the same data to make a comparison. > We don't know - nobody tried it yet. How would you even train a network unsupervised so that it can play chess from a text description? Nobody knows. That's the point. >"can you?" That doesn't matter, Tolstoy could, therefore it's within human capability.
    - lostmsu 5y ago
      > Obviously, it has to be the same data to make a comparison. No, it is very much not obvious. There's quite a bit of research showing models, that receive X samples of type A and Y samples on type B might be better in tasks on A, than models that are trained just on X.