4 ms·

But I have learned to program and do complex math having read and analyzed ridiculously less source material on those subjects (and in fact in the case of progr

• vidarh 4y ago
  We are trained in a way that involves far more back and forth, giving corrections to specific failure modes in our thinking, though. It'll be fascinating to see to what extent interleaving "independent study" (training on large chunks of data) with interaction fed back in as training data could reduce the amount of training needed.
• xtreme 4y ago
  Before you learned how to code from a book, you had to learn how to read and write English. You also had to learn how to follow instructions, how to imbibe and compose information etc. How many books and hours of instruction did that take?
  • depr 4y ago
    Not as much as it took GPT to process all its input. >Let us consider the GPT-3 model with 𝑃 =175 billion parameters as an example. This model was trained on 𝑇 = 300 billion tokens. On 𝑛 = 1024 A100 GPUs using batch-size 1536, we achieve 𝑋 = 140 teraFLOP/s per GPU. As a result, the time required to train this model is 34 days. https://arxiv.org/pdf/2104.04473.pdf https://arxiv.org/pdf/2104.04473.pdf I'm not sure expressing brain capacity in FLOPs makes much sense, but I'm sure if it can be expressed in FLOPs, the amount of FLOPs going to learning for a normal human is less than that.