5 ms·

This is very cool and useful, thanks. However, saying: "Suppose you wanted to train a world-class GPT model, just like OpenAI. How? You have no data. Now you d

This is very cool and useful, thanks. However, saying: "Suppose you wanted to train a world-class GPT model, just like OpenAI. How? You have no data. Now you do. Now everyone does" Is quite a stretch. You need much, much more that 36GB of quality data to train a world-class GPT model.

zurfer 6y ago
No, You don't need more. This is compressed text. All of English Wikipedia is roughly 20 GB. You can train amazing and state of the art text models with this data in addition. Openai states themselves that they trained on "40GB of internet text" https://openai.com/blog/better-language-models/ https://openai.com/blog/better-language-models/
- Voloskaya 6y ago
  You can indeed train state of the art text models with this data provided you come up with some better architecture that what we currently have. What I am saying is that you cannot train state of the ART "GPT" models like the link is saying. Your link is for GPT-2, GPT-3 used much, much more data. GPT-3 was trained in part on "books2" corpus which is not public but seems to basically be the same thing as this: 200k books * 100k words per book on average * ~3 token per words = 60B tokens, books2 is 55B tokens so it checks out. The total amount of tokens that GPT-3 was trained on from all sources is a combined 500B tokens, this is merely 10% of what they have. https://arxiv.org/pdf/2005.14165.pdf https://arxiv.org/pdf/2005.14165.pdf
bigbubba 6y ago
The GPT approach must be very poor; with only a fraction of that information you can train a human to perform better than any GPT model shown thus far. When you look at power burned doing it, the gap is even wider. Obviously our wetware is optimized for it and we aren't true blank slates, but the enormous magnitude of the discrepancy makes me think we're on the wrong path.
- monsieurbanana 6y ago
  I'm sure it's much more probable than we're in the wrong path, rather than the good one, but I don't think the amount of information needed by a GPT model can give insights to that. Instead, as you pointed out, it's just proof of how efficient our wetware is.
- gamblor956 6y ago
  GPT and most modern "AI" approaches suffer from the same problem: they're just brute force algorithmic approaches that don't mimic "intelligence" in any way. Since there is no understanding, they can't learn in the same way humans or other living creatures do (in general terms, by extrapolation). The ironic thing is that ML became popular because we didn't have the technology back then to properly model actual neuronal activity. Now we do, but ML is so established that modeling "true" intelligence is no longer de jure.
  - nialv7 6y ago
    That's a bold claim to make, given we ourselves don't even understand what understanding, or intelligence is. So how can you support your claim that what GPT-3 does is not intelligence.
    - gamblor956 6y ago
      I'm not the only one. It's industry consensus. https://www.technologyreview.com/2020/08/22/1007539/gpt3-openai-language-generator-artificial-intelligence-ai-opinion/ https://www.technologyreview.com/2020/08/22/1007539/gpt3-ope...
      - nialv7 6y ago
        That doesn't answer the question.
    - bigbubba 6y ago
      Personally I think our common definitions of intelligence are too narrowly defined, we have a bias towards recognizing intelligence similar to our own and a blindspot for intelligent systems unlike our own. I agree with Paul Stamets that mycelia networks may have a form of intelligence quite unlike our own. Large corporations likely have intelligence of their own ('slow AIs') and may in aggregate pursue goals that no individual participant in the organization agrees with (preference falsification demonstrates how such a thing might occur.) Closer to the traditional, I think reptiles may be far more intelligent that we usually give them credit for since we have a mammalian bias against their kind. Mammalians seem to have empathy circuits that fire more often for other mammals, and this inclines us to not consider the possible intelligence of non-mammals. So, GPT? I think that's probably a form of intelligence too, but not one that's particularly similar to our own. I also doubt that we'll have much luck getting it to ever perform at near-human levels, when you consider the data and power requirements. I think it's not simply a matter of silicon vs wetware; I think the GPT approach is substantially more different from our own than the hardware it runs on. It's a form of intelligence but that doesn't mean it is like our own.
minimaxir 6y ago
AI models suffer massive diminishing returns on both model size and amount of input text, which fortunately gives fresh models a chance to compete (the benefits of GPT-2/Transformer architectures is that they are capable of scaling indefinitely, but they still have diminishing returns)
- Voloskaya 6y ago
  I completly agree, it's a diminishing return but a return nonetheles, so to have a "world class GPT" model with 10% of the data, and 10% (optimistically) of the compute that OpenAI used, you will need some very very impressive innovation elsewhere.
asutekku 6y ago
36gb of compressed text is a lot though.
- Voloskaya 6y ago
  That's about 50B tokens. It can be a lot depending on what you want to do, but It's 10% of what GPT-3 (the definition of world class GPT model) used.
  - sillysaurusx 6y ago
    For what it's worth, we're serious about replicating GPT-3. books3 is just one piece. You will notice I never claimed equivalency to GPT-3's training data. books3 may be 10%, but The Pile is building the rest: https://twitter.com/arankomatsuzaki/status/1320414141895487490 https://twitter.com/arankomatsuzaki/status/13204141418954874... https://github.com/EleutherAI/The-Pile https://github.com/EleutherAI/The-Pile https://www.eleuther.ai/get-involved https://www.eleuther.ai/get-involved https://media.discordapp.net/attachments/735217892517216366/769820126466932756/unknown.png https://media.discordapp.net/attachments/735217892517216366/... https://media.discordapp.net/attachments/735217892517216366/769820362267295744/unknown.png https://media.discordapp.net/attachments/735217892517216366/...
    - Voloskaya 6y ago
      And again, I find books3 extremely cool and important work on your part, looking forward to the rest. I just have a minor gripe with saying that "now we can train world class GPT model" thanks to that, as you said it's just one piece, and as a typicial HNist I had to point it out :).
      - sillysaurusx 6y ago
        Believe it or not, I appreciate and relate to that sentiment. But after spending roughly one year acquiring knowledge related to this work, I feel I can say with a fairly high degree of certainty that this dataset alone is enough to train a model that will achieve "world class" status in some area. Writing books, perhaps. Which part of my logic do you feel is mistaken, and why? I am actually quite interested to hear thoughts from someone who is very pedantic about such things.

zurfer 6y ago

No, You don't need more. This is compressed text. All of English Wikipedia is roughly 20 GB. You can train amazing and state of the art text models with this data in addition. Openai states themselves that they trained on "40GB of internet text" https://openai.com/blog/better-language-models/ https://openai.com/blog/better-language-models/

Voloskaya 6y ago
You can indeed train state of the art text models with this data provided you come up with some better architecture that what we currently have. What I am saying is that you cannot train state of the ART "GPT" models like the link is saying. Your link is for GPT-2, GPT-3 used much, much more data. GPT-3 was trained in part on "books2" corpus which is not public but seems to basically be the same thing as this: 200k books * 100k words per book on average * ~3 token per words = 60B tokens, books2 is 55B tokens so it checks out. The total amount of tokens that GPT-3 was trained on from all sources is a combined 500B tokens, this is merely 10% of what they have. https://arxiv.org/pdf/2005.14165.pdf https://arxiv.org/pdf/2005.14165.pdf

bigbubba 6y ago

The GPT approach must be very poor; with only a fraction of that information you can train a human to perform better than any GPT model shown thus far. When you look at power burned doing it, the gap is even wider. Obviously our wetware is optimized for it and we aren't true blank slates, but the enormous magnitude of the discrepancy makes me think we're on the wrong path.

monsieurbanana 6y ago
I'm sure it's much more probable than we're in the wrong path, rather than the good one, but I don't think the amount of information needed by a GPT model can give insights to that. Instead, as you pointed out, it's just proof of how efficient our wetware is.
gamblor956 6y ago
GPT and most modern "AI" approaches suffer from the same problem: they're just brute force algorithmic approaches that don't mimic "intelligence" in any way. Since there is no understanding, they can't learn in the same way humans or other living creatures do (in general terms, by extrapolation). The ironic thing is that ML became popular because we didn't have the technology back then to properly model actual neuronal activity. Now we do, but ML is so established that modeling "true" intelligence is no longer de jure.
- nialv7 6y ago
  That's a bold claim to make, given we ourselves don't even understand what understanding, or intelligence is. So how can you support your claim that what GPT-3 does is not intelligence.
  - gamblor956 6y ago
    I'm not the only one. It's industry consensus. https://www.technologyreview.com/2020/08/22/1007539/gpt3-openai-language-generator-artificial-intelligence-ai-opinion/ https://www.technologyreview.com/2020/08/22/1007539/gpt3-ope...
    - nialv7 6y ago
      That doesn't answer the question.
  - bigbubba 6y ago
    Personally I think our common definitions of intelligence are too narrowly defined, we have a bias towards recognizing intelligence similar to our own and a blindspot for intelligent systems unlike our own. I agree with Paul Stamets that mycelia networks may have a form of intelligence quite unlike our own. Large corporations likely have intelligence of their own ('slow AIs') and may in aggregate pursue goals that no individual participant in the organization agrees with (preference falsification demonstrates how such a thing might occur.) Closer to the traditional, I think reptiles may be far more intelligent that we usually give them credit for since we have a mammalian bias against their kind. Mammalians seem to have empathy circuits that fire more often for other mammals, and this inclines us to not consider the possible intelligence of non-mammals. So, GPT? I think that's probably a form of intelligence too, but not one that's particularly similar to our own. I also doubt that we'll have much luck getting it to ever perform at near-human levels, when you consider the data and power requirements. I think it's not simply a matter of silicon vs wetware; I think the GPT approach is substantially more different from our own than the hardware it runs on. It's a form of intelligence but that doesn't mean it is like our own.

minimaxir 6y ago

AI models suffer massive diminishing returns on both model size and amount of input text, which fortunately gives fresh models a chance to compete (the benefits of GPT-2/Transformer architectures is that they are capable of scaling indefinitely, but they still have diminishing returns)

Voloskaya 6y ago
I completly agree, it's a diminishing return but a return nonetheles, so to have a "world class GPT" model with 10% of the data, and 10% (optimistically) of the compute that OpenAI used, you will need some very very impressive innovation elsewhere.

asutekku 6y ago

36gb of compressed text is a lot though.

Voloskaya 6y ago
That's about 50B tokens. It can be a lot depending on what you want to do, but It's 10% of what GPT-3 (the definition of world class GPT model) used.
- sillysaurusx 6y ago
  For what it's worth, we're serious about replicating GPT-3. books3 is just one piece. You will notice I never claimed equivalency to GPT-3's training data. books3 may be 10%, but The Pile is building the rest: https://twitter.com/arankomatsuzaki/status/1320414141895487490 https://twitter.com/arankomatsuzaki/status/13204141418954874... https://github.com/EleutherAI/The-Pile https://github.com/EleutherAI/The-Pile https://www.eleuther.ai/get-involved https://www.eleuther.ai/get-involved https://media.discordapp.net/attachments/735217892517216366/769820126466932756/unknown.png https://media.discordapp.net/attachments/735217892517216366/... https://media.discordapp.net/attachments/735217892517216366/769820362267295744/unknown.png https://media.discordapp.net/attachments/735217892517216366/...
  - Voloskaya 6y ago
    And again, I find books3 extremely cool and important work on your part, looking forward to the rest. I just have a minor gripe with saying that "now we can train world class GPT model" thanks to that, as you said it's just one piece, and as a typicial HNist I had to point it out :).
    - sillysaurusx 6y ago
      Believe it or not, I appreciate and relate to that sentiment. But after spending roughly one year acquiring knowledge related to this work, I feel I can say with a fairly high degree of certainty that this dataset alone is enough to train a model that will achieve "world class" status in some area. Writing books, perhaps. Which part of my logic do you feel is mistaken, and why? I am actually quite interested to hear thoughts from someone who is very pedantic about such things.