4 ms·

Probably not an issue just yet, think of how much data is generated by Twitter on a daily basis for example.

teraflop 5y ago
On the other hand, consider the difficulty of taking massive amounts of data from the modern web and filtering out the subset that was actually generated by humans, rather than previous generations of language models.
- adamsmith143 5y ago
  Definitely an interesting future problem. I'm sure OpenAI and others are thinking about it but I don't think these models are ubiquitous enough to have much impact just yet.
axg11 5y ago
Some estimates: - 500M tweets per day - 30 words/tokens per tweet - 40% of all tweets thrown away due to being duplicate/spam/bots = 9B tokens generated per day
zarzavat 5y ago
If you want to teach your kid to learn English, and they came back to you and said "Dad/mum, I finished reading the entire internet but I still don't understand English fully", would you say "OK son, now go and stare at the Twitter firehouse until you grok perfect English" ? It's clear that these models have orders of magnitude too much data already. It somewhat reminds me of the proposals for larger and larger colliders in the hopes of seeing new physics that is always one collider in the future.
- mijoharas 5y ago
  > It somewhat reminds me of the proposals for larger and larger colliders in the hopes of seeing new physics that is always one collider in the future. I agree with your main point, but think this analogy isn't an apt one. If you want to see what particles are created at higher energies you kinda need the bigger particle accelerators. (This isn't to say that we shouldn't be investigating lower energy collisions, but at a certain point you do need "bigger colliders" to see new things)
- lostmsu 5y ago
  I disagree with this take because you grok English not only from the text you read, but also from the context of physical world around you. And that context is enormous: assuming 8000x8000x2 vision with 3 color 1 byte channels at 24fps without compression, you get 3e+17 bytes (300 petabytes) of data along with your reading per year.
  - ralfd 5y ago
    Blind children can learn english fine though. And there are areas highly unmaterial (mathematics) which people still reason about.
    - lostmsu 5y ago
      You ignored the point. I only brought sight as an example (though, admittedly, it is the largest data inflow).
      - hugh-avherald 5y ago
        Humans have to learn sight while learning speech though. I don't mean knowing that this is a dog, I mean making sense of noisy vision inputs.
        lostmsu 5y ago
        So what? There's no indication that one hinders the other much, and may even improve generalization.
- adamsmith143 5y ago
  The general point is that there is a huge volume of training data generated daily not that Twitter is a great source of it. Though I believe that GPT-3 for example was trained on the Common Crawl dataset which would contain both Twitter and Reddit. >It's clear that these models have orders of magnitude too much data already. Seems like a strange claim. The scaling laws are showing that you can still make gains with more data and more parameters. >It somewhat reminds me of the proposals for larger and larger colliders in the hopes of seeing new physics that is always one collider in the future. This is literally true though, couldn't find the Higgs without the LHC and most GUT candidates would only start being ruled out at high energy levels.
  - zarzavat 5y ago
    > Seems like a strange claim. The scaling laws are showing that you can still make gains with more data and more parameters. But then we’ve given up on matching human intelligence which is all about working efficiently with small training data, and certainly training a human does not need anywhere near as much data as GPT-3. GPT-3 was interesting as a proof-of-concept of what happens when you use a gigantic amount of training data. We don’t need a bigger one until we can figure out how to make a smaller one that is just as effective. If scaling laws are telling us to keep putting even more training data into the thing, then the conclusion should be that the architecture is just not working out.
    - adamsmith143 5y ago
      >But then we’ve given up on matching human intelligence which is all about working efficiently with small training data, and certainly training a human does not need anywhere near as much data as GPT-3. I don't think we should really take so much inspiration from the brain. We didn't make airplanes work by building bird machines so why should we do that here. >GPT-3 was interesting as a proof-of-concept of what happens when you use a gigantic amount of training data. We don’t need a bigger one until we can figure out how to make a smaller one that is just as effective. This feels like a non sequitor. We can certainly keep making larger models and we will, because we can continue to make performance gains doing so. >If scaling laws are telling us to keep putting even more training data into the thing, then the conclusion should be that the architecture is just not working out. I don't think anyone in the field would agree to this point. Researchers see an easy avenue to gain better performance so they take it. Deepmind's model shows you can get similar results with more refined architecture, but this was released well after GPT-3. When teams significantly advance the state of the art with a much smaller model I think we should take notice but that hasn't happened yet.
- nullc 5y ago
  > It's clear that these models have orders of magnitude too much data already. I have a toy disproof for your claim that this is clear. Imagine that you are training a ML system using oracle access to Mum. The ML training system can request 10 million representative samples of Mum output, and then we could judge if the ML system has adequately reproduced Mum. Now also imagine that Mum frequently tells people that Mum knows a 23 letter secret and while mum won't tell people what is outright, she'll answer queries like if a guess is lexographically higher or lower. We could even imagine that the ML has seen Mum's side of some interactions with her doing that. Would the ML know Mum's secret? No. Would a child that could interact with Mum? Yes-- after at most ceil(log_alphabet(23)) queries at most, if the child is efficient. Learning in an interactive context is not the same as learning from written material, so you can't be sure that the fact that children learn english from less text means that a non-interactive ML system could english from the same amount. Q.E.D. Now, if someone figures out how to efficiently train these natural language models with reinforcement learning...

On the other hand, consider the difficulty of taking massive amounts of data from the modern web and filtering out the subset that was actually generated by humans, rather than previous generations of language models.

adamsmith143 5y ago
Definitely an interesting future problem. I'm sure OpenAI and others are thinking about it but I don't think these models are ubiquitous enough to have much impact just yet.

axg11 5y ago

Some estimates: - 500M tweets per day - 30 words/tokens per tweet - 40% of all tweets thrown away due to being duplicate/spam/bots = 9B tokens generated per day

zarzavat 5y ago

If you want to teach your kid to learn English, and they came back to you and said "Dad/mum, I finished reading the entire internet but I still don't understand English fully", would you say "OK son, now go and stare at the Twitter firehouse until you grok perfect English" ? It's clear that these models have orders of magnitude too much data already. It somewhat reminds me of the proposals for larger and larger colliders in the hopes of seeing new physics that is always one collider in the future.

mijoharas 5y ago
> It somewhat reminds me of the proposals for larger and larger colliders in the hopes of seeing new physics that is always one collider in the future. I agree with your main point, but think this analogy isn't an apt one. If you want to see what particles are created at higher energies you kinda need the bigger particle accelerators. (This isn't to say that we shouldn't be investigating lower energy collisions, but at a certain point you do need "bigger colliders" to see new things)
lostmsu 5y ago
I disagree with this take because you grok English not only from the text you read, but also from the context of physical world around you. And that context is enormous: assuming 8000x8000x2 vision with 3 color 1 byte channels at 24fps without compression, you get 3e+17 bytes (300 petabytes) of data along with your reading per year.
- ralfd 5y ago
  Blind children can learn english fine though. And there are areas highly unmaterial (mathematics) which people still reason about.
  - lostmsu 5y ago
    You ignored the point. I only brought sight as an example (though, admittedly, it is the largest data inflow).
    - hugh-avherald 5y ago
      Humans have to learn sight while learning speech though. I don't mean knowing that this is a dog, I mean making sense of noisy vision inputs.
      - lostmsu 5y ago
        So what? There's no indication that one hinders the other much, and may even improve generalization.
adamsmith143 5y ago
The general point is that there is a huge volume of training data generated daily not that Twitter is a great source of it. Though I believe that GPT-3 for example was trained on the Common Crawl dataset which would contain both Twitter and Reddit. >It's clear that these models have orders of magnitude too much data already. Seems like a strange claim. The scaling laws are showing that you can still make gains with more data and more parameters. >It somewhat reminds me of the proposals for larger and larger colliders in the hopes of seeing new physics that is always one collider in the future. This is literally true though, couldn't find the Higgs without the LHC and most GUT candidates would only start being ruled out at high energy levels.
- zarzavat 5y ago
  > Seems like a strange claim. The scaling laws are showing that you can still make gains with more data and more parameters. But then we’ve given up on matching human intelligence which is all about working efficiently with small training data, and certainly training a human does not need anywhere near as much data as GPT-3. GPT-3 was interesting as a proof-of-concept of what happens when you use a gigantic amount of training data. We don’t need a bigger one until we can figure out how to make a smaller one that is just as effective. If scaling laws are telling us to keep putting even more training data into the thing, then the conclusion should be that the architecture is just not working out.
  - adamsmith143 5y ago
    >But then we’ve given up on matching human intelligence which is all about working efficiently with small training data, and certainly training a human does not need anywhere near as much data as GPT-3. I don't think we should really take so much inspiration from the brain. We didn't make airplanes work by building bird machines so why should we do that here. >GPT-3 was interesting as a proof-of-concept of what happens when you use a gigantic amount of training data. We don’t need a bigger one until we can figure out how to make a smaller one that is just as effective. This feels like a non sequitor. We can certainly keep making larger models and we will, because we can continue to make performance gains doing so. >If scaling laws are telling us to keep putting even more training data into the thing, then the conclusion should be that the architecture is just not working out. I don't think anyone in the field would agree to this point. Researchers see an easy avenue to gain better performance so they take it. Deepmind's model shows you can get similar results with more refined architecture, but this was released well after GPT-3. When teams significantly advance the state of the art with a much smaller model I think we should take notice but that hasn't happened yet.
nullc 5y ago
> It's clear that these models have orders of magnitude too much data already. I have a toy disproof for your claim that this is clear. Imagine that you are training a ML system using oracle access to Mum. The ML training system can request 10 million representative samples of Mum output, and then we could judge if the ML system has adequately reproduced Mum. Now also imagine that Mum frequently tells people that Mum knows a 23 letter secret and while mum won't tell people what is outright, she'll answer queries like if a guess is lexographically higher or lower. We could even imagine that the ML has seen Mum's side of some interactions with her doing that. Would the ML know Mum's secret? No. Would a child that could interact with Mum? Yes-- after at most ceil(log_alphabet(23)) queries at most, if the child is efficient. Learning in an interactive context is not the same as learning from written material, so you can't be sure that the fact that children learn english from less text means that a non-interactive ML system could english from the same amount. Q.E.D. Now, if someone figures out how to efficiently train these natural language models with reinforcement learning...