3 ms·

The general point is that there is a huge volume of training data generated daily not that Twitter is a great source of it. Though I believe that GPT-3 for exam

The general point is that there is a huge volume of training data generated daily not that Twitter is a great source of it. Though I believe that GPT-3 for example was trained on the Common Crawl dataset which would contain both Twitter and Reddit. >It's clear that these models have orders of magnitude too much data already. Seems like a strange claim. The scaling laws are showing that you can still make gains with more data and more parameters. >It somewhat reminds me of the proposals for larger and larger colliders in the hopes of seeing new physics that is always one collider in the future. This is literally true though, couldn't find the Higgs without the LHC and most GUT candidates would only start being ruled out at high energy levels.

zarzavat 5y ago
> Seems like a strange claim. The scaling laws are showing that you can still make gains with more data and more parameters. But then we’ve given up on matching human intelligence which is all about working efficiently with small training data, and certainly training a human does not need anywhere near as much data as GPT-3. GPT-3 was interesting as a proof-of-concept of what happens when you use a gigantic amount of training data. We don’t need a bigger one until we can figure out how to make a smaller one that is just as effective. If scaling laws are telling us to keep putting even more training data into the thing, then the conclusion should be that the architecture is just not working out.
- adamsmith143 5y ago
  >But then we’ve given up on matching human intelligence which is all about working efficiently with small training data, and certainly training a human does not need anywhere near as much data as GPT-3. I don't think we should really take so much inspiration from the brain. We didn't make airplanes work by building bird machines so why should we do that here. >GPT-3 was interesting as a proof-of-concept of what happens when you use a gigantic amount of training data. We don’t need a bigger one until we can figure out how to make a smaller one that is just as effective. This feels like a non sequitor. We can certainly keep making larger models and we will, because we can continue to make performance gains doing so. >If scaling laws are telling us to keep putting even more training data into the thing, then the conclusion should be that the architecture is just not working out. I don't think anyone in the field would agree to this point. Researchers see an easy avenue to gain better performance so they take it. Deepmind's model shows you can get similar results with more refined architecture, but this was released well after GPT-3. When teams significantly advance the state of the art with a much smaller model I think we should take notice but that hasn't happened yet.
  - zarzavat 5y ago
    > I don't think we should really take so much inspiration from the brain. We didn't make airplanes work by building bird machines so why should we do that here. It’s not that we should mimic the brain’s implementation, but we should certainly strive to match the brain’s capabilities. One of its outwardly observable capabilities is that it is extremely efficient in the size of the training data set it requires. Efficiency isn’t an implementation detail, it’s definitional to what “highly intelligent” means. GPT-3 is not an airplane, it’s a zeppelin. Zeppelins also have scaling laws dictating that a zeppelin should be very very large. Building bigger and bigger zeppelins is one thing, justifying expending resources on gigantic zeppelins by stating the scaling law and concluding that a jet aircraft will magically pop out if you build a big enough zeppelin is quite another.
    - adamsmith143 5y ago
      Your earlier analogy kind of feels like saying that because you can go further by adding more fuel to a jet engines fuel tank that you have failed at efficiency and should redesign the engine. But generally I think the better analogy is a rocket ship. If we can still go higher and faster with more fuel we should try to do that before we worry about engine efficiency. You have to get to the moon before you can colonize the galaxy.
gwern 5y ago
Common Crawl actually does not contain Twitter, you can go check the indexes https://github.com/ikreymer/cdx-index-client https://github.com/ikreymer/cdx-index-client . Twitter is extremely aggressive about scraping/caching, and I guess that blocks CC. Models like GPT-3 still know a decent amount of Twitter material, and I figure that this is due to tweets being excerpts or mirrored manually in non-Twitter.com URLs (eg all the Twitter-mirroring bots on Reddit).

zarzavat 5y ago

> Seems like a strange claim. The scaling laws are showing that you can still make gains with more data and more parameters. But then we’ve given up on matching human intelligence which is all about working efficiently with small training data, and certainly training a human does not need anywhere near as much data as GPT-3. GPT-3 was interesting as a proof-of-concept of what happens when you use a gigantic amount of training data. We don’t need a bigger one until we can figure out how to make a smaller one that is just as effective. If scaling laws are telling us to keep putting even more training data into the thing, then the conclusion should be that the architecture is just not working out.

adamsmith143 5y ago
>But then we’ve given up on matching human intelligence which is all about working efficiently with small training data, and certainly training a human does not need anywhere near as much data as GPT-3. I don't think we should really take so much inspiration from the brain. We didn't make airplanes work by building bird machines so why should we do that here. >GPT-3 was interesting as a proof-of-concept of what happens when you use a gigantic amount of training data. We don’t need a bigger one until we can figure out how to make a smaller one that is just as effective. This feels like a non sequitor. We can certainly keep making larger models and we will, because we can continue to make performance gains doing so. >If scaling laws are telling us to keep putting even more training data into the thing, then the conclusion should be that the architecture is just not working out. I don't think anyone in the field would agree to this point. Researchers see an easy avenue to gain better performance so they take it. Deepmind's model shows you can get similar results with more refined architecture, but this was released well after GPT-3. When teams significantly advance the state of the art with a much smaller model I think we should take notice but that hasn't happened yet.
- zarzavat 5y ago
  > I don't think we should really take so much inspiration from the brain. We didn't make airplanes work by building bird machines so why should we do that here. It’s not that we should mimic the brain’s implementation, but we should certainly strive to match the brain’s capabilities. One of its outwardly observable capabilities is that it is extremely efficient in the size of the training data set it requires. Efficiency isn’t an implementation detail, it’s definitional to what “highly intelligent” means. GPT-3 is not an airplane, it’s a zeppelin. Zeppelins also have scaling laws dictating that a zeppelin should be very very large. Building bigger and bigger zeppelins is one thing, justifying expending resources on gigantic zeppelins by stating the scaling law and concluding that a jet aircraft will magically pop out if you build a big enough zeppelin is quite another.
  - adamsmith143 5y ago
    Your earlier analogy kind of feels like saying that because you can go further by adding more fuel to a jet engines fuel tank that you have failed at efficiency and should redesign the engine. But generally I think the better analogy is a rocket ship. If we can still go higher and faster with more fuel we should try to do that before we worry about engine efficiency. You have to get to the moon before you can colonize the galaxy.

gwern 5y ago

Common Crawl actually does not contain Twitter, you can go check the indexes https://github.com/ikreymer/cdx-index-client https://github.com/ikreymer/cdx-index-client . Twitter is extremely aggressive about scraping/caching, and I guess that blocks CC. Models like GPT-3 still know a decent amount of Twitter material, and I figure that this is due to tweets being excerpts or mirrored manually in non-Twitter.com URLs (eg all the Twitter-mirroring bots on Reddit).