3 ms·

>But then we’ve given up on matching human intelligence which is all about working efficiently with small training data, and certainly training a human does not

>But then we’ve given up on matching human intelligence which is all about working efficiently with small training data, and certainly training a human does not need anywhere near as much data as GPT-3. I don't think we should really take so much inspiration from the brain. We didn't make airplanes work by building bird machines so why should we do that here. >GPT-3 was interesting as a proof-of-concept of what happens when you use a gigantic amount of training data. We don’t need a bigger one until we can figure out how to make a smaller one that is just as effective. This feels like a non sequitor. We can certainly keep making larger models and we will, because we can continue to make performance gains doing so. >If scaling laws are telling us to keep putting even more training data into the thing, then the conclusion should be that the architecture is just not working out. I don't think anyone in the field would agree to this point. Researchers see an easy avenue to gain better performance so they take it. Deepmind's model shows you can get similar results with more refined architecture, but this was released well after GPT-3. When teams significantly advance the state of the art with a much smaller model I think we should take notice but that hasn't happened yet.

zarzavat 5y ago
> I don't think we should really take so much inspiration from the brain. We didn't make airplanes work by building bird machines so why should we do that here. It’s not that we should mimic the brain’s implementation, but we should certainly strive to match the brain’s capabilities. One of its outwardly observable capabilities is that it is extremely efficient in the size of the training data set it requires. Efficiency isn’t an implementation detail, it’s definitional to what “highly intelligent” means. GPT-3 is not an airplane, it’s a zeppelin. Zeppelins also have scaling laws dictating that a zeppelin should be very very large. Building bigger and bigger zeppelins is one thing, justifying expending resources on gigantic zeppelins by stating the scaling law and concluding that a jet aircraft will magically pop out if you build a big enough zeppelin is quite another.
- adamsmith143 5y ago
  Your earlier analogy kind of feels like saying that because you can go further by adding more fuel to a jet engines fuel tank that you have failed at efficiency and should redesign the engine. But generally I think the better analogy is a rocket ship. If we can still go higher and faster with more fuel we should try to do that before we worry about engine efficiency. You have to get to the moon before you can colonize the galaxy.

zarzavat 5y ago

> I don't think we should really take so much inspiration from the brain. We didn't make airplanes work by building bird machines so why should we do that here. It’s not that we should mimic the brain’s implementation, but we should certainly strive to match the brain’s capabilities. One of its outwardly observable capabilities is that it is extremely efficient in the size of the training data set it requires. Efficiency isn’t an implementation detail, it’s definitional to what “highly intelligent” means. GPT-3 is not an airplane, it’s a zeppelin. Zeppelins also have scaling laws dictating that a zeppelin should be very very large. Building bigger and bigger zeppelins is one thing, justifying expending resources on gigantic zeppelins by stating the scaling law and concluding that a jet aircraft will magically pop out if you build a big enough zeppelin is quite another.

adamsmith143 5y ago
Your earlier analogy kind of feels like saying that because you can go further by adding more fuel to a jet engines fuel tank that you have failed at efficiency and should redesign the engine. But generally I think the better analogy is a rocket ship. If we can still go higher and faster with more fuel we should try to do that before we worry about engine efficiency. You have to get to the moon before you can colonize the galaxy.