3 ms·

AGI is probably pretty close, investment keeps ramping up in the space, and we continue to see advancements in a positive direction. My one critique: scale is

AGI is probably pretty close, investment keeps ramping up in the space, and we continue to see advancements in a positive direction. My one critique: scale is absolutely part of the answer. Before big transformer models, people thought that few shot learning would require fundamentally different architectures. GPT3 changed all that which is why its white paper is named as such. Few shot learning is actually emergent out of its size. Which was surprising to a lot of people. Few shot learning is an incredible capability which is a big step toward AGI, so I’m not sure I buy the case that it’s not important.

the_af 5y ago
What is "pretty close"? A few years? Decades? A century? I don't think AGI is close, nor is the amount of investment a good indication of nearness (or even, actual possibility that it will ever happen).
- mountainriver 5y ago
  The investment continues to make progress, and the progress we are seeing is starting to look very human. GPT4 will probably be multimodal solving some of its grounding issues and will be able to fool most people in conversations. That’s probably just years away. GPT6? The lines will be very blurry as to what we call AGI
mkaic 5y ago
Hmm, I think you're right -- I should have phrased things differently. It's not that I don't think AGI won't require a very large model, it's more that I think that we're already pretty close to the scale we need, so OpenAI's goal of scaling another 1000x isn't really the direction I think we should be heading. I honestly think 175B parameters could very well be enough for an AGI if they were organized efficiently. It's definitely one of the weak points of the article, though, as it's more based on my own opinions, and isn't really empirical. My post is mostly just wild speculation for the sake of speculation anyway :) Thank you for your comments!
- mountainriver 5y ago
  Okay yeah totally, that makes sense. We also have stopped seeing huge gains in scaling beyond GPT3 sized models which would indicate we have hit some maximum there. Although we thought we had hit a maximum with deep learning before the transformer came around so it could be misleading. Distributing the computation to many models like what Google did with GLaM could very well be the future of AGI. Economies of models rather than one big model.

the_af 5y ago

What is "pretty close"? A few years? Decades? A century? I don't think AGI is close, nor is the amount of investment a good indication of nearness (or even, actual possibility that it will ever happen).

mountainriver 5y ago
The investment continues to make progress, and the progress we are seeing is starting to look very human. GPT4 will probably be multimodal solving some of its grounding issues and will be able to fool most people in conversations. That’s probably just years away. GPT6? The lines will be very blurry as to what we call AGI

mkaic 5y ago

Hmm, I think you're right -- I should have phrased things differently. It's not that I don't think AGI won't require a very large model, it's more that I think that we're already pretty close to the scale we need, so OpenAI's goal of scaling another 1000x isn't really the direction I think we should be heading. I honestly think 175B parameters could very well be enough for an AGI if they were organized efficiently. It's definitely one of the weak points of the article, though, as it's more based on my own opinions, and isn't really empirical. My post is mostly just wild speculation for the sake of speculation anyway :) Thank you for your comments!

mountainriver 5y ago
Okay yeah totally, that makes sense. We also have stopped seeing huge gains in scaling beyond GPT3 sized models which would indicate we have hit some maximum there. Although we thought we had hit a maximum with deep learning before the transformer came around so it could be misleading. Distributing the computation to many models like what Google did with GLaM could very well be the future of AGI. Economies of models rather than one big model.