4 ms·

Right, but is any serious researcher insisting that just scaling will be enough to achieve AGI? Because while I'm a layperson on this stuff, I don't get that im

Right, but is any serious researcher insisting that just scaling will be enough to achieve AGI? Because while I'm a layperson on this stuff, I don't get that impression, meaning this criticism from Marcus is mostly attacking a strawman. And he's not the only one peddling it. The understanding I have is that scalability of LLMs with data took even their developers by surprise. They kept at it because empirically, so far, it's worked. But nobody assumes it'll keep working indefinitely or that it'll lead to AGI alone. If they did, OpenAI, Google, etc. would have fired most of their researchers and would simply focus everything they have on scaling.

thiago_fm 3y ago
There are plenty of leaders in SV, one example being Sam Altman that is preaching that hardware is key. Just take a look at nVidia's $2t market cap. When we all know it isn't, and this can have the side effect of creating a bubble. I would be concerned with this as it can have terrible effects to AI research long-term.
albertzeyer 3y ago
When you look at the history of scaling up Transformers, e.g. GPT2, most people in the community were in fact quite surprised that just scaling up those models worked so well. And every time you read a lot of opinions that this is the end and scaling them up further will not give (much) improvements. This criticism is lower now but still there. Scaling self-attention has also worked much further than it was initially believed it could work. Initially it was always believed to be the main bottleneck, but this turned out wrong. The feed-forward layers are much more the bottleneck when looking at where most of the compute is spent. It's still a bit unclear how much further we can take the scaling. We are mostly at the limit of what is financial feasible today but as long as some form of generalized Moore's law continues (e.g. number of transistors per dollar), it becomes financial feasible to scale further. At some point we also hit a limit of available data. But maybe self-play (or variants) might solve this. I guess most researchers agree that just scaling up further is maybe not optimal (w.r.t. reaching AGI), or also will not be enough, but it's a somewhat open question.
xiphias2 3y ago
The word scaling has multiple definitions. It's not just the same hardware getting cheaper over time, there have been many architectural improvements: - Better training algorithms for training with lower precision - fastattention, fastattention 2 for decreasing bandwidth inside the GPU - ring attention for decreasing bandwidth across GPUs - algorithmic improvements in handling long context window - MoE - Lots of algorithmic improvements in fine tuning / alignment - GROQ hardware architecture (deterministic hardware, storing all data in SRAM for inferencing instead of using cache hierarchies) - Improvements in tokenization So far softmax(K * Q')*V is the only thing that hasn't (yet) been touched. If the question is that ,,just'' improving LLM perplexity by further algorithmic and hardware improvements will lead to AGI, at this point many researchers believe that the answer is yes (and many others that the answer is no :) ).

thiago_fm 3y ago

There are plenty of leaders in SV, one example being Sam Altman that is preaching that hardware is key. Just take a look at nVidia's $2t market cap. When we all know it isn't, and this can have the side effect of creating a bubble. I would be concerned with this as it can have terrible effects to AI research long-term.

albertzeyer 3y ago

When you look at the history of scaling up Transformers, e.g. GPT2, most people in the community were in fact quite surprised that just scaling up those models worked so well. And every time you read a lot of opinions that this is the end and scaling them up further will not give (much) improvements. This criticism is lower now but still there. Scaling self-attention has also worked much further than it was initially believed it could work. Initially it was always believed to be the main bottleneck, but this turned out wrong. The feed-forward layers are much more the bottleneck when looking at where most of the compute is spent. It's still a bit unclear how much further we can take the scaling. We are mostly at the limit of what is financial feasible today but as long as some form of generalized Moore's law continues (e.g. number of transistors per dollar), it becomes financial feasible to scale further. At some point we also hit a limit of available data. But maybe self-play (or variants) might solve this. I guess most researchers agree that just scaling up further is maybe not optimal (w.r.t. reaching AGI), or also will not be enough, but it's a somewhat open question.

xiphias2 3y ago

The word scaling has multiple definitions. It's not just the same hardware getting cheaper over time, there have been many architectural improvements: - Better training algorithms for training with lower precision - fastattention, fastattention 2 for decreasing bandwidth inside the GPU - ring attention for decreasing bandwidth across GPUs - algorithmic improvements in handling long context window - MoE - Lots of algorithmic improvements in fine tuning / alignment - GROQ hardware architecture (deterministic hardware, storing all data in SRAM for inferencing instead of using cache hierarchies) - Improvements in tokenization So far softmax(K * Q')*V is the only thing that hasn't (yet) been touched. If the question is that ,,just'' improving LLM perplexity by further algorithmic and hardware improvements will lead to AGI, at this point many researchers believe that the answer is yes (and many others that the answer is no :) ).