4 ms·

AI has seen rapid efficiency improvements every year for the last decade, but the amount of computation used has gone up, not down. The improvements always go i

AI has seen rapid efficiency improvements every year for the last decade, but the amount of computation used has gone up, not down. The improvements always go into making larger and more powerful models, not into just running the same models but at a lower cost per hour. https://epochai.org/blog/algorithmic-progress-in-language-models https://epochai.org/blog/algorithmic-progress-in-language-mo...

nyxtom 2y ago
This ^, the models are getting larger and we don't appear to have scratched the surface on Video / Vision models yet. Energy consumption is accelerating along side that at 28% YOY https://www.weforum.org/agenda/2024/04/how-to-manage-ais-energy-demand-today-tomorrow-and-in-the-future/#:~:text=AI%20and%20energy%20demand&text=The%20energy%20required%20to%20run,of%20Iceland%20used%20in%202021 https://www.weforum.org/agenda/2024/04/how-to-manage-ais-ene....
- zarzavat 2y ago
  While vision-vision models are certainly cool, I don’t think that they are as economically valuable as vision-speech or text-text. Humans don’t have vision output. Computation may be increasing, but that is a statement about the short-term not the long-term. If we want to predict the future then we care about: how many capabilities can you fit on a phone-sized computer? And I believe that the answer is: a lot.
SirMaster 2y ago
But doesn't it feel like the larger models are seeing diminished returns compared to the smaller open models? Feels like they will eventually converge unless some big breakthrough happens that only benefits the large models.
- nyxtom 2y ago
  Diffusion models still have plenty of room to grow (vision/video is orders of magnitude more expensive and larger) and we're only beginning to experiment with agent (AI -> AI Agent) workflow communication and automation.
- int_19h 2y ago
  Not at all. There's definitely a lot of hype about how "our 7B model is almost as good as GPT-4" etc, but upon close inspection, it turns out to be cherry picked meaningless numbers every single time. At the same time, there's no evidence that even at the scale of GPT-4 they are seeing diminishing returns. I don't see why that would be the case, either, so long as there's more data to feed into the models. IMO the cliff will occur once the models become large enough that no amount of naturally sourced data is enough to saturate them, but, well, we aren't even at 1% of the Internet yet.

nyxtom 2y ago

This ^, the models are getting larger and we don't appear to have scratched the surface on Video / Vision models yet. Energy consumption is accelerating along side that at 28% YOY https://www.weforum.org/agenda/2024/04/how-to-manage-ais-energy-demand-today-tomorrow-and-in-the-future/#:~:text=AI%20and%20energy%20demand&text=The%20energy%20required%20to%20run,of%20Iceland%20used%20in%202021 https://www.weforum.org/agenda/2024/04/how-to-manage-ais-ene....

zarzavat 2y ago
While vision-vision models are certainly cool, I don’t think that they are as economically valuable as vision-speech or text-text. Humans don’t have vision output. Computation may be increasing, but that is a statement about the short-term not the long-term. If we want to predict the future then we care about: how many capabilities can you fit on a phone-sized computer? And I believe that the answer is: a lot.

SirMaster 2y ago

But doesn't it feel like the larger models are seeing diminished returns compared to the smaller open models? Feels like they will eventually converge unless some big breakthrough happens that only benefits the large models.

nyxtom 2y ago
Diffusion models still have plenty of room to grow (vision/video is orders of magnitude more expensive and larger) and we're only beginning to experiment with agent (AI -> AI Agent) workflow communication and automation.
int_19h 2y ago
Not at all. There's definitely a lot of hype about how "our 7B model is almost as good as GPT-4" etc, but upon close inspection, it turns out to be cherry picked meaningless numbers every single time. At the same time, there's no evidence that even at the scale of GPT-4 they are seeing diminishing returns. I don't see why that would be the case, either, so long as there's more data to feed into the models. IMO the cliff will occur once the models become large enough that no amount of naturally sourced data is enough to saturate them, but, well, we aren't even at 1% of the Internet yet.