3 ms·

The market cap is irrational and I submit as evidence Llama 3. AI won’t need huge amounts of computing resources. It will need moderate amounts of computing re

The market cap is irrational and I submit as evidence Llama 3. AI won’t need huge amounts of computing resources. It will need moderate amounts of computing resources. The huge amount of computing resources seen today are the result of immature technology that uses resources inefficiently. As the technology improves, the capability-per-FLOP will increase and AI will move on-device.

apsec112 2y ago
AI has seen rapid efficiency improvements every year for the last decade, but the amount of computation used has gone up, not down. The improvements always go into making larger and more powerful models, not into just running the same models but at a lower cost per hour. https://epochai.org/blog/algorithmic-progress-in-language-models https://epochai.org/blog/algorithmic-progress-in-language-mo...
- nyxtom 2y ago
  This ^, the models are getting larger and we don't appear to have scratched the surface on Video / Vision models yet. Energy consumption is accelerating along side that at 28% YOY https://www.weforum.org/agenda/2024/04/how-to-manage-ais-energy-demand-today-tomorrow-and-in-the-future/#:~:text=AI%20and%20energy%20demand&text=The%20energy%20required%20to%20run,of%20Iceland%20used%20in%202021 https://www.weforum.org/agenda/2024/04/how-to-manage-ais-ene....
  - zarzavat 2y ago
    While vision-vision models are certainly cool, I don’t think that they are as economically valuable as vision-speech or text-text. Humans don’t have vision output. Computation may be increasing, but that is a statement about the short-term not the long-term. If we want to predict the future then we care about: how many capabilities can you fit on a phone-sized computer? And I believe that the answer is: a lot.
- SirMaster 2y ago
  But doesn't it feel like the larger models are seeing diminished returns compared to the smaller open models? Feels like they will eventually converge unless some big breakthrough happens that only benefits the large models.
  - nyxtom 2y ago
    Diffusion models still have plenty of room to grow (vision/video is orders of magnitude more expensive and larger) and we're only beginning to experiment with agent (AI -> AI Agent) workflow communication and automation.
  - int_19h 2y ago
    Not at all. There's definitely a lot of hype about how "our 7B model is almost as good as GPT-4" etc, but upon close inspection, it turns out to be cherry picked meaningless numbers every single time. At the same time, there's no evidence that even at the scale of GPT-4 they are seeing diminishing returns. I don't see why that would be the case, either, so long as there's more data to feed into the models. IMO the cliff will occur once the models become large enough that no amount of naturally sourced data is enough to saturate them, but, well, we aren't even at 1% of the Internet yet.
YetAnotherNick 2y ago
> I submit as evidence Llama 3. Llama 3 has likely costed billions to Meta and most of it went to Nvidia. What are you talking about. As much as what you think inference costs, it is tiny in comparison to training the model.
- zarzavat 2y ago
  Pre-training happens once then it’s done. You can’t make a multi-trillion dollar business out of providing hardware for that. Inference is where the money is because it’s recurring.
  - YetAnotherNick 2y ago
    No, while what you are saying sounds intuitive on one hand, OpenAI is building $100b supercomputer for training new models. As long as people want to train better models than a year before(which seems to be a good assumption for new few years), training will keep on giving good revenue to Nvidia. Inference requires 1/3rd the FLOPs than training per token and current models are already trained on entirety of internet. You can generate 3 times the internet for 1 training run.
lxgr 2y ago
Are you suggesting that there will be some future, more efficient architecture of machine learning that'll not be able to make use of extra compute at all (or at least have greatly diminishing returns while doing so)? Otherwise, I don't see why it wouldn't make sense to continue grabbing as many GPUs as feasible for the foreseeable future, at least until LLMs and other GenAI models have been determined to have plateaued without much doubt.
paulmd 2y ago
Really? What hardware was llama3 trained on? now realize there’s 15 other models you haven’t heard of trained on the same brand’s hardware that didn’t work out / weren’t impressive enough to release. you are literally describing Jevons paradox in action - the more efficient and practical you make the tech, the more of it we will consume. And sure, inference is easy, but training is still uniformly done on nvidia hardware, and the “software” is an incredibly non-trivial moat. AMD has been trying for literally 15 years and they’re on their third or fourth ground-up attempt to displace it… the most recent (ROCm) being over 5 years old at this point and still utterly non-competitive. Leave it to HN to imply that millions of person-hours of ecosystem building can basically be replicated in a long weekend. (also, reminder that ollama3 has a 70b model too… and it’s a lot better than the 8b one still!)

apsec112 2y ago

AI has seen rapid efficiency improvements every year for the last decade, but the amount of computation used has gone up, not down. The improvements always go into making larger and more powerful models, not into just running the same models but at a lower cost per hour. https://epochai.org/blog/algorithmic-progress-in-language-models https://epochai.org/blog/algorithmic-progress-in-language-mo...

nyxtom 2y ago
This ^, the models are getting larger and we don't appear to have scratched the surface on Video / Vision models yet. Energy consumption is accelerating along side that at 28% YOY https://www.weforum.org/agenda/2024/04/how-to-manage-ais-energy-demand-today-tomorrow-and-in-the-future/#:~:text=AI%20and%20energy%20demand&text=The%20energy%20required%20to%20run,of%20Iceland%20used%20in%202021 https://www.weforum.org/agenda/2024/04/how-to-manage-ais-ene....
- zarzavat 2y ago
  While vision-vision models are certainly cool, I don’t think that they are as economically valuable as vision-speech or text-text. Humans don’t have vision output. Computation may be increasing, but that is a statement about the short-term not the long-term. If we want to predict the future then we care about: how many capabilities can you fit on a phone-sized computer? And I believe that the answer is: a lot.
SirMaster 2y ago
But doesn't it feel like the larger models are seeing diminished returns compared to the smaller open models? Feels like they will eventually converge unless some big breakthrough happens that only benefits the large models.
- nyxtom 2y ago
  Diffusion models still have plenty of room to grow (vision/video is orders of magnitude more expensive and larger) and we're only beginning to experiment with agent (AI -> AI Agent) workflow communication and automation.
- int_19h 2y ago
  Not at all. There's definitely a lot of hype about how "our 7B model is almost as good as GPT-4" etc, but upon close inspection, it turns out to be cherry picked meaningless numbers every single time. At the same time, there's no evidence that even at the scale of GPT-4 they are seeing diminishing returns. I don't see why that would be the case, either, so long as there's more data to feed into the models. IMO the cliff will occur once the models become large enough that no amount of naturally sourced data is enough to saturate them, but, well, we aren't even at 1% of the Internet yet.

YetAnotherNick 2y ago

> I submit as evidence Llama 3. Llama 3 has likely costed billions to Meta and most of it went to Nvidia. What are you talking about. As much as what you think inference costs, it is tiny in comparison to training the model.

zarzavat 2y ago
Pre-training happens once then it’s done. You can’t make a multi-trillion dollar business out of providing hardware for that. Inference is where the money is because it’s recurring.
- YetAnotherNick 2y ago
  No, while what you are saying sounds intuitive on one hand, OpenAI is building $100b supercomputer for training new models. As long as people want to train better models than a year before(which seems to be a good assumption for new few years), training will keep on giving good revenue to Nvidia. Inference requires 1/3rd the FLOPs than training per token and current models are already trained on entirety of internet. You can generate 3 times the internet for 1 training run.

lxgr 2y ago

Are you suggesting that there will be some future, more efficient architecture of machine learning that'll not be able to make use of extra compute at all (or at least have greatly diminishing returns while doing so)? Otherwise, I don't see why it wouldn't make sense to continue grabbing as many GPUs as feasible for the foreseeable future, at least until LLMs and other GenAI models have been determined to have plateaued without much doubt.

paulmd 2y ago

Really? What hardware was llama3 trained on? now realize there’s 15 other models you haven’t heard of trained on the same brand’s hardware that didn’t work out / weren’t impressive enough to release. you are literally describing Jevons paradox in action - the more efficient and practical you make the tech, the more of it we will consume. And sure, inference is easy, but training is still uniformly done on nvidia hardware, and the “software” is an incredibly non-trivial moat. AMD has been trying for literally 15 years and they’re on their third or fourth ground-up attempt to displace it… the most recent (ROCm) being over 5 years old at this point and still utterly non-competitive. Leave it to HN to imply that millions of person-hours of ecosystem building can basically be replicated in a long weekend. (also, reminder that ollama3 has a 70b model too… and it’s a lot better than the 8b one still!)