4 ms·

(even if you know LLMs are the future, please try to...) Take the pessimistic view on LLMs for a moment. That this doesn't really work out and the expensive co

(even if you know LLMs are the future, please try to...) Take the pessimistic view on LLMs for a moment. That this doesn't really work out and the expensive computers are a somewhat embarrassing misstep. In that world, at least a few companies will find themselves with multibillion dollar high performance computers running on site. With capex and power numbers to scare the accountants and financial analysts. Not running LLMs. That's a really fast computer. Computers can do stuff. If not LLMs, it's going to do other things. To confidently sit out this capability acquisition round you need to be sure that LLMs are grossly overrated and also that all the competition will fail to find anything else to do with their GPU supercomputers. Oh, and that your own staff would also fail to find anything (else) useful to do with one. I am completely comfortable expecting datacenter scale supercomputers with previously unimaginable compute to do interesting things. I wouldn't want to be the megacorp missing the revolution because I spent the money on dividends instead.

madaxe_again 2y ago
LLMs are a subset of AI/ML. They are not the totality of the sphere, nor are they what is being worked on exclusively.
justincormack 2y ago
The price of these computers is high now, due to supply constraints, so buying now isnt necessarily the best time for this outcome. You can buy some later... The things these can do is (a) memory bandwidth, Nvidia GPUs have been described as "the worlds most expensive memory controller". Generally very useful anywhere. (b) fast low precision floating point. Historically most of the uses of supercomputers wanted high precision, eg for physics etc but people are now experimenting with using lower precision given availability (c) fast networking. generally useful. We were seeing a spread of GPU into other applications pre AI, like GPU driven databases, I think this stopped a bit just because of pricing and availability.
- JonChesterfield 2y ago
  Lots of memory bandwidth, low latency high bandwidth connections between nodes, lots of floating point. The MI300 looking a bit like HPC being repurposed for AI is probably a win in terms of using the hardware for other things in the future. I've had to double check the low precision point qualifier. I expected MI300X to run f32 and f64 at the same speed (and it does) and nvidia to have slower f64 (which it does). It looks like H100 is 250TF at half, 60TF at float, 30TF at double. MI300X is 650TF at half, 80TF at float or double. Very fast F16 indeed. (sourced from https://www.techpowerup.com/gpu-specs/radeon-instinct-mi300x.c4179 https://www.techpowerup.com/gpu-specs/radeon-instinct-mi300x... and https://www.techpowerup.com/gpu-specs/h100-sxm5-96-gb.c3974 https://www.techpowerup.com/gpu-specs/h100-sxm5-96-gb.c3974, I might have the wrong SKUs for comparison)
richardw 2y ago
Or they invested in GPU’s to mine bitcoin and the ASIC’s are just starting to arrive. At least for inference, GPU’s are not the end game. And who knows if training will continue to need them. If so there could be another round of investment at a better performance per watt, and specialised hardware will be the key. They’ll all be building custom chips.
- JonChesterfield 2y ago
  The specialised ASIC approach is less likely to be useful for other things though, whereas GPUs are starting to threaten to run arbitrary software well.
  - richardw 2y ago
    Sure. My base case is that LLM’s find uses and that compute gets cheaper and more efficient, which helps make more business cases possible. Investing in current state GPU’s doesn’t seem that useful in that world. Nvidia is priced for a monopoly and I’m not sure it is one.
wrasee 2y ago
Imagine if LLMs didn't exist, tech companies hadn't massively invested in compute and money was put elsewhere, perhaps dividends. In that world, a tech company could arbitrarily invest in multibillion dollar high performance computers for no particular reason. Following the argument, with all that compute on site with nothing to do engineers would would find something interesting and all other companies that didn't invest would be missing out on that revolution. So by the argument, any tech company could gain a competitive advantage by any non-strategic investment on the assumption that it will always work out. But of course, this only works if you have infinite money and infinite opportunity to speculate. As soon as your resources are constrained, then strategic choice becomes the dominant factor and your CEO has a lot of explaining to do.
- JonChesterfield 2y ago
  What we have here though is the fear of being the company that didn't make that investment while your peers did. That has a completely different risk schedule on it. If it's a mistake, your excuse is that your competition did it too and you wanted to guard against being left behind. If it works, everyone is happy. If you're the only company making that speculative investment, it's great if it works and you might be fired by the board if it doesn't.
  - wrasee 2y ago
    But the risk/reward is the same regardless of the frame of reference. It's the difference in the perceived risk between gaining $10 and not loosing $10, even when the odds are the same. And we know that while rationally its the same, _loss aversion_ is a cognitive bias that makes people behave differently in these two cases. But FOMO is something we should be mitigating against, since our fear alone doesn't change the actual odds. But I will take your point that you have more cover for making a bad decision if you know all your competitors are making the same bad decision. But you still missed the opportunity to not make a bad decision and therefore get ahead. The actual risk remains the same.
    - JonChesterfield 2y ago
stkdump 2y ago
I guess if you are a multibillion dollar company, you can keep that in the bank (stock buybacks or whatever) or invest. So the question isn't really if AI is overhyped or not at the moment. The question is really do you know anything better to do with all that money to make your company future-proof.

madaxe_again 2y ago

LLMs are a subset of AI/ML. They are not the totality of the sphere, nor are they what is being worked on exclusively.

justincormack 2y ago

The price of these computers is high now, due to supply constraints, so buying now isnt necessarily the best time for this outcome. You can buy some later... The things these can do is (a) memory bandwidth, Nvidia GPUs have been described as "the worlds most expensive memory controller". Generally very useful anywhere. (b) fast low precision floating point. Historically most of the uses of supercomputers wanted high precision, eg for physics etc but people are now experimenting with using lower precision given availability (c) fast networking. generally useful. We were seeing a spread of GPU into other applications pre AI, like GPU driven databases, I think this stopped a bit just because of pricing and availability.

JonChesterfield 2y ago
Lots of memory bandwidth, low latency high bandwidth connections between nodes, lots of floating point. The MI300 looking a bit like HPC being repurposed for AI is probably a win in terms of using the hardware for other things in the future. I've had to double check the low precision point qualifier. I expected MI300X to run f32 and f64 at the same speed (and it does) and nvidia to have slower f64 (which it does). It looks like H100 is 250TF at half, 60TF at float, 30TF at double. MI300X is 650TF at half, 80TF at float or double. Very fast F16 indeed. (sourced from https://www.techpowerup.com/gpu-specs/radeon-instinct-mi300x.c4179 https://www.techpowerup.com/gpu-specs/radeon-instinct-mi300x... and https://www.techpowerup.com/gpu-specs/h100-sxm5-96-gb.c3974 https://www.techpowerup.com/gpu-specs/h100-sxm5-96-gb.c3974, I might have the wrong SKUs for comparison)

richardw 2y ago

Or they invested in GPU’s to mine bitcoin and the ASIC’s are just starting to arrive. At least for inference, GPU’s are not the end game. And who knows if training will continue to need them. If so there could be another round of investment at a better performance per watt, and specialised hardware will be the key. They’ll all be building custom chips.

JonChesterfield 2y ago
The specialised ASIC approach is less likely to be useful for other things though, whereas GPUs are starting to threaten to run arbitrary software well.
- richardw 2y ago
  Sure. My base case is that LLM’s find uses and that compute gets cheaper and more efficient, which helps make more business cases possible. Investing in current state GPU’s doesn’t seem that useful in that world. Nvidia is priced for a monopoly and I’m not sure it is one.

wrasee 2y ago

Imagine if LLMs didn't exist, tech companies hadn't massively invested in compute and money was put elsewhere, perhaps dividends. In that world, a tech company could arbitrarily invest in multibillion dollar high performance computers for no particular reason. Following the argument, with all that compute on site with nothing to do engineers would would find something interesting and all other companies that didn't invest would be missing out on that revolution. So by the argument, any tech company could gain a competitive advantage by any non-strategic investment on the assumption that it will always work out. But of course, this only works if you have infinite money and infinite opportunity to speculate. As soon as your resources are constrained, then strategic choice becomes the dominant factor and your CEO has a lot of explaining to do.

JonChesterfield 2y ago
What we have here though is the fear of being the company that didn't make that investment while your peers did. That has a completely different risk schedule on it. If it's a mistake, your excuse is that your competition did it too and you wanted to guard against being left behind. If it works, everyone is happy. If you're the only company making that speculative investment, it's great if it works and you might be fired by the board if it doesn't.
- wrasee 2y ago
  But the risk/reward is the same regardless of the frame of reference. It's the difference in the perceived risk between gaining $10 and not loosing $10, even when the odds are the same. And we know that while rationally its the same, _loss aversion_ is a cognitive bias that makes people behave differently in these two cases. But FOMO is something we should be mitigating against, since our fear alone doesn't change the actual odds. But I will take your point that you have more cover for making a bad decision if you know all your competitors are making the same bad decision. But you still missed the opportunity to not make a bad decision and therefore get ahead. The actual risk remains the same.
  - JonChesterfield 2y ago

stkdump 2y ago

I guess if you are a multibillion dollar company, you can keep that in the bank (stock buybacks or whatever) or invest. So the question isn't really if AI is overhyped or not at the moment. The question is really do you know anything better to do with all that money to make your company future-proof.