3 ms·

What's baffling is the hardware is actually pretty good (from direct experience), and the software stack wouldn't take much effort to bring up to par either, at

by ein0p 2y ago

What's baffling is the hardware is actually pretty good (from direct experience), and the software stack wouldn't take much effort to bring up to par either, at least for inference. Perf/$ is much better than NVIDIA. More HBM than NVIDIA, too. And somehow nobody gives a shit.

mjevans 2y ago
Doesn't AMD have a similar issue, with the 'secret sauce' really that entire NV toolchain that is practically a monopoly?
- gdiamos 2y ago
  The problem is no differentiation If you have to do more work to port for no benefit, it doesn’t make sense except to commoditize it
  - dilyevsky 2y ago
    Better price and better lead time, if achieved, seems like enough differentiation to me, at least to enter the market successfully
alecco 2y ago
Is it? Good luck loading large models over PCIe 4.0. Or running things over multiple cards. And it has older HBM2e which is ~2x slower than H100 HBM3. https://en.wikipedia.org/wiki/High_Bandwidth_Memory#HBM2E https://en.wikipedia.org/wiki/High_Bandwidth_Memory#HBM2E "N5, PCIe 4.0, and HBM2e. This chip was probably delayed two years." -wmf
- wmf 2y ago
  Gaudi 3 at least has PCIe 5.0 but for some reason they stuck with HBM2e.
- ein0p 2y ago
  Once loaded, weights just stay in HBM. And PCIe throughput is not at all an issue - the cards communicate over a large number of very high speed ROCE links both inside and outside the server. HBM throughput is not as much of an issue either, since during inference you are mostly just reading weights, not also writing out intermediate results and gradients. HBM2e is actually a smart decision on Intel's part IMO - it's not as supply constrained, so they can have more of it. Gaudi2 has 96GB of HBM2e. Gaudi3 has 144GB IIRC, as well as about 4x more compute and even faster interconnect. This means you can do longer context and/or larger models without splitting it across hosts where bandwidth is at somewhat of a premium. Though Gaudi3 is also less price competitive, I must add. No relation to Intel - I have never worked for them and hold no position in their stock. I just had pretty good experience working with Gaudi2.

by ein0p 2y ago

mjevans 2y ago

Doesn't AMD have a similar issue, with the 'secret sauce' really that entire NV toolchain that is practically a monopoly?

gdiamos 2y ago
The problem is no differentiation If you have to do more work to port for no benefit, it doesn’t make sense except to commoditize it
- dilyevsky 2y ago
  Better price and better lead time, if achieved, seems like enough differentiation to me, at least to enter the market successfully

alecco 2y ago

Is it? Good luck loading large models over PCIe 4.0. Or running things over multiple cards. And it has older HBM2e which is ~2x slower than H100 HBM3. https://en.wikipedia.org/wiki/High_Bandwidth_Memory#HBM2E https://en.wikipedia.org/wiki/High_Bandwidth_Memory#HBM2E "N5, PCIe 4.0, and HBM2e. This chip was probably delayed two years." -wmf

wmf 2y ago
Gaudi 3 at least has PCIe 5.0 but for some reason they stuck with HBM2e.
ein0p 2y ago
Once loaded, weights just stay in HBM. And PCIe throughput is not at all an issue - the cards communicate over a large number of very high speed ROCE links both inside and outside the server. HBM throughput is not as much of an issue either, since during inference you are mostly just reading weights, not also writing out intermediate results and gradients. HBM2e is actually a smart decision on Intel's part IMO - it's not as supply constrained, so they can have more of it. Gaudi2 has 96GB of HBM2e. Gaudi3 has 144GB IIRC, as well as about 4x more compute and even faster interconnect. This means you can do longer context and/or larger models without splitting it across hosts where bandwidth is at somewhat of a premium. Though Gaudi3 is also less price competitive, I must add. No relation to Intel - I have never worked for them and hold no position in their stock. I just had pretty good experience working with Gaudi2.