3 ms·

The whole situation with Intel and AI is also baffling to me. They have an excellent product in this space - Gaudi2. Faster than A100 and very attractively pric

by ein0p 2y ago

The whole situation with Intel and AI is also baffling to me. They have an excellent product in this space - Gaudi2. Faster than A100 and very attractively priced. I’ve tried it, it works fine. Gaudi3 is also about to come out, and it’s twice as fast as Gaudi2. Yet nobody is buying. I get why you wouldn’t want this for training - it requires some minor code modifications and your Triton kernels are worthless. But for generative inference this is just what the doctor ordered.

soulbadguy 2y ago
> generative inference this is just what the doctor ordered. Naive question, wouldn't you need a descent tool chain for inference as well ?
- ein0p 2y ago
  Assuming you mean “decent” toolchain, it’s actually pretty decent. Could it use some polish? Yes. But any decent ML engineer would be able to get a high performance server (or a batch job) running in a relatively short time. Or in almost no time at all if using a lot of the FOSS models. You just basically create a model in PyTorch and then hand it over to Gaudi stuff which patches it with optimized, Gaudi specific ops and converts things into an inference graph. “Closeness to CUDA” is less important for inference because all the experimentation is already done by then, and if need be you could just implement the model using Gaudi ops to begin with, in a span of a few days including tuning and debugging.
alecco 2y ago
Gaudi was made by an Israeli company Intel acquired in 2019 (not an internal project). Gaudi 3 has PCIe 4.0 (vs. H100 PCIe 5.0, so 2x the bandwidth). It's strange for Intel, of all vendors, to lag behind in PCIe. And Nvidia has SXM for larger models (5x bandwidth). "N5, PCIe 4.0, and HBM2e. This chip was probably delayed two years." -wmf
- ein0p 2y ago
  PCIe doesn't matter - these accelerators talk to one another, and Gaudi is outstanding in this regard. HBM2e also doesn't matter if you run decent sized batches (which you should, for throughput). In fact, HBM2e, being far less supply constrained, might even be an advantage.
  - alecco 2y ago
    Besides that it does matter (e.g. loading/saving LLMs), it speaks how Gaudi is not even using Intel's latest technology. They are clearly not in the main production pipeline. So Gaudi is not going to save Intel.
    - ein0p 2y ago
      But you don’t “load” models during inference. They are already on-device.

by ein0p 2y ago

soulbadguy 2y ago

> generative inference this is just what the doctor ordered. Naive question, wouldn't you need a descent tool chain for inference as well ?

ein0p 2y ago
Assuming you mean “decent” toolchain, it’s actually pretty decent. Could it use some polish? Yes. But any decent ML engineer would be able to get a high performance server (or a batch job) running in a relatively short time. Or in almost no time at all if using a lot of the FOSS models. You just basically create a model in PyTorch and then hand it over to Gaudi stuff which patches it with optimized, Gaudi specific ops and converts things into an inference graph. “Closeness to CUDA” is less important for inference because all the experimentation is already done by then, and if need be you could just implement the model using Gaudi ops to begin with, in a span of a few days including tuning and debugging.

alecco 2y ago

Gaudi was made by an Israeli company Intel acquired in 2019 (not an internal project). Gaudi 3 has PCIe 4.0 (vs. H100 PCIe 5.0, so 2x the bandwidth). It's strange for Intel, of all vendors, to lag behind in PCIe. And Nvidia has SXM for larger models (5x bandwidth). "N5, PCIe 4.0, and HBM2e. This chip was probably delayed two years." -wmf

ein0p 2y ago
PCIe doesn't matter - these accelerators talk to one another, and Gaudi is outstanding in this regard. HBM2e also doesn't matter if you run decent sized batches (which you should, for throughput). In fact, HBM2e, being far less supply constrained, might even be an advantage.
- alecco 2y ago
  Besides that it does matter (e.g. loading/saving LLMs), it speaks how Gaudi is not even using Intel's latest technology. They are clearly not in the main production pipeline. So Gaudi is not going to save Intel.
  - ein0p 2y ago
    But you don’t “load” models during inference. They are already on-device.