3 ms·
PCIe doesn't matter - these accelerators talk to one another, and Gaudi is outstanding in this regard. HBM2e also doesn't matter if you run decent sized batches
by ein0p 2y ago
PCIe doesn't matter - these accelerators talk to one another, and Gaudi is outstanding in this regard. HBM2e also doesn't matter if you run decent sized batches (which you should, for throughput). In fact, HBM2e, being far less supply constrained, might even be an advantage.
- alecco 2y agoBesides that it does matter (e.g. loading/saving LLMs), it speaks how Gaudi is not even using Intel's latest technology. They are clearly not in the main production pipeline. So Gaudi is not going to save Intel.
- ein0p 2y agoBut you don’t “load” models during inference. They are already on-device.