Once loaded, weights just stay in HBM. And PCIe throughput is not at all an issue - the cards communicate over a large number of very high speed ROCE links both

Once loaded, weights just stay in HBM. And PCIe throughput is not at all an issue - the cards communicate over a large number of very high speed ROCE links both inside and outside the server. HBM throughput is not as much of an issue either, since during inference you are mostly just reading weights, not also writing out intermediate results and gradients. HBM2e is actually a smart decision on Intel's part IMO - it's not as supply constrained, so they can have more of it. Gaudi2 has 96GB of HBM2e. Gaudi3 has 144GB IIRC, as well as about 4x more compute and even faster interconnect. This means you can do longer context and/or larger models without splitting it across hosts where bandwidth is at somewhat of a premium. Though Gaudi3 is also less price competitive, I must add. No relation to Intel - I have never worked for them and hold no position in their stock. I just had pretty good experience working with Gaudi2.