4 ms·

Interestingly the "4th generation Intel Xeon Scalable processors" themselves have up to 2.45 TBps in memory bandwidth, with the 8-socket configuration, or 2 TBp

by zrm 3y ago

Interestingly the "4th generation Intel Xeon Scalable processors" themselves have up to 2.45 TBps in memory bandwidth, with the 8-socket configuration, or 2 TBps with 2-socket Xeon Max and HBM. If they'd make an 8-socket Xeon Max it would have 8 TBps. Considering that the Xeon Max 9462 is $8000 vs. the H100 going for north of $40,000, that could be interesting.

jocaal 3y ago
The throughput these gpu's have make the price pretty competitive, but I think AMD is working on a APU in their instinct lineup. That could be pretty competitive since Nvidia is overcharging for memory and you could just use sticks instead
- smoldesu 3y ago
  Nvidia is also working on a tightly integrated datacenter solution, FWIW: https://www.nvidia.com/en-us/data-center/grace-cpu/ https://www.nvidia.com/en-us/data-center/grace-cpu/
- zrm 3y ago
  A lot of this is workload-dependent. LLMs for example seem to be memory-bound, so a fast CPU with HBM or a large number of memory channels should do well. Socket SP5 has 12 channels, which is 461 GBps per socket at DDR5-4800. Intel is getting 1 TBps from HBM, but then you're paying for HBM. $8000 for the cheapest Xeon Max vs. $3000 for the Epyc 9334 with the same number of cores or ~$1000 for the least expensive thing that will fit in the 12-channel socket. CPUs also have a cost advantage because then you don't need a CPU and a GPU. Other things might be more compute bound. Then a fast GPU in a socket with a lot of memory channels worth of cheap sticks should be fun.
- aseipp 3y ago
  Only if you're purely 100% compute bound by a wide margin versus the size of your working set. But in that scenario, you can just widen the memory interface, lower the clocks speeds, and you'll normally still come out ahead in efficiency. Most datacenter parts are going to prefer such a route. The physical integrity needed for extremely high bandwidth interfaces is just really tough to achieve on a DIMM-like slot without really advanced high-channel socket topologies. Those numbers listed before aren't for nothing; 2.4TBps bandwith for an 8-socket Xeon vs 2.0Tbps with a 2-socket Xeon using HBM2 is a very significant improvement in overall efficiency.

by zrm 3y ago

jocaal 3y ago

The throughput these gpu's have make the price pretty competitive, but I think AMD is working on a APU in their instinct lineup. That could be pretty competitive since Nvidia is overcharging for memory and you could just use sticks instead

smoldesu 3y ago
Nvidia is also working on a tightly integrated datacenter solution, FWIW: https://www.nvidia.com/en-us/data-center/grace-cpu/ https://www.nvidia.com/en-us/data-center/grace-cpu/
zrm 3y ago
A lot of this is workload-dependent. LLMs for example seem to be memory-bound, so a fast CPU with HBM or a large number of memory channels should do well. Socket SP5 has 12 channels, which is 461 GBps per socket at DDR5-4800. Intel is getting 1 TBps from HBM, but then you're paying for HBM. $8000 for the cheapest Xeon Max vs. $3000 for the Epyc 9334 with the same number of cores or ~$1000 for the least expensive thing that will fit in the 12-channel socket. CPUs also have a cost advantage because then you don't need a CPU and a GPU. Other things might be more compute bound. Then a fast GPU in a socket with a lot of memory channels worth of cheap sticks should be fun.
aseipp 3y ago
Only if you're purely 100% compute bound by a wide margin versus the size of your working set. But in that scenario, you can just widen the memory interface, lower the clocks speeds, and you'll normally still come out ahead in efficiency. Most datacenter parts are going to prefer such a route. The physical integrity needed for extremely high bandwidth interfaces is just really tough to achieve on a DIMM-like slot without really advanced high-channel socket topologies. Those numbers listed before aren't for nothing; 2.4TBps bandwith for an 8-socket Xeon vs 2.0Tbps with a 2-socket Xeon using HBM2 is a very significant improvement in overall efficiency.