5 ms·
Wait, PCIe bandwidth is higher than memory bandwidth now? That's bonkers, when did that happen? I haven't been keeping up. Just looked at the i9-14900k and I g
by modeless 1y ago
Wait, PCIe bandwidth is higher than memory bandwidth now? That's bonkers, when did that happen? I haven't been keeping up.
Just looked at the i9-14900k and I guess it's true, but only if you add all the PCIe lanes together. I'm sure there are other chips where it's even more true. Crazy!
- adgjlsfhk1 1y agoon server chips it's kind of ridiculous. 5th gen Epyc has 128 lanes of PCIEx5 for over 1TB/s of pcie bandwith (compared to 600GB/s RAM bandwidth from 12 channel ddr5 at 6400)
- andersa 1y agoYour math is a bit off. 128 lanes gen5 is 8 times x16, which has a combined theoretical bandwidth of 512GB/s, and more like 440GB/s in practice after protocol overhead. Unless we are considering both read and write bandwidth, but that seems strange to compare to memory read bandwidth.
- pclmulqdq 1y agoPeople like to add read and write bandwidth for some silly reason. Your units are off, too, though: gen 5 is 32 GT/s, meaning 64 GB/s (or 512 gigabits per second) each direction on an x16 link.
- andersa 1y agoI meant for all 128 lanes being used, not each x16. Then you get 512GB/s.
- wmf 1y agoPCIe is full duplex while DDR5 is half duplex so in theory PCIe is higher. It's rare to max out PCIe in both directions though.
- mrcode007 1y agohappens frequently in fact when training neural nets on modern hw
- DiabloD3 1y ago"No." DDR5-8000 is 64GB/s per channel. Desktop CPUs have two channels. PCI-E 5.0 in x16 is 64GB/s. Desktops have one x16.
- modeless 1y agoHmm, Intel specs the max memory bandwidth as 89.6 GB/s. DDR5-8000 would be out of spec. But I guess it's pretty common to run higher specced memory, while you can't overclock PCIe (AFAIK?). Even so I guess my math was wrong, it doesn't quite add up to more than memory bandwidth. But it's pretty darn close!
- DiabloD3 1y agoThere is a difference between recommended and max achievable. Zen 5 can hit that (and that's what I run), and Arrow Lake can also. The recommended from AMD on Zen 4 and 5 is 6000 (or 48x2), for Arrow Lake is 6400 (or 51.2x2); both of them continue increase in performance up to 8000, both of them have extreme trouble going past 8000 and getting a stable machine.
- pseudosavant 1y agoOne x16 slot. They'll use PCIe lanes in other slots (x4, x1, M2 SSDs) and also for devices off the chipset (network, USB, etc). The current top AMD/Intel CPUs can do ~100GB/sec over 28 lanes of mostly PCIe 5.
- immibis 1y agoBut my Threadripper has 4 channels of DDR5, and the equivalent of 4.25 x16 PCIe 5. You know what adds up to an even bigger number though? Using both.
- AnthonyMouse 1y ago> Wait, PCIe bandwidth is higher than memory bandwidth now? Hmm. Somebody make me a PCIe card with RDIMM slots on it.
- thulle 1y agohttps://www.servethehome.com/inventec-96-dimm-cxl-expansion-box-at-ocp-summit-2024-for-tbs-of-memory-astera-labs-intel-xeon-6/ https://www.servethehome.com/inventec-96-dimm-cxl-expansion-... https://www.servethehome.com/micron-cz120-cxl-memory-module-spotted-at-intel-vision-2024/ https://www.servethehome.com/micron-cz120-cxl-memory-module-...
- rwmj 1y agoThat's the promise (or requirement?) of CXL - have your memory managed centrally, servers access it over PCIe. https://en.wikipedia.org/wiki/Compute_Express_Link https://en.wikipedia.org/wiki/Compute_Express_Link I wonder how many are actually using CXL. I haven't heard of any customers deploying it so far.