6 ms·

The Changing Relationship Between Memory, I/O, and Network Bandwidth

22 points by sorenbs 3y ago

gautamcgoel 3y ago
This is interesting, but I think there are a lot of caveats. 800G Ethernet NICs are very expensive, and PCI 5.0 SSDs are much more pricey than PCI 4.0 SSDs. Also they used DDR5 4800 in their comparison instead of DDR5 6400. All of this is just to say than on a "typical" server/workstation, the relative speeds of Ethernet, RAM, and dusk might look very different than what the authors suggest.
- CHY872 3y ago
  The other element which appears unsaid is that in a typical datacentre, your bisection bandwidth is typically << (num computers * network bandwidth per computer). Or in other words, even if your computation is bandwidth and not latency starved, you're not obviously going to be able to do a gigantic data shuffle quickly unless it's within a rack. This is to say, I'm not sure _just how much_ this would affect your overall target system architecture at present. Once you're switching at a couple of terabits things are likely quite different in those terms. The other element which is a bit scary is that it's fairly rare these days for mass market companies to index deeply on tech which isn't available in public clouds, so until AWS supports this sort of thing, it's unlikely that many folks will target it. You can kind of see this with the Optane PDIMMs - they looked absolutely fantastic, but given you couldn't get them on any AWS instance there wasn't much point actually trying to use them outside of very specific applications - as a software engineer this hardware lets me build my software very differently and in a simpler way, but how can I possibly risk architecting based on that if it then cannot support a customer's cloud migration? Obviously v different in HPC contexts. And it should always be said - latency is very different between these. Memory latency still measured in nanoseconds, PCIe latency still measured in 10s of microseconds, about 3 orders of magnitude difference.
  - touisteur 3y ago
    I can see many reasons why Optane PDIMMs didn't take off, other than not being cloud-available (which Intel might have subsidized with the help of a big software player and a series of killer apps). Price was excruciating, number of write cycles (so lifetime of machine, architecture) not very clear and upgrade path / exit hatch was also not clear. Also the programming model wasn't very clear and not many big important applications had migrated to use them efficiently or in an interesting manner (we were still in the 'crazy interesting papers' cycle). It still grates me that the only interface we ended up with for high speed durable data is nvme through pcie and regret the lost promise of fast byte-addressable persistent memory, but once again, worse is better seems to have won?
    - wtallis 3y ago
      > number of write cycles (so lifetime of machine, architecture) not very clear I thought Intel was pretty clear by saying they warranted the modules for five years of continuous operation regardless of workload.
    - CHY872 3y ago
      right - why won't 'networking w/ 800GbE' be different (in terms of weird programming model, crazy interesting papers, etc)?
      - touisteur 3y ago
        Very good question. Being up to my eyeballs on 400G, dpdk, gpudirect, spdk right now I see a way forward, but also I don't see how we 'progress' more without them, where's the 'other' path? Integrated chips and like Apple's m1/2 or NVIDIA's GH and AMD's MI300X?
- wtallis 3y ago
  > Also they used DDR5 4800 in their comparison instead of DDR5 6400. Are there server processors on the market yet that support DDR5-6400? Intel Sapphire Rapids and AMD Genoa only support DDR5-4800. Faster DIMMs are available (especially in the consumer market segments) but it seems quite reasonable to base this kind of analysis on what's actually officially supported and widely available.
vlovich123 3y ago
I was running memtest recently on my brand new 13900 with DDR5-6000 and the stock performance is 20 GB/s. Overclocking from motherboard stock setting of 4ghz to 6ghz still only meets me 25 GB/s. Is it the 38 GB/s for DDR5-4800 in the article hypothetical, my motherboard / CPU is a bottleneck somehow, or that memtest86 bundled with my motherboard isn’t capable of calculating bandwidth correctly for my system?
- effie 3y ago
  Check that you have installed one module, or two modules, one per each memory channel. If two modules are installed in wrong slots, they can use the same channel and this can result in a slowdown. One a 13900 core, with default clock, should be able to read at max. rate 89.2/2 = 44.8 GB/s (small gigabytes = 10^9). One DDR5 module running at 6000 MT/s has maximum throughput 48 GB/s. Granting some unknown slowdowns due to error corrections and other things I would be expecting to see at least above 40 GB/s. With two RAM modules and two or more cores working, one should be able to get above 80GB/s. Perhaps memtest does something that limits its memory performance (maybe it uses old less-efficient instructions for reading memory instead of SSE/AVX) or maybe it makes a factor 2 error in calculation of the speed.
  - vlovich123 3y ago
    I have two slots installed correctly. The bandwidth tool in AUR reports ~20-35 GB/s for sequential reads (although weird spikes of 70 gb/s) i.e. which is consistent with memtest86's ~25 GB/s (the discrepancy can be explained away by memtest probably not using AVX instructions). Writes seem a lot better at ~58 GB/s bypassing cache. That's still a large discrepancy though where I'm typically 30% away from the nominal speed.
    - effie 3y ago
      That's really weird, the author of the bandwidth tool has line graphs with >100GB/s on weaker hardware. https://zsmith.co/bandwidth.php https://zsmith.co/bandwidth.php Make sure all power saving stuff is disabled in BIOS, that XMP is being used, and Linux is not booting with some weird ACPI/APIC flags. Try ganged/unganged mode (DCT) in BIOS.

22 points by sorenbs 3y ago

This is interesting, but I think there are a lot of caveats. 800G Ethernet NICs are very expensive, and PCI 5.0 SSDs are much more pricey than PCI 4.0 SSDs. Also they used DDR5 4800 in their comparison instead of DDR5 6400. All of this is just to say than on a "typical" server/workstation, the relative speeds of Ethernet, RAM, and dusk might look very different than what the authors suggest.

CHY872 3y ago
The other element which appears unsaid is that in a typical datacentre, your bisection bandwidth is typically << (num computers * network bandwidth per computer). Or in other words, even if your computation is bandwidth and not latency starved, you're not obviously going to be able to do a gigantic data shuffle quickly unless it's within a rack. This is to say, I'm not sure _just how much_ this would affect your overall target system architecture at present. Once you're switching at a couple of terabits things are likely quite different in those terms. The other element which is a bit scary is that it's fairly rare these days for mass market companies to index deeply on tech which isn't available in public clouds, so until AWS supports this sort of thing, it's unlikely that many folks will target it. You can kind of see this with the Optane PDIMMs - they looked absolutely fantastic, but given you couldn't get them on any AWS instance there wasn't much point actually trying to use them outside of very specific applications - as a software engineer this hardware lets me build my software very differently and in a simpler way, but how can I possibly risk architecting based on that if it then cannot support a customer's cloud migration? Obviously v different in HPC contexts. And it should always be said - latency is very different between these. Memory latency still measured in nanoseconds, PCIe latency still measured in 10s of microseconds, about 3 orders of magnitude difference.
- touisteur 3y ago
  I can see many reasons why Optane PDIMMs didn't take off, other than not being cloud-available (which Intel might have subsidized with the help of a big software player and a series of killer apps). Price was excruciating, number of write cycles (so lifetime of machine, architecture) not very clear and upgrade path / exit hatch was also not clear. Also the programming model wasn't very clear and not many big important applications had migrated to use them efficiently or in an interesting manner (we were still in the 'crazy interesting papers' cycle). It still grates me that the only interface we ended up with for high speed durable data is nvme through pcie and regret the lost promise of fast byte-addressable persistent memory, but once again, worse is better seems to have won?
  - wtallis 3y ago
    > number of write cycles (so lifetime of machine, architecture) not very clear I thought Intel was pretty clear by saying they warranted the modules for five years of continuous operation regardless of workload.
  - CHY872 3y ago
    right - why won't 'networking w/ 800GbE' be different (in terms of weird programming model, crazy interesting papers, etc)?
    - touisteur 3y ago
      Very good question. Being up to my eyeballs on 400G, dpdk, gpudirect, spdk right now I see a way forward, but also I don't see how we 'progress' more without them, where's the 'other' path? Integrated chips and like Apple's m1/2 or NVIDIA's GH and AMD's MI300X?
wtallis 3y ago
> Also they used DDR5 4800 in their comparison instead of DDR5 6400. Are there server processors on the market yet that support DDR5-6400? Intel Sapphire Rapids and AMD Genoa only support DDR5-4800. Faster DIMMs are available (especially in the consumer market segments) but it seems quite reasonable to base this kind of analysis on what's actually officially supported and widely available.

vlovich123 3y ago

I was running memtest recently on my brand new 13900 with DDR5-6000 and the stock performance is 20 GB/s. Overclocking from motherboard stock setting of 4ghz to 6ghz still only meets me 25 GB/s. Is it the 38 GB/s for DDR5-4800 in the article hypothetical, my motherboard / CPU is a bottleneck somehow, or that memtest86 bundled with my motherboard isn’t capable of calculating bandwidth correctly for my system?

effie 3y ago
Check that you have installed one module, or two modules, one per each memory channel. If two modules are installed in wrong slots, they can use the same channel and this can result in a slowdown. One a 13900 core, with default clock, should be able to read at max. rate 89.2/2 = 44.8 GB/s (small gigabytes = 10^9). One DDR5 module running at 6000 MT/s has maximum throughput 48 GB/s. Granting some unknown slowdowns due to error corrections and other things I would be expecting to see at least above 40 GB/s. With two RAM modules and two or more cores working, one should be able to get above 80GB/s. Perhaps memtest does something that limits its memory performance (maybe it uses old less-efficient instructions for reading memory instead of SSE/AVX) or maybe it makes a factor 2 error in calculation of the speed.
- vlovich123 3y ago
  I have two slots installed correctly. The bandwidth tool in AUR reports ~20-35 GB/s for sequential reads (although weird spikes of 70 gb/s) i.e. which is consistent with memtest86's ~25 GB/s (the discrepancy can be explained away by memtest probably not using AVX instructions). Writes seem a lot better at ~58 GB/s bypassing cache. That's still a large discrepancy though where I'm typically 30% away from the nominal speed.
  - effie 3y ago
    That's really weird, the author of the bandwidth tool has line graphs with >100GB/s on weaker hardware. https://zsmith.co/bandwidth.php https://zsmith.co/bandwidth.php Make sure all power saving stuff is disabled in BIOS, that XMP is being used, and Linux is not booting with some weird ACPI/APIC flags. Try ganged/unganged mode (DCT) in BIOS.