3 ms·

I think they’re playing it safe with their points of comparison. DDR5 supports 64,000 MB/s per channel and 4x NVMe PCIe5 SSDs support 10,000 MB/s. Depending on

I think they’re playing it safe with their points of comparison. DDR5 supports 64,000 MB/s per channel and 4x NVMe PCIe5 SSDs support 10,000 MB/s. Depending on how many memory channels and what RAID you use, I think 10x improvement over high performance storage is unimpeachable. Memory latency being better than SSD latency will really benefit memory depending on workload, but I don’t think you can just throw one number out there to represent that. Now if you’re comparing to spinning rust, memory is definitely going to blow it away, but commodity hardware isn’t running tens or hundreds of TBs of memory. Memory to SSD comparisons seem right.

convolvatron 3y ago
its also may be limited on iops/s. you _can_ really just work with latency, but you need to consider the pipeline depth.
semi-extrinsic 3y ago
For example the latest AMD Genoa has 12 memory channels per socket, at dual socket and with enough DIMMs that's a 75x speed advantage even if you compare with RAID0 of high perfomance NVMe.
- wtallis 3y ago
  > even if you compare with RAID0 of high perfomance NVMe. Do you mean a RAID0 of just two or four NVMe SSDs? It's absolutely ridiculous to count aggregate DRAM bandwidth across two CPU sockets and not do the same for PCIe lanes. A fair comparison is that Genoa has about twice the DRAM bandwidth as it has PCIe bandwidth, though in a fully-loaded database server some portion of the PCIe bandwidth will be used for networking rather than storage.
  - zamadatix 3y ago
    FWIW I think you have the aggregation backwards, dual socket Genoa would have 24 channels of DRAM but sacrifice some of the PCIe lanes for the interconnect. Your numbers work out right though as single socket is actually about 1:1 in RAM vs PCIe bandwidth so dual socket would still come out roughly 2:1. I think 10x is a fair rough number though, depending on your access pattern.
    - wtallis 3y ago
      Dual-socket Genoa would be 24 channels at DDR5-4800 (38.4 GB/s) for a total of ~921.6 GB/s. Typical PCIe configurations are 64 or 80 lanes per socket for dual-socket, so 128 or 160 lanes total, at PCIe 5.0 speed that's ~504 GB/s for 128 lanes total or ~630 GB/s for 160 lane configurations. Single-socket Genoa would be 12 channels of DRAM (~460.8 GB/s) and 128 lanes of PCIe 5.0 (~504 GB/s), but none of the previous comments were specifically about single-socket Genoa and I wasn't going to silently switch from considering dual-socket in one sentence to single-socket in the next sentence.
      - zamadatix 3y ago
        Ah yes, I forgot about the 48 lane 2 socket interconnect mode which does allow you to still aggregate some additional lanes.

convolvatron 3y ago

its also may be limited on iops/s. you _can_ really just work with latency, but you need to consider the pipeline depth.

semi-extrinsic 3y ago

For example the latest AMD Genoa has 12 memory channels per socket, at dual socket and with enough DIMMs that's a 75x speed advantage even if you compare with RAID0 of high perfomance NVMe.

wtallis 3y ago
> even if you compare with RAID0 of high perfomance NVMe. Do you mean a RAID0 of just two or four NVMe SSDs? It's absolutely ridiculous to count aggregate DRAM bandwidth across two CPU sockets and not do the same for PCIe lanes. A fair comparison is that Genoa has about twice the DRAM bandwidth as it has PCIe bandwidth, though in a fully-loaded database server some portion of the PCIe bandwidth will be used for networking rather than storage.
- zamadatix 3y ago
  FWIW I think you have the aggregation backwards, dual socket Genoa would have 24 channels of DRAM but sacrifice some of the PCIe lanes for the interconnect. Your numbers work out right though as single socket is actually about 1:1 in RAM vs PCIe bandwidth so dual socket would still come out roughly 2:1. I think 10x is a fair rough number though, depending on your access pattern.
  - wtallis 3y ago
    Dual-socket Genoa would be 24 channels at DDR5-4800 (38.4 GB/s) for a total of ~921.6 GB/s. Typical PCIe configurations are 64 or 80 lanes per socket for dual-socket, so 128 or 160 lanes total, at PCIe 5.0 speed that's ~504 GB/s for 128 lanes total or ~630 GB/s for 160 lane configurations. Single-socket Genoa would be 12 channels of DRAM (~460.8 GB/s) and 128 lanes of PCIe 5.0 (~504 GB/s), but none of the previous comments were specifically about single-socket Genoa and I wasn't going to silently switch from considering dual-socket in one sentence to single-socket in the next sentence.
    - zamadatix 3y ago
      Ah yes, I forgot about the 48 lane 2 socket interconnect mode which does allow you to still aggregate some additional lanes.