4 ms·

Author (of the RAM article) here: Indeed, you can go further, but got to plan for other bandwidth needed by other peripherals, data movement and inter-CPU band

Author (of the RAM article) here: Indeed, you can go further, but got to plan for other bandwidth needed by other peripherals, data movement and inter-CPU bandwidth (NUMA) and intra-CPU-core bandwidth limitations too (AMD's infinity fabric is point to point between chiplets, but intel has some ring-bus architecture for moving bits between CPU cores). I got my Lenovo ThinkStation P620 workstation (with AMD Zen-2 ThreadRipper Pro WX, 8-memory channels like EPYC) to scan 10 x PCIe 4.0 SSDs at 66 GB/s (I had to move SSD cards around so they'd use separate PCIe root complexes to avoid a PCIe <-> CPU data transfer bottleneck. And even with doing I/O through 3 PCIe root complexes (out of 4 connected to that CPU), I seem to be hitting some inter-CPU-core bandwidth limitation. The throughput differs depending on which specific CPU cores happen to run the processes doing I/Os against different SSDs. Planning to publish some blog entries about these I/O tests but a teaser tweet is here (11M IOPS with a single-socket ThreadRipper workstation - it's not even a NUMA server! :-) https://twitter.com/TanelPoder/status/1352329243070504964 https://twitter.com/TanelPoder/status/1352329243070504964

MayeulC 6y ago
That bandwidth limitation could be due to Infinity Fabric, which seems to be rated at 42GBps (x2 as full duplex, though)? https://en.wikichip.org/wiki/amd/infinity_fabric https://en.wikichip.org/wiki/amd/infinity_fabric
- tanelpoder 6y ago
  Yes, that's what I'm suspecting too, although with higher clocked RAM, I should have somewhat more bandwidth. My DIMMs are 3200 MT, so should be running at 1600 MHz. But I saw a note (not sure where) that Infinity Fabric can run up to 2933 MT on my machine and it would run in sync with memory with DIMMs only up to 2933 MT. Unfortunately my BIOS doesn't allow to downgrade the RAM "clock" from 3200 MT to 2933, thus Ininity Fabric is running "out of sync" with my RAM. This should mean non-ideal memory access latency at least, not sure how it affects throughput of large sequential transfers. I'm planning to come up with some additional tests and hopefully write up a "part 2" too.
  - 1996 6y ago
    > Unfortunately my BIOS doesn't allow to downgrade the RAM "clock" How deep are you willing to go? The RAM clock is controlled by the memory training algorithms. They use data from the XMP, which can be edited. The simplest is to reflash your memory sticks to alter their XMP, so the training algorithm will reach the conclusions you want. There's some Windows software to do that. You could also implement your own MRC, something done by coreboot and the likes.
    - tanelpoder 6y ago
      Ha, thanks for the idea! I was briefly thinking of buying 2933 "MHz" RAM for the test (as I later would put it into my other workstation that can go up to 2600 "MHz" only), but then I realized I don't have time for this right now (will do my throughput, performance stability tests first and maybe look into getting the most out of the latency later).
  - deleted 6y ago
    [deleted]
1996 6y ago
> I had to move SSD cards around so they'd use separate PCIe root complexes to avoid a PCIe <-> CPU data transfer bottleneck I am doing similar things. Have you considered looking at how to control by software the PCI lanes assignment? Intel HSIO seems to be software configurable - except that usually, it's all done just by the bios. But as PCI specs allow for both device-side and host-side negotiations, it should be doable without "moving SSDs around" > The throughput differs depending on which specific CPU cores happen to run the processes doing I/Os against different SSDs. That strikes me as odd. I would check the detail of the PCI lanes and their routing. You could have something funky going on. My first guess would be that it's slow on one core because it's also handling something else, by design or by accident. There're some bad hardware designs out there. But thanks to stuff like HSIO, it should now be possible to fix the worst ones by software (how else would the bios do it otherwise!) just like in the old days of isapnptools!
- tanelpoder 6y ago
  As this is an AMD machine - and as it's a workstation, not server, perhaps this is why they've restricted it in BIOS. I'm not too much of an expert in PCI express - but if this workstation has 4 PCIe root complexes/host bridges, each capable of x32 PCIe 4.0 lanes - and there are no multi-root PCIe switches, wouldn't a lane physically have to communicate with just one PCIe root complex/CPU "port"?
  - 1996 6y ago
    In theory yes. In practice many things can be different. Carefully check what's happening under the hood. Hopefully, manual configuration of the PCIe will become more commonplace, as most bioses are badly broken each in their own unique way - and unfixable.
fomine3 6y ago
Changing NUMA per socket(NPS) config would be interesting. EPYC Rome (or equivalent TR) was advertised as unified compared to Naples, but actually it's a bit NUMA / NUPA? (Peripheral Access, anyone knows the correct word?) so it has "Quadrant".

MayeulC 6y ago

That bandwidth limitation could be due to Infinity Fabric, which seems to be rated at 42GBps (x2 as full duplex, though)? https://en.wikichip.org/wiki/amd/infinity_fabric https://en.wikichip.org/wiki/amd/infinity_fabric

tanelpoder 6y ago
Yes, that's what I'm suspecting too, although with higher clocked RAM, I should have somewhat more bandwidth. My DIMMs are 3200 MT, so should be running at 1600 MHz. But I saw a note (not sure where) that Infinity Fabric can run up to 2933 MT on my machine and it would run in sync with memory with DIMMs only up to 2933 MT. Unfortunately my BIOS doesn't allow to downgrade the RAM "clock" from 3200 MT to 2933, thus Ininity Fabric is running "out of sync" with my RAM. This should mean non-ideal memory access latency at least, not sure how it affects throughput of large sequential transfers. I'm planning to come up with some additional tests and hopefully write up a "part 2" too.
- 1996 6y ago
  > Unfortunately my BIOS doesn't allow to downgrade the RAM "clock" How deep are you willing to go? The RAM clock is controlled by the memory training algorithms. They use data from the XMP, which can be edited. The simplest is to reflash your memory sticks to alter their XMP, so the training algorithm will reach the conclusions you want. There's some Windows software to do that. You could also implement your own MRC, something done by coreboot and the likes.
  - tanelpoder 6y ago
    Ha, thanks for the idea! I was briefly thinking of buying 2933 "MHz" RAM for the test (as I later would put it into my other workstation that can go up to 2600 "MHz" only), but then I realized I don't have time for this right now (will do my throughput, performance stability tests first and maybe look into getting the most out of the latency later).
- deleted 6y ago
  [deleted]

1996 6y ago

> I had to move SSD cards around so they'd use separate PCIe root complexes to avoid a PCIe <-> CPU data transfer bottleneck I am doing similar things. Have you considered looking at how to control by software the PCI lanes assignment? Intel HSIO seems to be software configurable - except that usually, it's all done just by the bios. But as PCI specs allow for both device-side and host-side negotiations, it should be doable without "moving SSDs around" > The throughput differs depending on which specific CPU cores happen to run the processes doing I/Os against different SSDs. That strikes me as odd. I would check the detail of the PCI lanes and their routing. You could have something funky going on. My first guess would be that it's slow on one core because it's also handling something else, by design or by accident. There're some bad hardware designs out there. But thanks to stuff like HSIO, it should now be possible to fix the worst ones by software (how else would the bios do it otherwise!) just like in the old days of isapnptools!

tanelpoder 6y ago
As this is an AMD machine - and as it's a workstation, not server, perhaps this is why they've restricted it in BIOS. I'm not too much of an expert in PCI express - but if this workstation has 4 PCIe root complexes/host bridges, each capable of x32 PCIe 4.0 lanes - and there are no multi-root PCIe switches, wouldn't a lane physically have to communicate with just one PCIe root complex/CPU "port"?
- 1996 6y ago
  In theory yes. In practice many things can be different. Carefully check what's happening under the hood. Hopefully, manual configuration of the PCIe will become more commonplace, as most bioses are badly broken each in their own unique way - and unfixable.

fomine3 6y ago

Changing NUMA per socket(NPS) config would be interesting. EPYC Rome (or equivalent TR) was advertised as unified compared to Naples, but actually it's a bit NUMA / NUPA? (Peripheral Access, anyone knows the correct word?) so it has "Quadrant".