3 ms·
meanwhile, disk is potentially getting to be as fast as ram, throughput wise. 128 lanes of pcie 4.0 is 256GBps iirc. epyc's 8 channel ddr4-3200 otoh is good fo
by rektide 6y ago
meanwhile, disk is potentially getting to be as fast as ram, throughput wise.
128 lanes of pcie 4.0 is 256GBps iirc. epyc's 8 channel ddr4-3200 otoh is good for 208GBps.
Let's Encrypt stopped a little short, using 24x nvme disks (it fits in a 2U though so that's nice)[1]. that could be up to 96 of 128 pcie links in use. with the right ssds, working on large data-objects, that'd be somewhere a bit under 192GBps versus the max 208GBps of their ram.
in truth, ram's random access capabilities are far better, there's much less overhead (although nvme is pretty good). and i'm not sure i've ever seen anyone try to confirm that those 128 lanes of pcie on epyc aren't oversubscribed, that devices really can push that much data around. note that this doesn't necessarily even have to mean using the cpu; pci p2p is where it's at for in-the-know folks doing nvme, network, and gpu data-pushing; epyc's io-die is acting like a data-packet switch in these conditions, rather than having the cpu process/crunch these peripheral's data.
[1] https://news.ycombinator.com/item?id=25861422 https://news.ycombinator.com/item?id=25861422
- tanelpoder 6y agoAuthor (of the RAM article) here: Indeed, you can go further, but got to plan for other bandwidth needed by other peripherals, data movement and inter-CPU bandwidth (NUMA) and intra-CPU-core bandwidth limitations too (AMD's infinity fabric is point to point between chiplets, but intel has some ring-bus architecture for moving bits between CPU cores). I got my Lenovo ThinkStation P620 workstation (with AMD Zen-2 ThreadRipper Pro WX, 8-memory channels like EPYC) to scan 10 x PCIe 4.0 SSDs at 66 GB/s (I had to move SSD cards around so they'd use separate PCIe root complexes to avoid a PCIe <-> CPU data transfer bottleneck. And even with doing I/O through 3 PCIe root complexes (out of 4 connected to that CPU), I seem to be hitting some inter-CPU-core bandwidth limitation. The throughput differs depending on which specific CPU cores happen to run the processes doing I/Os against different SSDs. Planning to publish some blog entries about these I/O tests but a teaser tweet is here (11M IOPS with a single-socket ThreadRipper workstation - it's not even a NUMA server! :-) https://twitter.com/TanelPoder/status/1352329243070504964 https://twitter.com/TanelPoder/status/1352329243070504964
- MayeulC 6y agoThat bandwidth limitation could be due to Infinity Fabric, which seems to be rated at 42GBps (x2 as full duplex, though)? https://en.wikichip.org/wiki/amd/infinity_fabric https://en.wikichip.org/wiki/amd/infinity_fabric
- tanelpoder 6y agoYes, that's what I'm suspecting too, although with higher clocked RAM, I should have somewhat more bandwidth. My DIMMs are 3200 MT, so should be running at 1600 MHz. But I saw a note (not sure where) that Infinity Fabric can run up to 2933 MT on my machine and it would run in sync with memory with DIMMs only up to 2933 MT. Unfortunately my BIOS doesn't allow to downgrade the RAM "clock" from 3200 MT to 2933, thus Ininity Fabric is running "out of sync" with my RAM. This should mean non-ideal memory access latency at least, not sure how it affects throughput of large sequential transfers. I'm planning to come up with some additional tests and hopefully write up a "part 2" too.
- 1996 6y ago> Unfortunately my BIOS doesn't allow to downgrade the RAM "clock" How deep are you willing to go? The RAM clock is controlled by the memory training algorithms. They use data from the XMP, which can be edited. The simplest is to reflash your memory sticks to alter their XMP, so the training algorithm will reach the conclusions you want. There's some Windows software to do that. You could also implement your own MRC, something done by coreboot and the likes.
- tanelpoder 6y agoHa, thanks for the idea! I was briefly thinking of buying 2933 "MHz" RAM for the test (as I later would put it into my other workstation that can go up to 2600 "MHz" only), but then I realized I don't have time for this right now (will do my throughput, performance stability tests first and maybe look into getting the most out of the latency later).
- deleted 6y ago