4 ms·

On a second thought, are you sure Infinity Fabric would be that slow? I would expect it to be compared to L3 bandwidth not RAM. Plus, the article you gave sugge

On a second thought, are you sure Infinity Fabric would be that slow? I would expect it to be compared to L3 bandwidth not RAM. Plus, the article you gave suggests 40 GB/s P2P IFOP with slower 2.6 GHz RAM. I can't find the right diagram , but as I remember, this CPU had 8 chiplets, still allowing for much more bandwidth, depending on the topology.

dragontamer 5y ago
> I would expect it to be compared to L3 bandwidth not RAM. Infinity Fabric is well known to be slower / lower-bandwidth than DRAM actually. Just one of the peculiarities of this architecture. ------ That's why understanding MESI is important. If Core#10 writes something to DRAM: then Core#45 will not be able to write to it until Core#10 releases it (aka: reaches the "invalid" state of MESI). In contrast, writing to DRAM that's "not owned" by anybody is one-and-done. No need to communicate to other cores or chiplets. ------ No _singular_ core can reach 204GB/s spec of the architecture. You only reach that 204GB/s spec if all the cores (or chiplets really) are visiting different sections of RAM independently. This happens in practice often: the whole EPYC / Threadripper design is really aimed for virtual machines doing independent tasks. If you're trying to write high-performance code on EPYC / Threadripper, its important to keep this model in your brain. ------- Once you fix this problem, your next problem is probably going to be the page-table / virtual memory system. Maybe you should be looking up huge-pages in preparation?
- ashvardanian 5y ago
  Somehow, similarly bad numbers were achieved with 56 cores working with disjoint parts of 512 GB array. I didnt pin RAM to cores, though. Maybe thats the missing piece, but it’s sad if thats the only way to meet the spec. Almost no application can do that in modern cloud deployment realities.
  - lostmsu 5y ago
    On the contrary. Having multiple independent services running on a single physical machine will likely give you full performance. The easiest way is probably just splitting the physical machine into the number of VMs/pods equal or larger than CCX count.
  - dragontamer 5y ago
    https://people.freebsd.org/~gallatin/talks/euro2019.pdf https://people.freebsd.org/~gallatin/talks/euro2019.pdf Here's a talk about a lot of the NUMA-issues brought up by Netflix + FreeBSD. Its 200 Gb/s (bits), not bytes, but I'd expect you to be running into a lot of the similar problems. Its 200 Gb/s because they're reading from SSDs. But still, careful attention to NUMA was the only way they could reach full PCIe-bandwidths, which is a similar issue to yours (where NUMA is needed to reach full Memory/DRAM bandwidth). -------- Did you set your BIOS into NUMA mode? If you're in default BIOS-settings, your RAM is being split up between all 4 nodes (Uniform-memory-access mode). At least, that's how it was for my Threadripper 1950x (I know you've got a totally different model, but who knows?) I forgot exactly the name of the BIOS setting. But you gotta boot your computer up with 4-nodes-per-socket (or was it 8-nodes per socket for your computer?) or some similar setting. There were some programs on Linux (and Windows) that help you see what the OS is seeing.
    - kristjansson 5y ago
      See also the a more recent talk[0] on progress made on the same topic. On EPYC too, so perhaps more applicable to OP? [0]: https://people.freebsd.org/~gallatin/talks/euro2021.pdf https://people.freebsd.org/~gallatin/talks/euro2021.pdf
      - dragontamer 5y ago
        The original talk was on 2019 era hardware, for a 2019-era talk. Xeon and EPYC are both discussed, albeit somewhat dated now but still relevant IMO. The newer talk is 2021 hardware. IMO, the newer talk seems to focus more on the specific details of kTLS and PCIe TLS... but there's still stuff in there about NUMA for sure. But yeah, the 2021 talk definitely is on more recent hardware: 2021 Xeons and EPYCs.
        kristjansson 5y ago
        Ah, dang, that's on me, I forgot the earlier talk discussed EPYC chips too.
    - 5y ago
  - dragontamer 5y ago
    > Somehow, similarly bad numbers were achieved with 56 cores working with disjoint parts of 512 GB array Oh, that's not NUMA at all, now that I'm more carefully reading your post. NUMA would involve a "copy" step, ensuring that those elements are in NUMA-local memory before reading. Much like how in GPU-programming, you have to worry about the physicality of memory, NUMA-aware programming you have to memcpy data to the right location before it achieves high speeds. Each of the 56-cores needs its ~10GBs in "NUMA-local" memory _BEFORE_ you start the benchmark. Yeah, I realize this isn't practical. But... who ever said that NUMA use cases are practical? Lol. A lot of cases, it makes more sense to just take advantage of infinity fabric for simplicity (although its slower, its definitely more convenient).
  - toast0 5y ago
    NUMA is part of the reality. There's lots of modern computing that doesn't acknowledge reality, but that doesn't tend to do good things for performance. Communication isn't cheap, and implicit communication makes it hard to tell what's expensive. Reading from a pointer could be from L1 cache, or it could be from a pci-e card attached to another socket. The code looks the same, but the cost is radically different. Organizing how your code operates in memory to avoid cross-numa and cross-cpu in general isn't always easy, but it's required if you want to get the performance you paid for. OTOH, smaller nodes get this for free: my dual core desktop 'servers' don't have to worry about NUMA and don't have a lot of cross-cpu contention either.
    - dragontamer 5y ago
      > Reading from a pointer could be from L1 cache, or it could be from a pci-e card attached to another socket. The fun one is TLB (translation lookaside buffers) and the virtual memory system. Today's AMD core's have more L3 cache than what the TLB can handle with 4k-pages. You need to enable 2MB hugepages or 1GB hugepages to even access L3 cache at full speeds in practice... EDIT: Milan-X has 96MB L3 cache per CCX. 4kB-pages would require 24,000 (24-thousand) TLB-entries. IIRC, Milan only has 2000-TLB-entries. Hurraaahhhhhh.... ------ CPUs are devilishly complicated. It makes optimization "fun". Apparently, running "memcpy" requires Ph.D levels of study before you can "memcpy" at full speeds these days.
      - my123 5y ago
        In the same kind of funny subject, GPUs nowadays have full MMUs, with TLBs and all present too…
      - ncmncm 5y ago
        Hugepages FTW
- namibj 5y ago
  A singular CCD on Zen3 can max out the read bandwidth of two DDR4 channels at the latency-optimal 1:1 gearing between the IFOP clock and the DDR4 clock (i.e., 1600 MHz clock for DDR4-3200), but it's write bandwidth is only half as much (IFOP has asymmetric bandwidth). I haven't looked into detailed limits of the server northbridge, but the consumer northbridge has heaps of PCIe bandwidth compared to the relatively low DDR4 bandwidth. Even though PCIe4 is full-duplex, just the read bandwidth (after accounting for (in practice) unavoidable losses from DDR4's address bus nature) of the x16+x4 northbridge PCIe4 lanes plus the x4 PCIe4 lanes connecting the southbridge to the northbridge (which are re-emitted on X570 with the southbridge acting as a PCIe switch) exceeds the DDR4-2666 bandwidth. So assuming that DDR4-2666 RAM with the infinity fabric geared 1:1 at 1333 MHz, a Ryzen9 5950X's two CCDs can (together/combined) read 42 GB/s from DRAM, read 42 GB/s from PCIe, and write 42 GB/s to PCIe. This should break even with the peak L1$ bandwidth if the core is running at half the IFOP clock. Granted, in reality DDR4-3800 runs fine on most R9-5950X's (some chips don't work at the 1800 MHz IFOP; those will still run at DDR4-3600 in almost all cases, given high-quality DRAM (16 Gbit Micron Rev.E dies are used in e.g. Kingston Server Premier 32GB 2Rx8 "DDR4-3200 CL22" ECC-UDIMMs and my two sticks work at 3600 CL18 normally, even reaching 3600 (with CL 30+; this was a signal integrity test) in the same channel suggesting DDR4-3600 should work with 128 GiB RAM)). And that memory/IFOP speed is 42% above the 2666/1333 speed I equated to the available PCIe4x4 bandwidth. It's still damming to see that for sheer full-duplex bandwidth, DDR4 quite easily wins, considering 16 lanes full-duplex match the practical full-duplex ceiling of 2 DDR4-3600 channels if you measure bandwidth at the CPU's L2$.
wmf 5y ago
AFAIK each CCD gets 50 GB/s over Infinity Fabric which adds up to 200 GB/s which is effectively the same as memory bandwidth. IF should only be a bottleneck if you want to get high bandwidth from few CCDs.

dragontamer 5y ago

> I would expect it to be compared to L3 bandwidth not RAM. Infinity Fabric is well known to be slower / lower-bandwidth than DRAM actually. Just one of the peculiarities of this architecture. ------ That's why understanding MESI is important. If Core#10 writes something to DRAM: then Core#45 will not be able to write to it until Core#10 releases it (aka: reaches the "invalid" state of MESI). In contrast, writing to DRAM that's "not owned" by anybody is one-and-done. No need to communicate to other cores or chiplets. ------ No _singular_ core can reach 204GB/s spec of the architecture. You only reach that 204GB/s spec if all the cores (or chiplets really) are visiting different sections of RAM independently. This happens in practice often: the whole EPYC / Threadripper design is really aimed for virtual machines doing independent tasks. If you're trying to write high-performance code on EPYC / Threadripper, its important to keep this model in your brain. ------- Once you fix this problem, your next problem is probably going to be the page-table / virtual memory system. Maybe you should be looking up huge-pages in preparation?

ashvardanian 5y ago
Somehow, similarly bad numbers were achieved with 56 cores working with disjoint parts of 512 GB array. I didnt pin RAM to cores, though. Maybe thats the missing piece, but it’s sad if thats the only way to meet the spec. Almost no application can do that in modern cloud deployment realities.
- lostmsu 5y ago
  On the contrary. Having multiple independent services running on a single physical machine will likely give you full performance. The easiest way is probably just splitting the physical machine into the number of VMs/pods equal or larger than CCX count.
- dragontamer 5y ago
  https://people.freebsd.org/~gallatin/talks/euro2019.pdf https://people.freebsd.org/~gallatin/talks/euro2019.pdf Here's a talk about a lot of the NUMA-issues brought up by Netflix + FreeBSD. Its 200 Gb/s (bits), not bytes, but I'd expect you to be running into a lot of the similar problems. Its 200 Gb/s because they're reading from SSDs. But still, careful attention to NUMA was the only way they could reach full PCIe-bandwidths, which is a similar issue to yours (where NUMA is needed to reach full Memory/DRAM bandwidth). -------- Did you set your BIOS into NUMA mode? If you're in default BIOS-settings, your RAM is being split up between all 4 nodes (Uniform-memory-access mode). At least, that's how it was for my Threadripper 1950x (I know you've got a totally different model, but who knows?) I forgot exactly the name of the BIOS setting. But you gotta boot your computer up with 4-nodes-per-socket (or was it 8-nodes per socket for your computer?) or some similar setting. There were some programs on Linux (and Windows) that help you see what the OS is seeing.
  - kristjansson 5y ago
    See also the a more recent talk[0] on progress made on the same topic. On EPYC too, so perhaps more applicable to OP? [0]: https://people.freebsd.org/~gallatin/talks/euro2021.pdf https://people.freebsd.org/~gallatin/talks/euro2021.pdf
    - dragontamer 5y ago
      The original talk was on 2019 era hardware, for a 2019-era talk. Xeon and EPYC are both discussed, albeit somewhat dated now but still relevant IMO. The newer talk is 2021 hardware. IMO, the newer talk seems to focus more on the specific details of kTLS and PCIe TLS... but there's still stuff in there about NUMA for sure. But yeah, the 2021 talk definitely is on more recent hardware: 2021 Xeons and EPYCs.
      - kristjansson 5y ago
        Ah, dang, that's on me, I forgot the earlier talk discussed EPYC chips too.
  - 5y ago
- dragontamer 5y ago
  > Somehow, similarly bad numbers were achieved with 56 cores working with disjoint parts of 512 GB array Oh, that's not NUMA at all, now that I'm more carefully reading your post. NUMA would involve a "copy" step, ensuring that those elements are in NUMA-local memory before reading. Much like how in GPU-programming, you have to worry about the physicality of memory, NUMA-aware programming you have to memcpy data to the right location before it achieves high speeds. Each of the 56-cores needs its ~10GBs in "NUMA-local" memory _BEFORE_ you start the benchmark. Yeah, I realize this isn't practical. But... who ever said that NUMA use cases are practical? Lol. A lot of cases, it makes more sense to just take advantage of infinity fabric for simplicity (although its slower, its definitely more convenient).
- toast0 5y ago
  NUMA is part of the reality. There's lots of modern computing that doesn't acknowledge reality, but that doesn't tend to do good things for performance. Communication isn't cheap, and implicit communication makes it hard to tell what's expensive. Reading from a pointer could be from L1 cache, or it could be from a pci-e card attached to another socket. The code looks the same, but the cost is radically different. Organizing how your code operates in memory to avoid cross-numa and cross-cpu in general isn't always easy, but it's required if you want to get the performance you paid for. OTOH, smaller nodes get this for free: my dual core desktop 'servers' don't have to worry about NUMA and don't have a lot of cross-cpu contention either.
  - dragontamer 5y ago
    > Reading from a pointer could be from L1 cache, or it could be from a pci-e card attached to another socket. The fun one is TLB (translation lookaside buffers) and the virtual memory system. Today's AMD core's have more L3 cache than what the TLB can handle with 4k-pages. You need to enable 2MB hugepages or 1GB hugepages to even access L3 cache at full speeds in practice... EDIT: Milan-X has 96MB L3 cache per CCX. 4kB-pages would require 24,000 (24-thousand) TLB-entries. IIRC, Milan only has 2000-TLB-entries. Hurraaahhhhhh.... ------ CPUs are devilishly complicated. It makes optimization "fun". Apparently, running "memcpy" requires Ph.D levels of study before you can "memcpy" at full speeds these days.
    - my123 5y ago
      In the same kind of funny subject, GPUs nowadays have full MMUs, with TLBs and all present too…
    - ncmncm 5y ago
      Hugepages FTW
namibj 5y ago
A singular CCD on Zen3 can max out the read bandwidth of two DDR4 channels at the latency-optimal 1:1 gearing between the IFOP clock and the DDR4 clock (i.e., 1600 MHz clock for DDR4-3200), but it's write bandwidth is only half as much (IFOP has asymmetric bandwidth). I haven't looked into detailed limits of the server northbridge, but the consumer northbridge has heaps of PCIe bandwidth compared to the relatively low DDR4 bandwidth. Even though PCIe4 is full-duplex, just the read bandwidth (after accounting for (in practice) unavoidable losses from DDR4's address bus nature) of the x16+x4 northbridge PCIe4 lanes plus the x4 PCIe4 lanes connecting the southbridge to the northbridge (which are re-emitted on X570 with the southbridge acting as a PCIe switch) exceeds the DDR4-2666 bandwidth. So assuming that DDR4-2666 RAM with the infinity fabric geared 1:1 at 1333 MHz, a Ryzen9 5950X's two CCDs can (together/combined) read 42 GB/s from DRAM, read 42 GB/s from PCIe, and write 42 GB/s to PCIe. This should break even with the peak L1$ bandwidth if the core is running at half the IFOP clock. Granted, in reality DDR4-3800 runs fine on most R9-5950X's (some chips don't work at the 1800 MHz IFOP; those will still run at DDR4-3600 in almost all cases, given high-quality DRAM (16 Gbit Micron Rev.E dies are used in e.g. Kingston Server Premier 32GB 2Rx8 "DDR4-3200 CL22" ECC-UDIMMs and my two sticks work at 3600 CL18 normally, even reaching 3600 (with CL 30+; this was a signal integrity test) in the same channel suggesting DDR4-3600 should work with 128 GiB RAM)). And that memory/IFOP speed is 42% above the 2666/1333 speed I equated to the available PCIe4x4 bandwidth. It's still damming to see that for sheer full-duplex bandwidth, DDR4 quite easily wins, considering 16 lanes full-duplex match the practical full-duplex ceiling of 2 DDR4-3600 channels if you measure bandwidth at the CPU's L2$.

wmf 5y ago

AFAIK each CCD gets 50 GB/s over Infinity Fabric which adds up to 200 GB/s which is effectively the same as memory bandwidth. IF should only be a bottleneck if you want to get high bandwidth from few CCDs.