6 ms·

You are missing one critical element: NUMA awareness. Because of the Chiplet design, Threadrippers and EPYC chips are basically NUMA systems, like you would wi

You are missing one critical element: NUMA awareness. Because of the Chiplet design, Threadrippers and EPYC chips are basically NUMA systems, like you would with a multi-socket motherboard. This has an impact on both memory latency and bandwidth. Redo your test, but pin each thread to a CPU thread, and its memory pool to a stick of ram attached directly to the CCX of said thread. The bandwidth you are measuring is the Infinity Fabric's bandwidth, NOT the DDR4 bandwidth. You can read more about infinity fabric and its bandwidth here: https://en.wikichip.org/wiki/amd/infinity_fabric https://en.wikichip.org/wiki/amd/infinity_fabric

ashvardanian 5y ago
Thanks, that's a valid suggestion! But in that case I would be forced to split my array into smaller chunks in different disjoint region of address space, right?
vardump 5y ago
Came to say exactly this. A lot of code is wasting a ton of performance on servers because of not being NUMA aware. We should also test high throughput/performance code on NUMA systems. Remember to allocate local memory for your threads — IOW, be NUMA aware. Going through interconnect(s) is slow.
- jjoonathan 5y ago
  How is that generally done? Platform specific libraries? Utilities that ask the OS to pin a program to a node?
  - pkaye 5y ago
    In Linux you can use numactl command to pin a process.
  - cyber_kinetist 5y ago
    libnuma probably does the job. https://github.com/numactl/numactl https://github.com/numactl/numactl
  - magicalhippo 5y ago
    Windows has a NUMA API[1] to help with these things. For pinning a thread you can for example use the SetThreadSelectedCpuSetMasks call[2] from what I can gather. [1]: https://docs.microsoft.com/en-us/windows/win32/procthread/numa-support https://docs.microsoft.com/en-us/windows/win32/procthread/nu... [2]: https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-setthreadselectedcpusetmasks https://docs.microsoft.com/en-us/windows/win32/api/processth...
  - profquail 5y ago
    If you want to do set thread-core affinity in a cross-platform way, hwloc is a good approach: https://www.open-mpi.org/projects/hwloc/doc/v2.7.0/a00166.php#gae42c01b2addcfbf6048b9a516dd7a906 https://www.open-mpi.org/projects/hwloc/doc/v2.7.0/a00166.ph...
- hardolaf 5y ago
  I'm in HFT and a huge part of our job for the FPGA drivers is managing the NUMA awareness of our software. And we have to constantly educate new devs coming into the low level software teams about this. Luckily, the SW teams are starting to do this training for us but it's always a major headache when there's issues. You can easily lose 50%+ of your performance by just failing to set a core affinity without even considering anything else.
bravetraveler 5y ago
Hadn't even opened the article yet and I was thinking exactly this, 'bet they forgot NUMA'
ashvardanian 5y ago
On a second thought, are you sure Infinity Fabric would be that slow? I would expect it to be compared to L3 bandwidth not RAM. Plus, the article you gave suggests 40 GB/s P2P IFOP with slower 2.6 GHz RAM. I can't find the right diagram , but as I remember, this CPU had 8 chiplets, still allowing for much more bandwidth, depending on the topology.
- dragontamer 5y ago
  > I would expect it to be compared to L3 bandwidth not RAM. Infinity Fabric is well known to be slower / lower-bandwidth than DRAM actually. Just one of the peculiarities of this architecture. ------ That's why understanding MESI is important. If Core#10 writes something to DRAM: then Core#45 will not be able to write to it until Core#10 releases it (aka: reaches the "invalid" state of MESI). In contrast, writing to DRAM that's "not owned" by anybody is one-and-done. No need to communicate to other cores or chiplets. ------ No _singular_ core can reach 204GB/s spec of the architecture. You only reach that 204GB/s spec if all the cores (or chiplets really) are visiting different sections of RAM independently. This happens in practice often: the whole EPYC / Threadripper design is really aimed for virtual machines doing independent tasks. If you're trying to write high-performance code on EPYC / Threadripper, its important to keep this model in your brain. ------- Once you fix this problem, your next problem is probably going to be the page-table / virtual memory system. Maybe you should be looking up huge-pages in preparation?
  - ashvardanian 5y ago
    Somehow, similarly bad numbers were achieved with 56 cores working with disjoint parts of 512 GB array. I didnt pin RAM to cores, though. Maybe thats the missing piece, but it’s sad if thats the only way to meet the spec. Almost no application can do that in modern cloud deployment realities.
    - lostmsu 5y ago
      On the contrary. Having multiple independent services running on a single physical machine will likely give you full performance. The easiest way is probably just splitting the physical machine into the number of VMs/pods equal or larger than CCX count.
lostmsu 5y ago
I upvoted this knowing the explanation but not feeling myself confident enough to actually write it down in the hope that someone more authoritative will. Thank you!
lmilcin 5y ago
I would add that seeing somebody "forgetting" NUMA makes it likely there is a lot of other problems. NUMA isn't something you "forget" when you try to test bandwidth limits. It suggest lack of understanding of how that stuff works. It is as if somebody failed to reach hard drive transfer speeds and they "forgot" they need to be looking at the access pattern.
- bravetraveler 5y ago
  I think that's fair - this reads heavily like someone that thinks throwing hardware at the problem is all it takes. The marketing materials are guarantees. That kind of thing. They even quoted direct AMD statements that would indicate the memory access pattern isn't uniform. They glossed over it when they saw numbers. I used the term 'forget' in one of my posts, but it's not a true representation - NUMA isn't the kind of thing you forget when you're actually doing HPC. It's the bread and butter. It's like your mechanic forgetting your vehicle has a motor that needs consideration...
tester756 5y ago
but it only proves his point
namibj 5y ago
No, these are Zen2. Both Zen2 and Zen3 single-socket systems are not NUMA systems in the traditional view (i.e., w.r.t. DRAM). The L3 is CCX-local, but with Zen2 they re-introduced a separate northbridge die for each socket, integrated with the CCDs as a separate chiplet. Dual-socket systems remain NUMA, and for workloads that feel L3$ more than DRAM, treating CCX-as-NUMA still helps (but that's a scheduler thing, where kernel-level threads/processes (Linux barely distinguishes) that do fine-grained shared memory with _collective_ temporal locality should be accessed from within a single CCX).
- hindsightbias 5y ago
  But isn’t Zen2 a MCM? It’s not a single uniform processor and there is another level of abstraction on top of a socket
  - fomine3 5y ago
    MCM but only one I/O die, but has multiple "quadrants" in single I/O die that makes multiple NUMA domain depending on configuration. See https://developer.amd.com/wp-content/resources/56827-1-0.pdf https://developer.amd.com/wp-content/resources/56827-1-0.pdf
- paulmd 5y ago
  This is not correct. Despite only having a single IO die, the Zen2/Zen3 IOD is actually divided into four quadrants, and each chipset (or pair of chipsets) has preferential access to that quadrant, and has to cross an interconnect to get to the other quadrants (and their memory). This is still a non-uniform topology. This is explicitly called out in the BIOS of these systems: the terminology is "NPS4 mode" (Nodes Per Socket) and is documented in AMD's reference materials. The system can also be run in what's called "NPS1 mode", where all memory channels are interleaved, but since this obviously requires waiting for all four quadrants to finish their memory access, this increases latency somewhat. https://developer.amd.com/wp-content/resources/56338_1.00_pub.pdf https://developer.amd.com/wp-content/resources/56338_1.00_pu... Anandtech did some really good coverage of this in their 3rd Gen Epyc (Milan) deep-dive, including latency measurements. Zen2 is largely similar architecturally in this area. You can see from comparing the NPS1 and NPS4 charts that Zen2 has around 12.8% higher memory latency in NPS1 mode and this is reduced to 6.4% in Zen3. https://www.anandtech.com/show/16529/amd-epyc-milan-review/4 https://www.anandtech.com/show/16529/amd-epyc-milan-review/4 If you ignore the physical placement of functionality and simply look at a birds-eye view of the data paths, it's not that different from Zen1. Zen1 had four "chiplets" that each have their own memory controller (and associated uncore). In Zen2, the uncore has simply been pulled from the chiplets onto the standalone IOD, but it is still implemented as four quadrants - just as Naples was four monolithic dies interconnected and packaged together. As mentioned, since effectively the entire uncore is divided into quadrants, this has a few other quirks. The one that most frequently comes up is populating memory channels - you really really want to populate all memory channels on Epyc, even when running in NPS1 mode. If you're not going to populate sets of 8, you really really want to make sure it's one of the "balanced" configurations, as otherwise some quadrants don't have access to memory at all, and the performance hit can be substantial. For example, Lenovo's documentation shows that populating only 6 out of 8 channels can result in a 29% performance hit (relative to the theoretical potential of a 6-DIMM configuration) even in the "correct" configuration (two quadrants each lose one channel), and an improper configuration (one quadrant with no attached memory) will have a 60% performance hit. Populating 7 channels will result in a 65% performance hit from the theoretical maximum - you are losing 2/3rds of your performance largely due to the NUMA topology! https://lenovopress.com/lp1268.pdf https://lenovopress.com/lp1268.pdf PCIe latency is also slightly higher when crossing quadrants. Not the sort of thing most people will be paying attention to, but to use an example here, the guy above who's doing HFT and paying attention to NUMA affinity is probably paying attention to what cores go to what PCIe lanes for talking to his FPGAs, because it does matter. Netflix also ran into similar issues around bandwidth - needlessly pushing data across the NUMA domains will eventually bottleneck performance, if you are pushing enough data. Keeping it inside the quadrant doesn't incur that bottleneck. It really is a testament to how well AMD made NUMA work that it doesn't "feel" like NUMA - and I think they even turn the NPS1 mode on by default now. But architecturally, it is NUMA underneath, and you can extract a small amount of additional performance by pulling the veneer of UMA away and addressing the hardware as it is actually implemented.

ashvardanian 5y ago

Thanks, that's a valid suggestion! But in that case I would be forced to split my array into smaller chunks in different disjoint region of address space, right?

vardump 5y ago

Came to say exactly this. A lot of code is wasting a ton of performance on servers because of not being NUMA aware. We should also test high throughput/performance code on NUMA systems. Remember to allocate local memory for your threads — IOW, be NUMA aware. Going through interconnect(s) is slow.

jjoonathan 5y ago
How is that generally done? Platform specific libraries? Utilities that ask the OS to pin a program to a node?
- pkaye 5y ago
  In Linux you can use numactl command to pin a process.
- cyber_kinetist 5y ago
  libnuma probably does the job. https://github.com/numactl/numactl https://github.com/numactl/numactl
- magicalhippo 5y ago
  Windows has a NUMA API[1] to help with these things. For pinning a thread you can for example use the SetThreadSelectedCpuSetMasks call[2] from what I can gather. [1]: https://docs.microsoft.com/en-us/windows/win32/procthread/numa-support https://docs.microsoft.com/en-us/windows/win32/procthread/nu... [2]: https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-setthreadselectedcpusetmasks https://docs.microsoft.com/en-us/windows/win32/api/processth...
- profquail 5y ago
  If you want to do set thread-core affinity in a cross-platform way, hwloc is a good approach: https://www.open-mpi.org/projects/hwloc/doc/v2.7.0/a00166.php#gae42c01b2addcfbf6048b9a516dd7a906 https://www.open-mpi.org/projects/hwloc/doc/v2.7.0/a00166.ph...
hardolaf 5y ago
I'm in HFT and a huge part of our job for the FPGA drivers is managing the NUMA awareness of our software. And we have to constantly educate new devs coming into the low level software teams about this. Luckily, the SW teams are starting to do this training for us but it's always a major headache when there's issues. You can easily lose 50%+ of your performance by just failing to set a core affinity without even considering anything else.

bravetraveler 5y ago

Hadn't even opened the article yet and I was thinking exactly this, 'bet they forgot NUMA'

ashvardanian 5y ago

On a second thought, are you sure Infinity Fabric would be that slow? I would expect it to be compared to L3 bandwidth not RAM. Plus, the article you gave suggests 40 GB/s P2P IFOP with slower 2.6 GHz RAM. I can't find the right diagram , but as I remember, this CPU had 8 chiplets, still allowing for much more bandwidth, depending on the topology.

dragontamer 5y ago
> I would expect it to be compared to L3 bandwidth not RAM. Infinity Fabric is well known to be slower / lower-bandwidth than DRAM actually. Just one of the peculiarities of this architecture. ------ That's why understanding MESI is important. If Core#10 writes something to DRAM: then Core#45 will not be able to write to it until Core#10 releases it (aka: reaches the "invalid" state of MESI). In contrast, writing to DRAM that's "not owned" by anybody is one-and-done. No need to communicate to other cores or chiplets. ------ No _singular_ core can reach 204GB/s spec of the architecture. You only reach that 204GB/s spec if all the cores (or chiplets really) are visiting different sections of RAM independently. This happens in practice often: the whole EPYC / Threadripper design is really aimed for virtual machines doing independent tasks. If you're trying to write high-performance code on EPYC / Threadripper, its important to keep this model in your brain. ------- Once you fix this problem, your next problem is probably going to be the page-table / virtual memory system. Maybe you should be looking up huge-pages in preparation?
- ashvardanian 5y ago
  Somehow, similarly bad numbers were achieved with 56 cores working with disjoint parts of 512 GB array. I didnt pin RAM to cores, though. Maybe thats the missing piece, but it’s sad if thats the only way to meet the spec. Almost no application can do that in modern cloud deployment realities.
  - lostmsu 5y ago
    On the contrary. Having multiple independent services running on a single physical machine will likely give you full performance. The easiest way is probably just splitting the physical machine into the number of VMs/pods equal or larger than CCX count.

lostmsu 5y ago

I upvoted this knowing the explanation but not feeling myself confident enough to actually write it down in the hope that someone more authoritative will. Thank you!

lmilcin 5y ago

I would add that seeing somebody "forgetting" NUMA makes it likely there is a lot of other problems. NUMA isn't something you "forget" when you try to test bandwidth limits. It suggest lack of understanding of how that stuff works. It is as if somebody failed to reach hard drive transfer speeds and they "forgot" they need to be looking at the access pattern.

bravetraveler 5y ago
I think that's fair - this reads heavily like someone that thinks throwing hardware at the problem is all it takes. The marketing materials are guarantees. That kind of thing. They even quoted direct AMD statements that would indicate the memory access pattern isn't uniform. They glossed over it when they saw numbers. I used the term 'forget' in one of my posts, but it's not a true representation - NUMA isn't the kind of thing you forget when you're actually doing HPC. It's the bread and butter. It's like your mechanic forgetting your vehicle has a motor that needs consideration...

tester756 5y ago

but it only proves his point

namibj 5y ago

No, these are Zen2. Both Zen2 and Zen3 single-socket systems are not NUMA systems in the traditional view (i.e., w.r.t. DRAM). The L3 is CCX-local, but with Zen2 they re-introduced a separate northbridge die for each socket, integrated with the CCDs as a separate chiplet. Dual-socket systems remain NUMA, and for workloads that feel L3$ more than DRAM, treating CCX-as-NUMA still helps (but that's a scheduler thing, where kernel-level threads/processes (Linux barely distinguishes) that do fine-grained shared memory with _collective_ temporal locality should be accessed from within a single CCX).

hindsightbias 5y ago
But isn’t Zen2 a MCM? It’s not a single uniform processor and there is another level of abstraction on top of a socket
- fomine3 5y ago
  MCM but only one I/O die, but has multiple "quadrants" in single I/O die that makes multiple NUMA domain depending on configuration. See https://developer.amd.com/wp-content/resources/56827-1-0.pdf https://developer.amd.com/wp-content/resources/56827-1-0.pdf
paulmd 5y ago
This is not correct. Despite only having a single IO die, the Zen2/Zen3 IOD is actually divided into four quadrants, and each chipset (or pair of chipsets) has preferential access to that quadrant, and has to cross an interconnect to get to the other quadrants (and their memory). This is still a non-uniform topology. This is explicitly called out in the BIOS of these systems: the terminology is "NPS4 mode" (Nodes Per Socket) and is documented in AMD's reference materials. The system can also be run in what's called "NPS1 mode", where all memory channels are interleaved, but since this obviously requires waiting for all four quadrants to finish their memory access, this increases latency somewhat. https://developer.amd.com/wp-content/resources/56338_1.00_pub.pdf https://developer.amd.com/wp-content/resources/56338_1.00_pu... Anandtech did some really good coverage of this in their 3rd Gen Epyc (Milan) deep-dive, including latency measurements. Zen2 is largely similar architecturally in this area. You can see from comparing the NPS1 and NPS4 charts that Zen2 has around 12.8% higher memory latency in NPS1 mode and this is reduced to 6.4% in Zen3. https://www.anandtech.com/show/16529/amd-epyc-milan-review/4 https://www.anandtech.com/show/16529/amd-epyc-milan-review/4 If you ignore the physical placement of functionality and simply look at a birds-eye view of the data paths, it's not that different from Zen1. Zen1 had four "chiplets" that each have their own memory controller (and associated uncore). In Zen2, the uncore has simply been pulled from the chiplets onto the standalone IOD, but it is still implemented as four quadrants - just as Naples was four monolithic dies interconnected and packaged together. As mentioned, since effectively the entire uncore is divided into quadrants, this has a few other quirks. The one that most frequently comes up is populating memory channels - you really really want to populate all memory channels on Epyc, even when running in NPS1 mode. If you're not going to populate sets of 8, you really really want to make sure it's one of the "balanced" configurations, as otherwise some quadrants don't have access to memory at all, and the performance hit can be substantial. For example, Lenovo's documentation shows that populating only 6 out of 8 channels can result in a 29% performance hit (relative to the theoretical potential of a 6-DIMM configuration) even in the "correct" configuration (two quadrants each lose one channel), and an improper configuration (one quadrant with no attached memory) will have a 60% performance hit. Populating 7 channels will result in a 65% performance hit from the theoretical maximum - you are losing 2/3rds of your performance largely due to the NUMA topology! https://lenovopress.com/lp1268.pdf https://lenovopress.com/lp1268.pdf PCIe latency is also slightly higher when crossing quadrants. Not the sort of thing most people will be paying attention to, but to use an example here, the guy above who's doing HFT and paying attention to NUMA affinity is probably paying attention to what cores go to what PCIe lanes for talking to his FPGAs, because it does matter. Netflix also ran into similar issues around bandwidth - needlessly pushing data across the NUMA domains will eventually bottleneck performance, if you are pushing enough data. Keeping it inside the quadrant doesn't incur that bottleneck. It really is a testament to how well AMD made NUMA work that it doesn't "feel" like NUMA - and I think they even turn the NPS1 mode on by default now. But architecturally, it is NUMA underneath, and you can extract a small amount of additional performance by pulling the veneer of UMA away and addressing the hardware as it is actually implemented.