6 ms·

Non-Uniform Memory Access (NUMA) is reshaping microservice placement

98 points by signa11 1y ago

ashvardanian 1y ago
I've been struggling to find the settings to control NUMA on public cloud instances for a long time. Those are typically configured to present a single socket as a single UMA node, even on huge EPYCs. If someone has a tip on where to find those, I'd appreciate a link!
throw0101c 1y ago
If you really want to get into it, then could start worrying about your I/O. In the AMD example here: * https://www.thomas-krenn.com/en/wiki/Display_Linux_CPU_topology_with_lstopo#Dual_AMD_EPYC_7313 https://www.thomas-krenn.com/en/wiki/Display_Linux_CPU_topol... you'll see some NUMA nodes with networking I/O attached to them, others with NVMe, and others with no I/O. So if you're really worried about network latency then you'd pin the process to that node, but if you want look at disk numbers (a database?) you'd be potentially looking at that node. In recent years there's also chiplet-level locality that may need to be considered as well. Examining this has been a thing in the HPC space for a decade or two now: * https://www.open-mpi.org/projects/hwloc/lstopo/ https://www.open-mpi.org/projects/hwloc/lstopo/ * https://slurm.schedmd.com/mc_support.html https://slurm.schedmd.com/mc_support.html
- sidewndr46 1y ago
  This "lstopo" tool is amazing and something I have been searching for a while.
pjmlp 1y ago
Even with SMP, back in the 2000's, the Windows NT/2000 scheduler wasn't that great, re-scheduling processes/threads across CPUs, already by making use of the processor affinity mask we managed a visible performance improvement. NUMA systems now make this even more obvious, when scheduling is not done properly.
stego-tech 1y ago
Solid writeup of NUMA, scheduling, and the need for pinning for folks who don’t spend a lot of time in the IT side of things (where we, unfortunately, have been wrangling with this for over a decade). The long and short of it is that if you’re building a HPC application, or are sensitive to throughput and latency on your cutting-edge/high-traffic system design, then you need to manually pin your workloads for optimal performance. One thing the writeup didn’t seem to get into is the lack of scalability of this approach (manual pinning). As core counts and chiplets continue to explode, we still need better ways of scaling manual pinning or building more NUMA-aware OSes/applications that can auto-schedule with minimal penalties. Don’t get me wrong, it’s a lot better than ye olden days of dual core, multi-socket servers and stern warnings against fussing with NUMA schedulers from vendors if you wanted to preserve basic functionality, but it’s not a solved problem just yet.
- uberduper 1y ago
  Kubernetes will handle this automatically if you configure the Topology Manager.
- jasonjayr 1y ago
  This strikes me as something that Kubernetes could handle if it could support it. You can use affinity to ensure workloads stay together on the same machines, if K8s was NUMA aware, you could extend that affinity/anti-affinity mechanism down to the core/socket level. EDIT: aaaand ... I commented before reading the article, which describes this very mechanism.
  - jauntywundrkind 1y ago
    It'd be great to see Kubernetes make more extensive use of croups & especially nested croups, imo. The cpuset affinity should build into that layer nicely, imo. More broadly, Kubernetes' desire to schedule everything itself, to fit the workloads intelligent to insure successful running, feels like an anti-partern when the kernel has a much more aggressive way to let you trade off and define priorities and bound resources; it sucks having the ultra lo-fi kube take. I want the kernels "let it fail" version where nested cgroups get to fight it out according to their allocations. Really enjoyed this amazing write up on how Kube does use cgroups. Seems like the QoS controls do give some top level cgroups, that pods then nest inside of. That's something. At least! https://martinheinz.dev/blog/91 https://martinheinz.dev/blog/91
jauntywundrkind 1y ago
There's a constant drum-beat of NUMA related work going by if you follow phoronix.com . https://www.phoronix.com/news/Linux-6.17-NUMA-Locality-Random https://www.phoronix.com/news/Linux-6.17-NUMA-Locality-Rando... https://www.phoronix.com/news/Linux-6.13-Sched_Ext https://www.phoronix.com/news/Linux-6.13-Sched_Ext https://www.phoronix.com/news/DAMON-Self-Tuned-Memory-Tiering https://www.phoronix.com/news/DAMON-Self-Tuned-Memory-Tierin... https://www.phoronix.com/news/Linux-6.14-FUSE https://www.phoronix.com/news/Linux-6.14-FUSE There's some big work I'm missing thats more recent too, again about allocating & scheduling IIRC. Still trying to find it. The third link is in DAMON, which is trying to do a lot to optimize; good thread to tug more on! I have this pocket belief that eventually we might see post NUMA post coherency architectures, where even a single chip acts more like multiple independent clusters, that use something more like networking (CXL or UltraEthernet or something) to allow RDMA, but without coherency. Even today, the title here is woefully under-describing the problem. A Epyc chip is actually multiple different compute die, each with their own NUMA zone and their own L3 and other caches. For now yes each socket's memory is all via a single IO die & semi uniform, but whether that holds is in question, and even today, the multiple NUMA zones on one socket already require careful tuning for efficient workload processing.
- Aurornis 1y ago
  Emulating NUMA on a single chip is already a known performance tweak on certain architecture. There are options in place to enable it: https://www.kernel.org/doc/html/v5.8/x86/x86_64/fake-numa-for-cpusets.html https://www.kernel.org/doc/html/v5.8/x86/x86_64/fake-numa-fo... Even the Raspberry Pi 5 benefits from NUMA emulation because it makes memory use patterns better match the memory controller’s parallelization capabilities.
- positron26 1y ago
  IMO, matter of time before x86 or RISCV extension will show up to begin the inevitable unification of GPU and SIMD in an ISA. NUMA work and clustering over CCXs and sockets is paving the way for the software support in the OS. Question is what makes as much of Vulkan, OpenCL, and CUDA go away as possible?
jwgehieir 1y ago
[flagged]
jwgehieir 1y ago
[flagged]
bboreham 1y ago
Very detailed and accurate description. The author clearly knows way more than I do, but I would venture a few notes: 1. In the cloud, it can be difficult to know the NUMA characteristics of your VMs. AWS, Google, etc., do not publish it. I found the ‘lscpu’ command helpful. 2. Tools like https://github.com/SoilRos/cpu-latency https://github.com/SoilRos/cpu-latency plot the core-to-core latency on a 2d grid. There are many example visualisations on that page; maybe you can find the chip you are using. 3. If you get to pick VM sizes, pick ones the same size as a NUMA node on the underlying hardware. Eg prefer 64-core m8g.16xlarge over 96-core m8g.24xlarge which will span two nodes.
- gmokki 1y ago
  I've used https://instaguide.io/info.html?type=c5a.24xlarge#tab=lstopo https://instaguide.io/info.html?type=c5a.24xlarge#tab=lstopo to browse the info. It is getting a bit old though.
  - bboreham 1y ago
    That is nice. But no detail after gen 5, so mostly historical interest.
- Twirrim 1y ago
  At OCI, our VM shapes are all single NUMA node by default. We only relatively recently added support for cross-NUMA instances, precisely because of the complications that NUMA introduces. There are so many performance quirks, and so much software doesn't account for it yet (in part, I'd bet, because most development environments don't have multiple NUMA domains.) Here's a fun example we found a few years ago, not sure if work has happened in the upstream kernel since: the Linux page cache wasn't fully NUMA aware, and spans NUMA nodes. Someone at work was specifically looking at NUMA performance, and chose to benchmark databases on different NUMA nodes, trying the client on the same NUMA node, and then cross NUMA node, using numactl to pin. After a bunch of tests it looked like with client and server in NUMA 0 it was appreciably faster than client and server in NUMA 1. After a reboot, and re running tests, it had flipped. NUMA 1 faster than NUMA 0. Eventually they worked out that the fast NUMA was whichever one was benchmarked first after a reboot, and from there figured out that when you ran fresh, the database client library ended up in the page cache in that NUMA domain. So if they benchmarked with server in 0, client in 1, and then benchmarked with server in 0, client in 0, that clients access to the client library ended up reaching across to the page cached version in 1, paying a nice latency penalty over and over. His solution was to run the client in a NUMA pinned docker container so that it was a unique file to the OS.
porridgeraisin 1y ago
Very nice article. Gonna try this out on our lab cluster and see what improvements it gives.

98 points by signa11 1y ago

I've been struggling to find the settings to control NUMA on public cloud instances for a long time. Those are typically configured to present a single socket as a single UMA node, even on huge EPYCs. If someone has a tip on where to find those, I'd appreciate a link!

throw0101c 1y ago

If you really want to get into it, then could start worrying about your I/O. In the AMD example here: * https://www.thomas-krenn.com/en/wiki/Display_Linux_CPU_topology_with_lstopo#Dual_AMD_EPYC_7313 https://www.thomas-krenn.com/en/wiki/Display_Linux_CPU_topol... you'll see some NUMA nodes with networking I/O attached to them, others with NVMe, and others with no I/O. So if you're really worried about network latency then you'd pin the process to that node, but if you want look at disk numbers (a database?) you'd be potentially looking at that node. In recent years there's also chiplet-level locality that may need to be considered as well. Examining this has been a thing in the HPC space for a decade or two now: * https://www.open-mpi.org/projects/hwloc/lstopo/ https://www.open-mpi.org/projects/hwloc/lstopo/ * https://slurm.schedmd.com/mc_support.html https://slurm.schedmd.com/mc_support.html

sidewndr46 1y ago
This "lstopo" tool is amazing and something I have been searching for a while.

pjmlp 1y ago

Even with SMP, back in the 2000's, the Windows NT/2000 scheduler wasn't that great, re-scheduling processes/threads across CPUs, already by making use of the processor affinity mask we managed a visible performance improvement. NUMA systems now make this even more obvious, when scheduling is not done properly.

stego-tech 1y ago

Solid writeup of NUMA, scheduling, and the need for pinning for folks who don’t spend a lot of time in the IT side of things (where we, unfortunately, have been wrangling with this for over a decade). The long and short of it is that if you’re building a HPC application, or are sensitive to throughput and latency on your cutting-edge/high-traffic system design, then you need to manually pin your workloads for optimal performance. One thing the writeup didn’t seem to get into is the lack of scalability of this approach (manual pinning). As core counts and chiplets continue to explode, we still need better ways of scaling manual pinning or building more NUMA-aware OSes/applications that can auto-schedule with minimal penalties. Don’t get me wrong, it’s a lot better than ye olden days of dual core, multi-socket servers and stern warnings against fussing with NUMA schedulers from vendors if you wanted to preserve basic functionality, but it’s not a solved problem just yet.

uberduper 1y ago
Kubernetes will handle this automatically if you configure the Topology Manager.
jasonjayr 1y ago
This strikes me as something that Kubernetes could handle if it could support it. You can use affinity to ensure workloads stay together on the same machines, if K8s was NUMA aware, you could extend that affinity/anti-affinity mechanism down to the core/socket level. EDIT: aaaand ... I commented before reading the article, which describes this very mechanism.
- jauntywundrkind 1y ago
  It'd be great to see Kubernetes make more extensive use of croups & especially nested croups, imo. The cpuset affinity should build into that layer nicely, imo. More broadly, Kubernetes' desire to schedule everything itself, to fit the workloads intelligent to insure successful running, feels like an anti-partern when the kernel has a much more aggressive way to let you trade off and define priorities and bound resources; it sucks having the ultra lo-fi kube take. I want the kernels "let it fail" version where nested cgroups get to fight it out according to their allocations. Really enjoyed this amazing write up on how Kube does use cgroups. Seems like the QoS controls do give some top level cgroups, that pods then nest inside of. That's something. At least! https://martinheinz.dev/blog/91 https://martinheinz.dev/blog/91

jauntywundrkind 1y ago

There's a constant drum-beat of NUMA related work going by if you follow phoronix.com . https://www.phoronix.com/news/Linux-6.17-NUMA-Locality-Random https://www.phoronix.com/news/Linux-6.17-NUMA-Locality-Rando... https://www.phoronix.com/news/Linux-6.13-Sched_Ext https://www.phoronix.com/news/Linux-6.13-Sched_Ext https://www.phoronix.com/news/DAMON-Self-Tuned-Memory-Tiering https://www.phoronix.com/news/DAMON-Self-Tuned-Memory-Tierin... https://www.phoronix.com/news/Linux-6.14-FUSE https://www.phoronix.com/news/Linux-6.14-FUSE There's some big work I'm missing thats more recent too, again about allocating & scheduling IIRC. Still trying to find it. The third link is in DAMON, which is trying to do a lot to optimize; good thread to tug more on! I have this pocket belief that eventually we might see post NUMA post coherency architectures, where even a single chip acts more like multiple independent clusters, that use something more like networking (CXL or UltraEthernet or something) to allow RDMA, but without coherency. Even today, the title here is woefully under-describing the problem. A Epyc chip is actually multiple different compute die, each with their own NUMA zone and their own L3 and other caches. For now yes each socket's memory is all via a single IO die & semi uniform, but whether that holds is in question, and even today, the multiple NUMA zones on one socket already require careful tuning for efficient workload processing.

Aurornis 1y ago
Emulating NUMA on a single chip is already a known performance tweak on certain architecture. There are options in place to enable it: https://www.kernel.org/doc/html/v5.8/x86/x86_64/fake-numa-for-cpusets.html https://www.kernel.org/doc/html/v5.8/x86/x86_64/fake-numa-fo... Even the Raspberry Pi 5 benefits from NUMA emulation because it makes memory use patterns better match the memory controller’s parallelization capabilities.
positron26 1y ago
IMO, matter of time before x86 or RISCV extension will show up to begin the inevitable unification of GPU and SIMD in an ISA. NUMA work and clustering over CCXs and sockets is paving the way for the software support in the OS. Question is what makes as much of Vulkan, OpenCL, and CUDA go away as possible?

jwgehieir 1y ago

[flagged]

jwgehieir 1y ago

[flagged]

bboreham 1y ago

Very detailed and accurate description. The author clearly knows way more than I do, but I would venture a few notes: 1. In the cloud, it can be difficult to know the NUMA characteristics of your VMs. AWS, Google, etc., do not publish it. I found the ‘lscpu’ command helpful. 2. Tools like https://github.com/SoilRos/cpu-latency https://github.com/SoilRos/cpu-latency plot the core-to-core latency on a 2d grid. There are many example visualisations on that page; maybe you can find the chip you are using. 3. If you get to pick VM sizes, pick ones the same size as a NUMA node on the underlying hardware. Eg prefer 64-core m8g.16xlarge over 96-core m8g.24xlarge which will span two nodes.

gmokki 1y ago
I've used https://instaguide.io/info.html?type=c5a.24xlarge#tab=lstopo https://instaguide.io/info.html?type=c5a.24xlarge#tab=lstopo to browse the info. It is getting a bit old though.
- bboreham 1y ago
  That is nice. But no detail after gen 5, so mostly historical interest.
Twirrim 1y ago
At OCI, our VM shapes are all single NUMA node by default. We only relatively recently added support for cross-NUMA instances, precisely because of the complications that NUMA introduces. There are so many performance quirks, and so much software doesn't account for it yet (in part, I'd bet, because most development environments don't have multiple NUMA domains.) Here's a fun example we found a few years ago, not sure if work has happened in the upstream kernel since: the Linux page cache wasn't fully NUMA aware, and spans NUMA nodes. Someone at work was specifically looking at NUMA performance, and chose to benchmark databases on different NUMA nodes, trying the client on the same NUMA node, and then cross NUMA node, using numactl to pin. After a bunch of tests it looked like with client and server in NUMA 0 it was appreciably faster than client and server in NUMA 1. After a reboot, and re running tests, it had flipped. NUMA 1 faster than NUMA 0. Eventually they worked out that the fast NUMA was whichever one was benchmarked first after a reboot, and from there figured out that when you ran fresh, the database client library ended up in the page cache in that NUMA domain. So if they benchmarked with server in 0, client in 1, and then benchmarked with server in 0, client in 0, that clients access to the client library ended up reaching across to the page cached version in 1, paying a nice latency penalty over and over. His solution was to run the client in a NUMA pinned docker container so that it was a unique file to the OS.

porridgeraisin 1y ago

Very nice article. Gonna try this out on our lab cluster and see what improvements it gives.