4 ms·
All modern CPUs have memory controllers built in as opposed to using a distinct "northbridge" chip. The RAM attaches directly to the CPU socket (or soldered BGA
by pmjordan 13y ago
All modern CPUs have memory controllers built in as opposed to using a distinct "northbridge" chip. The RAM attaches directly to the CPU socket (or soldered BGA). In multi-socket systems, this means there are memory buses coming off each CPU, and for one CPU to reach memory attached to the other CPU, it needs to go through that CPU via the interconnect. Here's some ASCII art of a system with 3 memory channels from each CPU (old Mac Pro):
+------+ +------+
| CPU0 +==interconnect==+ CPU1 |
+-+-+-++ ++-+-+-+
| \ \ / / |
MEM0 | \ / | MEM5
MEM1 MEM2 MEM3 MEM4
Going via the interconnect introduces extra latency and consumes its bandwidth. Therefore, it's more efficient to pin threads and/or processes to specific CPU sockets, and to allocate memory for it from that socket's associated physical memory. This requires kernel support, and for absolute top performance you also want a user space API, so you don't need to rely on the in-kernel heuristics.
Lacking software support means you have to fall back to whatever the hardware does automatically. Current multi-socket x86 CPUs all offer an "interlaced" mode for this (sort of like RAID0), where they dispatch to physical socket based on some physical address bits, e.g.:
0x00000000-0x0000003f: CPU0's RAM
0x00000040-0x0000007f: CPU1's RAM
0x00000080-0x000000bf: CPU0's RAM
0x000000c0-0x000000ff: CPU1's RAM
and so on. This spreads the extra latency evenly. Makes for consistent but poor memory performance. The mode is selected during system initialisation, i.e. is the responsibility of the Firmware (BIOS/EFI). Non-Apple workstations and servers typically let you select this as part of the pre-boot system setup UI.
It depends on the motherboard layout, but you'll find that there are groups of DIMM slots located near each CPU socket in multi-socket systems. Physical proximity helps keep speeds high.
- KaiserPro 13y agoThe other thing to mention here is with PCIe 3, the controller for the bus is now in the CPU, which means that if you have two GPUs, you'll start to see GPU affinity problems as well. In previous systems you'd have one bus attached to a southbridge, which is then shared by both CPUs* I've seen this with linux and nvidia cards, with two CPUs we would get a maximum CPU->GPU transfer of around 7-8gigs, take out a CPU and it'd bump up to 10-13. The z620 has the second CPU on a daughter card, so its simple to do. *my memory is a bit hazy, I'm open to corrections
- pmjordan 13y agoYou should at least be able to control that with regular CPU affinity from software.
- seanmcdirmid 13y agoI get it, I haven't studied NUMA much since 10 years ago, when we were looking in it in terms of Sparc interconnects and expensive workstation stuff. From what I've heard from my colleagues who do HPC, NUMA just isn't that effective. The sweet spot right now is either with GPUs or MPI; the shared memory abstraction that NUMA provides is just not very tenable. Where is NUMA being used as a best (rather than legacy) solution right now?
- pmjordan 13y agoThe question is not so much "where is NUMA effective," but, "given that for highest speeds memory needs to be attached directly to the CPU, how do we deal effectively with the case of multiple CPUs?" If you want multi-socket SMP, these days you need to deal with its inherent NUMA architecture. Or sweep it under the rug as Apple did with the old Mac Pro and sacrifice performance.