5 ms·

The Mac pro only has one CPU because that's all they could fit in. I suspect the real reason is that Apple still {can't be bothered|can't figure out how} to ad

The Mac pro only has one CPU because that's all they could fit in. I suspect the real reason is that Apple still {can't be bothered|can't figure out how} to add NUMA support to OS X. The existing 2-socket Mac Pros have been crippled by the interleaved-mode NUMA. (which, interestingly enough, is even hardcoded in the firmware, so when running OSes that support NUMA on a Mac Pro, you can't benefit from that support, presumably so the benchmarks don't shine a bad light on OS X - of course it's blindingly obvious when compared to a same-spec workstation from HP, Dell, etc.) In fairness, adding NUMA to xnu is probably quite an undertaking. I wouldn't be surprised if they've previously tried and got in a terrible mess. The task/thread management is scattered across the Mach and BSD subsystems, and the virtual memory system handles some conditions (such as low memory) terribly, and seems to scale badly to large amounts of RAM and 64 bits.

leoc 13y ago
Getting rid of user-accessible PCI has a similar benefit for Apple - no more hassle having to support a zoo of PCI cards.
- pmjordan 13y ago
  Except there are Thunderbolt-to-PCIe enclosures, which come with their own set of problems (latency, hotplugging, I/O memory range allocation, bandwidth, not to mention cost). But I guess Apple can just say "we don't support those, take it up with the enclosure manufacturer". Drivers for Thunderbolt devices still are necessarily kexts and can thus take down the system without an obvious culprit. For this reason alone I'm surprised they went with it rather than supporting USB3 early on or that they didn't come up with something similar but altogether new, like they did with Lightning. They seem to be trying to cut down third party kext development, and aren't opening the sources for most of their own new kexts. Of course, with IOMMUs available in all Thunderbolt systems, they could have brought back the microkernel aspect of Mach for device drivers, but that would have been even more work.
seanmcdirmid 13y ago
What does NUMA have to do with SMP setups? NUMA would be about CPUs with separate physical memories acting as one weird processing resource, not a Mac Pro with multiple processes connected to the same memory. What am I missing here?
- pmjordan 13y ago
  All modern CPUs have memory controllers built in as opposed to using a distinct "northbridge" chip. The RAM attaches directly to the CPU socket (or soldered BGA). In multi-socket systems, this means there are memory buses coming off each CPU, and for one CPU to reach memory attached to the other CPU, it needs to go through that CPU via the interconnect. Here's some ASCII art of a system with 3 memory channels from each CPU (old Mac Pro): +------+ +------+ | CPU0 +==interconnect==+ CPU1 | +-+-+-++ ++-+-+-+ | \ \ / / | MEM0 | \ / | MEM5 MEM1 MEM2 MEM3 MEM4 Going via the interconnect introduces extra latency and consumes its bandwidth. Therefore, it's more efficient to pin threads and/or processes to specific CPU sockets, and to allocate memory for it from that socket's associated physical memory. This requires kernel support, and for absolute top performance you also want a user space API, so you don't need to rely on the in-kernel heuristics. Lacking software support means you have to fall back to whatever the hardware does automatically. Current multi-socket x86 CPUs all offer an "interlaced" mode for this (sort of like RAID0), where they dispatch to physical socket based on some physical address bits, e.g.: 0x00000000-0x0000003f: CPU0's RAM 0x00000040-0x0000007f: CPU1's RAM 0x00000080-0x000000bf: CPU0's RAM 0x000000c0-0x000000ff: CPU1's RAM and so on. This spreads the extra latency evenly. Makes for consistent but poor memory performance. The mode is selected during system initialisation, i.e. is the responsibility of the Firmware (BIOS/EFI). Non-Apple workstations and servers typically let you select this as part of the pre-boot system setup UI. It depends on the motherboard layout, but you'll find that there are groups of DIMM slots located near each CPU socket in multi-socket systems. Physical proximity helps keep speeds high.
  - KaiserPro 13y ago
    The other thing to mention here is with PCIe 3, the controller for the bus is now in the CPU, which means that if you have two GPUs, you'll start to see GPU affinity problems as well. In previous systems you'd have one bus attached to a southbridge, which is then shared by both CPUs* I've seen this with linux and nvidia cards, with two CPUs we would get a maximum CPU->GPU transfer of around 7-8gigs, take out a CPU and it'd bump up to 10-13. The z620 has the second CPU on a daughter card, so its simple to do. *my memory is a bit hazy, I'm open to corrections
sebbean 13y ago
Am I the only one curious as to what NUMA stands for?
- LukeHoersten 13y ago
  Non-uniform memory access http://en.m.wikipedia.org/wiki/Non-Uniform_Memory_Access http://en.m.wikipedia.org/wiki/Non-Uniform_Memory_Access

leoc 13y ago

Getting rid of user-accessible PCI has a similar benefit for Apple - no more hassle having to support a zoo of PCI cards.

pmjordan 13y ago
Except there are Thunderbolt-to-PCIe enclosures, which come with their own set of problems (latency, hotplugging, I/O memory range allocation, bandwidth, not to mention cost). But I guess Apple can just say "we don't support those, take it up with the enclosure manufacturer". Drivers for Thunderbolt devices still are necessarily kexts and can thus take down the system without an obvious culprit. For this reason alone I'm surprised they went with it rather than supporting USB3 early on or that they didn't come up with something similar but altogether new, like they did with Lightning. They seem to be trying to cut down third party kext development, and aren't opening the sources for most of their own new kexts. Of course, with IOMMUs available in all Thunderbolt systems, they could have brought back the microkernel aspect of Mach for device drivers, but that would have been even more work.

seanmcdirmid 13y ago

What does NUMA have to do with SMP setups? NUMA would be about CPUs with separate physical memories acting as one weird processing resource, not a Mac Pro with multiple processes connected to the same memory. What am I missing here?

pmjordan 13y ago
All modern CPUs have memory controllers built in as opposed to using a distinct "northbridge" chip. The RAM attaches directly to the CPU socket (or soldered BGA). In multi-socket systems, this means there are memory buses coming off each CPU, and for one CPU to reach memory attached to the other CPU, it needs to go through that CPU via the interconnect. Here's some ASCII art of a system with 3 memory channels from each CPU (old Mac Pro): +------+ +------+ | CPU0 +==interconnect==+ CPU1 | +-+-+-++ ++-+-+-+ | \ \ / / | MEM0 | \ / | MEM5 MEM1 MEM2 MEM3 MEM4 Going via the interconnect introduces extra latency and consumes its bandwidth. Therefore, it's more efficient to pin threads and/or processes to specific CPU sockets, and to allocate memory for it from that socket's associated physical memory. This requires kernel support, and for absolute top performance you also want a user space API, so you don't need to rely on the in-kernel heuristics. Lacking software support means you have to fall back to whatever the hardware does automatically. Current multi-socket x86 CPUs all offer an "interlaced" mode for this (sort of like RAID0), where they dispatch to physical socket based on some physical address bits, e.g.: 0x00000000-0x0000003f: CPU0's RAM 0x00000040-0x0000007f: CPU1's RAM 0x00000080-0x000000bf: CPU0's RAM 0x000000c0-0x000000ff: CPU1's RAM and so on. This spreads the extra latency evenly. Makes for consistent but poor memory performance. The mode is selected during system initialisation, i.e. is the responsibility of the Firmware (BIOS/EFI). Non-Apple workstations and servers typically let you select this as part of the pre-boot system setup UI. It depends on the motherboard layout, but you'll find that there are groups of DIMM slots located near each CPU socket in multi-socket systems. Physical proximity helps keep speeds high.
- KaiserPro 13y ago
  The other thing to mention here is with PCIe 3, the controller for the bus is now in the CPU, which means that if you have two GPUs, you'll start to see GPU affinity problems as well. In previous systems you'd have one bus attached to a southbridge, which is then shared by both CPUs* I've seen this with linux and nvidia cards, with two CPUs we would get a maximum CPU->GPU transfer of around 7-8gigs, take out a CPU and it'd bump up to 10-13. The z620 has the second CPU on a daughter card, so its simple to do. *my memory is a bit hazy, I'm open to corrections

sebbean 13y ago

Am I the only one curious as to what NUMA stands for?

LukeHoersten 13y ago
Non-uniform memory access http://en.m.wikipedia.org/wiki/Non-Uniform_Memory_Access http://en.m.wikipedia.org/wiki/Non-Uniform_Memory_Access