3 ms·

One Interesting quirk we found with Nvidia cards was the CPU-GPU transfer speed was affected by how many physical CPUs you had. I assumed it was because with n

One Interesting quirk we found with Nvidia cards was the CPU-GPU transfer speed was affected by how many physical CPUs you had. I assumed it was because with newer intel CPUs the PCI bridge moved to the die, away from the motherboard, so there was CPU affinity for the GPU. The difference in bandwidth was around 15%

dom0 10y ago
There is also a latency penalty for accessing PCIe devices that are attached to a foreign root complex. This is exacerbated if the QPI links are already busy.
zbjornson 10y ago
Which way was the association, more CPUs -> faster CPU-GPU transfer, or slower? By "physical CPUs" I assume you mean cores in a single socket, but in pre-Haswell multi-socket systems, the uncore clock speed was tied to the core clock speed, so idle CPUs that enter a low power state can actually cause horrible performance because cache coherence snoop signals have to wait for the low-power/slow CPUs to respond. See https://software.intel.com/en-us/forums/intel-moderncode-for-parallel-architectures/topic/379378 https://software.intel.com/en-us/forums/intel-moderncode-for...
- KaiserPro 10y ago
  sorry, yes, that was the most important part! single cpu was faster than dual. This was a xeon x56 vs an e5-26 On the z620 the second proc was on a removable daughter board, so it was simple test. on the e5-26 it was most pronounced, the x56 was less obvious
qb45 10y ago
> I assumed it was because with newer intel CPUs the PCI bridge moved to the die, away from the motherboard, so there was CPU affinity for the GPU. Actually, the PCIe controller doesn't even have to be on-die. The same issue existed in earlier systems where PCIe was in the northbridge, which was attached to the first socket. Sometimes other sockets had their own NBs and own PCI buses too and things were even more interesting. I heard years ago about people pinning their GPU-accelerated applications to the "right" CPUs.

dom0 10y ago

There is also a latency penalty for accessing PCIe devices that are attached to a foreign root complex. This is exacerbated if the QPI links are already busy.

zbjornson 10y ago

Which way was the association, more CPUs -> faster CPU-GPU transfer, or slower? By "physical CPUs" I assume you mean cores in a single socket, but in pre-Haswell multi-socket systems, the uncore clock speed was tied to the core clock speed, so idle CPUs that enter a low power state can actually cause horrible performance because cache coherence snoop signals have to wait for the low-power/slow CPUs to respond. See https://software.intel.com/en-us/forums/intel-moderncode-for-parallel-architectures/topic/379378 https://software.intel.com/en-us/forums/intel-moderncode-for...

KaiserPro 10y ago
sorry, yes, that was the most important part! single cpu was faster than dual. This was a xeon x56 vs an e5-26 On the z620 the second proc was on a removable daughter board, so it was simple test. on the e5-26 it was most pronounced, the x56 was less obvious

qb45 10y ago

> I assumed it was because with newer intel CPUs the PCI bridge moved to the die, away from the motherboard, so there was CPU affinity for the GPU. Actually, the PCIe controller doesn't even have to be on-die. The same issue existed in earlier systems where PCIe was in the northbridge, which was attached to the first socket. Sometimes other sockets had their own NBs and own PCI buses too and things were even more interesting. I heard years ago about people pinning their GPU-accelerated applications to the "right" CPUs.