The other thing to mention here is with PCIe 3, the controller for the bus is now in the CPU, which means that if you have two GPUs, you'll start to see GPU aff

The other thing to mention here is with PCIe 3, the controller for the bus is now in the CPU, which means that if you have two GPUs, you'll start to see GPU affinity problems as well. In previous systems you'd have one bus attached to a southbridge, which is then shared by both CPUs* I've seen this with linux and nvidia cards, with two CPUs we would get a maximum CPU->GPU transfer of around 7-8gigs, take out a CPU and it'd bump up to 10-13. The z620 has the second CPU on a daughter card, so its simple to do. *my memory is a bit hazy, I'm open to corrections