3 ms·
Just as information for some later readers, but this is applicable not just to Epyc but also the consumer version Threadripper. My understanding is that this is
by Ataraxic 5y ago
Just as information for some later readers, but this is applicable not just to Epyc but also the consumer version Threadripper. My understanding is that this is an example of a Non-Uniform Memory Access (NUMA) which was also used to link multiple cpus in different sockets together for a long time now, but now they've been integrated into a cpu that fits in one socket.
This actually has an importance if you are running a VM on such a system since you'll run into things like the actual RAM (not l3 cache) is often directly linked to a particular NUMA node. For example accessing memory in the first ram stick vs the second will give different latencies as it goes from ccx1 => ccx2 => stick2 versus ccx1 => stick1. This is applicable to I think 2XXX versions and earlier for threadripper. My understanding is that they solved this in later versions using the infinity fabric (IO die) so now all ccx's go through the IO die.
I ran into all of this trying to run an ubuntu machine that ran windows using KVM while passing through my nvidia graphics card.
- dagmx 5y agoThis also applies to Ryzen as well. The first gen Ryzen was plagued by software and schedulers that couldn't handle the ccx affinity
- reilly3000 5y agoThis sheds some light on why I had so much trouble running GPU passthrough with Unraid (KVM) on my 1950X w/ Nvida. Those were dark days.
- rob_c 5y agoFrankly that was probably 200% unraid ...
- masklinn 5y ago> Just as information for some later readers, but this is applicable not just to Epyc but also the consumer version Threadripper. It's applicable to any Zen design with more than one CCX, which is... any Zen 3 CPU of more than 8 cores (in Zen 2 it was 4). The wiki has the explanation under the "Core config" entry of the Zen CPUs, but for the 5000s it's all the 59xx (12 and 16 cores). Zen 3 APU are all single-CCX, though there are Zen 2 parts in the 5000 range which are multi-CCX (because why not confuse people): the 6-cores 5500U is a 2x3 and the 8-core 5700U is a 2x4. The rest is either low-core enough to be single-CCX Zen 2 (5300U) or topping out at a single 8-cores CCX (everything else).
- moonchild 5y ago> NUMA If this is NUMA, then so is any CPU with hyperthreading, as the hyperthreaded cores share L1.