4 ms·
NUMA is part of the reality. There's lots of modern computing that doesn't acknowledge reality, but that doesn't tend to do good things for performance. Commun
by toast0 5y ago
NUMA is part of the reality. There's lots of modern computing that doesn't acknowledge reality, but that doesn't tend to do good things for performance.
Communication isn't cheap, and implicit communication makes it hard to tell what's expensive. Reading from a pointer could be from L1 cache, or it could be from a pci-e card attached to another socket. The code looks the same, but the cost is radically different. Organizing how your code operates in memory to avoid cross-numa and cross-cpu in general isn't always easy, but it's required if you want to get the performance you paid for. OTOH, smaller nodes get this for free: my dual core desktop 'servers' don't have to worry about NUMA and don't have a lot of cross-cpu contention either.
- dragontamer 5y ago> Reading from a pointer could be from L1 cache, or it could be from a pci-e card attached to another socket. The fun one is TLB (translation lookaside buffers) and the virtual memory system. Today's AMD core's have more L3 cache than what the TLB can handle with 4k-pages. You need to enable 2MB hugepages or 1GB hugepages to even access L3 cache at full speeds in practice... EDIT: Milan-X has 96MB L3 cache per CCX. 4kB-pages would require 24,000 (24-thousand) TLB-entries. IIRC, Milan only has 2000-TLB-entries. Hurraaahhhhhh.... ------ CPUs are devilishly complicated. It makes optimization "fun". Apparently, running "memcpy" requires Ph.D levels of study before you can "memcpy" at full speeds these days.