3 ms·
Most devs I know are creating async workloads which don't require cache coherency, as they use parallelism to parallel process separate requests and workloads.
by Nursie 7y ago
Most devs I know are creating async workloads which don't require cache coherency, as they use parallelism to parallel process separate requests and workloads. I can see things being pretty linear in that sort of space.
- gmueckl 7y agoExcept that they are typically not. The Zen architectures are NUMA and controlling where memory is allocated is key to decent threaded performance. You may even have to do seemingly counterintuitive things like duplicating central data structures across nodes and other tricks from the distributed systems playbook.
- coder543 7y agoEpyc 2's memory layout is not like Epyc 1. Epyc 2 is very simple.
- jdsully 7y agoYup everything is equally slow now. Kinda sad, but the original NUMA design was treated as a glass half empty situation instead of AMD letting people maximize performance. This change lets them avoid the bad press and everyone is happier despite the final design being slower than it could have been.
- gmueckl 7y agoEpyc 2 has different memory latencies within and across NUMA nodes according to the infirmation I have. So it is not equally slow for all memory. Can you point me to a source that says otherwise? Edit: my source is this German article: https://www.heise.de/newsticker/meldung/AMD-Server-CPUs-Epyc-7002-Mit-Zen-2-an-Intel-vorbei-4489732.html https://www.heise.de/newsticker/meldung/AMD-Server-CPUs-Epyc...
- jdsully 7y agoSee the architecture diagram here: https://www.anandtech.com/show/14694/amd-rome-epyc-2nd-gen/2 https://www.anandtech.com/show/14694/amd-rome-epyc-2nd-gen/2 Everything goes through the central crossbar on the I/O die, where Zen1 had memory attached directly to each CPU chiplet which would relay as necessary. On Zen1 if you accessed direct attached memory you wouldn't pay the latency penalty from relaying the data. In Zen2 all data is relayed via the I/O die with the associated delay that entails.
- gmueckl 7y agoI did some more digging. It seems like the Linux NUMA topology shown in the anandtech article is a deliberate lie. There are different latencies between cores and memory comtrollers on the same socket, but these are deemed to be insignificant enough to not expose them in the reported NUMA topology.
- jdsully 7y agoThat is true with Intel chips as well. In the HFT space people actively work with Intel to determine which cores they should pin tasks to. The speed of light is constant, and some cores will always be a little closer to various resources.
- shaklee3 7y agoThat was true before Skylake, but is no longer true since they moved away from the multi ring architecture.
- jdsully 7y agoEven with the mesh the number of hops is variable based on which core is requesting and the physical geometry of the chip. The cores right beside the IMC will have the lowest latency. See this diagram: https://en.wikichip.org/wiki/intel/mesh_interconnect_architecture https://en.wikichip.org/wiki/intel/mesh_interconnect_archite... The main improvement is the max number of hops is log(n) instead N/2.
- cjhanks 7y agoThey are not linear unless all requests take an identical amount of time OR the system is not oversubscribed (common in many workloads) - and even then, the current linux CFS scheduler has a complexity of `O(log N)`. When you have variable length requests, you will find cores will not always be balanced, it is simply a statistical reality. And in those cases, the kernel will have to migrate your process to a different core, and if you have 256 cores, that core might be really far away.