3 ms·
My masters thesis is building a super low-latency, lock-less IPC mechanism for passing cache lines between cores. NUMA systems are crazy, for numerous reasons.
by Moral_ 11y ago
My masters thesis is building a super low-latency, lock-less IPC mechanism for passing cache lines between cores. NUMA systems are crazy, for numerous reasons. For instance requesting remote cache lines across sockets have asymmetric latencies. For example on the r820 server we use for testing requesting a cache line from socket 1 to socket 0 (Socket 1 receives the cache line from a core on socket 0) the latency is 528 cycles. But if you reverse the experiment and socket 0 receives a cache line from socket 1 the latency is 384 cycles. So when building an application, if you're into reducing cycles things like this creep up on you and become very confusing until you do experiments like above.
- vardump 11y agoAre you sure you measured correctly without systematic errors caused by out of order execution? Did you serialize execution by using RDTSCP or CPUID? Any chance you accidentally compared TSC counts between two separate sockets? Did you keep in mind that each CPU socket has a different unsynchronized TSC. Not accounting for those could give you an illusion of asymmetry.
- Moral_ 11y agoYeah, when we got results we didn't expect we made sure that the experiment was setup correctly, and actually made sense. We use the timing instructions illustrated in this intel white paper, which does do serialization http://www.intel.com/content/www/us/en/embedded/training/ia-32-ia-64-benchmark-code-execution-paper.html http://www.intel.com/content/www/us/en/embedded/training/ia-.... More over we tried a few different programs, CCbench, custom home-brewed, and BenchIT, all giving the same results. There's a paper about Asymmetry: Thread and Memory Placement on NUMA Systems: Asymmetry Matters https://www.usenix.org/system/files/conference/atc15/atc15-paper-lepers.pdf https://www.usenix.org/system/files/conference/atc15/atc15-p...