3 ms·
> The cross socket NUMA bandwidth is like a tiny straw. And the Internet is a series of tubes!
by redis_mlc 6y ago
> The cross socket NUMA bandwidth is like a tiny straw.
And the Internet is a series of tubes!
- StillBored 6y agoHow about a ballpark. The unidirectional bandwidth across a two socket system is roughly equal to a single DDR channel. Which is why you see postings on the intel forums about people complaining about their perf falling off a cliff if the hardware decides to flag the directory as requiring cross socket snoop (or whatever its actually doing). So even if you have done a reasonable job keeping your thread+data locally, doing a remote reference, or accidentally running the thread for a short while on the wrong socket can permanently lower the performance. See https://software.intel.com/en-us/forums/software-tuning-performance-optimization-platform-monitoring/topic/814772 https://software.intel.com/en-us/forums/software-tuning-perf... for an example about a contented cache line. Thankfully the latest products are mostly 10.4GT (gold and above) as it appears intel isn't doing as much product segmentation based on UPI link speeds anymore (silver/bronze are still slower though). https://www.microway.com/knowledge-center-articles/performance-characteristics-of-common-transports-buses/ https://www.microway.com/knowledge-center-articles/performan...
- kllrnohj 6y ago> The unidirectional bandwidth across a two socket system is roughly equal to a single DDR channel Depends on the implementation. Epyc Rome's 2P socket design is 64 PCI-E 4.0 lanes between the two CPUs. That's 128 GB/s bandwidth, which is roughly equivalent to 6-channel DDR4 2666.
- StillBored 6y agoYes, I should have been clearer that I was speaking of intel products (which is what the article was talking about) not NUMA in general.