3 ms·
NUMA pin the DMA buffers too. I didn't see that in the article. With the app I worked on getting the PCI bus/CPU/memory/interrupt all pinned to the "right" node
by StillBored 11y ago
NUMA pin the DMA buffers too. I didn't see that in the article. With the app I worked on getting the PCI bus/CPU/memory/interrupt all pinned to the "right" node gave us a good 30% performance improvement.
Put another way, identify which socket the PCIe bridge and device are attached to, then make sure the DMA buffers, and interrupt handler are pinned to that node.
Also, don't touch the buffers from the other socket. Strangely enough touching the DMA buffers from the other node just once was enough to cause a performance fall off.
- monocasa 11y agoProbably because the cache line changes from Exclusive to Owned in MOESI.
- StillBored 11y agoYah, that is the obvious choice, but it wasn't as clear cut. Especially with the behavior we were seeing where the performance problems would remain long after the lines were flushed from the CPU caches. I spent a fair amount of time looking at Intel docs on snoop filters and the like and never really satisfied myself that I knew exactly what was causing the problem.
- wyldfire 11y agoYeah, with previous generations of chipsets/CPUs, I couldn't saturate quad-10GbE receive streams unless I bound the buffers to particular nodes and IRQs to corresponding cores.