3 ms·
> We have NICs with 32 or 64 queues in boxes having 88 or 128 CPUs. Doesn't that mean that you should have more NICs? From my recollection, the main thing tha
by derefr 7y ago
> We have NICs with 32 or 64 queues in boxes having 88 or 128 CPUs.
Doesn't that mean that you should have more NICs?
From my recollection, the main thing that's made a computer a server or mainframe for the last 50 years has been that such systems have enough IO channels to saturate the CPU (i.e. they're intended to be CPU-bound, not IO-bound, under heavily-concurrent workloads.) If you take all the IO-offload cards out of a mainframe, it's just a regular computer.
- xxpor 7y agoPCIe Gen 3 is becoming a real problem here. There's only so many lanes you get per socket, it's completely possible today to end up being IO bound just due to PCIe bandwidth, not network speeds or CPU processing power. For example, this Cascade Lake Xeon has 48 lanes of PCIe gen 3: https://ark.intel.com/content/www/us/en/ark/compare.html?productIds=192458 https://ark.intel.com/content/www/us/en/ark/compare.html?pro... If you give yourself 32 of 48 lanes for networking, that's only 256 gbits of PCIe bandwidth. So each 16x slot is limited to a 100 gbit card. And that's a 22 core processor. With DPDK, coming up with a application that could process 200 gbit is almost trivial.
- magicalhippo 7y agoNot really my area of expertise by any means, but I got a bit excited by this talk[1]. The idea is that a PCIe switch can be used to send data directly from PCIe to PCIe device without hitting the host controller. NICs are mentioned as a use-case. While the talk is centered around RISC-V, there's nothing RISC-V specific about this, just that it enables whimpy cores to control multi-Gbps traffic. [1]: https://www.youtube.com/watch?v=LDOlqgUZtHE https://www.youtube.com/watch?v=LDOlqgUZtHE (Accelerating Computational Storage Over NVMe with RISC V)
- jeffbee 7y agoMaybe, but I don’t see why you’d want so many queues. If you are receiving 1 million frames per second, each 10kb long (10gb/s) and you spread that over 64 receive queues you will process exactly one packet per interrupt, maximizing the cost. If you have only four receive queues (one for every cache domain on a Zen2 CPU socket as an example) then if you set your interrupt mitigation to something modest like 10 microseconds then you will receive several frames per interrupt, and additionally you will not be polluting the L1i cache of the other 60 CPU cores.