4 ms·

The paper talks a lot about pushing Linux out of the way in the receive path (including to avoid copies), only to have a demultiplexing thread steer packets onc

by _wmd 11y ago

The paper talks a lot about pushing Linux out of the way in the receive path (including to avoid copies), only to have a demultiplexing thread steer packets once they arrive in userspace. Linux has PACKET_MMAP for a long time and modern linux has RPS, which would seem to exactly duplicate this functionality without the copies or single point of contention (one thread running on one core, vs. N interrupt queues directly waking threads pinned on N cores). Can anyone comment on why they aren't using it? > Because the NIC is no longer the bottleneck, this figure shows the upper bound of Maglev throughput with the current hardware, which is slightly higher than 15Mpps. In fact, the bottleneck here is the Maglev steering mod- ule, which will be our focus of optimization when we switch to 40Gbps NICs in the future Sounds like they could use RPS

asuffield 11y ago
(Tedious disclaimer: my opinion only, not speaking for anybody else. I'm an SRE at Google, and I'm oncall for this service.) On the understanding that I can't tell you any more than what it already says in the paper, the key detail that you're looking for is section 4.1.2: """ Maglev originally used the Linux kernel network stack for packet processing. It had to interact with the NIC using kernel sockets, which brought significant overhead to packet processing including hardware and software interrupts, context switches and system calls [26]. Each packet also had to be copied from kernel to userspace and back again, which incurred additional overhead. Maglev does not require a TCP/IP stack, but only needs to find a proper backend for each packet and encapsulate it using GRE. Therefore we lost no functionality and greatly improved performance when we introduced the kernel bypass mechanism – the throughput of each Maglev machine is improved by more than a factor of five. """ My understanding is that RPS just steers packets into the Linux TCP stack. Nothing on the maglev machine wants to process the TCP layer; that workload can be distributed over the backends.
thrownaway2424 11y ago
Google contributed RPS to Linux, so it's safe to assume they are aware of these facts and have chosen to do it this way despite that knowledge.
eisenbud 11y ago
Lots of people are doing userspace networking to go faster. See netmap, DPDK, for other examples, or look at https://www.cl.cam.ac.uk/research/security/ctsrd/pdfs/201408-sigcomm2014-specialization.pdf https://www.cl.cam.ac.uk/research/security/ctsrd/pdfs/201408... In our case the kernel buys us very little but costs a lot, as others correctly pointed out in this thread. We spent a lot of time working on our performance via the kernel, with moderate success, but userspace talking directly to the NIC was a huge leap in performance.

by _wmd 11y ago

asuffield 11y ago

(Tedious disclaimer: my opinion only, not speaking for anybody else. I'm an SRE at Google, and I'm oncall for this service.) On the understanding that I can't tell you any more than what it already says in the paper, the key detail that you're looking for is section 4.1.2: """ Maglev originally used the Linux kernel network stack for packet processing. It had to interact with the NIC using kernel sockets, which brought significant overhead to packet processing including hardware and software interrupts, context switches and system calls [26]. Each packet also had to be copied from kernel to userspace and back again, which incurred additional overhead. Maglev does not require a TCP/IP stack, but only needs to find a proper backend for each packet and encapsulate it using GRE. Therefore we lost no functionality and greatly improved performance when we introduced the kernel bypass mechanism – the throughput of each Maglev machine is improved by more than a factor of five. """ My understanding is that RPS just steers packets into the Linux TCP stack. Nothing on the maglev machine wants to process the TCP layer; that workload can be distributed over the backends.

thrownaway2424 11y ago

Google contributed RPS to Linux, so it's safe to assume they are aware of these facts and have chosen to do it this way despite that knowledge.

eisenbud 11y ago

Lots of people are doing userspace networking to go faster. See netmap, DPDK, for other examples, or look at https://www.cl.cam.ac.uk/research/security/ctsrd/pdfs/201408-sigcomm2014-specialization.pdf https://www.cl.cam.ac.uk/research/security/ctsrd/pdfs/201408... In our case the kernel buys us very little but costs a lot, as others correctly pointed out in this thread. We spent a lot of time working on our performance via the kernel, with moderate success, but userspace talking directly to the NIC was a huge leap in performance.