4 ms·

What tasks of a TCP stack actually need to coordinate between processes, or in other words, actually need a kernel? From what I understand, complete kernel byp

What tasks of a TCP stack actually need to coordinate between processes, or in other words, actually need a kernel? From what I understand, complete kernel bypass networking is only really feasible if you only have a single process that is using the network. And with a monolithic kernel, the TCP stack needs the kernel for port allocation and load balancing the demands on the network interface. Okay, that makes sense, but why can't most of the processing be offloaded to userspace, with the kernel just doing those specific tasks? Port allocation should happen very rarely, and throttling should only ever happen when the network interface is saturated. Wouldn't it be better to limit the kernel to just those tasks, and offload the rest to userspace?

MikeTheGreat 6y ago
I don't have great references, but I know that this has been researched in the past. For example: "U-Net: A User-Level Network Interface for Parallel and Distributed Computing" [0] ------ [0] https://www.cs.rice.edu/~alc/old/comp520/papers/U-Net%20A%20User.pdf https://www.cs.rice.edu/~alc/old/comp520/papers/U-Net%20A%20...
tsimionescu 6y ago
There are many ways to look at this. One of the simpler ones is looking at the Rx side: a frame has arrived on the network interface. Something needs to decode that and decide what to do with it, perhaps dropping it immediately. The TCP/IP stack will interpret the Ethernet headers, drop the frame if it is not addressed to any of the machines, check checksums, handle ARP requests etc. Also, the stack would buffer the frames until it can reassemble an entire IPv4 datagram out of them. Then, it will perform similar checks on the datagram destination address as above. Finally, with the IP datagram in hand, we can decide whether to hand the payload over to the TCP handler, the UDP handler, the ICMP handler, or perhaps others. Of course, we need some TCP knowledge as well, to allow multiple TCP handlers to co-exist, choosing the right one to forward the packet to by port. At all of these steps there are other concerns that need to be handled. Network security tools should get to look at the IP addresses and MACs to decide if the packet is accepted at all. Raw sockets created by user space can receive the Ethernet frames or IP datagrams directly, before other processing. There can also be contention on those as well (e.g. Maybe 2 processes both open a raw socket on the same NIC - who gets the bytes?). Multiple NICs on the same machine also need to be shared and load-balanced. The Linux kernel for example handles routing between these, even applying networking decisions such as advertising IPs from one NIC on other NICs when it thinks it can improve delivery rates. Now, all of these can and have been done in user-space for various reasons. But it's important to understand that there is a large amount of processing. Also, given the importance of networking to IPC and security, there are also security considerations at almost every step of the way. For example, if two privileged processes on the same machine are communicating over a virtual NIC with static IP allocation, can they expect their communication to be isolated from a non-privileged process running on the same machine, or do they need to use an encrypted&authenticated scheme? A centralized TCP/IP stack can make that guarantee, a per-process stack can't.
wmf 6y ago
Some classic work on this is the MIT Exokernel project. They wanted to use Solarflare-style NICs to steer connections but since those weren't invented yet they used BPF-style code. https://pdos.csail.mit.edu/archive/exo/ https://pdos.csail.mit.edu/archive/exo/ https://pdos.csail.mit.edu/6.828/2008/readings/engler95exokernel.pdf https://pdos.csail.mit.edu/6.828/2008/readings/engler95exoke...
vlovich123 6y ago
To do user space IP you need raw sockets (to build the TCP headers). Raw sockets require root access or special privileges for your process as the kernel typically is the only one allowed to build such headers as TCP requires good network behavior. You don’t want trivial compromises of user-space apps to suddenly give malicious code to send arbitrary TCP packets that may be exploiting some property of TCP stacks to DDOS more effectively or to misbehave on the behavioral side of TCP things like ignore flow control. That’s a huge part of why the kernel implements it. The other part is that network stacks typically can be more tightly coupled with network drivers for performance (again trusted context so less/no validation needed for the buffers being shared and whatnot). Now there are obviously cases where user space can do a better job of buffer management for better latency/throughout so the perf aspect isn’t that conclusive.
tenebrisalietum 6y ago
Sequencing is important. IP doesn't guarantee order; the kernel takes the segments of a TCP connection and then delivers them in order to the application as it reads from the socket. It will hold segments that arrive early and block if a segment is missing. If you don't care about the order you are receiving things then you want UDP. Is it better for userspace to do the sequencing? Maybe given NUMA - a process might be "closer" to its memory than the kernel and therefore could eat its traffic in the correct order locally faster than being fed by the kernel.
Matthias247 6y ago
> What tasks of a TCP stack actually need to coordinate between processes, or in other words, actually need a kernel? In short: You want to have something that does the multiplexing and demultiplexing of packet streams that belong to different applications. E.g. all packets arrive from a common NIC, but depending on the target port they have to a different application. And on the sending side you want to have some fairness, and common congestion control. If all applications could write directly to the TX path of the NIC it could easily be that packets from one particular application which has less CPU time would commonly get dropped. In addition for that having a common place like the Kernel allows for some nice monitoring/instrumentation purposes. And it makes sure peers get a reasonable response (even if that's only a RST) if the application is not started.

MikeTheGreat 6y ago

I don't have great references, but I know that this has been researched in the past. For example: "U-Net: A User-Level Network Interface for Parallel and Distributed Computing" [0] ------ [0] https://www.cs.rice.edu/~alc/old/comp520/papers/U-Net%20A%20User.pdf https://www.cs.rice.edu/~alc/old/comp520/papers/U-Net%20A%20...

tsimionescu 6y ago

There are many ways to look at this. One of the simpler ones is looking at the Rx side: a frame has arrived on the network interface. Something needs to decode that and decide what to do with it, perhaps dropping it immediately. The TCP/IP stack will interpret the Ethernet headers, drop the frame if it is not addressed to any of the machines, check checksums, handle ARP requests etc. Also, the stack would buffer the frames until it can reassemble an entire IPv4 datagram out of them. Then, it will perform similar checks on the datagram destination address as above. Finally, with the IP datagram in hand, we can decide whether to hand the payload over to the TCP handler, the UDP handler, the ICMP handler, or perhaps others. Of course, we need some TCP knowledge as well, to allow multiple TCP handlers to co-exist, choosing the right one to forward the packet to by port. At all of these steps there are other concerns that need to be handled. Network security tools should get to look at the IP addresses and MACs to decide if the packet is accepted at all. Raw sockets created by user space can receive the Ethernet frames or IP datagrams directly, before other processing. There can also be contention on those as well (e.g. Maybe 2 processes both open a raw socket on the same NIC - who gets the bytes?). Multiple NICs on the same machine also need to be shared and load-balanced. The Linux kernel for example handles routing between these, even applying networking decisions such as advertising IPs from one NIC on other NICs when it thinks it can improve delivery rates. Now, all of these can and have been done in user-space for various reasons. But it's important to understand that there is a large amount of processing. Also, given the importance of networking to IPC and security, there are also security considerations at almost every step of the way. For example, if two privileged processes on the same machine are communicating over a virtual NIC with static IP allocation, can they expect their communication to be isolated from a non-privileged process running on the same machine, or do they need to use an encrypted&authenticated scheme? A centralized TCP/IP stack can make that guarantee, a per-process stack can't.

wmf 6y ago

Some classic work on this is the MIT Exokernel project. They wanted to use Solarflare-style NICs to steer connections but since those weren't invented yet they used BPF-style code. https://pdos.csail.mit.edu/archive/exo/ https://pdos.csail.mit.edu/archive/exo/ https://pdos.csail.mit.edu/6.828/2008/readings/engler95exokernel.pdf https://pdos.csail.mit.edu/6.828/2008/readings/engler95exoke...

vlovich123 6y ago

To do user space IP you need raw sockets (to build the TCP headers). Raw sockets require root access or special privileges for your process as the kernel typically is the only one allowed to build such headers as TCP requires good network behavior. You don’t want trivial compromises of user-space apps to suddenly give malicious code to send arbitrary TCP packets that may be exploiting some property of TCP stacks to DDOS more effectively or to misbehave on the behavioral side of TCP things like ignore flow control. That’s a huge part of why the kernel implements it. The other part is that network stacks typically can be more tightly coupled with network drivers for performance (again trusted context so less/no validation needed for the buffers being shared and whatnot). Now there are obviously cases where user space can do a better job of buffer management for better latency/throughout so the perf aspect isn’t that conclusive.

tenebrisalietum 6y ago

Sequencing is important. IP doesn't guarantee order; the kernel takes the segments of a TCP connection and then delivers them in order to the application as it reads from the socket. It will hold segments that arrive early and block if a segment is missing. If you don't care about the order you are receiving things then you want UDP. Is it better for userspace to do the sequencing? Maybe given NUMA - a process might be "closer" to its memory than the kernel and therefore could eat its traffic in the correct order locally faster than being fed by the kernel.

Matthias247 6y ago

> What tasks of a TCP stack actually need to coordinate between processes, or in other words, actually need a kernel? In short: You want to have something that does the multiplexing and demultiplexing of packet streams that belong to different applications. E.g. all packets arrive from a common NIC, but depending on the target port they have to a different application. And on the sending side you want to have some fairness, and common congestion control. If all applications could write directly to the TX path of the NIC it could easily be that packets from one particular application which has less CPU time would commonly get dropped. In addition for that having a common place like the Kernel allows for some nice monitoring/instrumentation purposes. And it makes sure peers get a reasonable response (even if that's only a RST) if the application is not started.