4 ms·

I wonder if we will see more of the network layer move into the application layer, like what I think happened with QUIC. I’m not really a network guy, but the

I wonder if we will see more of the network layer move into the application layer, like what I think happened with QUIC. I’m not really a network guy, but the thing I find interesting about QUIC is that it’s upgradable. Normalising regular browser updates was a godsend for the web, while TCP/IP remains hard to upgrade because you still, in 2023, have to upgrade your whole OS. Protocols like TLS that sit outside the OS seem to have been much more dynamic, and that too is the promise of QUIC. It might be interesting to find operating system primitives that enable use of the networking hardware without having to implement the protocol itself. Although that stuff is above my pay grade.

pclmulqdq 3y ago
This is where a lot of high-performance stuff went in the last decade or so. HPC, HFT, AI training, several storage services, and clouds pretty much all do their networking in something other than a kernel. It turns out you can get huge performance gains by not using a one-size-fits-all networking system. It only takes a few more steps for this model to trickle down to general applications, as HTTP and RPC stacks pick it up.
ElectricalUnion 3y ago
Or the kernels becomes even less of a monolith on all fronts and more userland-like. Both io_uring and eBPF would allow us to handle more complex tasks inside the kernel, reducing context switch losses yet still allowing the benefits of a upgrade-able structure without having to upgrade the OS or even the kernel itself.
- doctor_eval 3y ago
  Yeah. It strikes me that the lack of a stable Linux kernel ABI is part of the reason why we can’t easily upgrade bits of it. Upgrading a kernel module would be a good solution (could even be done by applications) but IIUC its infeasible. But still, it seems conceivable that a networking kernel module could talk exclusively to an adaptor shim that is part of the kernel. idk, I actually haven’t so much as compiled a kernel since the late 90s, so I’m pretty much talking out the wrong end :)
phicoh 3y ago
The need to upgrade the OS is only a small part of the problem. On the one hand, the fact the TCP is unchanging has led to network cards supporting TCP directly. You can shove a large segment of data in the network card and it will split the data up in TCP segments and transmit them. There are also big content companies that let the kernel do all TLS encryption, in some cases that can also be offloaded to hardware. The big problem for TCP is all the boxes on the internet that understand TCP and don't understand TCP extensions. That could be home routers doing NAT, firewalls, etc. Those boxes are not only hard to upgrade, they also break things. QUIC fixes this by making sure that as much as possible is encrypted. Some ISPs monitor TCP headers and from retransmissions they can figure out where problems are in the network. QUIC will take that ability away. So QUIC is a mixed blessing.
- doctor_eval 3y ago
  I disagree. If there’s a financial incentive (better performance or whatever) then the server side will upgrade; it’s a financial and timing decision with strong motivators. But the client side problem is 5-6 orders of magnitude larger than the server side problem - and to make things worse, they basically don’t care. People seem happy enough to upgrade their browser, but upgrading their OS is a big deal. Even I don’t like doing it, and I consider it important. So I think a reasonable rule of thumb would be, the server side will take care of itself. But if you need your clients to reboot their computer to upgrade the network stack to improve your server performance, well, it ain’t gonna happen.
LaLaLand122 3y ago
> I wonder if we will see more of the network layer move into the application layer, like what I think happened with QUIC. https://lwn.net/Articles/169961/ https://lwn.net/Articles/169961/

pclmulqdq 3y ago

This is where a lot of high-performance stuff went in the last decade or so. HPC, HFT, AI training, several storage services, and clouds pretty much all do their networking in something other than a kernel. It turns out you can get huge performance gains by not using a one-size-fits-all networking system. It only takes a few more steps for this model to trickle down to general applications, as HTTP and RPC stacks pick it up.

ElectricalUnion 3y ago

Or the kernels becomes even less of a monolith on all fronts and more userland-like. Both io_uring and eBPF would allow us to handle more complex tasks inside the kernel, reducing context switch losses yet still allowing the benefits of a upgrade-able structure without having to upgrade the OS or even the kernel itself.

doctor_eval 3y ago
Yeah. It strikes me that the lack of a stable Linux kernel ABI is part of the reason why we can’t easily upgrade bits of it. Upgrading a kernel module would be a good solution (could even be done by applications) but IIUC its infeasible. But still, it seems conceivable that a networking kernel module could talk exclusively to an adaptor shim that is part of the kernel. idk, I actually haven’t so much as compiled a kernel since the late 90s, so I’m pretty much talking out the wrong end :)

phicoh 3y ago

The need to upgrade the OS is only a small part of the problem. On the one hand, the fact the TCP is unchanging has led to network cards supporting TCP directly. You can shove a large segment of data in the network card and it will split the data up in TCP segments and transmit them. There are also big content companies that let the kernel do all TLS encryption, in some cases that can also be offloaded to hardware. The big problem for TCP is all the boxes on the internet that understand TCP and don't understand TCP extensions. That could be home routers doing NAT, firewalls, etc. Those boxes are not only hard to upgrade, they also break things. QUIC fixes this by making sure that as much as possible is encrypted. Some ISPs monitor TCP headers and from retransmissions they can figure out where problems are in the network. QUIC will take that ability away. So QUIC is a mixed blessing.

doctor_eval 3y ago
I disagree. If there’s a financial incentive (better performance or whatever) then the server side will upgrade; it’s a financial and timing decision with strong motivators. But the client side problem is 5-6 orders of magnitude larger than the server side problem - and to make things worse, they basically don’t care. People seem happy enough to upgrade their browser, but upgrading their OS is a big deal. Even I don’t like doing it, and I consider it important. So I think a reasonable rule of thumb would be, the server side will take care of itself. But if you need your clients to reboot their computer to upgrade the network stack to improve your server performance, well, it ain’t gonna happen.

LaLaLand122 3y ago

> I wonder if we will see more of the network layer move into the application layer, like what I think happened with QUIC. https://lwn.net/Articles/169961/ https://lwn.net/Articles/169961/