6 ms·

Can we please stop running network drivers and network stacks in kernel mode by default? It's 2022 and we've got more than enough compute power nowadays that th

by boricj 4y ago

Can we please stop running network drivers and network stacks in kernel mode by default? It's 2022 and we've got more than enough compute power nowadays that the performance hit for running these in user-land is negligible for most use cases. Smartphone, tablet or laptop users usually do not need the level of performance that requires running that stuff in the kernel when browsing the web. I get that there are some use cases where performance really matters to the point where kernel network stack and drivers make a difference (high-throughput and/or low-latency services running on servers, high-performance routers...), but that should not be the default for everyone.

alfiedotwtf 4y ago
Obligatory comment: https://www.oreilly.com/openbook/opensources/book/appa.html https://www.oreilly.com/openbook/opensources/book/appa.html
- boricj 4y ago
  Note that the Tanenbaum-Torvalds debate was in 1992, over thirty years ago. The security of computer systems might have improved since then, but the fallout of security issues has massively increased. Managing your bank accounts wirelessly on the Internet from anywhere in the world with a thin, battery-powered device that fits inside your pocket was a pipe dream (and no one could've possibly imagined that an Internet-ready smart toaster could compromise it and steal your money on your accounts).
dijit 4y ago
1) Many will cry about that performance hit (including me). For over a decade our computers have gotten faster marginally, but our software has gotten slower at a greater rate. You can barely navigate the web now with a new low end computer (that isn't a Chromebook). Most on this site won't care though because our machines cost $2,000+ and the web is Fine(tm); many folk aren't buying anything over $300 though. 2) These are memory bugs, so the introduction of Rust into the kernel could help us here potentially, no need for an architectural revolution.
- boricj 4y ago
  > Many will cry about that performance hit (including me) because for over a decade our computers have gotten faster marginally, but our software has gotten slower and bloatier at an increasingly rapid pace. We're talking about network stacks and network drivers, not web browsers. Migrating the network stack from the kernel to a user-land process is not going to measurably slow down web browsers, especially on modern systems with gigabytes of RAM, multiple cores, IOMMUs and whatnots. > These are memory bugs, so the introduction of Rust into the kernel could help us here potentially, no need for an architectural revolution. That would require rewriting the network stack and network drivers in Rust (driver code is much more likely to have bugs than the rest of the kernel) for this to be effective, otherwise you'll still have a lot of C code in the network path. I'd argue that this would be a bigger architectural revolution than porting the existing code and running it in user-land. MINIX3 went through such a change when drivers were removed from the kernel (can't find the publication about it right now) and they only required reasonably small changes when porting these to user-land, there were not rewritten from scratch. But this is not just about memory safety, Rust code can still be vulnerable in many other ways (memory leaks, unsafe blocks, wrong assumptions, incorrect algorithm implementation, buggy/compromised toolchains...). Code running inside the trusted computing base of a system is a liability, enforcing privilege separation and principle of least authority reduces it.
  - dijit 4y ago
    > We're talking about network stacks and network drivers, not web browsers. Ah yes, the magic web-browser that doesn't do any kind of networking at all. > Migrating the network stack from the kernel to a user-land process is not going to measurably slow down web browsers, especially on modern systems with IOMMUs and whatnots. I don't know how you can possibly assert that, it's contradicting computer sciences' current understanding of operating system design as it relates to kernelmode/usermode switching, unless you're doing weird shared-memory things in userspace... which is terrifying. > That would require rewriting the network stack and network drivers in Rust Not really, C and Rust can interop just fine, you can have network drivers that are rust but the actual networking stack itself can remain C, if you want. > but this is not just about memory safety, Rust code can still be vulnerable in many other ways The post is literally memory safety bugs.
the_duke 4y ago
You really need to back up these assertions with some evidence. Do you have benchmarks that show the impact of switching to userspace on a typical, loaded desktop system with all kinds of workloads? Or are you just guessing?
- boricj 4y ago
  I did not anticipate a Hacker News discussion about a remotely exploitable Linux kernel WiFi vulnerability requiring some network benchmarks on an unusual network stack architecture, but I'll oblige: This publication (http://www.minix3.org/docs/jorrit-herder/asci06.pdf http://www.minix3.org/docs/jorrit-herder/asci06.pdf) claims that MINIX3 could saturate a 1 Gb/s Ethernet link with an user-space network stack, with separate processes for the stack and the driver, on a rusty 32-bit micro-kernel that can't do SMP. In 2006.
314 4y ago
What kind of latency would this introduce on packet processing? A context switch used to be measured in microseconds. I don't have a linux system here to run lmbench on, and I wonder what the level of latency would be on a modern system.
- deleted 4y ago
  [deleted]
- fulafel 4y ago
  Current everyday networking has tens-of-microseconds latency[1] and on 2018 era CPUs the context switch overhead seems to be 1-2.5 microseconds[2]. So in benchmark and specialized use cases, context switch overhead would be measurable but many orders of magntude away from relevant when talking about communication over the internet. [1] https://blog.cloudflare.com/how-to-achieve-low-latency/ https://blog.cloudflare.com/how-to-achieve-low-latency/ - 30-45 microseconds on plain 10G ethernet - faster ethernet probably wouldn't improve much on this [2] https://eli.thegreenplace.net/2018/measuring-context-switching-and-memory-overheads-for-linux-threads/ https://eli.thegreenplace.net/2018/measuring-context-switchi...
dogleash 4y ago
> It's 2022 So someone else should have done it for you by now? Be the change you want to see in the world. I'm sure you have an excuse for not doing it personally. Just as I'm sure the person who you've mentally assigned responsibility has at least as good of an excuse too.
- boricj 4y ago
  > Be the change you want to see in the world. I have made dozens of commits to MINIX3, including a brand-new ISO 9660 file system implementation (https://github.com/Stichting-MINIX-Research-Foundation/minix/commits?author=boricj https://github.com/Stichting-MINIX-Research-Foundation/minix...). I have made more than a hundred commits to SerenityOS (https://github.com/SerenityOS/serenity/commits?author=boricj https://github.com/SerenityOS/serenity/commits?author=boricj). Just because I deplore the general state of security in mainstream operating systems doesn't mean that I demand that someone else does something about it for free. I'm not paid to fix security bugs in the Linux kernel, do you expect me to fix these myself for free just because you want to? No one is entitled to my own free time spent hacking on random stuff.
fulafel 4y ago
Interestingly the user space way is also used in the performance absolutist end of the spectrum, so userspace can talk to the HW without kernel involvement (Snabb, dpdk, etc). See eg https://talawah.io/blog/linux-kernel-vs-dpdk-http-performance-showdown/#and-the-winner-is https://talawah.io/blog/linux-kernel-vs-dpdk-http-performanc...
- hansel_der 4y ago
  indeed, but afaik these more or less depend on dedicating a cpu-thread to the software-thread because a cache eviction would completely wreck the performance
  - fulafel 4y ago
    I think with networking appliances that are built from eg DPDK building blocks the main motivation for pinning and isolcpus usage is largely that users want to have a fixed topology of CPU cores, PCIe devices, NUMA memory pools/hugepages. But this doesn't really mean it would be necessary if we started using userspace networking drivers for general purpouse linux workloads.
- legulere 4y ago
  For performance you want to reduce context switches between processes including the kernel. With kernel bypass you do everything in one process in the user space, but you are also losing out on features like sharing hardware resources between processes. With general user space drivers you will gain additional context switches. There are also ways to reduce the cost of those context switches though, like IO-uring that Linux recently got.
deleted 4y ago
[deleted]
yardstick 4y ago
> I get that there are some use cases where performance really matters to the point where kernel network stack and drivers make a difference (high-throughput and/or low-latency services running on servers, high-performance routers...), but that should not be the default for everyone. I’m on board so long as there is a choice. Routers with crappy hardware need as much help as possible. Also tangentially this is why the current darling of VPN tech, WireGuard, is implemented in kernel not userspace.
nordis8 4y ago
> It's 2022 and we've got more than enough compute power nowadays to run in user-land Oh hell no! User-land driver can barely handle WiFi 4 speeds (72Mbps), with terrible CPU performance (1000's of context switches and interrupts per second). For WiFi 5 (ac), WiFi 6 (ac), you need heavy WiFi firmware involvement, multiple DMA queues and kernel driver, and even with all that it requires special care to reach target performance. There is no chance in hell to reach that kind of performance in user-land.
chasil 4y ago
Good gracious. "WEINBERG'S SECOND LAW: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization."
- meltyness 4y ago
  This is more of a disparaging analogy than a law.
mmis1000 4y ago
In the last decade, we have bump link speed from 100m 1G to some board even have 2.5G rj45 port now. I am not sure if move data between kernel and userland at 2.5G/s is even a good idea. And even worse, because kernel still have to distribute data to other userland program, you actually need another round trip so the impact need to multiply by two.
- zekica 4y ago
  Worse, IP package sizes didn't grow at all. So it would require 200 000 context switches per second if you don't want to add code bundling packets kernel-side and dispatching them a-la Nagle's algorithm. This is both error-prone and adds latency. Even worse, trampolines and other mitigations make it so switches clear CPU cache.
- legulere 4y ago
  The trick is to not copy the data, but to pass pages around.
- fulafel 4y ago
  It's the opposite: In the last decade networking connection speeds have plateaued. In the 90s we had 100M ethernet, in the 00's 1G ethernet, in the 2010's 10G ethernet, and it stuck there for a long time. We had much much less CPU cycles available per packet when 1G came, for example.
  - mmis1000 4y ago
    I don't think I have seen 10G ethernet network card in consumer grade hardware even recently. 10G is there in core infrastructure for a long time. But only until recently you see 2.5Gbps in endpoint devices. (Probably 2.5 makes more sense to a RJ45 head cable) And also I don't think 10G routers/switchs ever use pure software based solution to handle the traffics. They are all hardware based or mixed solutions. It's amazing that memory bandwidth / cpu speed / core counts grows so much that makes this even possible. But it still isn't a good idea.
throw10920 4y ago
"Drivers are exploitable so we should run them in userspace" is a hack, and not a good one. The problem is that drivers are exploitable in the first place, so the solution is that we should make them not exploitable (using Rust, or a better language than Rust that fixes some of its problems) and try to preserve our performance that is rapidly being stolen away by bloated userspace software, rather than just shrug our shoulders and say "oh well, I guess that drivers are just intrinsically insecure".
- legulere 4y ago
  It's not a hack. Virtual address spaces and process isolation were built exactly for misbehaving code.
Manu40 4y ago
Uhm... no? 1. Date of year has nothing to do with doing things correctly, ever. 2. Ironically, pulling it out of the kernel and running it in user-land will probably bring about more bugs and issues. I would much rather we just fix the problem where it is, and leave it at that, instead of potentially introducing new problems, like backdoors and exploits in the software provided. Not saying it WOULD happen, but the potential for it alone is just not worth the risk in my honest opinion. Let's just fix the right way, and be done with it.
matheusmoreira 4y ago
> Can we please stop running network drivers and network stacks in kernel mode by default? No. I want the kernel to have as much functionality as possible. I have some zero dependency freestanding software that I boot Linux directly into. I really don't want to have to maintain additional user space in the form of C libraries. If I need to manage wifi connections, I should be able to make some system calls and be done with it without having to link to anything else. Anything related to hardware belongs in the kernel so that all software can access it via Linux's amazing language agnostic system interface. If there are security problems, then that process should be improved without screwing up the interface by replacing it with user space C libraries. We have enough of that in the graphics stack.
GrabbinD33ze69 4y ago
I only have a very surface level understanding of linux, how does code running in user mode incur a bit of overhead as opposed to Kernel?

by boricj 4y ago

alfiedotwtf 4y ago

Obligatory comment: https://www.oreilly.com/openbook/opensources/book/appa.html https://www.oreilly.com/openbook/opensources/book/appa.html

boricj 4y ago
Note that the Tanenbaum-Torvalds debate was in 1992, over thirty years ago. The security of computer systems might have improved since then, but the fallout of security issues has massively increased. Managing your bank accounts wirelessly on the Internet from anywhere in the world with a thin, battery-powered device that fits inside your pocket was a pipe dream (and no one could've possibly imagined that an Internet-ready smart toaster could compromise it and steal your money on your accounts).

dijit 4y ago

1) Many will cry about that performance hit (including me). For over a decade our computers have gotten faster marginally, but our software has gotten slower at a greater rate. You can barely navigate the web now with a new low end computer (that isn't a Chromebook). Most on this site won't care though because our machines cost $2,000+ and the web is Fine(tm); many folk aren't buying anything over $300 though. 2) These are memory bugs, so the introduction of Rust into the kernel could help us here potentially, no need for an architectural revolution.

boricj 4y ago
> Many will cry about that performance hit (including me) because for over a decade our computers have gotten faster marginally, but our software has gotten slower and bloatier at an increasingly rapid pace. We're talking about network stacks and network drivers, not web browsers. Migrating the network stack from the kernel to a user-land process is not going to measurably slow down web browsers, especially on modern systems with gigabytes of RAM, multiple cores, IOMMUs and whatnots. > These are memory bugs, so the introduction of Rust into the kernel could help us here potentially, no need for an architectural revolution. That would require rewriting the network stack and network drivers in Rust (driver code is much more likely to have bugs than the rest of the kernel) for this to be effective, otherwise you'll still have a lot of C code in the network path. I'd argue that this would be a bigger architectural revolution than porting the existing code and running it in user-land. MINIX3 went through such a change when drivers were removed from the kernel (can't find the publication about it right now) and they only required reasonably small changes when porting these to user-land, there were not rewritten from scratch. But this is not just about memory safety, Rust code can still be vulnerable in many other ways (memory leaks, unsafe blocks, wrong assumptions, incorrect algorithm implementation, buggy/compromised toolchains...). Code running inside the trusted computing base of a system is a liability, enforcing privilege separation and principle of least authority reduces it.
- dijit 4y ago
  > We're talking about network stacks and network drivers, not web browsers. Ah yes, the magic web-browser that doesn't do any kind of networking at all. > Migrating the network stack from the kernel to a user-land process is not going to measurably slow down web browsers, especially on modern systems with IOMMUs and whatnots. I don't know how you can possibly assert that, it's contradicting computer sciences' current understanding of operating system design as it relates to kernelmode/usermode switching, unless you're doing weird shared-memory things in userspace... which is terrifying. > That would require rewriting the network stack and network drivers in Rust Not really, C and Rust can interop just fine, you can have network drivers that are rust but the actual networking stack itself can remain C, if you want. > but this is not just about memory safety, Rust code can still be vulnerable in many other ways The post is literally memory safety bugs.

the_duke 4y ago

You really need to back up these assertions with some evidence. Do you have benchmarks that show the impact of switching to userspace on a typical, loaded desktop system with all kinds of workloads? Or are you just guessing?

boricj 4y ago
I did not anticipate a Hacker News discussion about a remotely exploitable Linux kernel WiFi vulnerability requiring some network benchmarks on an unusual network stack architecture, but I'll oblige: This publication (http://www.minix3.org/docs/jorrit-herder/asci06.pdf http://www.minix3.org/docs/jorrit-herder/asci06.pdf) claims that MINIX3 could saturate a 1 Gb/s Ethernet link with an user-space network stack, with separate processes for the stack and the driver, on a rusty 32-bit micro-kernel that can't do SMP. In 2006.

314 4y ago

What kind of latency would this introduce on packet processing? A context switch used to be measured in microseconds. I don't have a linux system here to run lmbench on, and I wonder what the level of latency would be on a modern system.

deleted 4y ago
[deleted]
fulafel 4y ago
Current everyday networking has tens-of-microseconds latency[1] and on 2018 era CPUs the context switch overhead seems to be 1-2.5 microseconds[2]. So in benchmark and specialized use cases, context switch overhead would be measurable but many orders of magntude away from relevant when talking about communication over the internet. [1] https://blog.cloudflare.com/how-to-achieve-low-latency/ https://blog.cloudflare.com/how-to-achieve-low-latency/ - 30-45 microseconds on plain 10G ethernet - faster ethernet probably wouldn't improve much on this [2] https://eli.thegreenplace.net/2018/measuring-context-switching-and-memory-overheads-for-linux-threads/ https://eli.thegreenplace.net/2018/measuring-context-switchi...

dogleash 4y ago

> It's 2022 So someone else should have done it for you by now? Be the change you want to see in the world. I'm sure you have an excuse for not doing it personally. Just as I'm sure the person who you've mentally assigned responsibility has at least as good of an excuse too.

boricj 4y ago
> Be the change you want to see in the world. I have made dozens of commits to MINIX3, including a brand-new ISO 9660 file system implementation (https://github.com/Stichting-MINIX-Research-Foundation/minix/commits?author=boricj https://github.com/Stichting-MINIX-Research-Foundation/minix...). I have made more than a hundred commits to SerenityOS (https://github.com/SerenityOS/serenity/commits?author=boricj https://github.com/SerenityOS/serenity/commits?author=boricj). Just because I deplore the general state of security in mainstream operating systems doesn't mean that I demand that someone else does something about it for free. I'm not paid to fix security bugs in the Linux kernel, do you expect me to fix these myself for free just because you want to? No one is entitled to my own free time spent hacking on random stuff.

fulafel 4y ago

Interestingly the user space way is also used in the performance absolutist end of the spectrum, so userspace can talk to the HW without kernel involvement (Snabb, dpdk, etc). See eg https://talawah.io/blog/linux-kernel-vs-dpdk-http-performance-showdown/#and-the-winner-is https://talawah.io/blog/linux-kernel-vs-dpdk-http-performanc...

hansel_der 4y ago
indeed, but afaik these more or less depend on dedicating a cpu-thread to the software-thread because a cache eviction would completely wreck the performance
- fulafel 4y ago
  I think with networking appliances that are built from eg DPDK building blocks the main motivation for pinning and isolcpus usage is largely that users want to have a fixed topology of CPU cores, PCIe devices, NUMA memory pools/hugepages. But this doesn't really mean it would be necessary if we started using userspace networking drivers for general purpouse linux workloads.
legulere 4y ago
For performance you want to reduce context switches between processes including the kernel. With kernel bypass you do everything in one process in the user space, but you are also losing out on features like sharing hardware resources between processes. With general user space drivers you will gain additional context switches. There are also ways to reduce the cost of those context switches though, like IO-uring that Linux recently got.

deleted 4y ago

[deleted]

yardstick 4y ago

> I get that there are some use cases where performance really matters to the point where kernel network stack and drivers make a difference (high-throughput and/or low-latency services running on servers, high-performance routers...), but that should not be the default for everyone. I’m on board so long as there is a choice. Routers with crappy hardware need as much help as possible. Also tangentially this is why the current darling of VPN tech, WireGuard, is implemented in kernel not userspace.

nordis8 4y ago

> It's 2022 and we've got more than enough compute power nowadays to run in user-land Oh hell no! User-land driver can barely handle WiFi 4 speeds (72Mbps), with terrible CPU performance (1000's of context switches and interrupts per second). For WiFi 5 (ac), WiFi 6 (ac), you need heavy WiFi firmware involvement, multiple DMA queues and kernel driver, and even with all that it requires special care to reach target performance. There is no chance in hell to reach that kind of performance in user-land.

chasil 4y ago

Good gracious. "WEINBERG'S SECOND LAW: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization."

meltyness 4y ago
This is more of a disparaging analogy than a law.

mmis1000 4y ago

In the last decade, we have bump link speed from 100m 1G to some board even have 2.5G rj45 port now. I am not sure if move data between kernel and userland at 2.5G/s is even a good idea. And even worse, because kernel still have to distribute data to other userland program, you actually need another round trip so the impact need to multiply by two.

zekica 4y ago
Worse, IP package sizes didn't grow at all. So it would require 200 000 context switches per second if you don't want to add code bundling packets kernel-side and dispatching them a-la Nagle's algorithm. This is both error-prone and adds latency. Even worse, trampolines and other mitigations make it so switches clear CPU cache.
legulere 4y ago
The trick is to not copy the data, but to pass pages around.
fulafel 4y ago
It's the opposite: In the last decade networking connection speeds have plateaued. In the 90s we had 100M ethernet, in the 00's 1G ethernet, in the 2010's 10G ethernet, and it stuck there for a long time. We had much much less CPU cycles available per packet when 1G came, for example.
- mmis1000 4y ago
  I don't think I have seen 10G ethernet network card in consumer grade hardware even recently. 10G is there in core infrastructure for a long time. But only until recently you see 2.5Gbps in endpoint devices. (Probably 2.5 makes more sense to a RJ45 head cable) And also I don't think 10G routers/switchs ever use pure software based solution to handle the traffics. They are all hardware based or mixed solutions. It's amazing that memory bandwidth / cpu speed / core counts grows so much that makes this even possible. But it still isn't a good idea.

throw10920 4y ago

"Drivers are exploitable so we should run them in userspace" is a hack, and not a good one. The problem is that drivers are exploitable in the first place, so the solution is that we should make them not exploitable (using Rust, or a better language than Rust that fixes some of its problems) and try to preserve our performance that is rapidly being stolen away by bloated userspace software, rather than just shrug our shoulders and say "oh well, I guess that drivers are just intrinsically insecure".

legulere 4y ago
It's not a hack. Virtual address spaces and process isolation were built exactly for misbehaving code.

Manu40 4y ago

Uhm... no? 1. Date of year has nothing to do with doing things correctly, ever. 2. Ironically, pulling it out of the kernel and running it in user-land will probably bring about more bugs and issues. I would much rather we just fix the problem where it is, and leave it at that, instead of potentially introducing new problems, like backdoors and exploits in the software provided. Not saying it WOULD happen, but the potential for it alone is just not worth the risk in my honest opinion. Let's just fix the right way, and be done with it.

matheusmoreira 4y ago

> Can we please stop running network drivers and network stacks in kernel mode by default? No. I want the kernel to have as much functionality as possible. I have some zero dependency freestanding software that I boot Linux directly into. I really don't want to have to maintain additional user space in the form of C libraries. If I need to manage wifi connections, I should be able to make some system calls and be done with it without having to link to anything else. Anything related to hardware belongs in the kernel so that all software can access it via Linux's amazing language agnostic system interface. If there are security problems, then that process should be improved without screwing up the interface by replacing it with user space C libraries. We have enough of that in the graphics stack.

GrabbinD33ze69 4y ago

I only have a very surface level understanding of linux, how does code running in user mode incur a bit of overhead as opposed to Kernel?