5 ms·

I've been using Linux since 2005, and I've loved it in almost every circumstance. But the drama over the last couple of years surrounding Rust in the kernel has

I've been using Linux since 2005, and I've loved it in almost every circumstance. But the drama over the last couple of years surrounding Rust in the kernel has really soured me on it, and I'm now very pessimistic about its future. But I think beyond the emotional outbursts of various personalities, I don't think that the problem is which side is "right". Both sides have extremely valid points. I don't think the problem is actually solvable, because managing a 40M+ SLoC codebase is barely tenable in general, and super duper untenable for something that we rely on for security while running in ring 0. My best hope is for replacement. I think we've finally hit the ceiling of where monolithic kernels can take us. The Linux kernel will continue to make extremely slow progress while it deals with internal politics fighting against an architecture that can only get bigger and less secure over time. But what could be the replacement? There's a handful of fairly mature microkernels out there, each with extremely immature userspaces. There doesn't seem to be any concerted efforts behind any of them. I have a lot of hope for SeL4, but progress there seems to be slow mostly because the security model has poor ergonomics. I'd love to see some sort of breakout here.

bigfatkitten 2y ago
> I have a lot of hope for SeL4, but progress there seems to be slow mostly because the security model has poor ergonomics. seL4 has its place, but that place is not as a Linux replacement. Modern general purpose computers (both their hardware, and their userspace ecosystems) have too much unverifiable complexity for a formally verified microkernel to be really worthwhile.
- darksaints 2y ago
  I agree that SeL4 won't replace Linux anytime soon, but I beg to differ on the benefits of a microkernel, formally verified or not. Any ordinary well-designed microkernel gives you a huge benefit: process isolation of core services and drivers. That means that even in the case of an insecure and unverified driver, you still have reasonable expectations of security. There was an analysis of Linux CVE's a while back and the vast majority of critical Linux CVEs to that date would either be eliminated or mitigated below critical level just by using a basic microkernel architecture (not even a verified microkernel). Only 4% would have remained critical. https://microkerneldude.org/2018/08/23/microkernels-really-do-improve-security/ https://microkerneldude.org/2018/08/23/microkernels-really-d... The benefit of a verified microkernel like SeL4 is merely an incremental one over a basic microkernel like L4, capable of capturing that last 4% and further mitigating others. You get more reliable guarantees regarding process isolation, but architecturally it's not much different from L4. There's a little bit of clunkiness for writing userpace drivers for SeL4 that you wouldn't have for L4. That's what the LionsOS project is aiming to fix.
  - zozbot234 2y ago
    Process isolation of drivers is just not very useful when the driver is interfacing with a device that has full access to system memory. Which is the case for many devices today unless you use IOMMU to prevent this.
    - darksaints 2y ago
      The SeL4 microkernel specification assumes the use of a memory management unit, and is required by default. https://docs.sel4.systems/projects/sel4/frequently-asked-questions.html https://docs.sel4.systems/projects/sel4/frequently-asked-que...
      - comex 2y ago
        IOMMU, not regular (CPU) MMU. The FAQ _does_ address this, but it's under "What about DMA?". In short: drivers have to be trusted for now, except that there's experimental support for x86 VT-d (which is a type of IOMMU).
        vacuity 2y ago
        I've been developing a notion that, in modern times, a microkernel is not the sole root of trust. It is just the most privileged component that glues other essential components. Without it, things fall apart, but we still need quality in the "business logic" components (everything else, from this view). So a user should deploy a trusted microkernel with trusted means of download and whatever opsec, and similarly for other crucial components like drivers. This is all essential trust anyways. The leaps and bounds we've achieved through hardware engineering have the burden that they aren't credible for security. You can use IOMMU, but perhaps I won't. Integrated co-development of hardware and software is ideal, but generally there is an adversarial relationship, and we must reflect that in the software. Trust and security are not yes/no questions. We have to keep pushing boundaries. seL4 is a good start; let's make more from it.
        darksaints 2y ago
        Ah yes, thanks for the correction.
- yencabulator 2y ago
  Oh don't worry, seL4 isn't formally proven on any multicore computer anyway. And the seL4 core architecture is fundamentally "one single big lock" and won't scale at all to modern machines. The intended design is that each core runs its own kernel with no coordination (multikernel a la Barrelfish) -- none of which is implemented. So as far as any computer with >4 cores is concerned, seL4 is not relevant at this time, and if you wish for that to happen your choice is really either funding the seL4 people or getting someone else to make a different microkernel (with hopefully a lot less CAmkES "all the world is C" mess).
  - vacuity 2y ago
    Barrelfish! My dream project is developing a multikernel with seL4's focus on assurance. I want to go even further than seL4's minimalism, particularly with regards to the scheduler. I thiiiiink it doesn't have to be bad for performance. But I've not materialized anything and so I am just delusional. And yes, I am thinking of doing it in Rust. For all of Rust's shortcomings, especially for kernel development, I think it has a lot of promise. I also have the already-loves-Rust cognitive bias. Not trying to somehow achieve seL4's massive verification effort. (Will gasp AI faciliate it? Not likely.) I am sad that Barrelfish hasn't gotten more attention. We need more OS research.
    - yencabulator 2y ago
      You didn't mention capabilities, otherwise very much yes. Though I have to say I think I'd vote EROS or Theseus harder than Barrelfish; Barrelfish is "just" a multikernel. All I've materialized is a bunch of notes and bookmarks. Kani et al are very interesting, but can't handle large codebases yet. I'm trying to write Rust in very compartmentalized, sans-IO etc, way to have small libraries that are fuzzable and more amenable to Kani verification.
      - vacuity 2y ago
        Capabilities are implied; seL4 did it (everyone cool does it) and Barrelfish designed their capability system off of seL4's. When I say multikernel, that is the main innovation of Barrelfish to take; the kernel design otherwise is more like seL4 or perhaps EROS. Barrelfish also has some interesting takes on other OS services that I want to use, but that's not kernel design. I assume Theseus is [0]. It requires the whole system to buy in to one language and model, so I consider it untenable for a general-purpose OS. I think they're planning to use WASM to increase flexibility, but eh. > I'm trying to write Rust in very compartmentalized, sans-IO etc, way to have small libraries that are fuzzable and more amenable to Kani verification. Good design even if Kani isn't used in the end. [0] https://www.usenix.org/conference/osdi20/presentation/boos https://www.usenix.org/conference/osdi20/presentation/boos
- EasyMark 2y ago
  I mean why does it have to be formally verified. Seems to me like the performance tradeoff for microkernels can be worth it to have drivers and other traditional kernel layer code, that don't bring down the system and can just be restarted in case of failures. Probably not something that will work for all hardware, but I would bet the majority would be fine with it.
  - kennysoona 2y ago
    > I mean why does it have to be formally verified. Because we can and the security advantages are worth it.
  - darksaints 2y ago
    At this point, even an unverified kernel would be a huge step up in terms of security and reliability. And the performance disadvantages of a microkernel are all overblown, if not outright false [1]. Sure, you have to make twice as many syscalls as a monolithic kernel, but you can do it with much better caching behavior, due to the significantly smaller size. The SeL4 kernel is small enough to fit entirely in many modern processors' L2 cache. It's entirely possible (some chip designers have hinted as much), that with enough adoption they could prioritize having dedicated caches for the OS kernel...something that could never be possible with any monolithic kernel. [1] https://trustworthy.systems/publications/theses_public/23/Parker%3Absc.pdf https://trustworthy.systems/publications/theses_public/23/Pa...
raspyberr 2y ago
GNU Mach! GNU Mach! GNU Mach! GNU Mach! GNU Mach! GNU Mach!
colonial 2y ago
Not a kernel guy, but - what's stopping a microkernel from emulating the Linux userspace? I know Microsoft had some success implementing the Linux ABI with WSL v1.0. I suppose the main objection to that is accepting some degree of lock-in with the existing userspace (systemd, FHS...) over exploring new ideas for userspace at the same time.
- anp 2y ago
  FWIW Fuchsia has a not-quite-a-microkernel and has been building a Linux binary compatibility layer: https://fuchsia.dev/fuchsia-src/concepts/starnix?hl=en https://fuchsia.dev/fuchsia-src/concepts/starnix?hl=en. (disclaimer: I work on Fuchsia, on Starnix specifically) EDIT: for extra HN karma and related to the topic of the posted email thread, Starnix (Fuchsia's Linux compat layer) is written in Rust. It does run on top of a kernel written in C++ but Zircon is much smaller than Linux.
  - nechuchelo 2y ago
    Nice to hear fuchsia is still being worked on. I was a bit concerned given there were no new changelogs published for half a year.
  - zamadatix 2y ago
    Fascinating area to work in! I've had a few curiosity things come to mind before: What's the driving use case for Starnix? Well, obviously "run Linux apps on Fuchsia" like the RFC for it says... but "very specific apps as part of a specific use case which might be timeboxed" or "any app for the foreseeable future"? How complete in app support do you currently consider it compared to something like WSL1? What are your thoughts about why WSL2 went the opposite direction? Thanks!
    - anp 2y ago
      > Fascinating area to work in! I agree! Lots of fun stuff to do. > What's the driving use case for Starnix? The Starnix code is open source like the rest of Fuchsia and anyone is obviously free to read it and form their own opinions about where it's useful or where it's headed, but as a mere corporate employee I can't comment on direction/strategy :(. > How complete in app support do you currently consider it compared to something like WSL1? I'm only familiar with WSL1 as an occasional user so I can't really say for sure. We run (and pass) a lot of tests compiled for Linux from the Linux Test Project, gVisor's compatibility test suite, and some other sources. There are still a lot of those tests that we don't yet pass :). > What are your thoughts about why WSL2 went the opposite direction? I don't know much about the history there. I've heard Nth-hand rumors that MS had a product strategy shift from Windows Phone Android compat (a relatively focused use case where edge cases might be acceptable) to trying to court developers (a broad use case where varying from their deployment environment might cause problems). I have no idea whether those rumors are accurate. I've also heard that it was hard to make Linux programs perform well on top of NTFS, and that virtualized ext4 actually worked better for Linux workloads where fs performance mattered at all. Something something dirent cache for stat()? Some of this is discussed on the WSL1 vs WSL2 web page[0]. [0] https://learn.microsoft.com/en-us/windows/wsl/compare-versions https://learn.microsoft.com/en-us/windows/wsl/compare-versio...
dralley 2y ago
Like 75% of those lines of code are in drivers or architecture-specific code (code that only runs for x86 or ARM or SPARC or POWER etc.) The amount of kernel code actually executing on any given machine at any given point in time is more likely to be around 9-12 million lines than anywhere near 40 million. And a replacement kernel won't eliminate the need for hardware drivers for a very wide range of hardware. Again, that's where the line count ramps up.not
- scns 2y ago
  The ADMgpu driver alone was over 5 million loc in 2023.
  - samus 2y ago
    Most of these are header files. I suspect most of its contents are constants and blobs autogenerated with some tool by AMD.
- darksaints 2y ago
  Yes, of course. But apart from the (current) disadvantage that those drivers don't exist yet, those are all positives in favor of microkernel architectures. All of the massive SLOC codebases run in usermode and with full process isolation, require no specific language compatibility and can be written in any language, do not require upstreaming, and do not require extensive security evaluations from highly capable maintainers who have their focus scattered across 40m lines of code.
timeon 2y ago
> But what could be the replacement? There's a handful of fairly mature microkernels out there Redox[0] has advantage that no-one will want to rewrite it in Rust. [0]: https://redox-os.org/ https://redox-os.org/
cardanome 2y ago
The rust drama is completely overblown considering rust is still years away from being a viable replacement. Sure it makes sense to start experimenting and maybe write a few drivers in rust but many features are still only available in nightly rust. I suspect many rust devs tend to be on the younger side, while the old C guard sees Linux development in terms of decades. Change takes time. Monolithic kernels are fine. The higher complexity and worse performance of a microkernel design are mostly not worth the theoretical architectural advantages. If you wanted to get out of the current local optimum you would have to think outside of the unix design. The main treat for Linux is the Linux Foundation that is controlled by big tech monopolists like Microsoft and only spends only a small fraction on actual Kernel development. It is embrace, extend, extinguish all over but people think Microsoft are the good guys now.
- chippiewill 2y ago
  > but many features are still only available in nightly rust. Nope. The features are all in stable releases (Since last Spring in fact). However some of the features are still marked as unstable/experimental and have to be opted-in (so could in theory have breaking changes still). They're entirely features that are specific to kernel development and are only needed in the rust bindings layer to provide safe abstractions in a kernel environment.

bigfatkitten 2y ago

> I have a lot of hope for SeL4, but progress there seems to be slow mostly because the security model has poor ergonomics. seL4 has its place, but that place is not as a Linux replacement. Modern general purpose computers (both their hardware, and their userspace ecosystems) have too much unverifiable complexity for a formally verified microkernel to be really worthwhile.

darksaints 2y ago
I agree that SeL4 won't replace Linux anytime soon, but I beg to differ on the benefits of a microkernel, formally verified or not. Any ordinary well-designed microkernel gives you a huge benefit: process isolation of core services and drivers. That means that even in the case of an insecure and unverified driver, you still have reasonable expectations of security. There was an analysis of Linux CVE's a while back and the vast majority of critical Linux CVEs to that date would either be eliminated or mitigated below critical level just by using a basic microkernel architecture (not even a verified microkernel). Only 4% would have remained critical. https://microkerneldude.org/2018/08/23/microkernels-really-do-improve-security/ https://microkerneldude.org/2018/08/23/microkernels-really-d... The benefit of a verified microkernel like SeL4 is merely an incremental one over a basic microkernel like L4, capable of capturing that last 4% and further mitigating others. You get more reliable guarantees regarding process isolation, but architecturally it's not much different from L4. There's a little bit of clunkiness for writing userpace drivers for SeL4 that you wouldn't have for L4. That's what the LionsOS project is aiming to fix.
- zozbot234 2y ago
  Process isolation of drivers is just not very useful when the driver is interfacing with a device that has full access to system memory. Which is the case for many devices today unless you use IOMMU to prevent this.
  - darksaints 2y ago
    The SeL4 microkernel specification assumes the use of a memory management unit, and is required by default. https://docs.sel4.systems/projects/sel4/frequently-asked-questions.html https://docs.sel4.systems/projects/sel4/frequently-asked-que...
    - comex 2y ago
      IOMMU, not regular (CPU) MMU. The FAQ _does_ address this, but it's under "What about DMA?". In short: drivers have to be trusted for now, except that there's experimental support for x86 VT-d (which is a type of IOMMU).
      - vacuity 2y ago
        I've been developing a notion that, in modern times, a microkernel is not the sole root of trust. It is just the most privileged component that glues other essential components. Without it, things fall apart, but we still need quality in the "business logic" components (everything else, from this view). So a user should deploy a trusted microkernel with trusted means of download and whatever opsec, and similarly for other crucial components like drivers. This is all essential trust anyways. The leaps and bounds we've achieved through hardware engineering have the burden that they aren't credible for security. You can use IOMMU, but perhaps I won't. Integrated co-development of hardware and software is ideal, but generally there is an adversarial relationship, and we must reflect that in the software. Trust and security are not yes/no questions. We have to keep pushing boundaries. seL4 is a good start; let's make more from it.
      - darksaints 2y ago
        Ah yes, thanks for the correction.
yencabulator 2y ago
Oh don't worry, seL4 isn't formally proven on any multicore computer anyway. And the seL4 core architecture is fundamentally "one single big lock" and won't scale at all to modern machines. The intended design is that each core runs its own kernel with no coordination (multikernel a la Barrelfish) -- none of which is implemented. So as far as any computer with >4 cores is concerned, seL4 is not relevant at this time, and if you wish for that to happen your choice is really either funding the seL4 people or getting someone else to make a different microkernel (with hopefully a lot less CAmkES "all the world is C" mess).
- vacuity 2y ago
  Barrelfish! My dream project is developing a multikernel with seL4's focus on assurance. I want to go even further than seL4's minimalism, particularly with regards to the scheduler. I thiiiiink it doesn't have to be bad for performance. But I've not materialized anything and so I am just delusional. And yes, I am thinking of doing it in Rust. For all of Rust's shortcomings, especially for kernel development, I think it has a lot of promise. I also have the already-loves-Rust cognitive bias. Not trying to somehow achieve seL4's massive verification effort. (Will gasp AI faciliate it? Not likely.) I am sad that Barrelfish hasn't gotten more attention. We need more OS research.
  - yencabulator 2y ago
    You didn't mention capabilities, otherwise very much yes. Though I have to say I think I'd vote EROS or Theseus harder than Barrelfish; Barrelfish is "just" a multikernel. All I've materialized is a bunch of notes and bookmarks. Kani et al are very interesting, but can't handle large codebases yet. I'm trying to write Rust in very compartmentalized, sans-IO etc, way to have small libraries that are fuzzable and more amenable to Kani verification.
    - vacuity 2y ago
      Capabilities are implied; seL4 did it (everyone cool does it) and Barrelfish designed their capability system off of seL4's. When I say multikernel, that is the main innovation of Barrelfish to take; the kernel design otherwise is more like seL4 or perhaps EROS. Barrelfish also has some interesting takes on other OS services that I want to use, but that's not kernel design. I assume Theseus is [0]. It requires the whole system to buy in to one language and model, so I consider it untenable for a general-purpose OS. I think they're planning to use WASM to increase flexibility, but eh. > I'm trying to write Rust in very compartmentalized, sans-IO etc, way to have small libraries that are fuzzable and more amenable to Kani verification. Good design even if Kani isn't used in the end. [0] https://www.usenix.org/conference/osdi20/presentation/boos https://www.usenix.org/conference/osdi20/presentation/boos
EasyMark 2y ago
I mean why does it have to be formally verified. Seems to me like the performance tradeoff for microkernels can be worth it to have drivers and other traditional kernel layer code, that don't bring down the system and can just be restarted in case of failures. Probably not something that will work for all hardware, but I would bet the majority would be fine with it.
- kennysoona 2y ago
  > I mean why does it have to be formally verified. Because we can and the security advantages are worth it.
- darksaints 2y ago
  At this point, even an unverified kernel would be a huge step up in terms of security and reliability. And the performance disadvantages of a microkernel are all overblown, if not outright false [1]. Sure, you have to make twice as many syscalls as a monolithic kernel, but you can do it with much better caching behavior, due to the significantly smaller size. The SeL4 kernel is small enough to fit entirely in many modern processors' L2 cache. It's entirely possible (some chip designers have hinted as much), that with enough adoption they could prioritize having dedicated caches for the OS kernel...something that could never be possible with any monolithic kernel. [1] https://trustworthy.systems/publications/theses_public/23/Parker%3Absc.pdf https://trustworthy.systems/publications/theses_public/23/Pa...

raspyberr 2y ago

GNU Mach! GNU Mach! GNU Mach! GNU Mach! GNU Mach! GNU Mach!

colonial 2y ago

Not a kernel guy, but - what's stopping a microkernel from emulating the Linux userspace? I know Microsoft had some success implementing the Linux ABI with WSL v1.0. I suppose the main objection to that is accepting some degree of lock-in with the existing userspace (systemd, FHS...) over exploring new ideas for userspace at the same time.

anp 2y ago
FWIW Fuchsia has a not-quite-a-microkernel and has been building a Linux binary compatibility layer: https://fuchsia.dev/fuchsia-src/concepts/starnix?hl=en https://fuchsia.dev/fuchsia-src/concepts/starnix?hl=en. (disclaimer: I work on Fuchsia, on Starnix specifically) EDIT: for extra HN karma and related to the topic of the posted email thread, Starnix (Fuchsia's Linux compat layer) is written in Rust. It does run on top of a kernel written in C++ but Zircon is much smaller than Linux.
- nechuchelo 2y ago
  Nice to hear fuchsia is still being worked on. I was a bit concerned given there were no new changelogs published for half a year.
- zamadatix 2y ago
  Fascinating area to work in! I've had a few curiosity things come to mind before: What's the driving use case for Starnix? Well, obviously "run Linux apps on Fuchsia" like the RFC for it says... but "very specific apps as part of a specific use case which might be timeboxed" or "any app for the foreseeable future"? How complete in app support do you currently consider it compared to something like WSL1? What are your thoughts about why WSL2 went the opposite direction? Thanks!
  - anp 2y ago
    > Fascinating area to work in! I agree! Lots of fun stuff to do. > What's the driving use case for Starnix? The Starnix code is open source like the rest of Fuchsia and anyone is obviously free to read it and form their own opinions about where it's useful or where it's headed, but as a mere corporate employee I can't comment on direction/strategy :(. > How complete in app support do you currently consider it compared to something like WSL1? I'm only familiar with WSL1 as an occasional user so I can't really say for sure. We run (and pass) a lot of tests compiled for Linux from the Linux Test Project, gVisor's compatibility test suite, and some other sources. There are still a lot of those tests that we don't yet pass :). > What are your thoughts about why WSL2 went the opposite direction? I don't know much about the history there. I've heard Nth-hand rumors that MS had a product strategy shift from Windows Phone Android compat (a relatively focused use case where edge cases might be acceptable) to trying to court developers (a broad use case where varying from their deployment environment might cause problems). I have no idea whether those rumors are accurate. I've also heard that it was hard to make Linux programs perform well on top of NTFS, and that virtualized ext4 actually worked better for Linux workloads where fs performance mattered at all. Something something dirent cache for stat()? Some of this is discussed on the WSL1 vs WSL2 web page[0]. [0] https://learn.microsoft.com/en-us/windows/wsl/compare-versions https://learn.microsoft.com/en-us/windows/wsl/compare-versio...

dralley 2y ago

Like 75% of those lines of code are in drivers or architecture-specific code (code that only runs for x86 or ARM or SPARC or POWER etc.) The amount of kernel code actually executing on any given machine at any given point in time is more likely to be around 9-12 million lines than anywhere near 40 million. And a replacement kernel won't eliminate the need for hardware drivers for a very wide range of hardware. Again, that's where the line count ramps up.not

scns 2y ago
The ADMgpu driver alone was over 5 million loc in 2023.
- samus 2y ago
  Most of these are header files. I suspect most of its contents are constants and blobs autogenerated with some tool by AMD.
darksaints 2y ago
Yes, of course. But apart from the (current) disadvantage that those drivers don't exist yet, those are all positives in favor of microkernel architectures. All of the massive SLOC codebases run in usermode and with full process isolation, require no specific language compatibility and can be written in any language, do not require upstreaming, and do not require extensive security evaluations from highly capable maintainers who have their focus scattered across 40m lines of code.

timeon 2y ago

> But what could be the replacement? There's a handful of fairly mature microkernels out there Redox[0] has advantage that no-one will want to rewrite it in Rust. [0]: https://redox-os.org/ https://redox-os.org/

cardanome 2y ago

The rust drama is completely overblown considering rust is still years away from being a viable replacement. Sure it makes sense to start experimenting and maybe write a few drivers in rust but many features are still only available in nightly rust. I suspect many rust devs tend to be on the younger side, while the old C guard sees Linux development in terms of decades. Change takes time. Monolithic kernels are fine. The higher complexity and worse performance of a microkernel design are mostly not worth the theoretical architectural advantages. If you wanted to get out of the current local optimum you would have to think outside of the unix design. The main treat for Linux is the Linux Foundation that is controlled by big tech monopolists like Microsoft and only spends only a small fraction on actual Kernel development. It is embrace, extend, extinguish all over but people think Microsoft are the good guys now.

chippiewill 2y ago
> but many features are still only available in nightly rust. Nope. The features are all in stable releases (Since last Spring in fact). However some of the features are still marked as unstable/experimental and have to be opted-in (so could in theory have breaking changes still). They're entirely features that are specific to kernel development and are only needed in the rust bindings layer to provide safe abstractions in a kernel environment.