20 ms·

Kerla: Monolithic kernel in Rust, aiming for Linux ABI compatibility

471 points by Klasiaster 5y ago

krylon 5y ago
This may be a naive question, but I cannot help but wonder: As C and Rust aim to be link-compatible, wouldn't it be easier to gradually replace parts of the Linux kernel with Rust code? The only technical problem I can think of is that Rust may not be available for all CPU architectures Linux supports, but this is just speculation on my part having done no research on the matter.
- gspr 5y ago
  A practical problem is that in order to gain the benefit of linking Rust and C code, one has to give up Rust's wonderful guarantees at the interface. So until islands of Rust meet up, these interact through a C ABI, and have to expose C-like behavior.
  - ls65536 5y ago
    But you still get those guarantees and benefits in the Rust implementation itself, even if not at the interface (to non-Rust code). By a similar argument, a standalone Rust program "gives up" its guarantees whenever the process makes a call to libc or to the operating system (syscalls), but this isn't really a practical problem.
    - gspr 5y ago
      Sure, I agree. The point is that if the total corpus of Rust code is too isolated into too small islands, then the boundary effects of each island may eat up a lot of the benefit. I still think the project is really cool and worthwhile.
  - jll29 5y ago
    The world really needs _two_ projects, a practical one that is interoperable with the here and now, to get incremental improvements in security from bits reimplemented in Rust. And then, in the longer term, someone ought to write a book about operating system implementation in Rust that ignores Linux interoperability and focuses on readability, maintainability by showing use of Rust's idiomatic style.
- wongarsu 5y ago
  That's what may happen with the Linux kernel over time, now that it allows parts written in Rust. The limitation of that approach is that all internal interfaces (which includes most data structures) have to be C-compatible. You can get much of the safety benefits of Rust, but lose a lot of Rust's ergonomics.
  - geewee 5y ago
    I don't think that the Linux kernel has decided (yet) to allow rust code into the kernel proper, has it?
    - fork-bomber 5y ago
      It hasn't. But the attitude towards using Rust for device driver development has now gotten very positive, especially post the recent kernel summit.
    - pjmlp 5y ago
      Google has decided to go forward anyway. Just like Android Linux compiles just fine with clang for the last five years or so, it now makes use of Rust. https://source.android.com/setup/build/rust/building-rust-modules/overview https://source.android.com/setup/build/rust/building-rust-mo... If upstream ever cares to support clang or Rust, that is another matter.
      - jstarks 5y ago
        Unless I'm missing something, that link refers to building Android user-mode components in Rust, not kernel components in Rust.
        steveklabnik 5y ago
        That's correct, but it came after them rebuilding the bluetooth stack https://news.ycombinator.com/item?id=26647981 https://news.ycombinator.com/item?id=26647981
    - 5y ago
- fork-bomber 5y ago
  The Rust for Linux kernel project aims to enable writing Linux kernel device drivers in Rust. See: https://lwn.net/Articles/862018/ https://lwn.net/Articles/862018/ The issue of Rust's LLVM based compiler toolchain not targeting all CPU archs is intended to be solved by the gcc-rs project. See: https://lwn.net/Articles/871283/ https://lwn.net/Articles/871283/ I think the approach the Rust for Linux project is taking is wise: Not about outright re-writes but more about focusing on those subsystems where Rust's intrinsic safety and security properties helps the most.
  - maleldil 5y ago
    Last I checked, one of the big problems was the Rust panics when you run out of memory, which is unacceptable in the kernel. Is there any progress on that?
    - jerven 5y ago
      Yes a lot. https://rust-for-linux.github.io/docs/alloc/alloc/index.html https://rust-for-linux.github.io/docs/alloc/alloc/index.html
    - tialaramex 5y ago
      Out of the box Rust doesn't provide any way to allocate heap memory, so, you can't run out if it. Your ordinary userspace apps use std (the standard library) which relies on the alloc crate, and that provides heap allocation which indeed panics if the allocation fails. Because it correctly guesses that your "clever" strategy to handle allocation failure actually isn't and will just triple fault anyway so it should cut to the chase. The kernel obviously doesn't have std, and Rust for Linux implements its own alloc crate. Linus' requirement that you can fail memory allocation just means all the calls in alloc that can actually allocate memory now return Result to indicate whether the allocation was successful. This isn't how you'd do things in userspace, but, this isn't userspace so fine.
      - abendstolz 5y ago
        Can you elaborate as to why "his isn't how you'd do things in userspace, but, this isn't userspace so fine" holds? Naive me - not a kernel dev at all - would argue that returning Result<Memory, AllocationError> is always better, even for userspace because it would allow me to additionally log something or gracefully deal with this. Even if I don't want to deal with it, I could just `.unwrap()` or `.expect('my error message')` it. Note: I am not trying to be snarky here, I genuinely don't know and would like to. If answering this is too complex, maybe you can point me in the right direction so I can ask the right questions to find answers myself? Thanks in any case!
- phendrenad2 5y ago
  The Linux kernel is massive, so writing any part of it in Rust is an enormous undertaking. Even a few functions at a time, we're talking decades.
  - krylon 5y ago
    Well, writing a kernel from scratch that supports as many CPU architectures and devices as Linux would take about as long, assuming one could attract a critical mass of developers. Using an incremental approach at least gives us some benefits in the near future. I'm not even saying a fresh start would be a bad idea. But incrementally replacing parts of Linux seems like a more promising approach, IMHO.
- pornel 5y ago
  Yes, gradual replacement is easier and more practical for large projects. However, typical C APIs and idioms are different from idiomatic Rust. Rust uses more type-system features, generics, iterators, RAII, and prefers tree-like data structures and far fewer pointers (no linked lists!). C rewritten in Rust is still very C-like, and requires refactorings that you can't do until it's all Rust. BTW: There are two GCC-based Rust implementations in the works, so compatibility with exotic platforms is going to be solved.
- amw-zero 5y ago
  It’s not naive, it’s just been speculated and talked about incessantly since Rust came out. The one naive part is underestimating how long that would take - Linux is one of the largest codebases on Earth. It would require massive coordination over years to rewrite, and in the end all you would get is the exact same product. It’s just not an interesting or unique perspective, and pretty silly to suggest seriously.
Klasiaster 5y ago
A minimal interactive demo running busybox is here: ssh root@kerla-demo.seiya.me
- codetrotter 5y ago
  That’s very cool! Could you share some details on how to set it up for oneself with AWS Firecracker the same way that this demo was set up? Also, as owner of the instances are you able to see what commands people are running? First thing I did was to type hello, and it said hello world to me :)
  - Klasiaster 5y ago
    I'm not the project author, maybe you can ask via e-mail, see https://seiya.me/ https://seiya.me/
    - codetrotter 5y ago
      Ok, thanks :)
  - nuta 5y ago
    Author here. I've made the demo system public [1] but it's written just for me so it should be painful to set up the same environment. The mechanism is pretty simple: a Node.js server (running on genuine Linux) listens on tcp:22. Once you connect, it boots a dedicated Firecracker microVM instance and forwards network packets between your SSH client and VM. Regarding the command history, others (including I) can't see what you type. If you could, it must be a vulnerability. [1] https://github.com/nuta/kerla-demo https://github.com/nuta/kerla-demo
    - codetrotter 5y ago
      Thank you :)
- thawkins 5y ago
  is there not an opportunity to implement the posix filesystems interface on top of something other than a block device system. That would be fun.
rvz 5y ago
A star has been born. A Linux alternative needs to be compatible with Linux binaries and also support modern architectures. If you want Linux to survive, it needs to be re-written from scratch, optimised in Rust and compatible with the existing Linux binaries or drivers. This project might have a chance. From: [0] > TL;DR: I'm writing a Linux clone in Rust just for fun. It does NOT aim to replace the Linux kernel. A certain someone thought that their hobby OS Minix clone won't be 'big or professional like GNU'. It's time for 'change'. [0] https://seiya.me/writing-linux-clone-in-rust https://seiya.me/writing-linux-clone-in-rust
- breakfastduck 5y ago
  You think Linux, which the vast majority of web servers run today, is going to die out if it's not rewritten from scratch in Rust? How on earth have you come to that one?
- tonetheman 5y ago
  sigh... Linux is going to survive. And no it does not need to be re-written from scratch in Rust. Someone lied to you at the church of rust.
  - thawkins 5y ago
    But its a fun thing to do, and that's a good enough reason to do it. After all is that not exactly how Linux came into being anyway.
    - simion314 5y ago
      Maybe I will write a TODO app in Rust, should it be on top of HN because is some unoriginal,incomplete,bug filled,toy stuff just because the reason is written in Rust? 1 My assumption is that all titles with Rust get blind upvotes, sure a new kernel is cool but would be more cool if is more then someone toy project, this should probably be posted and upvoted in Rust forums so it gets support from the fans and hopefully the Rust guys will decide what kernel should they focus on and stop promoting every student weekend project here.
gabrielhidasy 5y ago
For each piece of software, there just be a version in Rust, and that version must advertise it's rustiness
- Klasiaster 5y ago
  The type of software makes the difference - does it process untrusted input, maybe even over the network? This here is a kernel including a memory-safe TCP/IP stack (https://github.com/smoltcp-rs/smoltcp/ https://github.com/smoltcp-rs/smoltcp/), and not having it crash or be full of security vulnerabilities due to preventable memory corruption is a quality beyond personal language preferences.
  - fivelessminutes 5y ago
    It is possible to overstate a valid case until it becomes meaningless... are modern OS IP stacks written in C really 'crash'ing or 'full of security vulnerabilities'? No.
    - Klasiaster 5y ago
      The TCP/IP stacks of the few popular OSes is not full of security issues, but that took a lot of effort that a small team can't do. Also, it's a matter of updating: after reviewing the CVE database, would you be comfortable running a 5 year old kernel without updates?
    - roblabla 5y ago
      There has actually been a few crashes in the core TCP/IP stack of both Linux and Windows in the last 5 years, though I'm not sure any of them were due to memory safety (most were logic bugs as far as I can tell). That said, rust's approach to error handling could help avoid crashes here as well. However, we have seen *many* memory-safety crashes in the drivers to talk to network interfaces. Those really are full of security vulnerabilities. The most prominent recent one was broadcom's wifi driver having heap buffer overflows allowing remote code execution by anyone within wifi range in 2019[0]. [0]: https://www.bleepingcomputer.com/news/security/broadcom-wifi-driver-flaws-expose-computers-phones-iot-to-rce-attacks/ https://www.bleepingcomputer.com/news/security/broadcom-wifi...
fork-bomber 5y ago
Very cool! I think a killer feature would be to emulate the Linux kernel device driver model such that existing C driver modules could be used with Kerla. Further, borrow (no pun intended!) the Linux device driver Rust wrappers from the Rust for Linux kernel project to then enable writing Linux device drivers in Rust. The question is whether emulating the Linux device driver model (by which I kind of mean mimicking the set of core kernel services needed to support drivers - kernel threads, sync primitives, allocators etc) ends up being a massive task akin to writing the core kernel itself. I think there might be reasonable trade offs that make the problem more tractable. Having the ability to have a significantly simpler kernel with Rust’s memory and thread safety guarantees plus the ability to use even a small set of Linux’s massive set of drivers would be very compelling.
- roblabla 5y ago
  Linux drivers don't have a fixed ABI (or even API), making such a feat somewhere between hard and impossible. You'd have to constantly play catch up with the latest kernel refactorings, which would require a tremendous amount of engineering effort. Heck, it'd probably be easier to emulate the Windows driver architecture - at least those have proper ABI compat.
  - fork-bomber 5y ago
    I asked this question during the recent Linux kernel plumbers conference: "How often in practice do the kernel A{B,P}Is fro device drivers change ?". I got a very mixed response. My take away was that while 'new' driver classes/subsystems do involve more churn in the core kernel support subsystems, there is a clear trend towards convergence in terms of significant change. If the change truly was significant and continuous then that would impede productisation with the Rust kernel. I'm not saying that the A{B,P}Is are in any way stable but I personally don't think that keeping up will be an intractable problem. In the spirit of Kerla's authors prime motive - which appears to be hacking for knowledge - this might be something to try, perhaps with a driver class that is known to be more stable.
  - krater23 5y ago
    >Heck, it'd probably be easier to emulate the Windows driver architecture - at least those have proper ABI compat. And better drivers.
nivenkos 5y ago
Post it when it's production ready. A big problem in the Rust ecosystem is the lack of continued work and support on libraries (understandably, since many are new and aren't commercially supported). For example, pyroute2 is still better for Linux admin that the equivalent Rust crates. Whilst the latter often fully support async, none of them cover all of the features of pyroute2 nor the easy-to-use API. Hopefully things will improve over time, it's always a shame to see great projects be abandoned.
- kryptiskt 5y ago
  > Post it when it's production ready. This is Hacker News, not Enterprise IT News.
  - nivenkos 5y ago
    I just meant that even big projects posted here in the past like the Headcrab debugger: https://github.com/headcrab-rs/headcrab https://github.com/headcrab-rs/headcrab have been seemingly abandoned now. Even Rocket https://github.com/SergioBenitez/Rocket https://github.com/SergioBenitez/Rocket seems to have greatly slowed development unfortunately.
    - fbanon 5y ago
      Maybe those projects are considered "done".
    - __s 5y ago
      Nowadays the Rust http libraries to use are warp or axum (but Rocket works as is) https://github.com/seanmonstar/warp https://github.com/seanmonstar/warp https://github.com/tokio-rs/axum https://github.com/tokio-rs/axum re headcrab, https://github.com/headcrab-rs/headcrab/issues/132 https://github.com/headcrab-rs/headcrab/issues/132
      - ibraheemdev 5y ago
        > Nowadays the Rust http libraries to use are warp or axum Warp and axum are certainly not the only choices, and saying they are the best choices is a matter of opinion. Actix-web, rocket, tide, warp, axum are all great web frameworks.
ordu 5y ago
Just to clarify goals of the project: "I completely agree that rewriting existing large, feature-rich, and robust operating system kernels is not a good idea. However, a question came to my mind: what are the pros and cons of writing a practical UNIX-like OS kernel in Rust from scratch? How does it look like? This is why I started this project."[1] [1] https://seiya.me/writing-linux-clone-in-rust https://seiya.me/writing-linux-clone-in-rust
nuta 5y ago
Author here. I'm surprised to see my hobby project on Hacker News. I know this kind of stuff spark the ``it's meaningless to rewrite everything (especially Linux) in Rust'' debate. I agree 100% that rewriting Linux in Rust (or your favorite language) is just a waste of time and such a project won't live long. That said, what I'd say here is that it's fun. Really fun. Implementing ABI compatibility requires you to understand under the hood of the kernel. You can learn how printf(3) works. You can learn what happens before main(). Especially, in Rust, it's inspiring to explore ways to write a kernel safely and expressively. One of my favorite part is the implementation of read system call [1]. Lastly, You can try this OS on SSH. Hope you enjoy :) ssh root@kerla-demo.seiya.me [1]: https://github.com/nuta/kerla/blob/main/kernel/syscalls/read.rs https://github.com/nuta/kerla/blob/main/kernel/syscalls/read...
- deleted 5y ago
  [deleted]
- rakoo 5y ago
  > I know this kind of stuff spark the ``it's meaningless to rewrite everything (especially Linux) in Rust'' debate. I agree 100% that rewriting Linux in Rust (or your favorite language) is just a waste of time and such a project won't live long. Considering that this is exactly how Linux was born (just a hobby project for fun), I wouldn't assume so fast that it's useless. Moreover, you don't need to justify yourself if you just want to have fun !
  - nuta 5y ago
    Thanks! I'll keep doing Just for Fun :D
  - uuddlrlr 5y ago
    Any rust project like this inevitably gets sardonic replies about rewrite-in-rust fanaticism. I agree they shouldn't have to justify themselves, but it is handy to preempt that.
    - Iv 5y ago
jacquesm 5y ago
Rewriting the Linux kernel in rust would be a rehash of why we ended up with Linux in the first place: rewriting an older OS from scratch. I would love to see a solid implementation of a real micro kernel based OS based on a message passing core in Rust to become a viable alternative to the now 50 year old Unix architecture. This would give us a possibility of some real progress on the security and process scheduling front, both of which are severely lacking in the Unix model.
- abendstolz 5y ago
  Just in case you don't know about it: https://redox-os.org/ https://redox-os.org/ could be interesting to you :)
- chasil 5y ago
  Fuchsia and QNX provide a lot of what you want. https://en.wikipedia.org/wiki/QNX https://en.wikipedia.org/wiki/QNX https://en.wikipedia.org/wiki/Fuchsia_(operating_system) https://en.wikipedia.org/wiki/Fuchsia_(operating_system) https://lobste.rs/s/p8bizb/getting_know_fuchsia_google_s_open_source https://lobste.rs/s/p8bizb/getting_know_fuchsia_google_s_ope...
  - jacquesm 5y ago
    I think I'm aware of QnX ;) (I wrote a clone).
- afr0ck 5y ago
  Why do people implicitly assume that micro kernels are the best OS design? In fact, micro kernels have a lot of issues and limit the possibilities of what an OS can do! One of the most important issues is they prevents resource sharing in a rich and efficient way between the components of the kernel. They make building rich relationships between kernel data structures and kernel subsystems very messy, complicated, impracticable and very inefficient. Micro kernels are delusional and people should stop implicitly assuming their superiority.
  - 5y ago
EdSchouten 5y ago
This is awesome. Imagine the potential if it was just complete enough to run Kubernetes’ kubelet on a cloud instance. Lots of security minded folks would love it.
- phkahler 5y ago
  That's IMHO how you get something like this off the ground. Make it useful for a simple use-case while offering something of value - in this case the security of Rust. It can then become a preferred solution to some problem and that will bring users and developers looking to incrementally expand it.
- JakaJancar 5y ago
  I wonder, if you run every container in it's own MicroVM a-la Firecracker like Fargate does (or Kerla's demo host), does the security of the kernel still matter as much?
exabrial 5y ago
Dunno: just a hobby, won't be big and professional like linux. :)
ativzzz 5y ago
What are some ways I can increase my knowledge in this domain that the OP is very skilled at, meaning low level OS development? I've taken an intro to OS class and am currently going through Linux From Scratch [1], which is interesting and is teaching me a lot, but it's more about how to setup a Linux distro using existing packages and not really about reading/writing the code involved. Any recommendations? [1] https://www.linuxfromscratch.org/ https://www.linuxfromscratch.org/
- 0x0203 5y ago
  One of the easiest ways to get into writing your own kernel is to start with the OS Dev wiki [0]. Both the wiki and their forums are a great place for getting your feet wet. For more general information, there are also a handful of articles and/or blog posts that I've come across when I was getting started that gave a very good introduction to many of the low-level components of how computers work and what the OS needs to do to run them. One of my favorites was Many But Finite [1]. Check out a bunch of his posts about the memory map, the computer boot process [2], cpu rings, etc... Not exhaustive by any means, but good, well-presented info. Many other such sources exist; maybe others will share their favorites. And if you'd just like to get a more gentle introduction into actual OS code than the monster that is Linux, maybe try looking at some of the BSD's. NetBSD [3] and OpenBSD [4] are remarkably easy to navigate and follow from a code flow perspective, and if you just want to see how things work in a real OS, it's both a good reference and a good place to start. [0] https://wiki.osdev.org/Main_Page https://wiki.osdev.org/Main_Page [1] https://manybutfinite.com/archives https://manybutfinite.com/archives [2] https://manybutfinite.com/post/how-computers-boot-up https://manybutfinite.com/post/how-computers-boot-up [3] http://netbsd.org/ http://netbsd.org/ [4] https://www.openbsd.org/ https://www.openbsd.org/
  - ativzzz 5y ago
    Thank you!
- fitzn 5y ago
  I think you're on the right track. I am similar to you and not so skilled at low level OS development. There is an educational OS called PIOS from Yale's CS department with specific parallelism goals in mind, but it boots from metal so the code is a great resource. Here's the code: https://github.com/bford/PIOS https://github.com/bford/PIOS Also, you could try reading the Plan 9 source code. (I would say the code is what you're after now, but in case you are interested in more of the theory of why it's designed that way, you can check out the research paper here: https://dedis.cs.yale.edu/2010/det/ https://dedis.cs.yale.edu/2010/det/)
jll29 5y ago
Kudos, it works and thee source code is a beautiful read. Thanks so much for sharing!

471 points by Klasiaster 5y ago

This may be a naive question, but I cannot help but wonder: As C and Rust aim to be link-compatible, wouldn't it be easier to gradually replace parts of the Linux kernel with Rust code? The only technical problem I can think of is that Rust may not be available for all CPU architectures Linux supports, but this is just speculation on my part having done no research on the matter.

gspr 5y ago
A practical problem is that in order to gain the benefit of linking Rust and C code, one has to give up Rust's wonderful guarantees at the interface. So until islands of Rust meet up, these interact through a C ABI, and have to expose C-like behavior.
- ls65536 5y ago
  But you still get those guarantees and benefits in the Rust implementation itself, even if not at the interface (to non-Rust code). By a similar argument, a standalone Rust program "gives up" its guarantees whenever the process makes a call to libc or to the operating system (syscalls), but this isn't really a practical problem.
  - gspr 5y ago
    Sure, I agree. The point is that if the total corpus of Rust code is too isolated into too small islands, then the boundary effects of each island may eat up a lot of the benefit. I still think the project is really cool and worthwhile.
- jll29 5y ago
  The world really needs _two_ projects, a practical one that is interoperable with the here and now, to get incremental improvements in security from bits reimplemented in Rust. And then, in the longer term, someone ought to write a book about operating system implementation in Rust that ignores Linux interoperability and focuses on readability, maintainability by showing use of Rust's idiomatic style.
wongarsu 5y ago
That's what may happen with the Linux kernel over time, now that it allows parts written in Rust. The limitation of that approach is that all internal interfaces (which includes most data structures) have to be C-compatible. You can get much of the safety benefits of Rust, but lose a lot of Rust's ergonomics.
- geewee 5y ago
  I don't think that the Linux kernel has decided (yet) to allow rust code into the kernel proper, has it?
  - fork-bomber 5y ago
    It hasn't. But the attitude towards using Rust for device driver development has now gotten very positive, especially post the recent kernel summit.
  - pjmlp 5y ago
    Google has decided to go forward anyway. Just like Android Linux compiles just fine with clang for the last five years or so, it now makes use of Rust. https://source.android.com/setup/build/rust/building-rust-modules/overview https://source.android.com/setup/build/rust/building-rust-mo... If upstream ever cares to support clang or Rust, that is another matter.
    - jstarks 5y ago
      Unless I'm missing something, that link refers to building Android user-mode components in Rust, not kernel components in Rust.
      - steveklabnik 5y ago
        That's correct, but it came after them rebuilding the bluetooth stack https://news.ycombinator.com/item?id=26647981 https://news.ycombinator.com/item?id=26647981
  - 5y ago
fork-bomber 5y ago
The Rust for Linux kernel project aims to enable writing Linux kernel device drivers in Rust. See: https://lwn.net/Articles/862018/ https://lwn.net/Articles/862018/ The issue of Rust's LLVM based compiler toolchain not targeting all CPU archs is intended to be solved by the gcc-rs project. See: https://lwn.net/Articles/871283/ https://lwn.net/Articles/871283/ I think the approach the Rust for Linux project is taking is wise: Not about outright re-writes but more about focusing on those subsystems where Rust's intrinsic safety and security properties helps the most.
- maleldil 5y ago
  Last I checked, one of the big problems was the Rust panics when you run out of memory, which is unacceptable in the kernel. Is there any progress on that?
  - jerven 5y ago
    Yes a lot. https://rust-for-linux.github.io/docs/alloc/alloc/index.html https://rust-for-linux.github.io/docs/alloc/alloc/index.html
  - tialaramex 5y ago
    Out of the box Rust doesn't provide any way to allocate heap memory, so, you can't run out if it. Your ordinary userspace apps use std (the standard library) which relies on the alloc crate, and that provides heap allocation which indeed panics if the allocation fails. Because it correctly guesses that your "clever" strategy to handle allocation failure actually isn't and will just triple fault anyway so it should cut to the chase. The kernel obviously doesn't have std, and Rust for Linux implements its own alloc crate. Linus' requirement that you can fail memory allocation just means all the calls in alloc that can actually allocate memory now return Result to indicate whether the allocation was successful. This isn't how you'd do things in userspace, but, this isn't userspace so fine.
    - abendstolz 5y ago
      Can you elaborate as to why "his isn't how you'd do things in userspace, but, this isn't userspace so fine" holds? Naive me - not a kernel dev at all - would argue that returning Result<Memory, AllocationError> is always better, even for userspace because it would allow me to additionally log something or gracefully deal with this. Even if I don't want to deal with it, I could just `.unwrap()` or `.expect('my error message')` it. Note: I am not trying to be snarky here, I genuinely don't know and would like to. If answering this is too complex, maybe you can point me in the right direction so I can ask the right questions to find answers myself? Thanks in any case!
phendrenad2 5y ago
The Linux kernel is massive, so writing any part of it in Rust is an enormous undertaking. Even a few functions at a time, we're talking decades.
- krylon 5y ago
  Well, writing a kernel from scratch that supports as many CPU architectures and devices as Linux would take about as long, assuming one could attract a critical mass of developers. Using an incremental approach at least gives us some benefits in the near future. I'm not even saying a fresh start would be a bad idea. But incrementally replacing parts of Linux seems like a more promising approach, IMHO.
pornel 5y ago
Yes, gradual replacement is easier and more practical for large projects. However, typical C APIs and idioms are different from idiomatic Rust. Rust uses more type-system features, generics, iterators, RAII, and prefers tree-like data structures and far fewer pointers (no linked lists!). C rewritten in Rust is still very C-like, and requires refactorings that you can't do until it's all Rust. BTW: There are two GCC-based Rust implementations in the works, so compatibility with exotic platforms is going to be solved.
amw-zero 5y ago
It’s not naive, it’s just been speculated and talked about incessantly since Rust came out. The one naive part is underestimating how long that would take - Linux is one of the largest codebases on Earth. It would require massive coordination over years to rewrite, and in the end all you would get is the exact same product. It’s just not an interesting or unique perspective, and pretty silly to suggest seriously.

Klasiaster 5y ago

A minimal interactive demo running busybox is here: ssh root@kerla-demo.seiya.me

codetrotter 5y ago
That’s very cool! Could you share some details on how to set it up for oneself with AWS Firecracker the same way that this demo was set up? Also, as owner of the instances are you able to see what commands people are running? First thing I did was to type hello, and it said hello world to me :)
- Klasiaster 5y ago
  I'm not the project author, maybe you can ask via e-mail, see https://seiya.me/ https://seiya.me/
  - codetrotter 5y ago
    Ok, thanks :)
- nuta 5y ago
  Author here. I've made the demo system public [1] but it's written just for me so it should be painful to set up the same environment. The mechanism is pretty simple: a Node.js server (running on genuine Linux) listens on tcp:22. Once you connect, it boots a dedicated Firecracker microVM instance and forwards network packets between your SSH client and VM. Regarding the command history, others (including I) can't see what you type. If you could, it must be a vulnerability. [1] https://github.com/nuta/kerla-demo https://github.com/nuta/kerla-demo
  - codetrotter 5y ago
    Thank you :)
thawkins 5y ago
is there not an opportunity to implement the posix filesystems interface on top of something other than a block device system. That would be fun.

rvz 5y ago

A star has been born. A Linux alternative needs to be compatible with Linux binaries and also support modern architectures. If you want Linux to survive, it needs to be re-written from scratch, optimised in Rust and compatible with the existing Linux binaries or drivers. This project might have a chance. From: [0] > TL;DR: I'm writing a Linux clone in Rust just for fun. It does NOT aim to replace the Linux kernel. A certain someone thought that their hobby OS Minix clone won't be 'big or professional like GNU'. It's time for 'change'. [0] https://seiya.me/writing-linux-clone-in-rust https://seiya.me/writing-linux-clone-in-rust

breakfastduck 5y ago
You think Linux, which the vast majority of web servers run today, is going to die out if it's not rewritten from scratch in Rust? How on earth have you come to that one?
tonetheman 5y ago
sigh... Linux is going to survive. And no it does not need to be re-written from scratch in Rust. Someone lied to you at the church of rust.
- thawkins 5y ago
  But its a fun thing to do, and that's a good enough reason to do it. After all is that not exactly how Linux came into being anyway.
  - simion314 5y ago
    Maybe I will write a TODO app in Rust, should it be on top of HN because is some unoriginal,incomplete,bug filled,toy stuff just because the reason is written in Rust? 1 My assumption is that all titles with Rust get blind upvotes, sure a new kernel is cool but would be more cool if is more then someone toy project, this should probably be posted and upvoted in Rust forums so it gets support from the fans and hopefully the Rust guys will decide what kernel should they focus on and stop promoting every student weekend project here.

gabrielhidasy 5y ago

For each piece of software, there just be a version in Rust, and that version must advertise it's rustiness

Klasiaster 5y ago
The type of software makes the difference - does it process untrusted input, maybe even over the network? This here is a kernel including a memory-safe TCP/IP stack (https://github.com/smoltcp-rs/smoltcp/ https://github.com/smoltcp-rs/smoltcp/), and not having it crash or be full of security vulnerabilities due to preventable memory corruption is a quality beyond personal language preferences.
- fivelessminutes 5y ago
  It is possible to overstate a valid case until it becomes meaningless... are modern OS IP stacks written in C really 'crash'ing or 'full of security vulnerabilities'? No.
  - Klasiaster 5y ago
    The TCP/IP stacks of the few popular OSes is not full of security issues, but that took a lot of effort that a small team can't do. Also, it's a matter of updating: after reviewing the CVE database, would you be comfortable running a 5 year old kernel without updates?
  - roblabla 5y ago
    There has actually been a few crashes in the core TCP/IP stack of both Linux and Windows in the last 5 years, though I'm not sure any of them were due to memory safety (most were logic bugs as far as I can tell). That said, rust's approach to error handling could help avoid crashes here as well. However, we have seen *many* memory-safety crashes in the drivers to talk to network interfaces. Those really are full of security vulnerabilities. The most prominent recent one was broadcom's wifi driver having heap buffer overflows allowing remote code execution by anyone within wifi range in 2019[0]. [0]: https://www.bleepingcomputer.com/news/security/broadcom-wifi-driver-flaws-expose-computers-phones-iot-to-rce-attacks/ https://www.bleepingcomputer.com/news/security/broadcom-wifi...

fork-bomber 5y ago

Very cool! I think a killer feature would be to emulate the Linux kernel device driver model such that existing C driver modules could be used with Kerla. Further, borrow (no pun intended!) the Linux device driver Rust wrappers from the Rust for Linux kernel project to then enable writing Linux device drivers in Rust. The question is whether emulating the Linux device driver model (by which I kind of mean mimicking the set of core kernel services needed to support drivers - kernel threads, sync primitives, allocators etc) ends up being a massive task akin to writing the core kernel itself. I think there might be reasonable trade offs that make the problem more tractable. Having the ability to have a significantly simpler kernel with Rust’s memory and thread safety guarantees plus the ability to use even a small set of Linux’s massive set of drivers would be very compelling.

roblabla 5y ago
Linux drivers don't have a fixed ABI (or even API), making such a feat somewhere between hard and impossible. You'd have to constantly play catch up with the latest kernel refactorings, which would require a tremendous amount of engineering effort. Heck, it'd probably be easier to emulate the Windows driver architecture - at least those have proper ABI compat.
- fork-bomber 5y ago
  I asked this question during the recent Linux kernel plumbers conference: "How often in practice do the kernel A{B,P}Is fro device drivers change ?". I got a very mixed response. My take away was that while 'new' driver classes/subsystems do involve more churn in the core kernel support subsystems, there is a clear trend towards convergence in terms of significant change. If the change truly was significant and continuous then that would impede productisation with the Rust kernel. I'm not saying that the A{B,P}Is are in any way stable but I personally don't think that keeping up will be an intractable problem. In the spirit of Kerla's authors prime motive - which appears to be hacking for knowledge - this might be something to try, perhaps with a driver class that is known to be more stable.
- krater23 5y ago
  >Heck, it'd probably be easier to emulate the Windows driver architecture - at least those have proper ABI compat. And better drivers.

nivenkos 5y ago

Post it when it's production ready. A big problem in the Rust ecosystem is the lack of continued work and support on libraries (understandably, since many are new and aren't commercially supported). For example, pyroute2 is still better for Linux admin that the equivalent Rust crates. Whilst the latter often fully support async, none of them cover all of the features of pyroute2 nor the easy-to-use API. Hopefully things will improve over time, it's always a shame to see great projects be abandoned.

kryptiskt 5y ago
> Post it when it's production ready. This is Hacker News, not Enterprise IT News.
- nivenkos 5y ago
  I just meant that even big projects posted here in the past like the Headcrab debugger: https://github.com/headcrab-rs/headcrab https://github.com/headcrab-rs/headcrab have been seemingly abandoned now. Even Rocket https://github.com/SergioBenitez/Rocket https://github.com/SergioBenitez/Rocket seems to have greatly slowed development unfortunately.
  - fbanon 5y ago
    Maybe those projects are considered "done".
  - __s 5y ago
    Nowadays the Rust http libraries to use are warp or axum (but Rocket works as is) https://github.com/seanmonstar/warp https://github.com/seanmonstar/warp https://github.com/tokio-rs/axum https://github.com/tokio-rs/axum re headcrab, https://github.com/headcrab-rs/headcrab/issues/132 https://github.com/headcrab-rs/headcrab/issues/132
    - ibraheemdev 5y ago
      > Nowadays the Rust http libraries to use are warp or axum Warp and axum are certainly not the only choices, and saying they are the best choices is a matter of opinion. Actix-web, rocket, tide, warp, axum are all great web frameworks.

ordu 5y ago

Just to clarify goals of the project: "I completely agree that rewriting existing large, feature-rich, and robust operating system kernels is not a good idea. However, a question came to my mind: what are the pros and cons of writing a practical UNIX-like OS kernel in Rust from scratch? How does it look like? This is why I started this project."[1] [1] https://seiya.me/writing-linux-clone-in-rust https://seiya.me/writing-linux-clone-in-rust

nuta 5y ago

Author here. I'm surprised to see my hobby project on Hacker News. I know this kind of stuff spark the ``it's meaningless to rewrite everything (especially Linux) in Rust'' debate. I agree 100% that rewriting Linux in Rust (or your favorite language) is just a waste of time and such a project won't live long. That said, what I'd say here is that it's fun. Really fun. Implementing ABI compatibility requires you to understand under the hood of the kernel. You can learn how printf(3) works. You can learn what happens before main(). Especially, in Rust, it's inspiring to explore ways to write a kernel safely and expressively. One of my favorite part is the implementation of read system call [1]. Lastly, You can try this OS on SSH. Hope you enjoy :) ssh root@kerla-demo.seiya.me [1]: https://github.com/nuta/kerla/blob/main/kernel/syscalls/read.rs https://github.com/nuta/kerla/blob/main/kernel/syscalls/read...

deleted 5y ago
[deleted]
rakoo 5y ago
> I know this kind of stuff spark the ``it's meaningless to rewrite everything (especially Linux) in Rust'' debate. I agree 100% that rewriting Linux in Rust (or your favorite language) is just a waste of time and such a project won't live long. Considering that this is exactly how Linux was born (just a hobby project for fun), I wouldn't assume so fast that it's useless. Moreover, you don't need to justify yourself if you just want to have fun !
- nuta 5y ago
  Thanks! I'll keep doing Just for Fun :D
- uuddlrlr 5y ago
  Any rust project like this inevitably gets sardonic replies about rewrite-in-rust fanaticism. I agree they shouldn't have to justify themselves, but it is handy to preempt that.
  - Iv 5y ago

jacquesm 5y ago

Rewriting the Linux kernel in rust would be a rehash of why we ended up with Linux in the first place: rewriting an older OS from scratch. I would love to see a solid implementation of a real micro kernel based OS based on a message passing core in Rust to become a viable alternative to the now 50 year old Unix architecture. This would give us a possibility of some real progress on the security and process scheduling front, both of which are severely lacking in the Unix model.

abendstolz 5y ago
Just in case you don't know about it: https://redox-os.org/ https://redox-os.org/ could be interesting to you :)
chasil 5y ago
Fuchsia and QNX provide a lot of what you want. https://en.wikipedia.org/wiki/QNX https://en.wikipedia.org/wiki/QNX https://en.wikipedia.org/wiki/Fuchsia_(operating_system) https://en.wikipedia.org/wiki/Fuchsia_(operating_system) https://lobste.rs/s/p8bizb/getting_know_fuchsia_google_s_open_source https://lobste.rs/s/p8bizb/getting_know_fuchsia_google_s_ope...
- jacquesm 5y ago
  I think I'm aware of QnX ;) (I wrote a clone).
afr0ck 5y ago
Why do people implicitly assume that micro kernels are the best OS design? In fact, micro kernels have a lot of issues and limit the possibilities of what an OS can do! One of the most important issues is they prevents resource sharing in a rich and efficient way between the components of the kernel. They make building rich relationships between kernel data structures and kernel subsystems very messy, complicated, impracticable and very inefficient. Micro kernels are delusional and people should stop implicitly assuming their superiority.
- 5y ago

EdSchouten 5y ago

This is awesome. Imagine the potential if it was just complete enough to run Kubernetes’ kubelet on a cloud instance. Lots of security minded folks would love it.

phkahler 5y ago
That's IMHO how you get something like this off the ground. Make it useful for a simple use-case while offering something of value - in this case the security of Rust. It can then become a preferred solution to some problem and that will bring users and developers looking to incrementally expand it.
JakaJancar 5y ago
I wonder, if you run every container in it's own MicroVM a-la Firecracker like Fargate does (or Kerla's demo host), does the security of the kernel still matter as much?

exabrial 5y ago

Dunno: just a hobby, won't be big and professional like linux. :)

ativzzz 5y ago

What are some ways I can increase my knowledge in this domain that the OP is very skilled at, meaning low level OS development? I've taken an intro to OS class and am currently going through Linux From Scratch [1], which is interesting and is teaching me a lot, but it's more about how to setup a Linux distro using existing packages and not really about reading/writing the code involved. Any recommendations? [1] https://www.linuxfromscratch.org/ https://www.linuxfromscratch.org/

0x0203 5y ago
One of the easiest ways to get into writing your own kernel is to start with the OS Dev wiki [0]. Both the wiki and their forums are a great place for getting your feet wet. For more general information, there are also a handful of articles and/or blog posts that I've come across when I was getting started that gave a very good introduction to many of the low-level components of how computers work and what the OS needs to do to run them. One of my favorites was Many But Finite [1]. Check out a bunch of his posts about the memory map, the computer boot process [2], cpu rings, etc... Not exhaustive by any means, but good, well-presented info. Many other such sources exist; maybe others will share their favorites. And if you'd just like to get a more gentle introduction into actual OS code than the monster that is Linux, maybe try looking at some of the BSD's. NetBSD [3] and OpenBSD [4] are remarkably easy to navigate and follow from a code flow perspective, and if you just want to see how things work in a real OS, it's both a good reference and a good place to start. [0] https://wiki.osdev.org/Main_Page https://wiki.osdev.org/Main_Page [1] https://manybutfinite.com/archives https://manybutfinite.com/archives [2] https://manybutfinite.com/post/how-computers-boot-up https://manybutfinite.com/post/how-computers-boot-up [3] http://netbsd.org/ http://netbsd.org/ [4] https://www.openbsd.org/ https://www.openbsd.org/
- ativzzz 5y ago
  Thank you!
fitzn 5y ago
I think you're on the right track. I am similar to you and not so skilled at low level OS development. There is an educational OS called PIOS from Yale's CS department with specific parallelism goals in mind, but it boots from metal so the code is a great resource. Here's the code: https://github.com/bford/PIOS https://github.com/bford/PIOS Also, you could try reading the Plan 9 source code. (I would say the code is what you're after now, but in case you are interested in more of the theory of why it's designed that way, you can check out the research paper here: https://dedis.cs.yale.edu/2010/det/ https://dedis.cs.yale.edu/2010/det/)

jll29 5y ago

Kudos, it works and thee source code is a beautiful read. Thanks so much for sharing!