10 ms·
The Case for Writing a Kernel in Rust [pdf]
- steveklabnik 9y agoNote that one of the co-authors of this paper, Amit, is the author of https://www.tockos.org/ https://www.tockos.org/ This paper is sort of a successor to https://sing.stanford.edu/site/publications/levy-plos15-tock.pdf https://sing.stanford.edu/site/publications/levy-plos15-tock... , a previous paper on this topic. After publication, Amit solved many of the issues presented there. It was previously discussed on HN https://news.ycombinator.com/item?id=14105884 https://news.ycombinator.com/item?id=14105884 Some more context https://www.reddit.com/r/rust/comments/655816/ownership_is_theft_experiences_building_an/dg7p2uf/ https://www.reddit.com/r/rust/comments/655816/ownership_is_t... Amit left even more context just now on /r/rust: https://www.reddit.com/r/rust/comments/75xwib/the_case_for_writing_an_os_in_rust/do9wk3g/ https://www.reddit.com/r/rust/comments/75xwib/the_case_for_w...
- frankmcsherry 9y agoSome minor pedantry that winds me up a bit, and that I thought I'd try to correct: > Note that one of the co-authors of this paper, Amit, is the author of https://www.tockos.org/ https://www.tockos.org/ The definite article "the" precludes the existence of other authors, who (from personal experience) can get wound up by language like this. You may have meant "author of the web site", in which case I have no opinion, but if you meant "author of the project" I'd totally use the indefinite article "an". Also all of the work is very interesting and people should read it.
- steveklabnik 9y agoYes, good call, thank you! I sometimes forget this distinction. You're totally correct that more people than Amit work on Tock, for sure, and I don't want to exclude their hard work.
- nickpsecurity 9y agoThis is good work. I enjoyed reading it. I wonder, though, why Muen separation kernel wasn't mentioned in related work on kernels in safe languages. It uses the SPARK language so that it's provably immune to a number of defects. The Ada compilers can also give you as little or much runtime as you want with several "profiles" available. Muen uses the "minimal, Zero-Runtime" profile. Rust improves on Ada and SPARK with the affine types giving temporal safety plus more flexible model for safe concurrency than Ravenscar. CompSci work shows one can probably do that and info-flow security with contracts but that would be extra, specialist work for Ada/SPARK versus what's built-in to Rust already. Verbose, too. https://muen.codelabs.ch/ https://muen.codelabs.ch/ When I about the Rust kernels, I assumed there would be some trusted components in there which could screw them up on assurance side. Aside from full formal verification, one thing someone might want to try is writing those pieces in SPARK like Muen does. Make a direct port from any proven SPARK to Rust source. Alternatively, compile the SPARK into executable code that Rust calls with its FFI since C code can call SPARK code. Then, those untrusted portions are shown safe with one set of rules while the rest is shown safe with Rust's compiler. Optionally, convert those to an intermediate language of CompCert or CakeML compilers for certified compilation. This is how my Brute-Force Assurance model would attempt to handle each's limitations in a kernel project. Note: Only problem with using SPARK for that is the person doing it probably has to buy a license if the Rust kernel in question isn't GPL. SPARK is only free for use with GPL code. Frama-C is another option one might use to avoid that but SPARK is stronger offering.
- pjmlp 9y agoIf you see from my discussions, many in the community (not the Rust designers) seem to be unaware, willing or not, of all the great work that came before Rust. Rust is making a great progress infecting other languages with the idea of having affine types for resource management and concurrency is possible, just that the ergonomics still need some fine tuning. No need to glace over the ideas of other strong type systems programming languages.
- pjmlp 9y agoOh well, not being a native speaker only now I realized that the last sentence is the complete opposite of what I intended to say, and the edit is button is gone now. What I actually wanted to say, is that it would be worthwhile for the community to learn what has been done before, like the examples referred by nicksecurity.
- naasking 9y agoThere's another OS that used some aspects of safe code injected into the kernel: Pebble [1] [1] https://www.usenix.org/conference/workshop-embedded-systems/pebble-component-based-operating-system-embedded-applications https://www.usenix.org/conference/workshop-embedded-systems/...
- jp_rider 9y agoIf you haven't already, check out Redox OS [1,2]. It's a microkernel written in Rust. Here's a good interview with the developer [3]. [1] https://www.reddit.com/r/Redox/ https://www.reddit.com/r/Redox/ [2] https://redox-os.org/ https://redox-os.org/ [3] https://www.youtube.com/watch?v=eH5JgMlNE8o https://www.youtube.com/watch?v=eH5JgMlNE8o
- Animats 9y agoThat's very nice. It's approaching the QNX level of microkernel. One unusual feature of QNX is that the kernel doesn't parse strings anywhere. There's a "resource manager", but that's a process. Programs register with the resource manager for a piece of the pathname namespace ("/dev", "/fs", etc.) and then get open requests sent to them when a pathname starts with their part of the namespace. Parsing creates a large attack surface, and getting it out of the kernel is a win. QNX tries to avoid variable-length objects in the kernel. Messages are variable length and copied by the kernel, but from one user space to another, not queued in kernel space. Most of the ways a kernel can run out of memory are avoided in QNX. If the kernel is out of resources, some system calls return errors, but the kernel doesn't crash. If you're doing a kernel in Rust, it's helpful to think that way. Rust doesn't handle out-of-memory conditions well.
- kbenson 9y ago> Rust doesn't handle out-of-memory conditions well. Do you happen to know languages that do handle out of memory conditions well? That seems like an interesting topic. If I understand how it's done in C, I wouldn't call that "well", but it does provide the mechanisms for doing it (which is more than can be said for all languages). Language level features (or coding styles in C that could be implemented at a language level elsewhere) that provide for increased and intuitive control would be interesting.
- Animats 9y agoAda is one of the few languages that takes out-of-memory conditions seriously. The exception Storage_Error is raised. Java, C# and Microsoft's common runtime have out-of-memory exceptions, but I'm not sure how reliable they are in a limited-memory environment. They're more like "GC isn't helping much" exceptions.
- mtgx 9y agoSomeone send this to the Google Fuchsia team.
- pjmlp 9y agoI guess Google is somehow aware of safety. Regarding Fuchsia, the C code is being migrated to modern C++, the TCP/IP stack and a few filesystem drivers are written in Go, and there are some other small pieces in Rust. Brillo was to be Android native layer + C++ Frameworks, it became Android Things instead, with the ability of writing userspace drivers in Java. ChromeOS requires disabling security to access the native layer. On Android the NDK main goals are only for performance or accessing libraries written in C or C++ into Java.
- Rusky 9y agoUnfortunately even modern C++ brings none of the benefits described in the article, because it defaults to unsafe code. Most of the Rust code from the Fuchsia team is to support Rust in applications, not in any code that needs to be trusted.
- pjmlp 9y agoWhen used together with another language high level language, C++ takes the role of low level unsafe blocks. So there wouldn't exist too much difference with a pure Rust application full of unsafe blocks. Which is the approach taken by Google on Android and ChromeOS. My understanding is that Fuchsia would follow the same approach, using only Dart for applications. Are there any examples of Rust applications in Fucshia?
- pcwalton 9y agoYou can't generally inline unsafe C++ into other languages (like, literally in the same file, or method) the way you can inline unsafe blocks into Rust. This increased friction invariably leads to C++ folks making a lot more code unsafe than needs to be.
- Sag0Sag0 9y agoI tried to write a kernel in rust but reverted back to c. I just feel that it is not very nice systems language. The entire time i felt like i was fighting to manipulate the language into doing something it wasn't supposed to do.
- gwerbin 9y agoWould you be able to go into more detail on this? An example maybe? It might be useful for someone like me who's "interested in Rust" but hasn't tried it (yet).
- steveklabnik 9y agoThere's also a lot of us who do enjoy writing kernels in Rust; https://os.phil-opp.com/ https://os.phil-opp.com/ goes through the start of ones for x86_64, and really shows off some nice features about Rust.
- Sag0Sag0 9y agoThe amount of unsafe code necessary for low level work, the difficulty in sending a stream of bytes to an io port and the innate complexity of the language. C's philosophy is everything is a series of bits which you can manipulate at will. Rust's is the same except you have to through a huge amount of hoops first to make things "safe". Things like converting a String to an i64 are a pain. There is large amount of boilerplate code needed. All in all this is a personal preference. I prefer some unsafeness in c to (in my opinion) the constant inconvenience of programming in rust.
- banachtarski 9y agoBe prepared for issues with ABI incompatibility and all that. Rust is a low level-language, but IMO not a systems language (disclaimer: I tried rust extensively over a 2 week period, background is a blend of C, C++ and Haskell from which it draws a lot of influence).
- bluejekyll 9y agoI’m sorry. But claiming Rust is not a systems language is akin to claiming a Volvo is not a car. If you don’t consider Rust to be a systems language, then you’ve got a very skewed perception of what a systems language is. You seem to be alluding to Rusts ABI potentially being an issue, and that’s a fair criticism. Where a stable ABI needs to be exported, Rust supports the C FFI, like almost all other languages.
- frankmcsherry 9y agoI've had a bunch of issues getting clean statements about the ABI that actually result in usable guarantees. For example, while you are advised to put `#[repr(C)]` on types you want to have a known layout for FFI, to the best of my understanding this decoration isn't included for generic types like tuples, slices, or `Vec`. Afaiu, Rust reserves the right to change the layout of `(T1, T2)` for `#[repr(C)]` types `T1` and `T2` on a build-by-build basis, which makes a lot of the core types unsuitable for FFI. Anyhow, that's my issue with it as a "systems" language. I can't (yet) write code that compiles down to the minimal operations I believe the computer can do, without re-inventing a non-trivial hunk of the standard library. Or, if I can, I can't find the language in the spec that ensures I won't have to fix things.
- pcwalton 9y agoI don't believe that Rust slices and tuples have unstable ABIs. I guess that isn't documented as well as it could be, though. File an issue perhaps? Isn't std::vector in C++ equally unspecified, though?
- 9y ago
- unlmtd1 9y agoCan we please just kill PDF? Have those ppl heard of HTML? Mind boggling!
- jpfr 9y agoThe easiest way forward for a rust kernel is not to start from scratch, but to port Minix3. Minix is an open source micro-kernel somewhat similar to QNX. The kernel itself is on the order of 4000 lines of code. [1] Though small, Minix has quite a long history, is super-stable and looks like a sweet spot in kernel development. The kernel could be ported piecemeal while always being able to boot during the transition. Similar to how remacs is slowly porting emacs over to rust [2]. And on top of that, you can use the rump kernel approach to run the device drivers, TCP/IP, etc. from NetBSD. So you immediately get a fully functional OS. [1] http://www.minix3.org/ http://www.minix3.org/ [2] https://github.com/Wilfred/remacs https://github.com/Wilfred/remacs
- eggestad 9y agoLet me be the contrarian to make the case for NOT writing a new kernel at all (no matter what language) Many a year ago (1997-2000) I was working on banking and finance software. I was a part of that breed of programmers that loath cobol and wanted everything to be in an object oriented language. (C++ was the poison of choice at the time.) The was a particular evangelical group of us in this company. With this group I found my self in a meeting with one of the old timers, business graduate, his jobs was to explain to us CS diaper babies how banking and finance actually worked when we wrote the software. (surprisingly NOT we're actually clueless about the bigger purpose of what a core banking system is fore (other than the obvious)). After patently listening to us with the facial expression that we're all diaper babies, he asked a rather pointed question: OK, supposed we did this, write a new system from scratch using OO programming and all the other industry standard components as of today, what new features to the end user can you build? Translated, when he was going to walk over to the bank to sell it, what would be his pitch? I got the message, what new pitch would the bank get to get new customers? Or simplified, when was the last time you went in to a prospecting new bank and asked 'What programming language or paradigm is your core banking system written in?' I don't give a shit about what language a kernel is written in, only what the system calls do. Please tell me what system call you can write in RUST that you can't write in C. The first time the problem arose when it was necessary to translate one HW assembly to a new CPU, the people at the time took a look at the problem and asked the question. What is easier, write an assembler A to assembler B translator, or write the kernel in the most minimal language possible that then can be compiled for an infinite HW architectures. The answer was simple; C. The answer is equally easy now. There is no system call you can implement in RUST that you can't implement in C. Just like core banking, finance and insurance systems are still in COBOL, the OS kernels will be for the foreseeable future in C. The case for writing a kernel in RUST should be because a need a fun hobby. The thing is that a proof of concept kernel is done when you've doen 10% of the work. The remaining 90% is just mind numbingly boring shit that you'll never complete. Even if you do, what's the end user's justification to switch? On the issue of micro kernel vs monolithic kernel, I'll most humbly refer to Linus Torvalds previous statements. TJ
- kbenson 9y agoI find it interesting that even though your story is about banking and finance software, you did not once mention safety or security. That doesn't magically make it a good idea to start from scratch, but unless you have a perfect security record, it should definitely be part of the deciding criteria. For something like the kernel, well they are still dealing with problems in C today. [1] 1: https://www.exploit-db.com/search/?action=search&q=kernel&g-recaptcha-response=&text=overflow&platform=16 https://www.exploit-db.com/search/?action=search&q=kernel&g-... (answer the captcha and search again)