24 ms·
Is it time to rewrite the operating system in Rust? [slides]
- bryanrasmussen 8y agohttp://www.erights.org/elang/intro/index.html http://www.erights.org/elang/intro/index.html
- tux1968 8y agoHadn't heard about this before, and it does look interesting. But not sure why you're linking to it in this context without comment? Especially since it seems to be a moribund project.
- bryanrasmussen 8y agoE had a pretty interesting security model and there were in various E writings expressions of pipe dreams that operating systems should be rewritten using it. I should probably have linked to one of those to make the connection more explicit, but really it was just a momentary bit of snarkiness that didn't deserve more than what I gave it.
- deleted 8y ago[deleted]
- exabrial 8y agoPresentation by the Joyent CTO btw, who hired a lot of the old Sun Solaris team.
- qaq 8y agoAmong other things because he is also was part of the old Solaris team?
- exabrial 8y agoHilarious example of why English is a terrible language: I meant for the the pronoun to refer to Joyent, not [necessarily] the CTO.
- twtw 8y agoOperator errors do not indicate the quality of a language.
- ryan93 8y agouse "which hired"
- exabrial 8y ago*whom
- magduf 8y agoNo, that's an example of someone not knowing the language very well. You should have written something like, "which is the company that...". I can't think offhand of any language that's really better here in a sense of being more efficient, except probably Classical Latin which is horribly complicated and of course a dead language.
- TomMarius 8y agoCzech allows you to specify that with a change of one letter depending on who you mean (the CTO - který, the company - která) - this depends on different genders of the two words, but is commonly used.
- jbb67 8y agono.
- drakenot 8y agoThe slide deck comes to this conclusion, at least for now. Rather, he discusses a hybrid approach of writing certain modules / drivers of a C-based kernel in Rust so it can be done a little bit at a time.
- keir-rex 8y agoI was expecting to see this type of response. I don't program at low levels but it seems like this would be the responding answer. Kernel programmers would be using something else if that's what they wanted to use. Decades of experience and hard learnt lessons down the drain.
- keir-rex 8y agoAlso, if you can abstract away at a sufficiently low level why not just do it in C where there's already a generation of experience and familiarity?
- RhodesianHunter 8y agoSure, why not just continue writing C for the rest of time?
- masklinn 8y agoThe entire point of the presentation is to discuss these issues, and recommend rust for more ancillary tasks (modules/drivers, firmware, utilities) than the kernel itself. > Kernel programmers would be using something else if that's what they wanted to use. The vast majority of the languages having arisen in the last 40 or so years are completely unsuitable for kernel development, so not necessarily.
- wwright 8y agoOn the other hand, the design of Rust is directly and deeply influenced by those very lessons. Less “down the drain,” possibly more close to “baked directly into the tools we use.”
- jorangreef 8y agoI think Zig would be better suited. See: https://github.com/ziglang/zig/wiki/Why-Zig-When-There-is-Already-CPP%2C-D%2C-and-Rust%3F https://github.com/ziglang/zig/wiki/Why-Zig-When-There-is-Al...
- Verdex_3 8y agoThis isn't quite what I was going to say, but it's close enough. I think parts of an OS can and should be written in something rust like, however the whole thing should probably not be written in rust. My personal thought is that C and C++ 98 era are bad, but they were what we needed at the time. Part of this badness was that you could do anything but exact facilities for specialized tasks had to be rolled from scratch. Libraries are possible, but not ideal (no namespaces for C and crazy header + recompile issues for both). In order to replace C and C++ we need more than just one language, we need an entire family of languages that all have the specific features implemented to support their domain. Rust can probably be used in game programming, but Zig (or in theory eventually Jai) is being specifically made with that goal in mind so they will probably be better. Rust itself will be much more useful for applications that cant stomach managed languages (ie web browsers, etc) because that's what they are specifically building it to do. It's got high level features in a form factor that you would expect from C#/Java with low runtime and no GC. We can probably write OSes in Rust or Zig, but ideally someone who really likes thinking about OSes will create a language specifically geared towards writing OSes that will be able to replace C.
- ansible 8y agoI took a brief look at Zig. The top slogan for Rust is "fearless concurrency", which ties back into the memory model, mutability and sharing references. This helps you prevent (at compile-time) data races and other interesting bugs. I didn't see any support for that in Zig at all.
- jungler 8y agoLast I checked Zig is more concerned about overflow conditions than Rust(numeric overflow, stack overflow, OOM etc.) and boasts features in that vein - so it competes, but not in the same territory. If you are writing a kernel it's highly likely that you'll end up in a lot of scenarios where you have to use unsafe Rust, in which case Zig would be the better language for the task. Admittedly, it would be nice to have some of both, and they both have room to get better. But I see Rust as more of a "libraries and applications" centric setup, where it's the right thing for a web browser and the wrong thing for a device driver.
- BruceM 8y agoIt'll be interesting to see how much use Rust gets within Fuchsia atop the Zircon microkernel.
- steveklabnik 8y agoThe current state is, a few hundred thousands of lines, and Google hiring more Rust engineers quite aggressively. I'm also excited to see how it turns out :)
- agumonkey 8y agoAny other high profile company/project using Rust ? (beside mozilla of course)
- steveklabnik 8y agoFacebook, Dropbox, cloudflare, fastly, tons of startups you’ve heard of...
- mepian 8y agoThe slide about the operating systems of the 1980s ignores Lisp machines. Genera was not a "research system" and it had good performance. There was also Xerox Star's OS written in Mesa.
- twoodfin 8y agoIs there a video of this presentation? bcantrill’s always a fun / interesting watch.
- justincormack 8y agoThere will be it was recorded. From Qcon SF this week.
- adamnemecek 8y agoRust allows for new architectures. Eg beos style os would be a match made in heaven as rust really shines when it comes to async code. Also maybe you could have a less complicated memory manager as you can deal with allocations statically.
- qaq 8y ago"rust really shines" they need to land async/await
- steveklabnik 8y agoWe really do. It’s in nightly; likely to hit stable early next year. One of its precursors is being stabilized as we speak.
- EatYourGreens 8y agoThese slides are quite high-level, so I've got to ask: what's the expected benefit of using Rust to implement a kernel? I somehow thought that nearly all interesting concurrency there would not fit into the paradigm of exclusive ownership. And if we are preaching for programming in unsafe Rust, then doesn't the message become less compelling?
- empath75 8y agoIn practice, even in redox there’s not as much unsafe code as you might think.
- eximius 8y agoThe purported benefit would be to wrap all of the unsafe bits in safe wrappers and then have a less bug prone, more secure OS, I suppose. This would be, to put it mildly, quite difficult. Even then, unsafe code is sometimes not written correctly which brings the whole thing down. I, personally, still think it's worth it. I think efforts like Redox OS can teach us a lot about what we're doing and offer a chance to collapse some of the layers of cruft existing OSes have accumulated.
- OoooooooO 8y agoHaving only a small part of the code with unsafe means you only have to check only a small % of the code for UB/security holes and not the whole code base like in C. It just limits the places shit can happen and which need to be closely reviewed which alone is a big help.
- cyphar 8y agoIt should be noted that you can have correctness bugs can occur in safe code if a Rust guarantee was violated in an unsafe block. This might seem obvious, but it does mean that when you hit one of these bugs you might start with debugging safe code and thus it's not as clean a separation as some Rust evangelists might imply.
- TheCoelacanth 8y ago
- jillesvangurp 8y agoI think the question is wrong. People are already writing new operating systems in Rust just because they can. The question is whether we will end up using their efforts or whether we keep on going back to the same monolith kernels we have been using for the last thirty years or so. Whether it is NT, Linux, or any of the BSD kernels, they each have decades of history behind them. These will of course still be around for the foreseeable future (i.e. decades). At the same time, we are seeing Rust pop up in a lot of places that used to be the exclusive domain of C. People are already re-implementing libraries, popular command line tools, etc. A lot of these implementations have clear merit in the sense that they are faster, safer to use, easier to scale, maintain, etc. IMHO it is just a matter of time before vendors start providing e.g. drivers written in Rust for their hardware. I could see linux evolve to a point where integrating drivers like that is both possible and common. Once that happens, it will be a hybrid kernel effectively. Rust is not the only thing moving that direction; wasm is also going that direction. People are talking about running that in the kernel. And of course Rust runs on top of that as well. So one outcome would be a lot of Rust code running in a wasm sandbox on top of legacy kernels.
- lachlan-sneff 8y agoThat's what nebulet is doing!
- gammagoblin 8y ago>The question is whether we will end up using their efforts or whether we keep on going back to the same monolith kernels we have been using for the last thirty years or so. Whether it is NT, Linux, or any of the BSD kernels, they each have decades of history behind them. Please just... stop. You are using terms that you clearly don't understand the meaning of. What is a "monolith" from your point of view? Because I can tell you that Windows NT is most certainly not a monolithic kernel. Its architecture is very much that of an impure hybrid kernel. You're using the term "monolith" as some sort of buzzword in a realm where that exact term already has a highly specific meaning. >At the same time, we are seeing Rust pop up in a lot of places that used to be the exclusive domain of C. People are already re-implementing libraries, popular command line tools, Yes, but some (e.g. ripgrep) don't actually do the same thing as the programs they are supposed to "re-implement". Most Rust "re-implementations" don't actually re-implement programs, they are new programs doing a similar-but-not-quite-the-same thing to the older C implementations. For example, ripgrep is neither GNU or POSIX compliant, and therefore it is not a re-implementation. >I could see linux evolve to a point where integrating drivers like that is both possible and common. Once that happens, it will be a hybrid kernel effectively. Hybrid kernels have absolutely nothing to do with the language they are written in beyond the fact that they're written in it. Monolithic, hybrid, microkernels and exokernels are all terms to describe the architecture of a kernel. >So one outcome would be a lot of Rust code running in a wasm sandbox on top of legacy kernels. So then Rust and wasm isn't a part of the kernel is what you're saying.
- otabdeveloper1 8y ago> Is it time to rewrite the operating system in Rust? No. Next question, please.
- albru123 8y agoHow constructive of you...
- deleted 8y ago[deleted]
- deleted 8y ago[deleted]
- tom_mellior 8y agoBit of a mixed bag. Slide 12: > Go etc. are garbage collected, making interacting with C either impossible or excruciatingly slow. "Impossible" is, of course, a lie. "Excruciatingly slow" may be the case, though it depends on the kind of interaction you are looking at. Without any context, this statement is overly general. Slide 14 can be taken out of context and misrepresented as showing that "Rust can be faster than C", although the programs being compared use different data structures, so it's inconclusive. (It's not clear to me whether the author wanted to make a broad claim like this. The title suggests so, but again, the difference in data structures suggests otherwise.) Slide 16: > every operating system retains some assembly for reasons of performance Is this really the case? I would be interested in examples. My impression was that assembly is needed for hardware interaction for which there are simply no C constructs available (setting timers, interrupts, system registers, and whatnot), and possibly highly space-constrained bootloader stuff, but performance? C compilers are not stupid. Slide 19: > in-kernel C tends to be de facto safe This is a bit of a stretch. I mean, yes, most lines of the Linux kernel have never been the cause of a CVE. But it's hard to tell which lines are the critical ones. Properly encapsulating safe/unsafe regions does seem like an improvement. (My understanding is that Rust's "unsafe" is not properly encapsulated: You might mess something up that will cause a segfault later on, in ostensibly "safe" code. Still, it's a step forward.)
- jcranmer 8y ago>> in-kernel C tends to be de facto safe > This is a bit of a stretch. Not "a bit;" quite a lot of a stretch. What makes C problematic is that the language is specified in terms of an abstract machine that looks absolutely nothing like real hardware, and executing instructions that make no sense in the abstract machine but are very well-defined in the real hardware is liable to cause problems for the optimizer in the compiler. The assumption that it's safe is built on the principle that a) compiler vendors aren't going to break the kernel build and b) it's possible to jam enough flags onto the compiler to make the abstract machine adhere to the real semantics. Both of these are dangerous assumptions. One of the more infamous compiler-writers-are-breaking-code-for-no-reason rants is the optimization that assumes that a dereferenced pointer cannot be null, so it deletes null checks. Of course, the actual code that caused that rant was this: static unsigned int tun_chr_poll(struct file *file, poll_table * wait) { struct tun_file *tfile = file->private_data; struct tun_struct *tun = __tun_get(tfile); struct sock *sk = tun->sk; unsigned int mask = 0; if (!tun) return POLLERR; This isn't some advanced compiler optimization breaking your code for being an idiot. That code would only work in the first place if you had compiler optimizations that pushed the dereference of tun below the check, which means there's no simple abstract machine interpretation that would make it work.
- eeZah7Ux 8y agoNO, and I wish HN stopped upvoting this sort of propaganda. There are hundreds of languages around and the slides did not even attempt to compare pros and cons across languages that would be suitable for kernels.
- discoball 8y agoNim is realtime garbage collected and compiles to optimized C and it can use the C ecosystem of libraries and tooling. It also produces smaller binaries than Rust and is faster for most tasks.
- m0th87 8y agoDespite what the Nim documentation says, realtime and garbage collection are mutually exclusive (unless you count non-deferred reference counting and give up on collecting cycles.) Setting maximum pause times doesn't change the fact that the garbage will be collected nondeterministically, which is a problem for kernels.
- deleted 8y ago[deleted]
- lucozade 8y agoThose are 3 orthogonal(ish) things. Realtime just means that actions triggered by event occur in an explicitly bounded time. It doesn't usually mean that they occur in the same time. So a GC with bounded pause could be used in a realtime system assuming you could guarantee the explicit bounds. I can't comment on whether or not this is true of Nim and pause time probably isn't sufficient (depends on how it's defined) but they're not mutually exclusive. And most kernels don't need to be realtime unless they are realtime kernels. Having said that, of course, the vast majority of GC systems are most definitely not realtime, or anywhere close. But that's just actually true, not theoretically true.
- m0th87 8y ago> So a GC with bounded pause could be used in a realtime system assuming you could guarantee the explicit bounds. You can't guarantee explicit bounds on when a given resource will be collected with GC. Bounded times can provide upper bounds on GC pass runtime, but they do that by potentially deferring further collection. Those deferrals make it nondeterministic. > And most kernels don't need to be realtime unless they are realtime kernels. That's not true. Most (all?) kernels have internal realtime needs, e.g. responding to interrupts. There is a way to write a kernel with a GC'd language, as Niklaus Wirth has demonstrated, by circumventing the GC where needed. But the critique I was bringing up was simply that there's no such thing as a realtime GC.
- orbifold 8y agoA bit surprising that he doesn't mention any other C++ based kernels such as https://github.com/l4ka/pistachio/ https://github.com/l4ka/pistachio/ and most of the fuchsia code base except for the lk micro kernel. In my opinion fuchsia points towards where things are going multi-language implementations (mostly c++) with an interface description language for interfacing components. From a point of view of postmodern c++, rust has little to no advantages left. It has a much more mature ecosystem and set of libraries, is just as fast and can be very safe if used correctly.
- skocznymroczny 8y ago"can be very safe if used correctly" the same bit applies to C, and we all know how it's working out
- taeric 8y agoWhat we don't know, is how it would have worked out if we didn't have C. There is also a bit that most of the bad security failures are not as sophisticated as the ones we mostly talk about. (Ironically, of course, the widest vulnerability lately was at the CPU level...)
- fredmorcos 8y ago> From a point of view of postmodern c++, rust has little to no advantages left. My entire adult life I've been writing C, I resisted Rust for a while but took the plunge when a C++ project came along that was in dire need of being reworked. I liked it, the language and the compiler and tooling around it helped me tremendously. I assume you're aware of all the things that Rust statically guarantees for you, and I'll also assume you know the difference between language complexity and language implementation complexity (eg, the compiler or some other tooling): I've been skimming through the C++ section of cppreference.com, and oh boy, there's heaps and heaps of... stuff... just stuff. The interplay between all that stuff is complex and it's hard to keep everything in mind. Keeping things in mind is important for correctness, or at least for having a reasonable amount of confidence in what you're writing. Information locality is also important for correctness, and C++ lacks both of those. I'll even go further and say that C++ is a write-only language, like a garbage bin of features and exceptions to each of those features. Rust has many many many clear advantages (and some disadvantages) compared to postmodern C++.
- mitchtbaum 8y agoAnother important and related question to ask is what role Lua should play in writing drivers and applications on top of a new kernel. NetBSD started on this direction, but exactly what an operating system looks like when it leverages both systems-level and scripting-level programmability waits to be seen. https://www.netbsd.org/~lneto/dls14.pdf https://www.netbsd.org/~lneto/dls14.pdf https://www.lua.org/wshop13/Cormack.pdf https://www.lua.org/wshop13/Cormack.pdf
- z3t4 8y agoI played with the idea of setting up a runtime for a scripting language such as JS, then I was thinking, what is a OS anyway ? If it can be written in Rust or any other high level language, it can also be written in a scripting language !...?
- Animats 8y agoin-kernel C tends to be de facto safe Huh?
- Animats 8y agoI'd really like to see an open-source QNX-type microkernel in Rust. Everybody is re-implementing early Unix in Rust, but that's not a good model. The great thing about the QNX model is that the tiny kernel barely changes from year to year, because everything is outside it and not very trusted. So it approaches being totally debugged. L4 is too low-level - you have to run another OS on top of it, usually Linux. QNX offers a POSIX interface.
- m0th87 8y agoWould it be possible to use rump kernels with this to get support for the myriad hardware out there from the get-go? Or are QNX-esque microkernels and rump kernels mutually exclusive for some reason?
- Animats 8y agoUsing the drivers from rump kernels with a QNX kernel might work. QNX runs drivers, file systems, and networking in userland. The rump kernel people have those components. The QNX kernel itself does only memory allocation, process management, CPU dispatching, timers, and inter-process messaging. So it's tiny, relatively trouble-free, and has a small attack surface. So if someone implemented a QNX-style microkernel in Rust, file systems and networking could be adapted from rump kernels. Drivers would require more porting. That would be a nice way to get a usable but small OS for embedded and IoT devices. Linux has more than you want in an embedded device.
- AnIdiotOnTheNet 8y ago> QNX offers a POSIX interface. You say that like its valuable. POSIX has a lot of problems, if you're going to go through the trouble of throwing away a mature kernel like Linux with all the hardware support that brings, you may as well also ditch your POSIX baggage.
- telotortium 8y agoWhat is the state of documentation on QNX? Is there sufficient documentation that it would be possible to write a QNX-compatible kernel from the documentation (plus access to a QNX machine, which has to be available somehow outside of industry, right?)?
- Leace 8y agoI'm surprised there are no mentions to Microsoft Research's Singularity, that used only software barriers, and the languages used there (Sing#, Spec#) had ownership concepts ([Claims] attribute).
- stefs 8y agoi think there is a case for a rewrite, apart from the memory safety guarantees. two things come to my mind: 1. elimination of backwards compatibility. this is true for all new operating systems, independently of the language. clean slate and throw away all the baggage that was reasonable 20 years ago, doesn't hold true today anymore but still is baked into the old architecture. this might be less relevant for all-purpose OSes, but might be a viable option for specialized systems (IoT, Network Appliances, ...). you remember what linus says when a kernel patch breaks buggy userland code? we now get a second chance to implement a new system that avoids whole classes for buggy code (while the underlying problem doesn't go away, it may get greatly reduced). 2. reduction of complexity. this _is_ a rust thing, and the story to back this up is [stylo](https://blog.rust-lang.org/2017/11/14/Fearless-Concurrency-In-Firefox-Quantum.html https://blog.rust-lang.org/2017/11/14/Fearless-Concurrency-I...), the parallel css engine used in firefox. i remember reading that google tried to implement concurrent styling in chrome but failed, because it got too complex in c++. human intellect is pretty much finite and doesn't scale well; we rely on better tools. so, rust might enable us to do things that were considered impossible before. the emergence of such an OS would be gradual of course, niche at first, then slowly growing until one day it's the new de-facto standard. lets say, game engines. you rarely write your own game engine, because that's too complex and ties up all the resources you need to actually implement your game. so you buy one and it's in C++, of course. because they all are, and thus your developers are fluent in C++. there are no game engines in rust and few developers who know the language. but over the years, more and more game devs will try rust in their spare time and like it and write game engines in rust for their side or indie projects, where it's still feasible. a few of those will grow and get more features and tooling and at _some point_, suddenly, rust will be a viable alternative. even though rusts strengths aren't really that important in game engines. performance happens on the graphic cards anyway. security is not as big of a deal. parallelism is quite constrained anyway. etc, etc. but if all else is equal, rust might be more productive. so, is it time to rewrite the OS in rust? sure, why not. it's just that we wont all switch to redox overnight.