5 ms·

What I would love to see is Rust in the kernel, with a caveat. That caveat is something akin to Erlang's OTP supervision trees. Erlang has a similar philosophy

What I would love to see is Rust in the kernel, with a caveat. That caveat is something akin to Erlang's OTP supervision trees. Erlang has a similar philosophy to Rust in that if something erroneous happens the process fails. The difference is that Erlang OTP is designed to have supervisor processes of the worker processes. The only thing the supervisors do is monitor the processes under them and control when, if, and how those processes are restarted on failure or other conditions. The supervisors themselves have supervisors, right up to the root supervisor for each Erlang application. Rust got the language and the tooling perfect, but Erlang got the services and service infrastructure perfect. The more I think about it the more I think I should shut up and put up. In other words, apply my knowledge of Erlang and OTP to create a gen_server in Rust as a jumping off point for a OTP like Rust framework, perhaps called OARS (Open Advanced Rust Services). This is definitely bigger than one person. If you'd like to join me on this journey then reply to this comment and I'll send you project details by the end of the weekend.

ta988 6y ago
I don't know enough Rust and Erlang for helping with that, but that definitively sounds great. What would you use for IPC?
gpm 6y ago
I don't really know much about erlang, but I think this may be along the lines of what you are thinking of: https://github.com/bastion-rs/bastion https://github.com/bastion-rs/bastion (I also don't really think the linux kernel people would be interested...)
weakfish 6y ago
If you don't mind a student rust enthusiast, I would love to take a crack at this. Email is john123allison AT gmail DOT com
asabil 6y ago
I would love to see the same thing, I always told people around me that Erlang/OTP is more akin to an OS than a traditional programming language. That being said, the key feature to enable what Erlang supports is asynchronous termination, which as far as I know is not possible in regular Rust.
- oconnor663 6y ago
  I think the current options for killing a generic task in Rust are either 1) make it an async task, which can be cancelled as long as it doesn't accidentally block a thread, or 2) make it a separate process, and have the OS kill it. Do either of those fit this use case?
jerf 6y ago
That is a good idea, but one thing I would advise, having both seen several attempts made at this sort of thing and having made one myself [1], try very hard to separate the accidental things Erlang brings to the idea from the fundamental things Erlang brings to the idea. Most attempts I've seen made at this flounder on this pretty hard by trying to port too directly the exact Erlang supervisor tree idea while grinding hard against the rest of the language, rather than porting the core functionality in in a way that integrates natively with the language in question as much as possible. For instance, one thing I found when I was writing my library that will probably apply to most other languages (probably including Rust) is that Erlang has a somewhat complicated setup step for running a gen_server, with an explicit setup call, a separate execution call, several bits and pieces for 'officially' communicating with a gen_server, etc. But a lot of these things are for dealing with the exact ways that Erlang interacts with processes, and you probably don't need most of them. Simply asking for a process that makes the subprocess "start" from scratch is probably enough, and letting that process use existing communication mechanisms already in the language rather than trying to directly port the Erlang stuff. Similarly, I found no value in trying to provide direct ports of all the different types of gen_server, which aren't so much about the supervision trees (even if that's where they seem to be located) as a set of standard APIs for working with those various things. They're superfluous in a language that already has other solutions for those problems. In addition to keeping an eye out for features you don't need from Erlang, keep an eye out for features in the host language that may be useful; e.g., the most recent suture integrates with the Go ecosystem's ever-increasing use of context.Contexts as a way to manage termination, which hasn't got a clear Erlang equivalent. (Linking to processes has some overlapping functionality but isn't exactly the same, both offering some additional functionality contexts don't have as well as missing some functionality contexts do have.) Erlang has a lot of good ideas that I'd love to see ported into more languages. But a lot of attempts to do so flounder on these issues, creating libraries so foreign to the host language that they have zero chance of uptake. The other thing I'd point out is that even in Go, to say nothing of Rust, crashing is actually fairly uncommon by Erlang standards. Many things that crash in Erlang are statically prevented at compile time in Go, and Rust statically precludes even more of them. However, I have found it OTP-esque supervision trees to be a very nice organizational structure to my code; I use suture in nearly every non-trivial Go program I write because it makes for a really nice modular approach for the question of "how do I start and stop persistent services?". I have seen it hold together runtime services that would otherwise be failing, the way it is supposed to, and that's nice, but the organization structure is still probably the larger benefit. (There is deep reason for the way Erlang is doing it the way it does, which is that a lot of Erlang's type system, or lack thereof, is for communicating between nodes, so even if you perfectly program Erlang, if two nodes running different versions of code try to communicate with each other and they've changed the protocol you might get a pattern matching fail on the messages flowing between versions. The Erlang way of doing cross-machine communication with this sort of automatic serialization at the language level has not caught on, and all modern languages have a relatively distinct serialization step where this sort of error is better handled, as you try to deserialize the remote message into your internal data structure.) Anyhow, the upshot is, you want to translate the functionality out of Erlang into other languages, not transliterate it. [1]: https://github.com/thejerf/suture https://github.com/thejerf/suture
tylerscott 6y ago
I love the idea of OARS and would like to help any way I can!
steveklabnik 6y ago
Doing this well requires a heavy enough runtime that it would disqualify it for this kind of work. Not on a technical level, but on a social one. While there have been operating systems created with this sort of runtime, they're not as well known or successful as ones that haven't. I would imagine this email would be a flat "no" if this were a core part of Rust, sadly.
- acomjean 6y ago
  From what I remember the Linux kernel doesn’t even use std.lib. It’s pretty straight up C with no dependencies (the os being low level). Makes kernel module programming difficult.
  - SubjectToChange 6y ago
    The Linux kernel is freestanding. But that's hardly the reason why kernel programming is difficult.
- staticassertion 6y ago
  > a heavy enough runtime Hey, asking out of pure ignorance - Why do you think this? My naive point of view is that the Linux kernel already has enough of a runtime to support such at hing - a scheduler, kernel threads, and interrupts. An actor system shouldn't be particularly heavy - you can probably implement an actor in just a couple of bytes. Agreed that it couldn't be a core part of Rust though. Maybe it's too off topic or whatever, just curious.
  - steveklabnik 6y ago
    Erlang makes heavy use of green threads to do this kind of work. You spin up thousands or hundreds of thousands of these. Kernel threads are too heavy weight to do so. And making them lighter weight has tradeoffs too. Erlang makes use of a GC and immutability to make tasks more restart-able; maybe Rust's memory safety features would let you do this sorta kinda, but I don't think it's been really demonstrated fully yet. Like sure, you could build out some of these features, but they only truly work really well if you extremely commit to the architecture, in my opinion. And the kernel isn't about to do that.
zozbot234 6y ago
You may want to write a proper user story for this and submit it to https://blog.rust-lang.org/2021/04/14/async-vision-doc-shiny-future.html https://blog.rust-lang.org/2021/04/14/async-vision-doc-shiny... (mentioned in the latest "This Week in Rust" development summary).
jb3689 6y ago
I don’t understand what you’re getting from this. Crash looping can still happen in OTP, the root supervisor can still die if the crash threshold is met in a small window. This would also be very heavy weight. IIUC the issue is not that errors occur, but that errors occur (and panic) and cannot be handled
nahuel0x 6y ago
You want a microkernel.
fortran77 6y ago
Why not write an entire OS in a message-passing VM based language with garbage collection? No crashes, ever! In fact, you can run Erlang directly on raw metal, or write an OS in Erlang: http://www.erlang-factory.com/static/upload/media/1498583896660864samwilliamshowtobuildanoswitherlang_awhistlestoptourofhydros.pdf http://www.erlang-factory.com/static/upload/media/1498583896...
- bitwize 6y ago
  Because borrow checking > GC. When you have a GC, you need several times more memory to run the same program with the same performance as without -- and usually, you have to say goodbye to any sort of determinism in execution time, as well. What Rust brings to the table is guaranteed memory safety without GC, and all memory is released in strictly deterministic time. So for an OS, it's much better to bring the good bits of Erlang to Rust.
scrubs 6y ago
I don't see how one can run and hide from Linus' point. Exception/panic based work is problematic in libraries, and has no place in kernels. Erlang's OTP supervision trees is as others have pointed out, a runtime issue that app-devs build on. Therefore it's an abstraction that's above the kernel and out of scope w.r.t. to kernel work.
- Communitivity 6y ago
  This is a fair and valid point. On reflection, what I am looking for is not something that would be in the kernel. It's more something that is an added layer on Rust, quite possibly with compiler support, to provide for OTP style services and supervisors.
deleted 6y ago
[deleted]
fortran77 6y ago
If it's based from OTP we should give it proper credit and call it OTP/OARS.

ta988 6y ago

I don't know enough Rust and Erlang for helping with that, but that definitively sounds great. What would you use for IPC?

gpm 6y ago

I don't really know much about erlang, but I think this may be along the lines of what you are thinking of: https://github.com/bastion-rs/bastion https://github.com/bastion-rs/bastion (I also don't really think the linux kernel people would be interested...)

weakfish 6y ago

If you don't mind a student rust enthusiast, I would love to take a crack at this. Email is john123allison AT gmail DOT com

asabil 6y ago

I would love to see the same thing, I always told people around me that Erlang/OTP is more akin to an OS than a traditional programming language. That being said, the key feature to enable what Erlang supports is asynchronous termination, which as far as I know is not possible in regular Rust.

oconnor663 6y ago
I think the current options for killing a generic task in Rust are either 1) make it an async task, which can be cancelled as long as it doesn't accidentally block a thread, or 2) make it a separate process, and have the OS kill it. Do either of those fit this use case?

jerf 6y ago

That is a good idea, but one thing I would advise, having both seen several attempts made at this sort of thing and having made one myself [1], try very hard to separate the accidental things Erlang brings to the idea from the fundamental things Erlang brings to the idea. Most attempts I've seen made at this flounder on this pretty hard by trying to port too directly the exact Erlang supervisor tree idea while grinding hard against the rest of the language, rather than porting the core functionality in in a way that integrates natively with the language in question as much as possible. For instance, one thing I found when I was writing my library that will probably apply to most other languages (probably including Rust) is that Erlang has a somewhat complicated setup step for running a gen_server, with an explicit setup call, a separate execution call, several bits and pieces for 'officially' communicating with a gen_server, etc. But a lot of these things are for dealing with the exact ways that Erlang interacts with processes, and you probably don't need most of them. Simply asking for a process that makes the subprocess "start" from scratch is probably enough, and letting that process use existing communication mechanisms already in the language rather than trying to directly port the Erlang stuff. Similarly, I found no value in trying to provide direct ports of all the different types of gen_server, which aren't so much about the supervision trees (even if that's where they seem to be located) as a set of standard APIs for working with those various things. They're superfluous in a language that already has other solutions for those problems. In addition to keeping an eye out for features you don't need from Erlang, keep an eye out for features in the host language that may be useful; e.g., the most recent suture integrates with the Go ecosystem's ever-increasing use of context.Contexts as a way to manage termination, which hasn't got a clear Erlang equivalent. (Linking to processes has some overlapping functionality but isn't exactly the same, both offering some additional functionality contexts don't have as well as missing some functionality contexts do have.) Erlang has a lot of good ideas that I'd love to see ported into more languages. But a lot of attempts to do so flounder on these issues, creating libraries so foreign to the host language that they have zero chance of uptake. The other thing I'd point out is that even in Go, to say nothing of Rust, crashing is actually fairly uncommon by Erlang standards. Many things that crash in Erlang are statically prevented at compile time in Go, and Rust statically precludes even more of them. However, I have found it OTP-esque supervision trees to be a very nice organizational structure to my code; I use suture in nearly every non-trivial Go program I write because it makes for a really nice modular approach for the question of "how do I start and stop persistent services?". I have seen it hold together runtime services that would otherwise be failing, the way it is supposed to, and that's nice, but the organization structure is still probably the larger benefit. (There is deep reason for the way Erlang is doing it the way it does, which is that a lot of Erlang's type system, or lack thereof, is for communicating between nodes, so even if you perfectly program Erlang, if two nodes running different versions of code try to communicate with each other and they've changed the protocol you might get a pattern matching fail on the messages flowing between versions. The Erlang way of doing cross-machine communication with this sort of automatic serialization at the language level has not caught on, and all modern languages have a relatively distinct serialization step where this sort of error is better handled, as you try to deserialize the remote message into your internal data structure.) Anyhow, the upshot is, you want to translate the functionality out of Erlang into other languages, not transliterate it. [1]: https://github.com/thejerf/suture https://github.com/thejerf/suture

tylerscott 6y ago

I love the idea of OARS and would like to help any way I can!

steveklabnik 6y ago

Doing this well requires a heavy enough runtime that it would disqualify it for this kind of work. Not on a technical level, but on a social one. While there have been operating systems created with this sort of runtime, they're not as well known or successful as ones that haven't. I would imagine this email would be a flat "no" if this were a core part of Rust, sadly.

acomjean 6y ago
From what I remember the Linux kernel doesn’t even use std.lib. It’s pretty straight up C with no dependencies (the os being low level). Makes kernel module programming difficult.
- SubjectToChange 6y ago
  The Linux kernel is freestanding. But that's hardly the reason why kernel programming is difficult.
staticassertion 6y ago
> a heavy enough runtime Hey, asking out of pure ignorance - Why do you think this? My naive point of view is that the Linux kernel already has enough of a runtime to support such at hing - a scheduler, kernel threads, and interrupts. An actor system shouldn't be particularly heavy - you can probably implement an actor in just a couple of bytes. Agreed that it couldn't be a core part of Rust though. Maybe it's too off topic or whatever, just curious.
- steveklabnik 6y ago
  Erlang makes heavy use of green threads to do this kind of work. You spin up thousands or hundreds of thousands of these. Kernel threads are too heavy weight to do so. And making them lighter weight has tradeoffs too. Erlang makes use of a GC and immutability to make tasks more restart-able; maybe Rust's memory safety features would let you do this sorta kinda, but I don't think it's been really demonstrated fully yet. Like sure, you could build out some of these features, but they only truly work really well if you extremely commit to the architecture, in my opinion. And the kernel isn't about to do that.

zozbot234 6y ago

You may want to write a proper user story for this and submit it to https://blog.rust-lang.org/2021/04/14/async-vision-doc-shiny-future.html https://blog.rust-lang.org/2021/04/14/async-vision-doc-shiny... (mentioned in the latest "This Week in Rust" development summary).

jb3689 6y ago

I don’t understand what you’re getting from this. Crash looping can still happen in OTP, the root supervisor can still die if the crash threshold is met in a small window. This would also be very heavy weight. IIUC the issue is not that errors occur, but that errors occur (and panic) and cannot be handled

nahuel0x 6y ago

You want a microkernel.

fortran77 6y ago

Why not write an entire OS in a message-passing VM based language with garbage collection? No crashes, ever! In fact, you can run Erlang directly on raw metal, or write an OS in Erlang: http://www.erlang-factory.com/static/upload/media/1498583896660864samwilliamshowtobuildanoswitherlang_awhistlestoptourofhydros.pdf http://www.erlang-factory.com/static/upload/media/1498583896...

bitwize 6y ago
Because borrow checking > GC. When you have a GC, you need several times more memory to run the same program with the same performance as without -- and usually, you have to say goodbye to any sort of determinism in execution time, as well. What Rust brings to the table is guaranteed memory safety without GC, and all memory is released in strictly deterministic time. So for an OS, it's much better to bring the good bits of Erlang to Rust.

scrubs 6y ago

I don't see how one can run and hide from Linus' point. Exception/panic based work is problematic in libraries, and has no place in kernels. Erlang's OTP supervision trees is as others have pointed out, a runtime issue that app-devs build on. Therefore it's an abstraction that's above the kernel and out of scope w.r.t. to kernel work.

Communitivity 6y ago
This is a fair and valid point. On reflection, what I am looking for is not something that would be in the kernel. It's more something that is an added layer on Rust, quite possibly with compiler support, to provide for OTP style services and supervisors.

deleted 6y ago

[deleted]

fortran77 6y ago

If it's based from OTP we should give it proper credit and call it OTP/OARS.