15 ms·

Rust in illumos

140 points by vermaden 2y ago

yjftsjthsd-h 2y ago
> The development model of illumos is different from Linux and thus there are no Rust drivers in upstream illumos yet. But that is to be expected for new things. In our model, we take the time to mature new tech in a fork, and for rust, the Oxide fork has taken that role. In there, we have several drivers for the Oxide Networking stack that are in rust. Based on that some experience could be gained. Soft forks are also good, but doesn't illumos have a stable driver ABI, such that you could just make your drivers completely independently? I thought that was how it had ex. Nvidia drivers ( https://docs.openindiana.org/dev/graphics-stack/#nvidia https://docs.openindiana.org/dev/graphics-stack/#nvidia )
- boricj 2y ago
  Even if there is a stable ABI driver to the point where upstream is oblivious of any Rust stuff going on inside drivers, that doesn't solve the problem of fragmentation amongst forks (assuming upstreaming to Illumos is a goal).
  - jclulow 2y ago
    I'm part of the illumos core team and I'm quite keen to use Rust in the base of illumos, FWIW. There several contexts in which we could make good use of it, and the primary challenges in most cases is either one of release engineering or the resultant binary sizes. I'm confident we'll be able to work them out, it'll just take more exploration and effort! The rough areas where I think things would be useful, and probably also roughly the order in which I would seek to do them: * Rust in the tools we use to build the software. We have a fair amount of Perl and Python and shell script that goes into doing things like making sure the build products are correctly formed, preparing things for packaging, diffing the binary artefacts from two different builds, etc. It would be pretty easy to use regular Cargo-driven Rust development for these tools, to the extent that they don't represent shipped artefacts. * Rust to make C-compatible shared libraries. We ship a lot of libraries in the base system, and I could totally see redoing some of the parts that are harder to get right (e.g., things that poke at cryptographic keys, or do a lot of string parsing, or complex maths!) in Rust. I suspect we would _not_ want to use Cargo for these bits, and probably try to minimise the dependencies and keep tight control on the size of the output binaries etc. * Rust to make kernel modules. Kernel modules are pretty similar to C shared libraries, in that we expect them to have few dependencies and probably not be built through Cargo using crates.io and so on. * Rust to make executable programs like daemons and command-line commands. I think here the temptation to use more dependencies will increase, and so this will be the hardest thing to figure out. The other thing beyond those challenges is that we want the build to work completely offline, _and_ we don't want to needlessly unpack and "vendor" (to use the vernacular) a lot of external files. So we'd probably be looking at using some of the facilities Cargo is growing to use an offline local clone of parts of a repository, and some way to reliably populate that while not inhibiting the ease of development, etc. Nothing insurmountable, just a lot of effort, like most things that are worth doing!
    - marvin-hansen 2y ago
      Have you explored building with Bazel? What you describe as a problem is roughly what Bazel solves: Polyglot complex builds from fully vendored deps. Just pointing this out because I had a fair share of issues with Cargo and ultimately moved to Bazel and a bit later to BuildBuddy as CI. Since then my builds are reliable, run a lot faster and even stuff like cross compilation in a cluster works flawlessly. Obviously there is some complexity implied when moving to Bazel, but the bigger question is whether the capabilities of your current build solution keep up with the complexity of your requirements?
      - steveklabnik 2y ago
        (Not Josh, not involved with illumos, do work at Oxide) For at least one project at Oxide we use buck2, which is conceptually in a similar place. I’d like to use it more but am still too new to wield it effectively. In general I would love to see more “here’s how to move to buck/bazel when you outgrow cargo) content.
        marvin-hansen 2y ago
        Here you go! https://github.com/bazelbuild/examples/tree/main/rust-examples https://github.com/bazelbuild/examples/tree/main/rust-exampl... I wrote all of those examples and contributed them back to Bazel because I've been there... Personally, I prefer the Bazel ecosystem by a wide margin over buck2. By technology alone, buck2 is better, but as my requirements were growing, I needed a lot more mature rule sets such as rules OCI to build and publish container images without Docker and buck2 simply doesn't have the ecosystem available to support complex builds beyond a certain level. It may get there one day.
        steveklabnik 2y ago
        Thanks! I fully agree with the ecosystem comments; I’m just a Buck fan because it’s in Rust and I like the “no built in rules” concept, but it’s true that it’s much younger and seemingly less widely used. Regardless I should spend some time with Bazel.
- cryptonector 2y ago
  I don't know if Illumos follows the Sun model of "interface contracts", which allows for downstreams to get timely notice of upcoming backwards-incompatible changes to unstable interfaces. Of course, Illumos can choose to make some intra-kernel interfaces stable.
  - jclulow 2y ago
    The answer is definitely not "no", but as far as I'm aware we haven't found any particular cases where that level of formalism and that specific style of arrangement would help. In an open source world where multiple shipping operating systems are based on (different snapshots of) the same core code and interfaces, it seems much more valuable to try and push straight to Committed interfaces (public, stable, documented) where we can. There are plenty of cases where people use Uncommitted interfaces today in things that they layer on top in appliances and so on, but mostly it's expected that those people are paying attention to what's being worked on and integrated. When folks pull new changes from illumos-gate into their release engineering branches or soft forks, they're generally doing that explicitly and testing that things still work for them. Also, unlike the distributions (who ship artefacts to users) it doesn't really make much sense for us to version illumos itself. This makes it a bit harder to reason about when it would be acceptable to renege on an interface contract, etc. In general we try to move things to Committed over time, and then generally try very hard never to break or remove those things after that point.
4ad 2y ago
> The development model of illumos is different from Linux and thus there are no Rust drivers in upstream illumos yet. But that is to be expected for new things. In our model, we take the time to mature new tech in a fork, and for rust, the Oxide fork has taken that role. This is silly. Virtually all illumos development is done by Oxide. There is no other upstream to speak of. As far as I am concerned, illumos is Oxide. Also, it's illumos, in lower case.
- dvtkrlbs 2y ago
  Illumos is not Oxide. Oxide is just using a in house distribution of illumos for their needs. Illumos created is created 14 years ago by Solaris developers to swap the closed source bits by open source implementations.
  - dvtkrlbs 2y ago
    Looking at the last commits it does not seems like majority upstream development is not done by Oxide (I did not look at it extensively)
    - yjftsjthsd-h 2y ago
      user@machine:~/illumos-gate$ git log --since 2023-09-10 | grep ^Author | awk -F @ '{print $2}' | sed 's/>$//' | sort | uniq -c | sort -n | tail 11 mnx.io 16 grumpf.hope-2000.org 18 gmail.com 21 oxide.computer 29 hamachi.org 36 richlowe.net 37 racktopsystems.com 71 fiddaman.net 91 fingolfin.org 125 me.com I'm not going to post names because that feels a touch icky, but a little bit of cross-comparison suggests that one of those domains should be combined with oxide (read: I search for names and they work there) in a way that does probably make Oxide more than 50% of commits in the last year. Though even if that's true it doesn't make them the only player or anything.
      - asveikau 2y ago
        Considering that Oxide's CTO is a longtime prominent Solaris guy, it's not shocking. I wouldn't surprised if that's the whole reason they use ilumos.
rtpg 2y ago
This article really comes at some of the crux of packaging pains for many languages. The "distro ships the software" idea is something so many modern language toolkits are pretty far away from (and, as someone constantly burned by Debian packaging choices, something I'm pretty happy with as well). But without a good answer, there's going to be frustration. When distros are charging themselves with things like security, shared libraries do become a bit load bearing. And for users, the idea that you update some lib once instead of "number of software vendor" times is tempting! But as a software developer, I really really enjoy how I can ship a thing with exactly a certain set of versions, so it's almost an anti-feature to have a distro swap out functionality out from under me. Of course there can be balances and degrees to this. But a part of me feels like the general trend of software packaging is leaning towards "you bundle one thing at a time", away from "you have a system running N things", and in that model I don't know where distro packagers are (at least for server software).
- oconnor663 2y ago
  I think it would be pretty natural for a distro to run it's own crates.io/PyPI/NPM mirror to build packages against, and have a unified set of library versions for the entire OS that way. Maybe not even a server, maybe just a giant Git repo or whatever, filtered down to the set of packages they actually use. Have any of them tried this?
  - steveklabnik 2y ago
    I’m not aware of any that have; this would require them to become experts in running parallel infra, in many languages and with varied setups, and would mean that they’re not longer self-contained. That feels antithetical to their goals to me at least.
  - rtpg 2y ago
    I mean Ubuntu has a bunch of python libs. The problem is that you need to make a universe of mutually compatible libs, so if _any_ of the libs have old-ish dependencies, that holds back so much. I think what happens in practice is package maintainers do the work to get things up to date, and so only include a subset.
turtle_heck 2y ago
> for one would love to have people help me with the new Installer[0] and with the package Forge[1] Those repos really need some basic high-level information about what they are and how they work, the Forge doesn't even have a README. [0] https://github.com/Toasterson/illumos-installer https://github.com/Toasterson/illumos-installer [1] https://github.com/toasterson/forge https://github.com/toasterson/forge
uitbouwcalculat 2y ago
[dead]
fake-name 2y ago
> The Debian approach of putting everything into one system and only having one version of each dependency is not feasible for a huge international community of people that develop together but never meet. I find this to be overbroad and incorrect in the larger case. I want one version that has been reasonably tested for everything but the parts I'm developing or working with. Having just one version for everyone generally means it's probably going to work, and maybe even has been tested. For my own stuff, I am happy to manage the dependencies myself, but moving away from the debian approach harms the system in general.
the8472 2y ago
> Missing support for shared libraries Well, rust does support shared libraries. Call things by their name: what distros want is for all libraries to have a stable ABI. And I guess some sort of segmentation that the source of one library doesn't bleed into the binary output of another, and I think the latter part is even more tricky than ABI stability. Generics are monomorphized in downstream crates (aren't C++ templates the same?), so I think you'd either have to ban generic functions and inlining from library APIs or do dependent-tree rebuilds anyway.
- silon42 2y ago
  Not necessarily ban, but any incompatible changes to those is an ABI break.
  - the8472 2y ago
    Which would require rebuilding all dependents, yeah, that was the second option mentioned. If you're already willing to rebuild-the-world then you could use dynamic libraries today. But you'd only get a shared memory / size benefit from that, not drop-in-updates.

140 points by vermaden 2y ago

> The development model of illumos is different from Linux and thus there are no Rust drivers in upstream illumos yet. But that is to be expected for new things. In our model, we take the time to mature new tech in a fork, and for rust, the Oxide fork has taken that role. In there, we have several drivers for the Oxide Networking stack that are in rust. Based on that some experience could be gained. Soft forks are also good, but doesn't illumos have a stable driver ABI, such that you could just make your drivers completely independently? I thought that was how it had ex. Nvidia drivers ( https://docs.openindiana.org/dev/graphics-stack/#nvidia https://docs.openindiana.org/dev/graphics-stack/#nvidia )

boricj 2y ago
Even if there is a stable ABI driver to the point where upstream is oblivious of any Rust stuff going on inside drivers, that doesn't solve the problem of fragmentation amongst forks (assuming upstreaming to Illumos is a goal).
- jclulow 2y ago
  I'm part of the illumos core team and I'm quite keen to use Rust in the base of illumos, FWIW. There several contexts in which we could make good use of it, and the primary challenges in most cases is either one of release engineering or the resultant binary sizes. I'm confident we'll be able to work them out, it'll just take more exploration and effort! The rough areas where I think things would be useful, and probably also roughly the order in which I would seek to do them: * Rust in the tools we use to build the software. We have a fair amount of Perl and Python and shell script that goes into doing things like making sure the build products are correctly formed, preparing things for packaging, diffing the binary artefacts from two different builds, etc. It would be pretty easy to use regular Cargo-driven Rust development for these tools, to the extent that they don't represent shipped artefacts. * Rust to make C-compatible shared libraries. We ship a lot of libraries in the base system, and I could totally see redoing some of the parts that are harder to get right (e.g., things that poke at cryptographic keys, or do a lot of string parsing, or complex maths!) in Rust. I suspect we would _not_ want to use Cargo for these bits, and probably try to minimise the dependencies and keep tight control on the size of the output binaries etc. * Rust to make kernel modules. Kernel modules are pretty similar to C shared libraries, in that we expect them to have few dependencies and probably not be built through Cargo using crates.io and so on. * Rust to make executable programs like daemons and command-line commands. I think here the temptation to use more dependencies will increase, and so this will be the hardest thing to figure out. The other thing beyond those challenges is that we want the build to work completely offline, _and_ we don't want to needlessly unpack and "vendor" (to use the vernacular) a lot of external files. So we'd probably be looking at using some of the facilities Cargo is growing to use an offline local clone of parts of a repository, and some way to reliably populate that while not inhibiting the ease of development, etc. Nothing insurmountable, just a lot of effort, like most things that are worth doing!
  - marvin-hansen 2y ago
    Have you explored building with Bazel? What you describe as a problem is roughly what Bazel solves: Polyglot complex builds from fully vendored deps. Just pointing this out because I had a fair share of issues with Cargo and ultimately moved to Bazel and a bit later to BuildBuddy as CI. Since then my builds are reliable, run a lot faster and even stuff like cross compilation in a cluster works flawlessly. Obviously there is some complexity implied when moving to Bazel, but the bigger question is whether the capabilities of your current build solution keep up with the complexity of your requirements?
    - steveklabnik 2y ago
      (Not Josh, not involved with illumos, do work at Oxide) For at least one project at Oxide we use buck2, which is conceptually in a similar place. I’d like to use it more but am still too new to wield it effectively. In general I would love to see more “here’s how to move to buck/bazel when you outgrow cargo) content.
      - marvin-hansen 2y ago
        Here you go! https://github.com/bazelbuild/examples/tree/main/rust-examples https://github.com/bazelbuild/examples/tree/main/rust-exampl... I wrote all of those examples and contributed them back to Bazel because I've been there... Personally, I prefer the Bazel ecosystem by a wide margin over buck2. By technology alone, buck2 is better, but as my requirements were growing, I needed a lot more mature rule sets such as rules OCI to build and publish container images without Docker and buck2 simply doesn't have the ecosystem available to support complex builds beyond a certain level. It may get there one day.
        steveklabnik 2y ago
        Thanks! I fully agree with the ecosystem comments; I’m just a Buck fan because it’s in Rust and I like the “no built in rules” concept, but it’s true that it’s much younger and seemingly less widely used. Regardless I should spend some time with Bazel.
cryptonector 2y ago
I don't know if Illumos follows the Sun model of "interface contracts", which allows for downstreams to get timely notice of upcoming backwards-incompatible changes to unstable interfaces. Of course, Illumos can choose to make some intra-kernel interfaces stable.
- jclulow 2y ago
  The answer is definitely not "no", but as far as I'm aware we haven't found any particular cases where that level of formalism and that specific style of arrangement would help. In an open source world where multiple shipping operating systems are based on (different snapshots of) the same core code and interfaces, it seems much more valuable to try and push straight to Committed interfaces (public, stable, documented) where we can. There are plenty of cases where people use Uncommitted interfaces today in things that they layer on top in appliances and so on, but mostly it's expected that those people are paying attention to what's being worked on and integrated. When folks pull new changes from illumos-gate into their release engineering branches or soft forks, they're generally doing that explicitly and testing that things still work for them. Also, unlike the distributions (who ship artefacts to users) it doesn't really make much sense for us to version illumos itself. This makes it a bit harder to reason about when it would be acceptable to renege on an interface contract, etc. In general we try to move things to Committed over time, and then generally try very hard never to break or remove those things after that point.

4ad 2y ago

dvtkrlbs 2y ago
Illumos is not Oxide. Oxide is just using a in house distribution of illumos for their needs. Illumos created is created 14 years ago by Solaris developers to swap the closed source bits by open source implementations.
- dvtkrlbs 2y ago
  Looking at the last commits it does not seems like majority upstream development is not done by Oxide (I did not look at it extensively)
  - yjftsjthsd-h 2y ago
    user@machine:~/illumos-gate$ git log --since 2023-09-10 | grep ^Author | awk -F @ '{print $2}' | sed 's/>$//' | sort | uniq -c | sort -n | tail 11 mnx.io 16 grumpf.hope-2000.org 18 gmail.com 21 oxide.computer 29 hamachi.org 36 richlowe.net 37 racktopsystems.com 71 fiddaman.net 91 fingolfin.org 125 me.com I'm not going to post names because that feels a touch icky, but a little bit of cross-comparison suggests that one of those domains should be combined with oxide (read: I search for names and they work there) in a way that does probably make Oxide more than 50% of commits in the last year. Though even if that's true it doesn't make them the only player or anything.
    - asveikau 2y ago
      Considering that Oxide's CTO is a longtime prominent Solaris guy, it's not shocking. I wouldn't surprised if that's the whole reason they use ilumos.

rtpg 2y ago

This article really comes at some of the crux of packaging pains for many languages. The "distro ships the software" idea is something so many modern language toolkits are pretty far away from (and, as someone constantly burned by Debian packaging choices, something I'm pretty happy with as well). But without a good answer, there's going to be frustration. When distros are charging themselves with things like security, shared libraries do become a bit load bearing. And for users, the idea that you update some lib once instead of "number of software vendor" times is tempting! But as a software developer, I really really enjoy how I can ship a thing with exactly a certain set of versions, so it's almost an anti-feature to have a distro swap out functionality out from under me. Of course there can be balances and degrees to this. But a part of me feels like the general trend of software packaging is leaning towards "you bundle one thing at a time", away from "you have a system running N things", and in that model I don't know where distro packagers are (at least for server software).

oconnor663 2y ago
I think it would be pretty natural for a distro to run it's own crates.io/PyPI/NPM mirror to build packages against, and have a unified set of library versions for the entire OS that way. Maybe not even a server, maybe just a giant Git repo or whatever, filtered down to the set of packages they actually use. Have any of them tried this?
- steveklabnik 2y ago
  I’m not aware of any that have; this would require them to become experts in running parallel infra, in many languages and with varied setups, and would mean that they’re not longer self-contained. That feels antithetical to their goals to me at least.
- rtpg 2y ago
  I mean Ubuntu has a bunch of python libs. The problem is that you need to make a universe of mutually compatible libs, so if _any_ of the libs have old-ish dependencies, that holds back so much. I think what happens in practice is package maintainers do the work to get things up to date, and so only include a subset.

turtle_heck 2y ago

> for one would love to have people help me with the new Installer[0] and with the package Forge[1] Those repos really need some basic high-level information about what they are and how they work, the Forge doesn't even have a README. [0] https://github.com/Toasterson/illumos-installer https://github.com/Toasterson/illumos-installer [1] https://github.com/toasterson/forge https://github.com/toasterson/forge

uitbouwcalculat 2y ago

[dead]

fake-name 2y ago

> The Debian approach of putting everything into one system and only having one version of each dependency is not feasible for a huge international community of people that develop together but never meet. I find this to be overbroad and incorrect in the larger case. I want one version that has been reasonably tested for everything but the parts I'm developing or working with. Having just one version for everyone generally means it's probably going to work, and maybe even has been tested. For my own stuff, I am happy to manage the dependencies myself, but moving away from the debian approach harms the system in general.

the8472 2y ago

> Missing support for shared libraries Well, rust does support shared libraries. Call things by their name: what distros want is for all libraries to have a stable ABI. And I guess some sort of segmentation that the source of one library doesn't bleed into the binary output of another, and I think the latter part is even more tricky than ABI stability. Generics are monomorphized in downstream crates (aren't C++ templates the same?), so I think you'd either have to ban generic functions and inlining from library APIs or do dependent-tree rebuilds anyway.

silon42 2y ago
Not necessarily ban, but any incompatible changes to those is an ABI break.
- the8472 2y ago
  Which would require rebuilding all dependents, yeah, that was the second option mentioned. If you're already willing to rebuild-the-world then you could use dynamic libraries today. But you'd only get a shared memory / size benefit from that, not drop-in-updates.