4 ms·

> does that actually matter to the customers who are buying these? It's not like we specifically say "oh btw there's illumos inside and that's why you should

> does that actually matter to the customers who are buying these? It's not like we specifically say "oh btw there's illumos inside and that's why you should buy the rack." It's not a customer-facing detail of the product. I'm sure most will never even know that this is the case. What customers do care about is that the rack is efficient, reliable, suits their needs, etc. Choosing illumos instead of Linux here is a choice made to help effectively deliver on that value. This does not mean that you couldn't build a similar product on top of Linux inherently, by the way, just that we decided illumos was more fit for purpose. This decision was made with the team, in the form of an RFD[1]. It's #26, though it is not currently public. The two choices that were seriously considered were KVM on Linux, and bhyve on illumos. It is pretty long. In the end, a path must be chosen, and we chose our path. I do not work on this part of the product, but I haven't seen any reason to believe it has been a hindrance, and probably is actually the right call. > the fact that this is fundamentally not-Linux (yes I know it runs Linux binaries unmodified) would be a reason against buying it, not for. I am curious why, if you feel like elaborating. EDIT: oh just saw your comment down here: https://news.ycombinator.com/item?id=39180814 https://news.ycombinator.com/item?id=39180814 1: https://rfd.shared.oxide.computer/ https://rfd.shared.oxide.computer/

wmf 3y ago
The Linux vs. Illumos decision seems to be downstream of a more fundamental decision to make VMs the narrow waist of the Oxide system. That's what I'm curious about.
- amluto 3y ago
  Especially since Oxide has a big fancy firmware stack. I would expect this stack to be able to do an excellent job of securely allocating bare-metal (i.e. VMX root on x86 or EL2 if Oxide ever goes ARM) resources. This would allow workloads on Oxide to run their own VMs, to safely use PCIe devices without dealing with interrupt redirection, etc.
  - wmf 3y ago
    I'm not affiliated with Oxide but I don't think you can put Crucible and VPC/OPTE in firmware. Without a DPU those components have to run in the hypervisor.
    - amluto 3y ago
      Possibly not. But I do wonder why cloud and cloud-like systems aren’t more aggressive about splitting the infrastructure and tenant portions of each server into different pieces of hardware, e.g. DPU. A DPU could look DPU could look like a PCIe target exposing NVMe and a NIC, for example. Obviously this would be an even more custom design than Oxide currently has, but Oxide doesn’t seem particularly shy about such things.
throwawaaarrgh 3y ago
A team should always pick the tools they are most familiar with. They will always have better results with that, than trying to use something they understand less. With this in mind, using their own stack is a perfectly adequate choice. Factors outside their team will determine if that works out in the long term.
- wmf 3y ago
  A handful of the team are more familiar with Illumos and the next hundred people they hire after that will be more familiar with Linux.
  - throwawaaarrgh 3y ago
    A lot of people out there claim to know Linux, yet few can prove it. OTOH, if they gain a cult following with lots of people using their stack, those people might become more familiar with their stack than most Linux people are with theirs. They could grow a captive base of prospective hires. That's not the big concern though. The big concern is whether vendor integration and certification becomes a stumbling block. You can hire any monkey to write good-enough code, but that doesn't give you millions in return. Partnerships with vendors and compliance certifications can give you hundreds of millions. The harder that is, the farther the money is. A totally custom, foreign stack can make it harder, or not; it depends how they allocate their human capital and business strategy, whether they can convince vendors to partner, and clients to buy in. Anything very different is a risk that's hard to ignore.
  - steveklabnik 3y ago
    To be clear, we at the time had already hired people with deep familiarity with Linux at the time this decision was made. In particular, Laura Abbott, as one example. It is true that the number of developers that know Linux is larger than the ones that know illumos. But this is also true of the number of developers who know C as the ones who know Rust. Just like some folks need to be onboarded to Rust, some will need to be onboarded to illumos. That is of course part of the tradeoff.
  - Jtsummers 3y ago
binjip 3y ago
It would be great if that RFD will become public someday, if it of course possible, especially if it's a long read.
apache8080 3y ago
Is there any chance Oxide is going to make more of these RFFs public? I think it would be a useful artifact to see why a company would choose to run a non-Linux OS. I also think there are other Oxide RFDs that would have a similar benefit e.g why Oxide decided to build dropshot instead of using one of the existing rust REST server crates?
- steveklabnik 3y ago
  Yes, for sure. We agree there's tons of value there, there's just so much to do it's easy to let things fall through the cracks. This thread has given some renewed energy to work on releasing some of them, so no promises but we'll see :)
networked 3y ago
If you publish the RFD, please submit it to HN. As a former (mostly former) FreeBSD user interested in bhyve, I would like to read the case for bhyve on illumos.

wmf 3y ago

The Linux vs. Illumos decision seems to be downstream of a more fundamental decision to make VMs the narrow waist of the Oxide system. That's what I'm curious about.

amluto 3y ago
Especially since Oxide has a big fancy firmware stack. I would expect this stack to be able to do an excellent job of securely allocating bare-metal (i.e. VMX root on x86 or EL2 if Oxide ever goes ARM) resources. This would allow workloads on Oxide to run their own VMs, to safely use PCIe devices without dealing with interrupt redirection, etc.
- wmf 3y ago
  I'm not affiliated with Oxide but I don't think you can put Crucible and VPC/OPTE in firmware. Without a DPU those components have to run in the hypervisor.
  - amluto 3y ago
    Possibly not. But I do wonder why cloud and cloud-like systems aren’t more aggressive about splitting the infrastructure and tenant portions of each server into different pieces of hardware, e.g. DPU. A DPU could look DPU could look like a PCIe target exposing NVMe and a NIC, for example. Obviously this would be an even more custom design than Oxide currently has, but Oxide doesn’t seem particularly shy about such things.

throwawaaarrgh 3y ago

A team should always pick the tools they are most familiar with. They will always have better results with that, than trying to use something they understand less. With this in mind, using their own stack is a perfectly adequate choice. Factors outside their team will determine if that works out in the long term.

wmf 3y ago
A handful of the team are more familiar with Illumos and the next hundred people they hire after that will be more familiar with Linux.
- throwawaaarrgh 3y ago
  A lot of people out there claim to know Linux, yet few can prove it. OTOH, if they gain a cult following with lots of people using their stack, those people might become more familiar with their stack than most Linux people are with theirs. They could grow a captive base of prospective hires. That's not the big concern though. The big concern is whether vendor integration and certification becomes a stumbling block. You can hire any monkey to write good-enough code, but that doesn't give you millions in return. Partnerships with vendors and compliance certifications can give you hundreds of millions. The harder that is, the farther the money is. A totally custom, foreign stack can make it harder, or not; it depends how they allocate their human capital and business strategy, whether they can convince vendors to partner, and clients to buy in. Anything very different is a risk that's hard to ignore.
- steveklabnik 3y ago
  To be clear, we at the time had already hired people with deep familiarity with Linux at the time this decision was made. In particular, Laura Abbott, as one example. It is true that the number of developers that know Linux is larger than the ones that know illumos. But this is also true of the number of developers who know C as the ones who know Rust. Just like some folks need to be onboarded to Rust, some will need to be onboarded to illumos. That is of course part of the tradeoff.
- Jtsummers 3y ago

binjip 3y ago

It would be great if that RFD will become public someday, if it of course possible, especially if it's a long read.

apache8080 3y ago

Is there any chance Oxide is going to make more of these RFFs public? I think it would be a useful artifact to see why a company would choose to run a non-Linux OS. I also think there are other Oxide RFDs that would have a similar benefit e.g why Oxide decided to build dropshot instead of using one of the existing rust REST server crates?

steveklabnik 3y ago
Yes, for sure. We agree there's tons of value there, there's just so much to do it's easy to let things fall through the cracks. This thread has given some renewed energy to work on releasing some of them, so no promises but we'll see :)

networked 3y ago

If you publish the RFD, please submit it to HN. As a former (mostly former) FreeBSD user interested in bhyve, I would like to read the case for bhyve on illumos.