6 ms·

I know they’re ex-Sun, but is there any real technical benefit for choosing not-Linux (for their business value prop)? I know of the technical benefits of illu

by sneak 3y ago

I know they’re ex-Sun, but is there any real technical benefit for choosing not-Linux (for their business value prop)? I know of the technical benefits of illumos over linux, but does that actually matter to the customers who are buying these? Aren’t they opening a whole can of worms for ideology/tradition that won’t sell any more computers? As someone who runs Linux container workloads, the fact that this is fundamentally not-Linux (yes I know it runs Linux binaries unmodified) would be a reason against buying it, not for.

kardianos 3y ago
In one podcast, the reason given was staff familiarity and owning the full stack, not just the kernel I believe.
vvern 3y ago
> yes I know it runs Linux binaries unmodified Is it that it runs Linux binaries unmodified or that it runs vms and manages VMs which run Linux, and as an end-user, that's what you run your software in?
- tonyarkles 3y ago
  As far as I recall it's not a VM. They run in "LX Branded Zones" which does require a Linux userland so that the binaries can find their libraries etc but Zones are more like "better cgroups than cgroups, a decade earlier" than VMs.
  - bcantrill 3y ago
    No, it's a VM, running a bhyve-based hypervisor, Propolis.[0] LX branded zones were/are great -- but for absolute fidelity one really needs VMs. [0] https://github.com/oxidecomputer/propolis https://github.com/oxidecomputer/propolis
    - bpye 3y ago
      Do you have a solution for running containers (Kubernetes, etc)? Are you spinning up a Linux VM to run the containers in there, doing VM per container, or something else?
      - panick21_ 3y ago
        Costumers can decide I would assume. Most likely you install you install some Kubernetes and then just have multible VMs distributed across the rack. And then run multible Pods in each node. VM per container seems like a waist unless you need that extra isolation.
        bpye 3y ago
        I wondered if there was any support for running containers built in - something like EKS/AKS/GKE/Cloud Run/etc - but looking at the docs it appears not. I agree that VM per container can be wasteful - though something like Firecracker at least helps with start time.
        panick21_ 3y ago
        From the podcast it seems that they want to deliver a minimal viable product. Their primary costumers already have a lot of their own higher level stack. They might get into adding more higher level software eventually depending on what costumers want.
- bcantrill 3y ago
  It runs VMs -- so it doesn't just run Linux binaries unmodified, it runs Linux kernels unmodified (and, for that matter, Windows, FreeBSD, OpenBSD, etc.).
pjmlp 3y ago
Their customers run virtualised OS on top of this. This is no different from Azure Host OS, Bottlerocket, Flatcar or whatever. This maters to them, as knowing the whole stack, some of the kernel code is still theirs from Sun days, and making it available matters to the customers that want source code access for security assement reasons.
skullone 3y ago
It seems healthy to have options, almost like the universe is healing a bit after oracle bought Sun. I can't imagine better hands bringing the oxide system together than that team. As an engineer who works entirely with Linux anymore, I pine for the days of another strong Unix in the mix to run high value workloads on. Comparing openvswitch on Linux, to say, the crossbow SDN facility on Solaris, I'd take crossbow any day. Nothing "wrong" with Linux, but it is sorely lacking in "master plan" levels of cohesion with all the tooling taking their own path, often bringing complexity that requires even for abstraction with yet more complicated tooling on top.
Extigy 3y ago
Perhaps Illumos is particularly well suited for a Hypervisor/Cloud platform due to work upstreamed by Joyent originally for SmartOS?
shusaaafuejdn 3y ago
As far as performance and feature set, probably not anymore (I would have answered differently 10 years ago, and if I am wrong today would love to be educated about it). However, if we are considering code quality, which I consider important if you are actually going to be maintaining it yourself as oxide will have to do since they need customizations, then most of the proprietary Unix sources are just superior imo. That is, they have better organization, more consistency in standards, etc. The BSDs are slightly better in this regard as well, it really isn't a proprietary vs open source issue, it's more about the insane size of the Linux kernel project making strict standards enforcement difficult if not impossible the further you get from the very core system components. Irregardless of them being ex-Sun (and I am not ex Sun), if I needed a custom OS for a product I was working on, Linux would be close to the last Unix based OS source tree I would try to do it with, only after all other options failed for whatever reason. And that's not even taking into account the licensing, which is a whole other can of worms.
bcantrill 3y ago
Keep in mind that Helios is really just an implementation detail of the rack; like Hubris[0], it's not something visible to the user or to applications. (The user of the rack provisions VMs.) As for why an illumos derivative and not something else, we expanded on this a bit in our Q&A when we shipped our first rack[1] -- and we will expand on it again in the (recorded) discussion that we will have later today.[2] [0] https://hubris.oxide.computer/ https://hubris.oxide.computer/ [1] https://www.youtube.com/watch?v=5P5Mk_IggE0&t=2556s https://www.youtube.com/watch?v=5P5Mk_IggE0&t=2556s [2] https://mastodon.social/@bcantrill/111840269356297809 https://mastodon.social/@bcantrill/111840269356297809
- kaliszad 3y ago
  Perhaps you could talk a bit about the distributed storage based on Crucible with ZFS as the backing storage tonight. I would really love to hear some of the details and challenges there.
  - bcantrill 3y ago
    Yes! Crucible[0] is on our list of upcoming episodes. We can touch on it tonight, but it's really deserving of its own deep dive! [0] https://github.com/oxidecomputer/crucible https://github.com/oxidecomputer/crucible
    - panick21_ 3y ago
      The timing of your podcast is the least convenient thing ever for us poor Europeans. And then the brutal wait the next day until its uploaded. The only thing I miss about Twitter Spaces is that you could listen the morning after.
      - kaliszad 3y ago
        Yes (hello from Czechia), however there will always be somebody who this is inconvenient for. Also, I have to confess I was at times immersed in other work that I made a few Oxide and Friends live. I might stay up tonight. I am looking forward to the crucible episode. It sounds like it could be a startup on its own, it wouldn't be the first distributed file/ storage system company.
quux 3y ago
Aren't they also ex-Joyent? Joyent ran customer VMs in prod on Illumos for many years so there's a lot of experience there.
- steveklabnik 3y ago
  Many people, including part of the founding team, are ex-Joyent, yes. Some also worked at Sun, on the operating systems that illumos is ultimately derived from.
- littlestymaar 3y ago
  bcantrill used to work at Sun then became CTO at Joyent, so the reason why Joyent ran Illumos is probably the same reason as why Oxide is, because Cantrill likes it and judges that it's a good fit for what they are doing.
  - steveklabnik 3y ago
    As I elaborated above, bcantrill did not decree that we must use illumos. Technical decisions are not handed down from above at Oxide.
    - littlestymaar 3y ago
      I saw your comment[1] after I wrote mine, but I'm not saying that he's forcing you guys to use it (that would not a good way of being a CTO at a start-up…), but that doesn't prevent him from advocating for solutions he believes in. Would you say that Oxide would have chosen Illumos if he wasn't part of the company? [1]: https://news.ycombinator.com/item?id=39180706 https://news.ycombinator.com/item?id=39180706
      - sunshowers 3y ago
        (I work at Oxide.) Bryan is just one out of several illumos experts here. If none of those were around, sure, maybe we wouldn't have picked illumos -- but then we'd be unrecognizably different. I came into Oxide with a Linux background and zero knowledge of illumos. Learning about DTrace especially has been great.
jeffbee 3y ago
Do you have the same gut reaction to ESXi?
- stonogo 3y ago
  I sure do. We've finally got to a place where we don't need weird hardware tricks to containerize workloads -- this is why a lot of shops pursue docker-like ops for production. When I buy hardware, long-term maintenance is a factor, and when my whole operations fleet relies on ESX, or in this case a Solaris fork, I'm now beholden to one company for support at that layer. Buying a rack of Supermicro gear and running RHEL or SLES with containerized orchestration on top means I can, in a pinch, hire experts anywhere to work on my systems. I have no reason to believe Oxide would be anything but responsive and effective in supporting their systems, but introducing bespoke software this deep in the stack severely curtails my options if things get bad.
  - apendleton 3y ago
    I think the value proposition they're offering is a carefully integrated system where everything has been thoroughly engineered/tested to work with everything else, down to writing custom firmware to guarantee that it's all ship-shape, so that customers don't have to touch any of the innards, and will probably just treat them as a black box. It seems like it's chock-full of stuff that they custom-built and that nobody else would be familiar with, by design. If that's not what you want, this probably isn't the product for you.
  - jeffbee 3y ago
    I can somewhat see your point, but in my experience you can't rely on RHEL or whatever vendor Linux to correctly bring up random OEM hardware. You will slowly discover all of the quirks, like it didn't initialize the platform EDAC the way you expected, or it didn't resolve some weird IRQ issue, etc. Nothing about my experience leads me to believe Linux will JFW on a given box, so I don't feel like Linux has an advantage in this regard, or that niche operating systems have a disadvantage. Certainly I feel like a first-party OS from the hardware vendor is going to have a lot of advantages.
steveklabnik 3y ago
> does that actually matter to the customers who are buying these? It's not like we specifically say "oh btw there's illumos inside and that's why you should buy the rack." It's not a customer-facing detail of the product. I'm sure most will never even know that this is the case. What customers do care about is that the rack is efficient, reliable, suits their needs, etc. Choosing illumos instead of Linux here is a choice made to help effectively deliver on that value. This does not mean that you couldn't build a similar product on top of Linux inherently, by the way, just that we decided illumos was more fit for purpose. This decision was made with the team, in the form of an RFD[1]. It's #26, though it is not currently public. The two choices that were seriously considered were KVM on Linux, and bhyve on illumos. It is pretty long. In the end, a path must be chosen, and we chose our path. I do not work on this part of the product, but I haven't seen any reason to believe it has been a hindrance, and probably is actually the right call. > the fact that this is fundamentally not-Linux (yes I know it runs Linux binaries unmodified) would be a reason against buying it, not for. I am curious why, if you feel like elaborating. EDIT: oh just saw your comment down here: https://news.ycombinator.com/item?id=39180814 https://news.ycombinator.com/item?id=39180814 1: https://rfd.shared.oxide.computer/ https://rfd.shared.oxide.computer/
- wmf 3y ago
  The Linux vs. Illumos decision seems to be downstream of a more fundamental decision to make VMs the narrow waist of the Oxide system. That's what I'm curious about.
  - amluto 3y ago
    Especially since Oxide has a big fancy firmware stack. I would expect this stack to be able to do an excellent job of securely allocating bare-metal (i.e. VMX root on x86 or EL2 if Oxide ever goes ARM) resources. This would allow workloads on Oxide to run their own VMs, to safely use PCIe devices without dealing with interrupt redirection, etc.
    - wmf 3y ago
      I'm not affiliated with Oxide but I don't think you can put Crucible and VPC/OPTE in firmware. Without a DPU those components have to run in the hypervisor.
thinkingkong 3y ago
This has been / will be the market education challenge; Its the same one Joyent had with SmartOS. Theyre correctly pointing out that the end user or operator will basically never interact with this layer, but it does cause some knee-jerk reactions. All that said, there are some pretty great technical benefits to using illumos derived systems the least of which is the teams familiarity and ability to do real diagnosis on production issues. I wont put words in anyones mouth but I suspect thats going to be critical for them as they support customer deployments w/o direct physical access.
spamizbad 3y ago
Seems strange to me too but it sounds like the end-users basically never interact with this - it's just firmware humming along in the background. As long as its open-source and reasonably well documented its already lightyears ahead of what else is out there.
greggyb 3y ago
If you're running in one of the big 3 cloud providers, the bottom-level hypervisors are not-linux. This is equivalent. Are you anti-AWS or anti-Azure for the same reason? This is the substrate upon which you will run any virtualized infrastructure.
- qmarchi 3y ago
  Small note, that's not true for Google Cloud, which runs on top of Linux, though modified. Disclaimer: Former Googler, Cloud Support
  - refulgentis 3y ago
    Another Xoogler here: any idea what they mean by it's not Linux at the bottom for other providers? Like, surely it's _some_ common OS? Either my binaries wouldn't run or AWS is reimplementing Linux so they can, which seems odd. Or are they just saying that the VM my binary runs on might be some predictable Linux version, but the underlying thing launching the VM could be anything?
    - qmarchi 3y ago
      Correct, that the Hypervisor isn't running Linux. I think the only provider where that would make sense would be Microsoft, where they have their own OS.
    - p_l 3y ago
      Old AWS used to be Xen, Nitro afaik uses customised VMM and I don't recall if it's not a custom OS or hosted on top of something. Azure is Hyper-V underneath IIRC, a custom variant at least (remember Windows Server Nano? IIRC it was the closest you could get to running it), with sometimes weird things like network cards running Linux and integrating with Windows' built-in SDN facility. Rest of the bigger ones is mainly Linux with occasional Xen and such, but sometimes you can encounter non-trivial VMware deployments.
      - zokier 3y ago
StillBored 3y ago
Linux is a nightmare in the embedded/appliance space because one ends up just having platform engineers who spend their day fixing problems with the latest kernels, drivers, core libraries, etc, that the actual application depends on. Or one goes the route of 99% of the IoT/etc vendors, and never update the base OS and pray that there aren't any active exploits targeting it. This is why a lot of medium-sized companies cried about Centos, which allowed them to largely stick to a fairly stable platform that was getting security updates without having to actually pay/run a full blown RHEL/etc install. Every ten years or so they had to revisit all the dependencies, but that is a far easier problem than dealing with a year or two update cycle, which is too short when the qualification timeframe for some of these systems is 6+ months long. So, this is almost exclusively a Linux problem; any of the *BSD/etc. alternatives give you almost all of what Linux provides without this constant breakage.
- bcantrill 3y ago
  This is a really, really good point -- and is a result of the model of Linux being only a kernel (and not system libraries, commands, etc.). It means that any real use of Linux is not merely signing up for kernel maintenance (which itself can be arduous) but also must make decisions around every other aspect of the system (each with its own communities, release management, etc.). This act is the act of creating a distribution -- and it's a huge burden to take on. Both illumos and the BSD derivatives make this significantly easier by simply including much more of the system within their scope: they are not merely kernels, but also system libraries and commands. This weighed heavily in our own calculus, so I'm glad you brought it up!
  - trhway 3y ago
    >including much more of the system within their scope: they are not merely kernels, but also system libraries and commands. giving limited resources of the dev team it may lead to limited support of the system outside of the narrow set of officially supported/certified hardware with that support falling behind on modern hardware, as it happened with Sun, and vendor lock-in as a result into overpriced and low performing hardware. There is a reason that back then at Solaris dev there was a joke about embedding Linux kernel as a universal driver for Solaris kernel in order to get reasonable support for the hardware around.
NexRebular 3y ago
Not everything needs to be linux. Besides, if monocultures are supposed to be harmful, why is linux being thrown to everything nowadays? Very dangerous to have a single point of failure in (critical) applications.
mardifoufs 3y ago
I think it's a good idea to have more choice, especially in OSS. A Linux mono culture isn't any better than a chromium mono culture. They might be able to do stuff that just isn't practical if they stuck with Linux. They are also probably more familiar with illumos, or at least familiar enough to know that they can use it to do more than with linux
moondev 3y ago
The main drawbacks to me are 1. No support for nested virtualization, so running a vm inside your vm is not available. This prevents use of projects such as kubevirt or firecracker on a Linux guest, and WSL2 on a Windows guest. 2. No GPU support If the base hypervisor was Linux, it would be way more capable for users it seems. I also wonder if internally Linux is used for development of the platform itself so they can create "virtual" racks to dogfood the product without full blown physical racks. With all that said, I do not know the roadmap and admittedly there are already quite a few existing platforms built on kvm, so as their hypervisor improves and becomes more capable it could potentially become strategic advantage.
- steveklabnik 3y ago
  > I also wonder if internally Linux is used for development of the platform itself Developers at Oxide work on whatever platform they'd like, as long as they can do their work. I will say I am in the minority as a Windows user though, most are on some form of Unix. > so they can create "virtual" racks to dogfood the product without full blown physical racks. So one of the reasons why Rust is such an advantage for us is its strong cross-platform support: you can run a simulated version of the control plane on Mac, Linux, and Illumos, without a physical rack. The non-simulated version must run on Helios. [1] That said we do have a rack in the office (literally named dogfood) that employees can use for various things if they wish. 1: https://github.com/oxidecomputer/omicron?tab=readme-ov-file#build-and-run https://github.com/oxidecomputer/omicron?tab=readme-ov-file#...
  - moondev 3y ago
    Interesting thanks for the insight. > I will say I am in the minority as a Windows user though, most are on some form of Unix. Now i'm imagining Helios inside WSI - Windows Subsystem for illumos
    - steveklabnik 3y ago
      You're welcome. I will give you one more fun anecdote here: when I came to Oxide, nobody in my corner of the company was using Windows. And hubris and humility almost Just Worked: we had one build system issue that was using strings instead of the path APIs, but as soon as I fixed those, it all worked. bcantrill remarked that if you had gone back in time and told him long ago that some of his code would Just Work on Windows, he would have called you a liar, and it's one of the things that validates our decisions to go with Rust over C as the default language for development inside Oxide. > Now i'm imagining Helios inside WSI - Windows Subsystem for illumos That would be pretty funny, ha! IIRC something about simulated omicron doesn't work inside WSL, but since I don't work on it actively, I haven't bothered to try and patch that up. I think I tried one time, I don't remember specifically what the issue was, as I don't generally use WSL for development, so it's a bit foreign to me as well.
sarlalian 3y ago
As a customer, I expect most of the technical advantages will be basically being a down stream consumer of ZFS. From a developer / maintainer of an OS, Dtrace and ZFS are large technical wins. Part of the overall value proposition of Oxide, is "correctness". You get an OS/Hardware stack that are designed to work together. You get 20 years of cruft thrown out. You get a lot of tooling, API's, etc written in a performant memory safe language (rust). Also you get a really fantastic podcast about the whole process. And as a customer you get a company that understands their stack from driver to VM and has a ton of internal expertise debugging production problems.

by sneak 3y ago

kardianos 3y ago

In one podcast, the reason given was staff familiarity and owning the full stack, not just the kernel I believe.

vvern 3y ago

> yes I know it runs Linux binaries unmodified Is it that it runs Linux binaries unmodified or that it runs vms and manages VMs which run Linux, and as an end-user, that's what you run your software in?

tonyarkles 3y ago
As far as I recall it's not a VM. They run in "LX Branded Zones" which does require a Linux userland so that the binaries can find their libraries etc but Zones are more like "better cgroups than cgroups, a decade earlier" than VMs.
- bcantrill 3y ago
  No, it's a VM, running a bhyve-based hypervisor, Propolis.[0] LX branded zones were/are great -- but for absolute fidelity one really needs VMs. [0] https://github.com/oxidecomputer/propolis https://github.com/oxidecomputer/propolis
  - bpye 3y ago
    Do you have a solution for running containers (Kubernetes, etc)? Are you spinning up a Linux VM to run the containers in there, doing VM per container, or something else?
    - panick21_ 3y ago
      Costumers can decide I would assume. Most likely you install you install some Kubernetes and then just have multible VMs distributed across the rack. And then run multible Pods in each node. VM per container seems like a waist unless you need that extra isolation.
      - bpye 3y ago
        I wondered if there was any support for running containers built in - something like EKS/AKS/GKE/Cloud Run/etc - but looking at the docs it appears not. I agree that VM per container can be wasteful - though something like Firecracker at least helps with start time.
        panick21_ 3y ago
        From the podcast it seems that they want to deliver a minimal viable product. Their primary costumers already have a lot of their own higher level stack. They might get into adding more higher level software eventually depending on what costumers want.
bcantrill 3y ago
It runs VMs -- so it doesn't just run Linux binaries unmodified, it runs Linux kernels unmodified (and, for that matter, Windows, FreeBSD, OpenBSD, etc.).

pjmlp 3y ago

Their customers run virtualised OS on top of this. This is no different from Azure Host OS, Bottlerocket, Flatcar or whatever. This maters to them, as knowing the whole stack, some of the kernel code is still theirs from Sun days, and making it available matters to the customers that want source code access for security assement reasons.

skullone 3y ago

It seems healthy to have options, almost like the universe is healing a bit after oracle bought Sun. I can't imagine better hands bringing the oxide system together than that team. As an engineer who works entirely with Linux anymore, I pine for the days of another strong Unix in the mix to run high value workloads on. Comparing openvswitch on Linux, to say, the crossbow SDN facility on Solaris, I'd take crossbow any day. Nothing "wrong" with Linux, but it is sorely lacking in "master plan" levels of cohesion with all the tooling taking their own path, often bringing complexity that requires even for abstraction with yet more complicated tooling on top.

Extigy 3y ago

Perhaps Illumos is particularly well suited for a Hypervisor/Cloud platform due to work upstreamed by Joyent originally for SmartOS?

shusaaafuejdn 3y ago

As far as performance and feature set, probably not anymore (I would have answered differently 10 years ago, and if I am wrong today would love to be educated about it). However, if we are considering code quality, which I consider important if you are actually going to be maintaining it yourself as oxide will have to do since they need customizations, then most of the proprietary Unix sources are just superior imo. That is, they have better organization, more consistency in standards, etc. The BSDs are slightly better in this regard as well, it really isn't a proprietary vs open source issue, it's more about the insane size of the Linux kernel project making strict standards enforcement difficult if not impossible the further you get from the very core system components. Irregardless of them being ex-Sun (and I am not ex Sun), if I needed a custom OS for a product I was working on, Linux would be close to the last Unix based OS source tree I would try to do it with, only after all other options failed for whatever reason. And that's not even taking into account the licensing, which is a whole other can of worms.

bcantrill 3y ago

Keep in mind that Helios is really just an implementation detail of the rack; like Hubris[0], it's not something visible to the user or to applications. (The user of the rack provisions VMs.) As for why an illumos derivative and not something else, we expanded on this a bit in our Q&A when we shipped our first rack[1] -- and we will expand on it again in the (recorded) discussion that we will have later today.[2] [0] https://hubris.oxide.computer/ https://hubris.oxide.computer/ [1] https://www.youtube.com/watch?v=5P5Mk_IggE0&t=2556s https://www.youtube.com/watch?v=5P5Mk_IggE0&t=2556s [2] https://mastodon.social/@bcantrill/111840269356297809 https://mastodon.social/@bcantrill/111840269356297809

kaliszad 3y ago
Perhaps you could talk a bit about the distributed storage based on Crucible with ZFS as the backing storage tonight. I would really love to hear some of the details and challenges there.
- bcantrill 3y ago
  Yes! Crucible[0] is on our list of upcoming episodes. We can touch on it tonight, but it's really deserving of its own deep dive! [0] https://github.com/oxidecomputer/crucible https://github.com/oxidecomputer/crucible
  - panick21_ 3y ago
    The timing of your podcast is the least convenient thing ever for us poor Europeans. And then the brutal wait the next day until its uploaded. The only thing I miss about Twitter Spaces is that you could listen the morning after.
    - kaliszad 3y ago
      Yes (hello from Czechia), however there will always be somebody who this is inconvenient for. Also, I have to confess I was at times immersed in other work that I made a few Oxide and Friends live. I might stay up tonight. I am looking forward to the crucible episode. It sounds like it could be a startup on its own, it wouldn't be the first distributed file/ storage system company.

quux 3y ago

Aren't they also ex-Joyent? Joyent ran customer VMs in prod on Illumos for many years so there's a lot of experience there.

steveklabnik 3y ago
Many people, including part of the founding team, are ex-Joyent, yes. Some also worked at Sun, on the operating systems that illumos is ultimately derived from.
littlestymaar 3y ago
bcantrill used to work at Sun then became CTO at Joyent, so the reason why Joyent ran Illumos is probably the same reason as why Oxide is, because Cantrill likes it and judges that it's a good fit for what they are doing.
- steveklabnik 3y ago
  As I elaborated above, bcantrill did not decree that we must use illumos. Technical decisions are not handed down from above at Oxide.
  - littlestymaar 3y ago
    I saw your comment[1] after I wrote mine, but I'm not saying that he's forcing you guys to use it (that would not a good way of being a CTO at a start-up…), but that doesn't prevent him from advocating for solutions he believes in. Would you say that Oxide would have chosen Illumos if he wasn't part of the company? [1]: https://news.ycombinator.com/item?id=39180706 https://news.ycombinator.com/item?id=39180706
    - sunshowers 3y ago
      (I work at Oxide.) Bryan is just one out of several illumos experts here. If none of those were around, sure, maybe we wouldn't have picked illumos -- but then we'd be unrecognizably different. I came into Oxide with a Linux background and zero knowledge of illumos. Learning about DTrace especially has been great.

jeffbee 3y ago

Do you have the same gut reaction to ESXi?

stonogo 3y ago
I sure do. We've finally got to a place where we don't need weird hardware tricks to containerize workloads -- this is why a lot of shops pursue docker-like ops for production. When I buy hardware, long-term maintenance is a factor, and when my whole operations fleet relies on ESX, or in this case a Solaris fork, I'm now beholden to one company for support at that layer. Buying a rack of Supermicro gear and running RHEL or SLES with containerized orchestration on top means I can, in a pinch, hire experts anywhere to work on my systems. I have no reason to believe Oxide would be anything but responsive and effective in supporting their systems, but introducing bespoke software this deep in the stack severely curtails my options if things get bad.
- apendleton 3y ago
  I think the value proposition they're offering is a carefully integrated system where everything has been thoroughly engineered/tested to work with everything else, down to writing custom firmware to guarantee that it's all ship-shape, so that customers don't have to touch any of the innards, and will probably just treat them as a black box. It seems like it's chock-full of stuff that they custom-built and that nobody else would be familiar with, by design. If that's not what you want, this probably isn't the product for you.
- jeffbee 3y ago
  I can somewhat see your point, but in my experience you can't rely on RHEL or whatever vendor Linux to correctly bring up random OEM hardware. You will slowly discover all of the quirks, like it didn't initialize the platform EDAC the way you expected, or it didn't resolve some weird IRQ issue, etc. Nothing about my experience leads me to believe Linux will JFW on a given box, so I don't feel like Linux has an advantage in this regard, or that niche operating systems have a disadvantage. Certainly I feel like a first-party OS from the hardware vendor is going to have a lot of advantages.

steveklabnik 3y ago

> does that actually matter to the customers who are buying these? It's not like we specifically say "oh btw there's illumos inside and that's why you should buy the rack." It's not a customer-facing detail of the product. I'm sure most will never even know that this is the case. What customers do care about is that the rack is efficient, reliable, suits their needs, etc. Choosing illumos instead of Linux here is a choice made to help effectively deliver on that value. This does not mean that you couldn't build a similar product on top of Linux inherently, by the way, just that we decided illumos was more fit for purpose. This decision was made with the team, in the form of an RFD[1]. It's #26, though it is not currently public. The two choices that were seriously considered were KVM on Linux, and bhyve on illumos. It is pretty long. In the end, a path must be chosen, and we chose our path. I do not work on this part of the product, but I haven't seen any reason to believe it has been a hindrance, and probably is actually the right call. > the fact that this is fundamentally not-Linux (yes I know it runs Linux binaries unmodified) would be a reason against buying it, not for. I am curious why, if you feel like elaborating. EDIT: oh just saw your comment down here: https://news.ycombinator.com/item?id=39180814 https://news.ycombinator.com/item?id=39180814 1: https://rfd.shared.oxide.computer/ https://rfd.shared.oxide.computer/

wmf 3y ago
The Linux vs. Illumos decision seems to be downstream of a more fundamental decision to make VMs the narrow waist of the Oxide system. That's what I'm curious about.
- amluto 3y ago
  Especially since Oxide has a big fancy firmware stack. I would expect this stack to be able to do an excellent job of securely allocating bare-metal (i.e. VMX root on x86 or EL2 if Oxide ever goes ARM) resources. This would allow workloads on Oxide to run their own VMs, to safely use PCIe devices without dealing with interrupt redirection, etc.
  - wmf 3y ago
    I'm not affiliated with Oxide but I don't think you can put Crucible and VPC/OPTE in firmware. Without a DPU those components have to run in the hypervisor.

thinkingkong 3y ago

This has been / will be the market education challenge; Its the same one Joyent had with SmartOS. Theyre correctly pointing out that the end user or operator will basically never interact with this layer, but it does cause some knee-jerk reactions. All that said, there are some pretty great technical benefits to using illumos derived systems the least of which is the teams familiarity and ability to do real diagnosis on production issues. I wont put words in anyones mouth but I suspect thats going to be critical for them as they support customer deployments w/o direct physical access.

spamizbad 3y ago

Seems strange to me too but it sounds like the end-users basically never interact with this - it's just firmware humming along in the background. As long as its open-source and reasonably well documented its already lightyears ahead of what else is out there.

greggyb 3y ago

If you're running in one of the big 3 cloud providers, the bottom-level hypervisors are not-linux. This is equivalent. Are you anti-AWS or anti-Azure for the same reason? This is the substrate upon which you will run any virtualized infrastructure.

qmarchi 3y ago
Small note, that's not true for Google Cloud, which runs on top of Linux, though modified. Disclaimer: Former Googler, Cloud Support
- refulgentis 3y ago
  Another Xoogler here: any idea what they mean by it's not Linux at the bottom for other providers? Like, surely it's _some_ common OS? Either my binaries wouldn't run or AWS is reimplementing Linux so they can, which seems odd. Or are they just saying that the VM my binary runs on might be some predictable Linux version, but the underlying thing launching the VM could be anything?
  - qmarchi 3y ago
    Correct, that the Hypervisor isn't running Linux. I think the only provider where that would make sense would be Microsoft, where they have their own OS.
  - p_l 3y ago
    Old AWS used to be Xen, Nitro afaik uses customised VMM and I don't recall if it's not a custom OS or hosted on top of something. Azure is Hyper-V underneath IIRC, a custom variant at least (remember Windows Server Nano? IIRC it was the closest you could get to running it), with sometimes weird things like network cards running Linux and integrating with Windows' built-in SDN facility. Rest of the bigger ones is mainly Linux with occasional Xen and such, but sometimes you can encounter non-trivial VMware deployments.
    - zokier 3y ago

StillBored 3y ago

Linux is a nightmare in the embedded/appliance space because one ends up just having platform engineers who spend their day fixing problems with the latest kernels, drivers, core libraries, etc, that the actual application depends on. Or one goes the route of 99% of the IoT/etc vendors, and never update the base OS and pray that there aren't any active exploits targeting it. This is why a lot of medium-sized companies cried about Centos, which allowed them to largely stick to a fairly stable platform that was getting security updates without having to actually pay/run a full blown RHEL/etc install. Every ten years or so they had to revisit all the dependencies, but that is a far easier problem than dealing with a year or two update cycle, which is too short when the qualification timeframe for some of these systems is 6+ months long. So, this is almost exclusively a Linux problem; any of the *BSD/etc. alternatives give you almost all of what Linux provides without this constant breakage.

bcantrill 3y ago
This is a really, really good point -- and is a result of the model of Linux being only a kernel (and not system libraries, commands, etc.). It means that any real use of Linux is not merely signing up for kernel maintenance (which itself can be arduous) but also must make decisions around every other aspect of the system (each with its own communities, release management, etc.). This act is the act of creating a distribution -- and it's a huge burden to take on. Both illumos and the BSD derivatives make this significantly easier by simply including much more of the system within their scope: they are not merely kernels, but also system libraries and commands. This weighed heavily in our own calculus, so I'm glad you brought it up!
- trhway 3y ago
  >including much more of the system within their scope: they are not merely kernels, but also system libraries and commands. giving limited resources of the dev team it may lead to limited support of the system outside of the narrow set of officially supported/certified hardware with that support falling behind on modern hardware, as it happened with Sun, and vendor lock-in as a result into overpriced and low performing hardware. There is a reason that back then at Solaris dev there was a joke about embedding Linux kernel as a universal driver for Solaris kernel in order to get reasonable support for the hardware around.

NexRebular 3y ago

Not everything needs to be linux. Besides, if monocultures are supposed to be harmful, why is linux being thrown to everything nowadays? Very dangerous to have a single point of failure in (critical) applications.

mardifoufs 3y ago

I think it's a good idea to have more choice, especially in OSS. A Linux mono culture isn't any better than a chromium mono culture. They might be able to do stuff that just isn't practical if they stuck with Linux. They are also probably more familiar with illumos, or at least familiar enough to know that they can use it to do more than with linux

moondev 3y ago

The main drawbacks to me are 1. No support for nested virtualization, so running a vm inside your vm is not available. This prevents use of projects such as kubevirt or firecracker on a Linux guest, and WSL2 on a Windows guest. 2. No GPU support If the base hypervisor was Linux, it would be way more capable for users it seems. I also wonder if internally Linux is used for development of the platform itself so they can create "virtual" racks to dogfood the product without full blown physical racks. With all that said, I do not know the roadmap and admittedly there are already quite a few existing platforms built on kvm, so as their hypervisor improves and becomes more capable it could potentially become strategic advantage.

steveklabnik 3y ago
> I also wonder if internally Linux is used for development of the platform itself Developers at Oxide work on whatever platform they'd like, as long as they can do their work. I will say I am in the minority as a Windows user though, most are on some form of Unix. > so they can create "virtual" racks to dogfood the product without full blown physical racks. So one of the reasons why Rust is such an advantage for us is its strong cross-platform support: you can run a simulated version of the control plane on Mac, Linux, and Illumos, without a physical rack. The non-simulated version must run on Helios. [1] That said we do have a rack in the office (literally named dogfood) that employees can use for various things if they wish. 1: https://github.com/oxidecomputer/omicron?tab=readme-ov-file#build-and-run https://github.com/oxidecomputer/omicron?tab=readme-ov-file#...
- moondev 3y ago
  Interesting thanks for the insight. > I will say I am in the minority as a Windows user though, most are on some form of Unix. Now i'm imagining Helios inside WSI - Windows Subsystem for illumos
  - steveklabnik 3y ago
    You're welcome. I will give you one more fun anecdote here: when I came to Oxide, nobody in my corner of the company was using Windows. And hubris and humility almost Just Worked: we had one build system issue that was using strings instead of the path APIs, but as soon as I fixed those, it all worked. bcantrill remarked that if you had gone back in time and told him long ago that some of his code would Just Work on Windows, he would have called you a liar, and it's one of the things that validates our decisions to go with Rust over C as the default language for development inside Oxide. > Now i'm imagining Helios inside WSI - Windows Subsystem for illumos That would be pretty funny, ha! IIRC something about simulated omicron doesn't work inside WSL, but since I don't work on it actively, I haven't bothered to try and patch that up. I think I tried one time, I don't remember specifically what the issue was, as I don't generally use WSL for development, so it's a bit foreign to me as well.

sarlalian 3y ago

As a customer, I expect most of the technical advantages will be basically being a down stream consumer of ZFS. From a developer / maintainer of an OS, Dtrace and ZFS are large technical wins. Part of the overall value proposition of Oxide, is "correctness". You get an OS/Hardware stack that are designed to work together. You get 20 years of cruft thrown out. You get a lot of tooling, API's, etc written in a performant memory safe language (rust). Also you get a really fantastic podcast about the whole process. And as a customer you get a company that understands their stack from driver to VM and has a ton of internal expertise debugging production problems.