18 ms·

Helios: A distribution of Illumos powering the Oxide Rack

418 points by eduction 3y ago

sneak 3y ago
I know they’re ex-Sun, but is there any real technical benefit for choosing not-Linux (for their business value prop)? I know of the technical benefits of illumos over linux, but does that actually matter to the customers who are buying these? Aren’t they opening a whole can of worms for ideology/tradition that won’t sell any more computers? As someone who runs Linux container workloads, the fact that this is fundamentally not-Linux (yes I know it runs Linux binaries unmodified) would be a reason against buying it, not for.
- kardianos 3y ago
  In one podcast, the reason given was staff familiarity and owning the full stack, not just the kernel I believe.
- vvern 3y ago
  > yes I know it runs Linux binaries unmodified Is it that it runs Linux binaries unmodified or that it runs vms and manages VMs which run Linux, and as an end-user, that's what you run your software in?
  - tonyarkles 3y ago
    As far as I recall it's not a VM. They run in "LX Branded Zones" which does require a Linux userland so that the binaries can find their libraries etc but Zones are more like "better cgroups than cgroups, a decade earlier" than VMs.
    - bcantrill 3y ago
      No, it's a VM, running a bhyve-based hypervisor, Propolis.[0] LX branded zones were/are great -- but for absolute fidelity one really needs VMs. [0] https://github.com/oxidecomputer/propolis https://github.com/oxidecomputer/propolis
      - bpye 3y ago
        Do you have a solution for running containers (Kubernetes, etc)? Are you spinning up a Linux VM to run the containers in there, doing VM per container, or something else?
        panick21_ 3y ago
        Costumers can decide I would assume. Most likely you install you install some Kubernetes and then just have multible VMs distributed across the rack. And then run multible Pods in each node. VM per container seems like a waist unless you need that extra isolation.
        bpye 3y ago
        I wondered if there was any support for running containers built in - something like EKS/AKS/GKE/Cloud Run/etc - but looking at the docs it appears not. I agree that VM per container can be wasteful - though something like Firecracker at least helps with start time.
        panick21_ 3y ago
        From the podcast it seems that they want to deliver a minimal viable product. Their primary costumers already have a lot of their own higher level stack. They might get into adding more higher level software eventually depending on what costumers want.
  - bcantrill 3y ago
    It runs VMs -- so it doesn't just run Linux binaries unmodified, it runs Linux kernels unmodified (and, for that matter, Windows, FreeBSD, OpenBSD, etc.).
- pjmlp 3y ago
  Their customers run virtualised OS on top of this. This is no different from Azure Host OS, Bottlerocket, Flatcar or whatever. This maters to them, as knowing the whole stack, some of the kernel code is still theirs from Sun days, and making it available matters to the customers that want source code access for security assement reasons.
- skullone 3y ago
  It seems healthy to have options, almost like the universe is healing a bit after oracle bought Sun. I can't imagine better hands bringing the oxide system together than that team. As an engineer who works entirely with Linux anymore, I pine for the days of another strong Unix in the mix to run high value workloads on. Comparing openvswitch on Linux, to say, the crossbow SDN facility on Solaris, I'd take crossbow any day. Nothing "wrong" with Linux, but it is sorely lacking in "master plan" levels of cohesion with all the tooling taking their own path, often bringing complexity that requires even for abstraction with yet more complicated tooling on top.
- Extigy 3y ago
  Perhaps Illumos is particularly well suited for a Hypervisor/Cloud platform due to work upstreamed by Joyent originally for SmartOS?
- shusaaafuejdn 3y ago
  As far as performance and feature set, probably not anymore (I would have answered differently 10 years ago, and if I am wrong today would love to be educated about it). However, if we are considering code quality, which I consider important if you are actually going to be maintaining it yourself as oxide will have to do since they need customizations, then most of the proprietary Unix sources are just superior imo. That is, they have better organization, more consistency in standards, etc. The BSDs are slightly better in this regard as well, it really isn't a proprietary vs open source issue, it's more about the insane size of the Linux kernel project making strict standards enforcement difficult if not impossible the further you get from the very core system components. Irregardless of them being ex-Sun (and I am not ex Sun), if I needed a custom OS for a product I was working on, Linux would be close to the last Unix based OS source tree I would try to do it with, only after all other options failed for whatever reason. And that's not even taking into account the licensing, which is a whole other can of worms.
- bcantrill 3y ago
  Keep in mind that Helios is really just an implementation detail of the rack; like Hubris[0], it's not something visible to the user or to applications. (The user of the rack provisions VMs.) As for why an illumos derivative and not something else, we expanded on this a bit in our Q&A when we shipped our first rack[1] -- and we will expand on it again in the (recorded) discussion that we will have later today.[2] [0] https://hubris.oxide.computer/ https://hubris.oxide.computer/ [1] https://www.youtube.com/watch?v=5P5Mk_IggE0&t=2556s https://www.youtube.com/watch?v=5P5Mk_IggE0&t=2556s [2] https://mastodon.social/@bcantrill/111840269356297809 https://mastodon.social/@bcantrill/111840269356297809
  - kaliszad 3y ago
    Perhaps you could talk a bit about the distributed storage based on Crucible with ZFS as the backing storage tonight. I would really love to hear some of the details and challenges there.
    - bcantrill 3y ago
      Yes! Crucible[0] is on our list of upcoming episodes. We can touch on it tonight, but it's really deserving of its own deep dive! [0] https://github.com/oxidecomputer/crucible https://github.com/oxidecomputer/crucible
      - panick21_ 3y ago
        The timing of your podcast is the least convenient thing ever for us poor Europeans. And then the brutal wait the next day until its uploaded. The only thing I miss about Twitter Spaces is that you could listen the morning after.
        kaliszad 3y ago
        Yes (hello from Czechia), however there will always be somebody who this is inconvenient for. Also, I have to confess I was at times immersed in other work that I made a few Oxide and Friends live. I might stay up tonight. I am looking forward to the crucible episode. It sounds like it could be a startup on its own, it wouldn't be the first distributed file/ storage system company.
- quux 3y ago
  Aren't they also ex-Joyent? Joyent ran customer VMs in prod on Illumos for many years so there's a lot of experience there.
  - steveklabnik 3y ago
    Many people, including part of the founding team, are ex-Joyent, yes. Some also worked at Sun, on the operating systems that illumos is ultimately derived from.
  - littlestymaar 3y ago
    bcantrill used to work at Sun then became CTO at Joyent, so the reason why Joyent ran Illumos is probably the same reason as why Oxide is, because Cantrill likes it and judges that it's a good fit for what they are doing.
    - steveklabnik 3y ago
      As I elaborated above, bcantrill did not decree that we must use illumos. Technical decisions are not handed down from above at Oxide.
      - littlestymaar 3y ago
        I saw your comment[1] after I wrote mine, but I'm not saying that he's forcing you guys to use it (that would not a good way of being a CTO at a start-up…), but that doesn't prevent him from advocating for solutions he believes in. Would you say that Oxide would have chosen Illumos if he wasn't part of the company? [1]: https://news.ycombinator.com/item?id=39180706 https://news.ycombinator.com/item?id=39180706
        sunshowers 3y ago
        (I work at Oxide.) Bryan is just one out of several illumos experts here. If none of those were around, sure, maybe we wouldn't have picked illumos -- but then we'd be unrecognizably different. I came into Oxide with a Linux background and zero knowledge of illumos. Learning about DTrace especially has been great.
- jeffbee 3y ago
  Do you have the same gut reaction to ESXi?
  - stonogo 3y ago
    I sure do. We've finally got to a place where we don't need weird hardware tricks to containerize workloads -- this is why a lot of shops pursue docker-like ops for production. When I buy hardware, long-term maintenance is a factor, and when my whole operations fleet relies on ESX, or in this case a Solaris fork, I'm now beholden to one company for support at that layer. Buying a rack of Supermicro gear and running RHEL or SLES with containerized orchestration on top means I can, in a pinch, hire experts anywhere to work on my systems. I have no reason to believe Oxide would be anything but responsive and effective in supporting their systems, but introducing bespoke software this deep in the stack severely curtails my options if things get bad.
    - apendleton 3y ago
      I think the value proposition they're offering is a carefully integrated system where everything has been thoroughly engineered/tested to work with everything else, down to writing custom firmware to guarantee that it's all ship-shape, so that customers don't have to touch any of the innards, and will probably just treat them as a black box. It seems like it's chock-full of stuff that they custom-built and that nobody else would be familiar with, by design. If that's not what you want, this probably isn't the product for you.
    - jeffbee 3y ago
      I can somewhat see your point, but in my experience you can't rely on RHEL or whatever vendor Linux to correctly bring up random OEM hardware. You will slowly discover all of the quirks, like it didn't initialize the platform EDAC the way you expected, or it didn't resolve some weird IRQ issue, etc. Nothing about my experience leads me to believe Linux will JFW on a given box, so I don't feel like Linux has an advantage in this regard, or that niche operating systems have a disadvantage. Certainly I feel like a first-party OS from the hardware vendor is going to have a lot of advantages.
- steveklabnik 3y ago
  > does that actually matter to the customers who are buying these? It's not like we specifically say "oh btw there's illumos inside and that's why you should buy the rack." It's not a customer-facing detail of the product. I'm sure most will never even know that this is the case. What customers do care about is that the rack is efficient, reliable, suits their needs, etc. Choosing illumos instead of Linux here is a choice made to help effectively deliver on that value. This does not mean that you couldn't build a similar product on top of Linux inherently, by the way, just that we decided illumos was more fit for purpose. This decision was made with the team, in the form of an RFD[1]. It's #26, though it is not currently public. The two choices that were seriously considered were KVM on Linux, and bhyve on illumos. It is pretty long. In the end, a path must be chosen, and we chose our path. I do not work on this part of the product, but I haven't seen any reason to believe it has been a hindrance, and probably is actually the right call. > the fact that this is fundamentally not-Linux (yes I know it runs Linux binaries unmodified) would be a reason against buying it, not for. I am curious why, if you feel like elaborating. EDIT: oh just saw your comment down here: https://news.ycombinator.com/item?id=39180814 https://news.ycombinator.com/item?id=39180814 1: https://rfd.shared.oxide.computer/ https://rfd.shared.oxide.computer/
  - wmf 3y ago
    The Linux vs. Illumos decision seems to be downstream of a more fundamental decision to make VMs the narrow waist of the Oxide system. That's what I'm curious about.
    - amluto 3y ago
      Especially since Oxide has a big fancy firmware stack. I would expect this stack to be able to do an excellent job of securely allocating bare-metal (i.e. VMX root on x86 or EL2 if Oxide ever goes ARM) resources. This would allow workloads on Oxide to run their own VMs, to safely use PCIe devices without dealing with interrupt redirection, etc.
      - wmf 3y ago
        I'm not affiliated with Oxide but I don't think you can put Crucible and VPC/OPTE in firmware. Without a DPU those components have to run in the hypervisor.
- thinkingkong 3y ago
  This has been / will be the market education challenge; Its the same one Joyent had with SmartOS. Theyre correctly pointing out that the end user or operator will basically never interact with this layer, but it does cause some knee-jerk reactions. All that said, there are some pretty great technical benefits to using illumos derived systems the least of which is the teams familiarity and ability to do real diagnosis on production issues. I wont put words in anyones mouth but I suspect thats going to be critical for them as they support customer deployments w/o direct physical access.
- spamizbad 3y ago
  Seems strange to me too but it sounds like the end-users basically never interact with this - it's just firmware humming along in the background. As long as its open-source and reasonably well documented its already lightyears ahead of what else is out there.
- greggyb 3y ago
  If you're running in one of the big 3 cloud providers, the bottom-level hypervisors are not-linux. This is equivalent. Are you anti-AWS or anti-Azure for the same reason? This is the substrate upon which you will run any virtualized infrastructure.
  - qmarchi 3y ago
    Small note, that's not true for Google Cloud, which runs on top of Linux, though modified. Disclaimer: Former Googler, Cloud Support
    - refulgentis 3y ago
      Another Xoogler here: any idea what they mean by it's not Linux at the bottom for other providers? Like, surely it's _some_ common OS? Either my binaries wouldn't run or AWS is reimplementing Linux so they can, which seems odd. Or are they just saying that the VM my binary runs on might be some predictable Linux version, but the underlying thing launching the VM could be anything?
      - qmarchi 3y ago
        Correct, that the Hypervisor isn't running Linux. I think the only provider where that would make sense would be Microsoft, where they have their own OS.
      - p_l 3y ago
        Old AWS used to be Xen, Nitro afaik uses customised VMM and I don't recall if it's not a custom OS or hosted on top of something. Azure is Hyper-V underneath IIRC, a custom variant at least (remember Windows Server Nano? IIRC it was the closest you could get to running it), with sometimes weird things like network cards running Linux and integrating with Windows' built-in SDN facility. Rest of the bigger ones is mainly Linux with occasional Xen and such, but sometimes you can encounter non-trivial VMware deployments.
        zokier 3y ago
- StillBored 3y ago
  Linux is a nightmare in the embedded/appliance space because one ends up just having platform engineers who spend their day fixing problems with the latest kernels, drivers, core libraries, etc, that the actual application depends on. Or one goes the route of 99% of the IoT/etc vendors, and never update the base OS and pray that there aren't any active exploits targeting it. This is why a lot of medium-sized companies cried about Centos, which allowed them to largely stick to a fairly stable platform that was getting security updates without having to actually pay/run a full blown RHEL/etc install. Every ten years or so they had to revisit all the dependencies, but that is a far easier problem than dealing with a year or two update cycle, which is too short when the qualification timeframe for some of these systems is 6+ months long. So, this is almost exclusively a Linux problem; any of the *BSD/etc. alternatives give you almost all of what Linux provides without this constant breakage.
  - bcantrill 3y ago
    This is a really, really good point -- and is a result of the model of Linux being only a kernel (and not system libraries, commands, etc.). It means that any real use of Linux is not merely signing up for kernel maintenance (which itself can be arduous) but also must make decisions around every other aspect of the system (each with its own communities, release management, etc.). This act is the act of creating a distribution -- and it's a huge burden to take on. Both illumos and the BSD derivatives make this significantly easier by simply including much more of the system within their scope: they are not merely kernels, but also system libraries and commands. This weighed heavily in our own calculus, so I'm glad you brought it up!
    - trhway 3y ago
      >including much more of the system within their scope: they are not merely kernels, but also system libraries and commands. giving limited resources of the dev team it may lead to limited support of the system outside of the narrow set of officially supported/certified hardware with that support falling behind on modern hardware, as it happened with Sun, and vendor lock-in as a result into overpriced and low performing hardware. There is a reason that back then at Solaris dev there was a joke about embedding Linux kernel as a universal driver for Solaris kernel in order to get reasonable support for the hardware around.
- NexRebular 3y ago
  Not everything needs to be linux. Besides, if monocultures are supposed to be harmful, why is linux being thrown to everything nowadays? Very dangerous to have a single point of failure in (critical) applications.
- mardifoufs 3y ago
  I think it's a good idea to have more choice, especially in OSS. A Linux mono culture isn't any better than a chromium mono culture. They might be able to do stuff that just isn't practical if they stuck with Linux. They are also probably more familiar with illumos, or at least familiar enough to know that they can use it to do more than with linux
- moondev 3y ago
  The main drawbacks to me are 1. No support for nested virtualization, so running a vm inside your vm is not available. This prevents use of projects such as kubevirt or firecracker on a Linux guest, and WSL2 on a Windows guest. 2. No GPU support If the base hypervisor was Linux, it would be way more capable for users it seems. I also wonder if internally Linux is used for development of the platform itself so they can create "virtual" racks to dogfood the product without full blown physical racks. With all that said, I do not know the roadmap and admittedly there are already quite a few existing platforms built on kvm, so as their hypervisor improves and becomes more capable it could potentially become strategic advantage.
  - steveklabnik 3y ago
    > I also wonder if internally Linux is used for development of the platform itself Developers at Oxide work on whatever platform they'd like, as long as they can do their work. I will say I am in the minority as a Windows user though, most are on some form of Unix. > so they can create "virtual" racks to dogfood the product without full blown physical racks. So one of the reasons why Rust is such an advantage for us is its strong cross-platform support: you can run a simulated version of the control plane on Mac, Linux, and Illumos, without a physical rack. The non-simulated version must run on Helios. [1] That said we do have a rack in the office (literally named dogfood) that employees can use for various things if they wish. 1: https://github.com/oxidecomputer/omicron?tab=readme-ov-file#build-and-run https://github.com/oxidecomputer/omicron?tab=readme-ov-file#...
    - moondev 3y ago
      Interesting thanks for the insight. > I will say I am in the minority as a Windows user though, most are on some form of Unix. Now i'm imagining Helios inside WSI - Windows Subsystem for illumos
      - steveklabnik 3y ago
        You're welcome. I will give you one more fun anecdote here: when I came to Oxide, nobody in my corner of the company was using Windows. And hubris and humility almost Just Worked: we had one build system issue that was using strings instead of the path APIs, but as soon as I fixed those, it all worked. bcantrill remarked that if you had gone back in time and told him long ago that some of his code would Just Work on Windows, he would have called you a liar, and it's one of the things that validates our decisions to go with Rust over C as the default language for development inside Oxide. > Now i'm imagining Helios inside WSI - Windows Subsystem for illumos That would be pretty funny, ha! IIRC something about simulated omicron doesn't work inside WSL, but since I don't work on it actively, I haven't bothered to try and patch that up. I think I tried one time, I don't remember specifically what the issue was, as I don't generally use WSL for development, so it's a bit foreign to me as well.
- sarlalian 3y ago
  As a customer, I expect most of the technical advantages will be basically being a down stream consumer of ZFS. From a developer / maintainer of an OS, Dtrace and ZFS are large technical wins. Part of the overall value proposition of Oxide, is "correctness". You get an OS/Hardware stack that are designed to work together. You get 20 years of cruft thrown out. You get a lot of tooling, API's, etc written in a performant memory safe language (rust). Also you get a really fantastic podcast about the whole process. And as a customer you get a company that understands their stack from driver to VM and has a ton of internal expertise debugging production problems.
milon 3y ago
I'm glad this is out, i'm going to deploy this locally and learn as much about it as possible. Oxide is pretty much the company I dream to work at, both for the tech stack, plus the people working there. Thank you Oxide team!
- refulgentis 3y ago
  Can you get me excited? I spent 20 seconds browsing the homepage and walked away with "so the idea is vertical integration for on-premise server purchases? On custom OS? Why? Why would people pay a premium?" But immediately got myself to "what does a server OS do anyway, doesn't it just launch VMs? You don't need Linux, just the ability to launch Linux VMs" Tell me more? :)
  - deleted 3y ago
    [deleted]
  - throwup238 3y ago
    The best elevator pitch I've heard is "AWS APIs for on-prem datacenters". They make turn-key managed racks that behave just like a commercial cloud would with all the APIs for VM, storage, and network provisioning and integration you'd expect from AWS, except made to deploy in your company's datacenter under your control.
    - kortilla 3y ago
      That’s the elevator pitch for open stack
      - steveklabnik 3y ago
        You are not wrong that OpenStack is sort of similar in a sense, but the difference is that Oxide is a hardware + software product, and OpenStack is purely software.
    - capitol_ 3y ago
      That just sounds like a bunch of api's on top of linux.
busterarm 3y ago
Not that I'm not rooting for Oxide, but their product is still so niche and early stage that I can't imagine any actual businesses buying their stuff for a long time. They only just shipped their first rack to their first customer at the end of last summer and it's Idaho National Laboratory. State research institutions are basically the only entities positioned to gamble on this right now.
- lijok 3y ago
  This describes every single product in its early days in existence. If you're planning to launch any other way, you've doomed the company before you even launched. Lucky few survive, in spite of, and that's what contributes to the 9/10 startups statistic. Lazer focus on the first set of customers that will help you cross the chasm. Only then mass market.
- __float 3y ago
  We have historically had private institutions with impactful research labs. Are there any of those still kicking?
- steveklabnik 3y ago
  Just a small note, but from when we announced this back in October, two customers were mentioned: https://oxide.computer/blog/oxide-unveils-the-worlds-first-commercial-cloud-computer https://oxide.computer/blog/oxide-unveils-the-worlds-first-c... > Oxide customers include the Idaho National Laboratory as well as a global financial services organization. Additional installments at Fortune 1000 enterprises will be completed in the coming months.
- elzbardico 3y ago
  Large financial institutions surprisingly are good customers for new, still-untested computing technology. I would not get surprised if Oxide next customers were a few giant banks and funds.
  - __d 3y ago
    In my experience, some financial institutions have a very good understanding of risk. They are able to identify, and most importantly, quantify risk in a way that many businesses cannot. Consequently, they're able to take risks with new hardware/software that other companies shy away from.
temptemptemp111 3y ago
What even is Oxide Computer? It makes no sense - it was publicized with all sorts of anti-blob, freedom, and posts about management engines and a sort of alternative to RaptorCS/IBM (which now has blobs again)... Yet most of that stuff is now buried/removed and Oxide Computer is just a hardware platform with unnecessary lock-in. For the bunker of the rich to be able to run their own mini-cloud? Sure. For anything else it seems like a bad design.
- throwawaaarrgh 3y ago
  It's a mainframe. You use it like you use mainframes, but probably easier, as they're adopting more modern functionality. You won't be aware you're using it, just like you aren't aware when you use a zSystem.
  - Anon_Is_Truth 3y ago
    [dead]
- steveklabnik 3y ago
  > Yet most of that stuff is now buried/removed Nothing has changed with regards to our anti-blob and pro-open source stances. I am not sure what you're referring to here. > with unnecessary lock-in. What lock-in are you referring to here? The way that things run on the rack is via virtual machines, you can run virtual machines on many providers. We even have a terraform provider so that you can use familiar tools instead of the API directly, if you believe that is lock-in (and that stuff is all also fully open source).
  - temptemptemp111 3y ago
    I don't expect anyone to see my comments unless they're really looking since I've been shadow banned for many years now - so I appreciate your reply. To be clearer regarding my questions: - What happened to Project X (supposedly coreboot++ for latest AMD CPUs)? It seems dead, despite being more reported on than Oxide's attempts in working with AMD (to achieve the same outcomes, presumably - what's the difference?). Loads of well meaning people have approached this with virtue, innocence and skills; perhaps another approach is needed that fully respects the dynamic between the user, the chip manufacturers and the governments and banks they're in debt to. - Does Oxide attempt to sandbox, completely remove or 'verify as benign' aspects like the PSP? For example, if someone could verify that the PSP cannot possibly be affected over the network, then peace of mind could be more affordable regarding things like supply chain attacks and bad actors with AMD/Intel/Apple management engine secrets. Not referring to software lock-in, just hardware. And it isn't very nefarious like other hardware lock-in (serialization, see Rossmann Group). Just hardware on the rack-level: replacing oxide gear & upgrading oxide gear (not sure about repair, that could be easy). And if the offering were of a less blobby architecture, then many of us would be happy to pay a bit more for the hardware as a system. However, if the hardware platform is FOSS, then it won't be unnecessarily difficult to mix and match and integrate the Oxide gear with other DC-class gear.
lifeisstillgood 3y ago
I would be interested in how did you first hear of Oxide. I somehow landed on their podcast because it covered <whatever the hell I thought was interesting at that moment>. The podcast is for me amazeballs marketing - it does everything but sell their product (might be a good idea to add a pitch in for each out-tro!) I mean they talk about it, like “we had such a tough time getting the compiler to do something something and then veer off to discuss back in the day stories. Ah never mind. Keep talking guys hope it works out
- Hackbraten 3y ago
  Was following @jessfraz on Twitter back then, so I got word of Oxide when they first announced it there.
  - __d 3y ago
    Between Jess, Bryan, and Adam, it was hard to miss :-)
- zengid 3y ago
  If you listen to their original podcast 'On The Metal' it was infamous for it's overly repeated use of 2 or 3 pre-recorded self promotions, so much so that a fan recorded their own commercial for them to air. 'Oxide and Friends' however isn't really what I would consider a podcast, but a recording of live "spaces" or group calls, beginning on Twitter and now happening in Discord. IMO it's not really best consumed as a podcast, but rather to participate in live. If you tune in live you'll pick up on the vibe of the recordings a lot better. https://oxide.computer/podcasts/oxide-and-friends https://oxide.computer/podcasts/oxide-and-friends
- ahmedfromtunis 3y ago
  For me, it was when Pentagram showcased their branding when Oxide was first announced.
nubinetwork 3y ago
I was hoping for this since they announced the server rack... nobody wants a paperweight if (God forbid) oxide were to go out of business.
- steveklabnik 3y ago
  To be clear about it, the "paperweight problem" is very important to us as well. It's worth remembering that the MPL doesn't care if a copy is posted openly on GitHub or not, and (I am not a lawyer!) we have obligations to our customers under it regardless if non-customers can browse the code.
alberth 3y ago
License MPL 2.0 is an interesting license choice, for an operating system. EDIT: why the downvotes?
- steveklabnik 3y ago
  Quoting from an RFD co-authored by bcantrill and myself describing Oxide's policies around open source: > For any new Oxide-created software, the MPL 2.0 should generally be the license of choice. The exception to this should be any software that is a part of a larger ecosystem that has a prevailing license, in which case that prevailing license may be used. EDIT: I also am confused about why you are downvoted. Are there any major operating systems distributions that are MPL licensed? I can't think of any off the top of my head. Beyond that it's a simple question.
  - alberth 3y ago
    > [if Oxide-created software] is a part of a larger ecosystem that has a prevailing license, in which case that prevailing license may be used How does that work if the prevailing is BSD/MIT/ISC? You're saying that Oxide can then be licensed under BSD/MIT/ISC?
    - steveklabnik 3y ago
      So I decided to cut off my quote but the next line has the answer: > For example, Rust crates are generally dual-licensed as MIT/Apache 2. We often produce components that we share with the broader open source world. For example, dropshot[1] is our in-house web framework, but we publish it as a standalone package. It is licensed under Apache-2.0 instead of MPL 2.0 because the norm in the Rust ecosystem is Apache and not MPL. > You're saying that Oxide can then be licensed under BSD/MIT/ISC? I am saying that we do not have one single license across the company. Some components are probably BSD/MIT/ISC licensed somewhere, and I guarantee that some third party dependencies we use are licensed under those licenses. That's different from "you could choose to take it under BSD," which I didn't mean to imply, sorry about that! 1: https://crates.io/crates/dropshot https://crates.io/crates/dropshot
  - 3y ago
codethief 3y ago
Can anyone ELI5 what Oxide's offer is? I've looked at their website and still got no clue. Is it hardware + software I can purchase and use on-premise? Is it a PaaS / yet another cloud provider?
- chrishare 3y ago
  On-prem, fully-integrated compute and storage solution with cloud-like APIs to provision resources, all with a commitment to open source.
  - mkoubaa 3y ago
    Mainframe 2.0
    - danpalmer 3y ago
      This is really not accurate in any way that matters I don't think. It's a mainframe in as much as you buy a rack and spec it out. It's not a mainframe in that the performance is typical server performance rather than the mainframe profile which is very different and requires different considerations, the compute model is typical server compute rather than the mainframe compute model which (aside from compatibility layers) is a radically different environment to build software for.
  - haolez 3y ago
    Do you know if they support GPUs or whatever is needed to host LLM models?
    - steveklabnik 3y ago
      The current product does not have any GPUs in it. https://news.ycombinator.com/item?id=39183072 https://news.ycombinator.com/item?id=39183072
- steveklabnik 3y ago
  I believe you're being downvoted because there is already a big thread about this here, though I think that's a bit unfair to you. I haven't posted in that thread yet because I wanted to let others say what is meaningful about the product to them, but this seems like a good place to put my reply. Regardless of all that: it is hardware + software you can purchase and use on-premise, that's correct. The differentiator from virtually all existing on-prem cloud products is that we are a single vendor who has designed the hardware and software (which is as open source as we can possibly make it, by the way, hence announcements like this) to work well together. Most products combine various other products from various vendors, and are effectively selling you integration. We believe that that leads to all kinds of problems that our product solves. Another factor here is that we only have two SKUs: a half rack and a full rack. You don't buy Oxide 1U at a time, you buy it a rack at a time. By designing the entire rack as a cohesive unit, we can do a lot of things that you simply cannot do in the 1U form factor. There is a running joke that we talk about our fans all the time, and it's true. Because our sleds have a larger form factor than a traditional 1U, we can use larger fans. This means we can run them at a lower RPM, which means power savings. That's the deliberate design choice. But we also have gained accidental benefits from doing things like this: lower RPM also means that our servers are way quieter than others. That's pretty neat. Some early prospective customers literally asked if the thing is on when it was demo'd to them, because it's so quiet. Is that a reason to buy a server? Not necessarily, but it's just a fun example of some of the things that end up happening when you re-think a product as a whole, rather than as an integration exercise.
criddell 3y ago
I’m unfamiliar with illumos so I went to their webpage and the very first thing it says is: > illumos is a Unix operating system Is illumos an actual Unix (like macOS) or a Unix-like OS (like GNU/Linux)?
- chucky_z 3y ago
  Actual Unix. I believe it is in the Solaris family.
- deleted 3y ago
  [deleted]
- steveklabnik 3y ago
  Actual Unix. Wikipedia is pretty good: https://en.wikipedia.org/wiki/Illumos https://en.wikipedia.org/wiki/Illumos > It is based on OpenSolaris, which was based on System V Release 4 (SVR4) and the Berkeley Software Distribution (BSD). Illumos comprises a kernel, device drivers, system libraries, and utility software for system administration. This core is now the base for many different open-sourced Illumos distributions, in a similar way in which the Linux kernel is used in different Linux distributions.
- Thoreandan 3y ago
  Nobody's paid to have it pass Open Group Unix Branding certification tests https://www.opengroup.org/openbrand/register/ https://www.opengroup.org/openbrand/register/ so it can't use the UNIX™ trade mark. But it's got the AT&T Unix kernel & userland sources contained in it. PDP-11 Unix System III: https://www.tuhs.org/cgi-bin/utree.pl?file=SysIII/usr/src/uts/pdp11/os/main.c https://www.tuhs.org/cgi-bin/utree.pl?file=SysIII/usr/src/ut... IllumOS: https://github.com/illumos/illumos-gate/blob/b8169dedfa435c059cef6fd836f19b9f1fb7d099/usr/src/uts/common/os/main.c#L122 https://github.com/illumos/illumos-gate/blob/b8169dedfa435c0...
- BirAdam 3y ago
  It was an open source branch of Solaris that Ian Murdock worked on while he was at Sun under the name Project Indiana. It descends from UNIX SVR4.
- 3y ago
internetter 3y ago
I really want one of these racks in my bedroom. Unfortunately, somehow I think I couldn't afford one ;)
ahmedfromtunis 3y ago
It is great that the software is open-source, but would be you useful to be deployed on other hardware? And what would happen if, for whatever reason, a company can no longer purchase Oxide racks, will it need to start over its infra, or can it just build around Oxide hardware?
- steveklabnik 3y ago
  It is not likely that it would be immediately useful outside of our hardware, but the main thing they're doing is deploying virtual machines. If they decided to no longer use the Oxide rack they have purchased, they would move their VMs to whatever infrastructure they choose to succeed it.
  - jclulow 3y ago
    Yeah, we definitely only intend Helios to run on either the Oxide rack or in service of software engineering work surrounding the rack (that's what we use the ISO installers and virtual machine images for). If you're interested in a distribution targeting end user use of illumos on servers, I would absolutely recommend looking at OmniOS! Helios is very closely based on OmniOS r151046 LTS, and we use that LTS of OmniOS release directly for non-Oxide-rack infrastructure systems inside Oxide as well.
mihaic 3y ago
I'm really curious: what kind of workload would companies want to run on a custom Unix that isn't Linux/Mac/BSD? I'm rooting for more mature OS diversity, I just have no idea who the end users would be and what their needs would look like.
- epistasis 3y ago
  ZFS is native on illumos, and the containerization equivalent, etc, is pretty great. There's a good argument that your servers in the cloud don't need to be on the same OS, as long as you can hire enough talent to work on them.
- steveklabnik 3y ago
  This is not a user-facing detail of the product. Customers run VMs on the rack, they do not build their applications for illumos. They're gonna run whatever operating system in those VMs that they need to accomplish their goals.
  - tw04 3y ago
    That being said - there's something to be said for enterprise support. Are there plans to support importing/converting/running third party OVAs? Many vendors will support something running in KVM, I can't recall the last time I saw b-hyve as a supported hypervisor. I'd imagine as broadcom slowly destroys vmware's market share vendors will look to alternatives, but I doubt b-hyve is even a blip on their radar at this point.
    - steveklabnik 3y ago
      I don't know the status of supporting OVA as a file format, but we absolutely support creating and uploading your own images. Here are the current docs on how to do so: https://docs.oxide.computer/guides/creating-and-sharing-images#_overview https://docs.oxide.computer/guides/creating-and-sharing-imag...
    - antranigv 3y ago
      OVAs are basically ZIP files with some XML. If you want, you can convert an OVA to RAW image or VMDK or whatever the latest fancy format is, and bhyve can boot that for you. Better to use Raw. bhyve, unlike other "famous" hypervisors is pretty stable, has good enough virtualized drivers (altho I'm sure Oxide has made it better) and can boot a VM with 1.5 TB of RAM and 240 vCPU[1]. Something I was not able to do with anything other than bhyve. I know this is HackerNews, so I have to say it, marketing != engineering. Just because the FreeBSD project's marketing suck, doesn't mean engineering is bad. usually it better than the mainstream ones. 1: https://antranigv.am/posts/2023/10/bhyve-cpu-allocation-256/ https://antranigv.am/posts/2023/10/bhyve-cpu-allocation-256/
mad_vill 3y ago
oh wow ... can't wait to waste a bunch of time trying to get this running in hyper-v.
- rhinoceraptor 3y ago
  I would recommend trying SmartOS or OmniOS instead, since the Oxide rack isn't filled with IBM compatible personal computers on sleds, and they have no BIOS or UEFI.
yjftsjthsd-h 3y ago
Sweet:) And a big thanks for writing what appears to be clear and straightforward documentation; IMO that's an area that the illumos community has historically struggled with. And seeing a new source release talking about consolidations gives me the warm fuzzies, even if this does seem to depart from the traditional gate paradigm unless I'm seriously misreading the repo organization here. Some (mostly tooling) questions: - Why gmake? Especially since dmake is needed later anyways? - Instructions say run rustup with bash explicitly; is that a defect in upstream, or is the local sh not completely posix compatible? - How is this developed internally? Do Oxide folks run illlmos workstations or is this all developed in Virtual machines or SSHed to servers? - Why MPL? GPL compatibility?
- steveklabnik 3y ago
  I can't answer all your questions, because I don't actually work on helios, but I do have an answer to some of them: > Do Oxide folks run illlmos workstations or is this all developed in Virtual machines or SSHed to servers? I wrote about this topic here: https://news.ycombinator.com/item?id=39181727 https://news.ycombinator.com/item?id=39181727 That said, some folks certainly run illumos on a workstation. > Why MPL? GPL compatibility? On MPL: https://news.ycombinator.com/item?id=39181844 https://news.ycombinator.com/item?id=39181844 That said in that comment I didn't really speak to the "why." We feel like it's a good compromise in the possibility space: more copyleft than BSD, but also less restrictive than the GPL.
- antranigv 3y ago
  > is that a defect in upstream, or is the local sh not completely posix compatible? AFAIK it's an issue with upstream. Just like most open-source projects, there is Linuxism/Bashism in there.
- jclulow 3y ago
  FWIW, though for historical reasons we use dmake to build the core operating system, I tend to recommend people use GNU make (gmake) for new Makefiles in other consolidations. It's broadly available (including on other platforms) and has more modern features.
deleted 3y ago
[deleted]
snvzz 3y ago
The name is, by unfortunate coincidence, also used by another operating system's microkernel[0]. 0. https://sr.ht/~sircmpwn/helios/ https://sr.ht/~sircmpwn/helios/
xer0x 3y ago
Oh the feels!
segmondy 3y ago
I really hope Oxide succeeds. I thought they were crazy when they announced what they were going to do. It's not the kind of play you see from a start up, but they were determined. Most folks that started the race at that time are out. I hope their computer is ready to deploy GPUs. Deploying multiple GPUs today is a freaking pain.

418 points by eduction 3y ago

I know they’re ex-Sun, but is there any real technical benefit for choosing not-Linux (for their business value prop)? I know of the technical benefits of illumos over linux, but does that actually matter to the customers who are buying these? Aren’t they opening a whole can of worms for ideology/tradition that won’t sell any more computers? As someone who runs Linux container workloads, the fact that this is fundamentally not-Linux (yes I know it runs Linux binaries unmodified) would be a reason against buying it, not for.

kardianos 3y ago
In one podcast, the reason given was staff familiarity and owning the full stack, not just the kernel I believe.
vvern 3y ago
> yes I know it runs Linux binaries unmodified Is it that it runs Linux binaries unmodified or that it runs vms and manages VMs which run Linux, and as an end-user, that's what you run your software in?
- tonyarkles 3y ago
  As far as I recall it's not a VM. They run in "LX Branded Zones" which does require a Linux userland so that the binaries can find their libraries etc but Zones are more like "better cgroups than cgroups, a decade earlier" than VMs.
  - bcantrill 3y ago
    No, it's a VM, running a bhyve-based hypervisor, Propolis.[0] LX branded zones were/are great -- but for absolute fidelity one really needs VMs. [0] https://github.com/oxidecomputer/propolis https://github.com/oxidecomputer/propolis
    - bpye 3y ago
      Do you have a solution for running containers (Kubernetes, etc)? Are you spinning up a Linux VM to run the containers in there, doing VM per container, or something else?
      - panick21_ 3y ago
        Costumers can decide I would assume. Most likely you install you install some Kubernetes and then just have multible VMs distributed across the rack. And then run multible Pods in each node. VM per container seems like a waist unless you need that extra isolation.
        bpye 3y ago
        I wondered if there was any support for running containers built in - something like EKS/AKS/GKE/Cloud Run/etc - but looking at the docs it appears not. I agree that VM per container can be wasteful - though something like Firecracker at least helps with start time.
        panick21_ 3y ago
        From the podcast it seems that they want to deliver a minimal viable product. Their primary costumers already have a lot of their own higher level stack. They might get into adding more higher level software eventually depending on what costumers want.
- bcantrill 3y ago
  It runs VMs -- so it doesn't just run Linux binaries unmodified, it runs Linux kernels unmodified (and, for that matter, Windows, FreeBSD, OpenBSD, etc.).
pjmlp 3y ago
Their customers run virtualised OS on top of this. This is no different from Azure Host OS, Bottlerocket, Flatcar or whatever. This maters to them, as knowing the whole stack, some of the kernel code is still theirs from Sun days, and making it available matters to the customers that want source code access for security assement reasons.
skullone 3y ago
It seems healthy to have options, almost like the universe is healing a bit after oracle bought Sun. I can't imagine better hands bringing the oxide system together than that team. As an engineer who works entirely with Linux anymore, I pine for the days of another strong Unix in the mix to run high value workloads on. Comparing openvswitch on Linux, to say, the crossbow SDN facility on Solaris, I'd take crossbow any day. Nothing "wrong" with Linux, but it is sorely lacking in "master plan" levels of cohesion with all the tooling taking their own path, often bringing complexity that requires even for abstraction with yet more complicated tooling on top.
Extigy 3y ago
Perhaps Illumos is particularly well suited for a Hypervisor/Cloud platform due to work upstreamed by Joyent originally for SmartOS?
shusaaafuejdn 3y ago
As far as performance and feature set, probably not anymore (I would have answered differently 10 years ago, and if I am wrong today would love to be educated about it). However, if we are considering code quality, which I consider important if you are actually going to be maintaining it yourself as oxide will have to do since they need customizations, then most of the proprietary Unix sources are just superior imo. That is, they have better organization, more consistency in standards, etc. The BSDs are slightly better in this regard as well, it really isn't a proprietary vs open source issue, it's more about the insane size of the Linux kernel project making strict standards enforcement difficult if not impossible the further you get from the very core system components. Irregardless of them being ex-Sun (and I am not ex Sun), if I needed a custom OS for a product I was working on, Linux would be close to the last Unix based OS source tree I would try to do it with, only after all other options failed for whatever reason. And that's not even taking into account the licensing, which is a whole other can of worms.
bcantrill 3y ago
Keep in mind that Helios is really just an implementation detail of the rack; like Hubris[0], it's not something visible to the user or to applications. (The user of the rack provisions VMs.) As for why an illumos derivative and not something else, we expanded on this a bit in our Q&A when we shipped our first rack[1] -- and we will expand on it again in the (recorded) discussion that we will have later today.[2] [0] https://hubris.oxide.computer/ https://hubris.oxide.computer/ [1] https://www.youtube.com/watch?v=5P5Mk_IggE0&t=2556s https://www.youtube.com/watch?v=5P5Mk_IggE0&t=2556s [2] https://mastodon.social/@bcantrill/111840269356297809 https://mastodon.social/@bcantrill/111840269356297809
- kaliszad 3y ago
  Perhaps you could talk a bit about the distributed storage based on Crucible with ZFS as the backing storage tonight. I would really love to hear some of the details and challenges there.
  - bcantrill 3y ago
    Yes! Crucible[0] is on our list of upcoming episodes. We can touch on it tonight, but it's really deserving of its own deep dive! [0] https://github.com/oxidecomputer/crucible https://github.com/oxidecomputer/crucible
    - panick21_ 3y ago
      The timing of your podcast is the least convenient thing ever for us poor Europeans. And then the brutal wait the next day until its uploaded. The only thing I miss about Twitter Spaces is that you could listen the morning after.
      - kaliszad 3y ago
        Yes (hello from Czechia), however there will always be somebody who this is inconvenient for. Also, I have to confess I was at times immersed in other work that I made a few Oxide and Friends live. I might stay up tonight. I am looking forward to the crucible episode. It sounds like it could be a startup on its own, it wouldn't be the first distributed file/ storage system company.
quux 3y ago
Aren't they also ex-Joyent? Joyent ran customer VMs in prod on Illumos for many years so there's a lot of experience there.
- steveklabnik 3y ago
  Many people, including part of the founding team, are ex-Joyent, yes. Some also worked at Sun, on the operating systems that illumos is ultimately derived from.
- littlestymaar 3y ago
  bcantrill used to work at Sun then became CTO at Joyent, so the reason why Joyent ran Illumos is probably the same reason as why Oxide is, because Cantrill likes it and judges that it's a good fit for what they are doing.
  - steveklabnik 3y ago
    As I elaborated above, bcantrill did not decree that we must use illumos. Technical decisions are not handed down from above at Oxide.
    - littlestymaar 3y ago
      I saw your comment[1] after I wrote mine, but I'm not saying that he's forcing you guys to use it (that would not a good way of being a CTO at a start-up…), but that doesn't prevent him from advocating for solutions he believes in. Would you say that Oxide would have chosen Illumos if he wasn't part of the company? [1]: https://news.ycombinator.com/item?id=39180706 https://news.ycombinator.com/item?id=39180706
      - sunshowers 3y ago
        (I work at Oxide.) Bryan is just one out of several illumos experts here. If none of those were around, sure, maybe we wouldn't have picked illumos -- but then we'd be unrecognizably different. I came into Oxide with a Linux background and zero knowledge of illumos. Learning about DTrace especially has been great.
jeffbee 3y ago
Do you have the same gut reaction to ESXi?
- stonogo 3y ago
  I sure do. We've finally got to a place where we don't need weird hardware tricks to containerize workloads -- this is why a lot of shops pursue docker-like ops for production. When I buy hardware, long-term maintenance is a factor, and when my whole operations fleet relies on ESX, or in this case a Solaris fork, I'm now beholden to one company for support at that layer. Buying a rack of Supermicro gear and running RHEL or SLES with containerized orchestration on top means I can, in a pinch, hire experts anywhere to work on my systems. I have no reason to believe Oxide would be anything but responsive and effective in supporting their systems, but introducing bespoke software this deep in the stack severely curtails my options if things get bad.
  - apendleton 3y ago
    I think the value proposition they're offering is a carefully integrated system where everything has been thoroughly engineered/tested to work with everything else, down to writing custom firmware to guarantee that it's all ship-shape, so that customers don't have to touch any of the innards, and will probably just treat them as a black box. It seems like it's chock-full of stuff that they custom-built and that nobody else would be familiar with, by design. If that's not what you want, this probably isn't the product for you.
  - jeffbee 3y ago
    I can somewhat see your point, but in my experience you can't rely on RHEL or whatever vendor Linux to correctly bring up random OEM hardware. You will slowly discover all of the quirks, like it didn't initialize the platform EDAC the way you expected, or it didn't resolve some weird IRQ issue, etc. Nothing about my experience leads me to believe Linux will JFW on a given box, so I don't feel like Linux has an advantage in this regard, or that niche operating systems have a disadvantage. Certainly I feel like a first-party OS from the hardware vendor is going to have a lot of advantages.
steveklabnik 3y ago
> does that actually matter to the customers who are buying these? It's not like we specifically say "oh btw there's illumos inside and that's why you should buy the rack." It's not a customer-facing detail of the product. I'm sure most will never even know that this is the case. What customers do care about is that the rack is efficient, reliable, suits their needs, etc. Choosing illumos instead of Linux here is a choice made to help effectively deliver on that value. This does not mean that you couldn't build a similar product on top of Linux inherently, by the way, just that we decided illumos was more fit for purpose. This decision was made with the team, in the form of an RFD[1]. It's #26, though it is not currently public. The two choices that were seriously considered were KVM on Linux, and bhyve on illumos. It is pretty long. In the end, a path must be chosen, and we chose our path. I do not work on this part of the product, but I haven't seen any reason to believe it has been a hindrance, and probably is actually the right call. > the fact that this is fundamentally not-Linux (yes I know it runs Linux binaries unmodified) would be a reason against buying it, not for. I am curious why, if you feel like elaborating. EDIT: oh just saw your comment down here: https://news.ycombinator.com/item?id=39180814 https://news.ycombinator.com/item?id=39180814 1: https://rfd.shared.oxide.computer/ https://rfd.shared.oxide.computer/
- wmf 3y ago
  The Linux vs. Illumos decision seems to be downstream of a more fundamental decision to make VMs the narrow waist of the Oxide system. That's what I'm curious about.
  - amluto 3y ago
    Especially since Oxide has a big fancy firmware stack. I would expect this stack to be able to do an excellent job of securely allocating bare-metal (i.e. VMX root on x86 or EL2 if Oxide ever goes ARM) resources. This would allow workloads on Oxide to run their own VMs, to safely use PCIe devices without dealing with interrupt redirection, etc.
    - wmf 3y ago
      I'm not affiliated with Oxide but I don't think you can put Crucible and VPC/OPTE in firmware. Without a DPU those components have to run in the hypervisor.
thinkingkong 3y ago
This has been / will be the market education challenge; Its the same one Joyent had with SmartOS. Theyre correctly pointing out that the end user or operator will basically never interact with this layer, but it does cause some knee-jerk reactions. All that said, there are some pretty great technical benefits to using illumos derived systems the least of which is the teams familiarity and ability to do real diagnosis on production issues. I wont put words in anyones mouth but I suspect thats going to be critical for them as they support customer deployments w/o direct physical access.
spamizbad 3y ago
Seems strange to me too but it sounds like the end-users basically never interact with this - it's just firmware humming along in the background. As long as its open-source and reasonably well documented its already lightyears ahead of what else is out there.
greggyb 3y ago
If you're running in one of the big 3 cloud providers, the bottom-level hypervisors are not-linux. This is equivalent. Are you anti-AWS or anti-Azure for the same reason? This is the substrate upon which you will run any virtualized infrastructure.
- qmarchi 3y ago
  Small note, that's not true for Google Cloud, which runs on top of Linux, though modified. Disclaimer: Former Googler, Cloud Support
  - refulgentis 3y ago
    Another Xoogler here: any idea what they mean by it's not Linux at the bottom for other providers? Like, surely it's _some_ common OS? Either my binaries wouldn't run or AWS is reimplementing Linux so they can, which seems odd. Or are they just saying that the VM my binary runs on might be some predictable Linux version, but the underlying thing launching the VM could be anything?
    - qmarchi 3y ago
      Correct, that the Hypervisor isn't running Linux. I think the only provider where that would make sense would be Microsoft, where they have their own OS.
    - p_l 3y ago
      Old AWS used to be Xen, Nitro afaik uses customised VMM and I don't recall if it's not a custom OS or hosted on top of something. Azure is Hyper-V underneath IIRC, a custom variant at least (remember Windows Server Nano? IIRC it was the closest you could get to running it), with sometimes weird things like network cards running Linux and integrating with Windows' built-in SDN facility. Rest of the bigger ones is mainly Linux with occasional Xen and such, but sometimes you can encounter non-trivial VMware deployments.
      - zokier 3y ago
StillBored 3y ago
Linux is a nightmare in the embedded/appliance space because one ends up just having platform engineers who spend their day fixing problems with the latest kernels, drivers, core libraries, etc, that the actual application depends on. Or one goes the route of 99% of the IoT/etc vendors, and never update the base OS and pray that there aren't any active exploits targeting it. This is why a lot of medium-sized companies cried about Centos, which allowed them to largely stick to a fairly stable platform that was getting security updates without having to actually pay/run a full blown RHEL/etc install. Every ten years or so they had to revisit all the dependencies, but that is a far easier problem than dealing with a year or two update cycle, which is too short when the qualification timeframe for some of these systems is 6+ months long. So, this is almost exclusively a Linux problem; any of the *BSD/etc. alternatives give you almost all of what Linux provides without this constant breakage.
- bcantrill 3y ago
  This is a really, really good point -- and is a result of the model of Linux being only a kernel (and not system libraries, commands, etc.). It means that any real use of Linux is not merely signing up for kernel maintenance (which itself can be arduous) but also must make decisions around every other aspect of the system (each with its own communities, release management, etc.). This act is the act of creating a distribution -- and it's a huge burden to take on. Both illumos and the BSD derivatives make this significantly easier by simply including much more of the system within their scope: they are not merely kernels, but also system libraries and commands. This weighed heavily in our own calculus, so I'm glad you brought it up!
  - trhway 3y ago
    >including much more of the system within their scope: they are not merely kernels, but also system libraries and commands. giving limited resources of the dev team it may lead to limited support of the system outside of the narrow set of officially supported/certified hardware with that support falling behind on modern hardware, as it happened with Sun, and vendor lock-in as a result into overpriced and low performing hardware. There is a reason that back then at Solaris dev there was a joke about embedding Linux kernel as a universal driver for Solaris kernel in order to get reasonable support for the hardware around.
NexRebular 3y ago
Not everything needs to be linux. Besides, if monocultures are supposed to be harmful, why is linux being thrown to everything nowadays? Very dangerous to have a single point of failure in (critical) applications.
mardifoufs 3y ago
I think it's a good idea to have more choice, especially in OSS. A Linux mono culture isn't any better than a chromium mono culture. They might be able to do stuff that just isn't practical if they stuck with Linux. They are also probably more familiar with illumos, or at least familiar enough to know that they can use it to do more than with linux
moondev 3y ago
The main drawbacks to me are 1. No support for nested virtualization, so running a vm inside your vm is not available. This prevents use of projects such as kubevirt or firecracker on a Linux guest, and WSL2 on a Windows guest. 2. No GPU support If the base hypervisor was Linux, it would be way more capable for users it seems. I also wonder if internally Linux is used for development of the platform itself so they can create "virtual" racks to dogfood the product without full blown physical racks. With all that said, I do not know the roadmap and admittedly there are already quite a few existing platforms built on kvm, so as their hypervisor improves and becomes more capable it could potentially become strategic advantage.
- steveklabnik 3y ago
  > I also wonder if internally Linux is used for development of the platform itself Developers at Oxide work on whatever platform they'd like, as long as they can do their work. I will say I am in the minority as a Windows user though, most are on some form of Unix. > so they can create "virtual" racks to dogfood the product without full blown physical racks. So one of the reasons why Rust is such an advantage for us is its strong cross-platform support: you can run a simulated version of the control plane on Mac, Linux, and Illumos, without a physical rack. The non-simulated version must run on Helios. [1] That said we do have a rack in the office (literally named dogfood) that employees can use for various things if they wish. 1: https://github.com/oxidecomputer/omicron?tab=readme-ov-file#build-and-run https://github.com/oxidecomputer/omicron?tab=readme-ov-file#...
  - moondev 3y ago
    Interesting thanks for the insight. > I will say I am in the minority as a Windows user though, most are on some form of Unix. Now i'm imagining Helios inside WSI - Windows Subsystem for illumos
    - steveklabnik 3y ago
      You're welcome. I will give you one more fun anecdote here: when I came to Oxide, nobody in my corner of the company was using Windows. And hubris and humility almost Just Worked: we had one build system issue that was using strings instead of the path APIs, but as soon as I fixed those, it all worked. bcantrill remarked that if you had gone back in time and told him long ago that some of his code would Just Work on Windows, he would have called you a liar, and it's one of the things that validates our decisions to go with Rust over C as the default language for development inside Oxide. > Now i'm imagining Helios inside WSI - Windows Subsystem for illumos That would be pretty funny, ha! IIRC something about simulated omicron doesn't work inside WSL, but since I don't work on it actively, I haven't bothered to try and patch that up. I think I tried one time, I don't remember specifically what the issue was, as I don't generally use WSL for development, so it's a bit foreign to me as well.
sarlalian 3y ago
As a customer, I expect most of the technical advantages will be basically being a down stream consumer of ZFS. From a developer / maintainer of an OS, Dtrace and ZFS are large technical wins. Part of the overall value proposition of Oxide, is "correctness". You get an OS/Hardware stack that are designed to work together. You get 20 years of cruft thrown out. You get a lot of tooling, API's, etc written in a performant memory safe language (rust). Also you get a really fantastic podcast about the whole process. And as a customer you get a company that understands their stack from driver to VM and has a ton of internal expertise debugging production problems.

milon 3y ago

I'm glad this is out, i'm going to deploy this locally and learn as much about it as possible. Oxide is pretty much the company I dream to work at, both for the tech stack, plus the people working there. Thank you Oxide team!

refulgentis 3y ago
Can you get me excited? I spent 20 seconds browsing the homepage and walked away with "so the idea is vertical integration for on-premise server purchases? On custom OS? Why? Why would people pay a premium?" But immediately got myself to "what does a server OS do anyway, doesn't it just launch VMs? You don't need Linux, just the ability to launch Linux VMs" Tell me more? :)
- deleted 3y ago
  [deleted]
- throwup238 3y ago
  The best elevator pitch I've heard is "AWS APIs for on-prem datacenters". They make turn-key managed racks that behave just like a commercial cloud would with all the APIs for VM, storage, and network provisioning and integration you'd expect from AWS, except made to deploy in your company's datacenter under your control.
  - kortilla 3y ago
    That’s the elevator pitch for open stack
    - steveklabnik 3y ago
      You are not wrong that OpenStack is sort of similar in a sense, but the difference is that Oxide is a hardware + software product, and OpenStack is purely software.
  - capitol_ 3y ago
    That just sounds like a bunch of api's on top of linux.

busterarm 3y ago

Not that I'm not rooting for Oxide, but their product is still so niche and early stage that I can't imagine any actual businesses buying their stuff for a long time. They only just shipped their first rack to their first customer at the end of last summer and it's Idaho National Laboratory. State research institutions are basically the only entities positioned to gamble on this right now.

lijok 3y ago
This describes every single product in its early days in existence. If you're planning to launch any other way, you've doomed the company before you even launched. Lucky few survive, in spite of, and that's what contributes to the 9/10 startups statistic. Lazer focus on the first set of customers that will help you cross the chasm. Only then mass market.
__float 3y ago
We have historically had private institutions with impactful research labs. Are there any of those still kicking?
steveklabnik 3y ago
Just a small note, but from when we announced this back in October, two customers were mentioned: https://oxide.computer/blog/oxide-unveils-the-worlds-first-commercial-cloud-computer https://oxide.computer/blog/oxide-unveils-the-worlds-first-c... > Oxide customers include the Idaho National Laboratory as well as a global financial services organization. Additional installments at Fortune 1000 enterprises will be completed in the coming months.
elzbardico 3y ago
Large financial institutions surprisingly are good customers for new, still-untested computing technology. I would not get surprised if Oxide next customers were a few giant banks and funds.
- __d 3y ago
  In my experience, some financial institutions have a very good understanding of risk. They are able to identify, and most importantly, quantify risk in a way that many businesses cannot. Consequently, they're able to take risks with new hardware/software that other companies shy away from.

temptemptemp111 3y ago

What even is Oxide Computer? It makes no sense - it was publicized with all sorts of anti-blob, freedom, and posts about management engines and a sort of alternative to RaptorCS/IBM (which now has blobs again)... Yet most of that stuff is now buried/removed and Oxide Computer is just a hardware platform with unnecessary lock-in. For the bunker of the rich to be able to run their own mini-cloud? Sure. For anything else it seems like a bad design.

throwawaaarrgh 3y ago
It's a mainframe. You use it like you use mainframes, but probably easier, as they're adopting more modern functionality. You won't be aware you're using it, just like you aren't aware when you use a zSystem.
- Anon_Is_Truth 3y ago
  [dead]
steveklabnik 3y ago
> Yet most of that stuff is now buried/removed Nothing has changed with regards to our anti-blob and pro-open source stances. I am not sure what you're referring to here. > with unnecessary lock-in. What lock-in are you referring to here? The way that things run on the rack is via virtual machines, you can run virtual machines on many providers. We even have a terraform provider so that you can use familiar tools instead of the API directly, if you believe that is lock-in (and that stuff is all also fully open source).
- temptemptemp111 3y ago
  I don't expect anyone to see my comments unless they're really looking since I've been shadow banned for many years now - so I appreciate your reply. To be clearer regarding my questions: - What happened to Project X (supposedly coreboot++ for latest AMD CPUs)? It seems dead, despite being more reported on than Oxide's attempts in working with AMD (to achieve the same outcomes, presumably - what's the difference?). Loads of well meaning people have approached this with virtue, innocence and skills; perhaps another approach is needed that fully respects the dynamic between the user, the chip manufacturers and the governments and banks they're in debt to. - Does Oxide attempt to sandbox, completely remove or 'verify as benign' aspects like the PSP? For example, if someone could verify that the PSP cannot possibly be affected over the network, then peace of mind could be more affordable regarding things like supply chain attacks and bad actors with AMD/Intel/Apple management engine secrets. Not referring to software lock-in, just hardware. And it isn't very nefarious like other hardware lock-in (serialization, see Rossmann Group). Just hardware on the rack-level: replacing oxide gear & upgrading oxide gear (not sure about repair, that could be easy). And if the offering were of a less blobby architecture, then many of us would be happy to pay a bit more for the hardware as a system. However, if the hardware platform is FOSS, then it won't be unnecessarily difficult to mix and match and integrate the Oxide gear with other DC-class gear.

lifeisstillgood 3y ago

I would be interested in how did you first hear of Oxide. I somehow landed on their podcast because it covered <whatever the hell I thought was interesting at that moment>. The podcast is for me amazeballs marketing - it does everything but sell their product (might be a good idea to add a pitch in for each out-tro!) I mean they talk about it, like “we had such a tough time getting the compiler to do something something and then veer off to discuss back in the day stories. Ah never mind. Keep talking guys hope it works out

Hackbraten 3y ago
Was following @jessfraz on Twitter back then, so I got word of Oxide when they first announced it there.
- __d 3y ago
  Between Jess, Bryan, and Adam, it was hard to miss :-)
zengid 3y ago
If you listen to their original podcast 'On The Metal' it was infamous for it's overly repeated use of 2 or 3 pre-recorded self promotions, so much so that a fan recorded their own commercial for them to air. 'Oxide and Friends' however isn't really what I would consider a podcast, but a recording of live "spaces" or group calls, beginning on Twitter and now happening in Discord. IMO it's not really best consumed as a podcast, but rather to participate in live. If you tune in live you'll pick up on the vibe of the recordings a lot better. https://oxide.computer/podcasts/oxide-and-friends https://oxide.computer/podcasts/oxide-and-friends
ahmedfromtunis 3y ago
For me, it was when Pentagram showcased their branding when Oxide was first announced.

nubinetwork 3y ago

I was hoping for this since they announced the server rack... nobody wants a paperweight if (God forbid) oxide were to go out of business.

steveklabnik 3y ago
To be clear about it, the "paperweight problem" is very important to us as well. It's worth remembering that the MPL doesn't care if a copy is posted openly on GitHub or not, and (I am not a lawyer!) we have obligations to our customers under it regardless if non-customers can browse the code.

alberth 3y ago

License MPL 2.0 is an interesting license choice, for an operating system. EDIT: why the downvotes?

steveklabnik 3y ago
Quoting from an RFD co-authored by bcantrill and myself describing Oxide's policies around open source: > For any new Oxide-created software, the MPL 2.0 should generally be the license of choice. The exception to this should be any software that is a part of a larger ecosystem that has a prevailing license, in which case that prevailing license may be used. EDIT: I also am confused about why you are downvoted. Are there any major operating systems distributions that are MPL licensed? I can't think of any off the top of my head. Beyond that it's a simple question.
- alberth 3y ago
  > [if Oxide-created software] is a part of a larger ecosystem that has a prevailing license, in which case that prevailing license may be used How does that work if the prevailing is BSD/MIT/ISC? You're saying that Oxide can then be licensed under BSD/MIT/ISC?
  - steveklabnik 3y ago
    So I decided to cut off my quote but the next line has the answer: > For example, Rust crates are generally dual-licensed as MIT/Apache 2. We often produce components that we share with the broader open source world. For example, dropshot[1] is our in-house web framework, but we publish it as a standalone package. It is licensed under Apache-2.0 instead of MPL 2.0 because the norm in the Rust ecosystem is Apache and not MPL. > You're saying that Oxide can then be licensed under BSD/MIT/ISC? I am saying that we do not have one single license across the company. Some components are probably BSD/MIT/ISC licensed somewhere, and I guarantee that some third party dependencies we use are licensed under those licenses. That's different from "you could choose to take it under BSD," which I didn't mean to imply, sorry about that! 1: https://crates.io/crates/dropshot https://crates.io/crates/dropshot
- 3y ago

codethief 3y ago

Can anyone ELI5 what Oxide's offer is? I've looked at their website and still got no clue. Is it hardware + software I can purchase and use on-premise? Is it a PaaS / yet another cloud provider?

chrishare 3y ago
On-prem, fully-integrated compute and storage solution with cloud-like APIs to provision resources, all with a commitment to open source.
- mkoubaa 3y ago
  Mainframe 2.0
  - danpalmer 3y ago
    This is really not accurate in any way that matters I don't think. It's a mainframe in as much as you buy a rack and spec it out. It's not a mainframe in that the performance is typical server performance rather than the mainframe profile which is very different and requires different considerations, the compute model is typical server compute rather than the mainframe compute model which (aside from compatibility layers) is a radically different environment to build software for.
- haolez 3y ago
  Do you know if they support GPUs or whatever is needed to host LLM models?
  - steveklabnik 3y ago
    The current product does not have any GPUs in it. https://news.ycombinator.com/item?id=39183072 https://news.ycombinator.com/item?id=39183072
steveklabnik 3y ago
I believe you're being downvoted because there is already a big thread about this here, though I think that's a bit unfair to you. I haven't posted in that thread yet because I wanted to let others say what is meaningful about the product to them, but this seems like a good place to put my reply. Regardless of all that: it is hardware + software you can purchase and use on-premise, that's correct. The differentiator from virtually all existing on-prem cloud products is that we are a single vendor who has designed the hardware and software (which is as open source as we can possibly make it, by the way, hence announcements like this) to work well together. Most products combine various other products from various vendors, and are effectively selling you integration. We believe that that leads to all kinds of problems that our product solves. Another factor here is that we only have two SKUs: a half rack and a full rack. You don't buy Oxide 1U at a time, you buy it a rack at a time. By designing the entire rack as a cohesive unit, we can do a lot of things that you simply cannot do in the 1U form factor. There is a running joke that we talk about our fans all the time, and it's true. Because our sleds have a larger form factor than a traditional 1U, we can use larger fans. This means we can run them at a lower RPM, which means power savings. That's the deliberate design choice. But we also have gained accidental benefits from doing things like this: lower RPM also means that our servers are way quieter than others. That's pretty neat. Some early prospective customers literally asked if the thing is on when it was demo'd to them, because it's so quiet. Is that a reason to buy a server? Not necessarily, but it's just a fun example of some of the things that end up happening when you re-think a product as a whole, rather than as an integration exercise.

criddell 3y ago

I’m unfamiliar with illumos so I went to their webpage and the very first thing it says is: > illumos is a Unix operating system Is illumos an actual Unix (like macOS) or a Unix-like OS (like GNU/Linux)?

chucky_z 3y ago
Actual Unix. I believe it is in the Solaris family.
deleted 3y ago
[deleted]
steveklabnik 3y ago
Actual Unix. Wikipedia is pretty good: https://en.wikipedia.org/wiki/Illumos https://en.wikipedia.org/wiki/Illumos > It is based on OpenSolaris, which was based on System V Release 4 (SVR4) and the Berkeley Software Distribution (BSD). Illumos comprises a kernel, device drivers, system libraries, and utility software for system administration. This core is now the base for many different open-sourced Illumos distributions, in a similar way in which the Linux kernel is used in different Linux distributions.
Thoreandan 3y ago
Nobody's paid to have it pass Open Group Unix Branding certification tests https://www.opengroup.org/openbrand/register/ https://www.opengroup.org/openbrand/register/ so it can't use the UNIX™ trade mark. But it's got the AT&T Unix kernel & userland sources contained in it. PDP-11 Unix System III: https://www.tuhs.org/cgi-bin/utree.pl?file=SysIII/usr/src/uts/pdp11/os/main.c https://www.tuhs.org/cgi-bin/utree.pl?file=SysIII/usr/src/ut... IllumOS: https://github.com/illumos/illumos-gate/blob/b8169dedfa435c059cef6fd836f19b9f1fb7d099/usr/src/uts/common/os/main.c#L122 https://github.com/illumos/illumos-gate/blob/b8169dedfa435c0...
BirAdam 3y ago
It was an open source branch of Solaris that Ian Murdock worked on while he was at Sun under the name Project Indiana. It descends from UNIX SVR4.
3y ago

internetter 3y ago

I really want one of these racks in my bedroom. Unfortunately, somehow I think I couldn't afford one ;)

ahmedfromtunis 3y ago

It is great that the software is open-source, but would be you useful to be deployed on other hardware? And what would happen if, for whatever reason, a company can no longer purchase Oxide racks, will it need to start over its infra, or can it just build around Oxide hardware?

steveklabnik 3y ago
It is not likely that it would be immediately useful outside of our hardware, but the main thing they're doing is deploying virtual machines. If they decided to no longer use the Oxide rack they have purchased, they would move their VMs to whatever infrastructure they choose to succeed it.
- jclulow 3y ago
  Yeah, we definitely only intend Helios to run on either the Oxide rack or in service of software engineering work surrounding the rack (that's what we use the ISO installers and virtual machine images for). If you're interested in a distribution targeting end user use of illumos on servers, I would absolutely recommend looking at OmniOS! Helios is very closely based on OmniOS r151046 LTS, and we use that LTS of OmniOS release directly for non-Oxide-rack infrastructure systems inside Oxide as well.

mihaic 3y ago

I'm really curious: what kind of workload would companies want to run on a custom Unix that isn't Linux/Mac/BSD? I'm rooting for more mature OS diversity, I just have no idea who the end users would be and what their needs would look like.

epistasis 3y ago
ZFS is native on illumos, and the containerization equivalent, etc, is pretty great. There's a good argument that your servers in the cloud don't need to be on the same OS, as long as you can hire enough talent to work on them.
steveklabnik 3y ago
This is not a user-facing detail of the product. Customers run VMs on the rack, they do not build their applications for illumos. They're gonna run whatever operating system in those VMs that they need to accomplish their goals.
- tw04 3y ago
  That being said - there's something to be said for enterprise support. Are there plans to support importing/converting/running third party OVAs? Many vendors will support something running in KVM, I can't recall the last time I saw b-hyve as a supported hypervisor. I'd imagine as broadcom slowly destroys vmware's market share vendors will look to alternatives, but I doubt b-hyve is even a blip on their radar at this point.
  - steveklabnik 3y ago
    I don't know the status of supporting OVA as a file format, but we absolutely support creating and uploading your own images. Here are the current docs on how to do so: https://docs.oxide.computer/guides/creating-and-sharing-images#_overview https://docs.oxide.computer/guides/creating-and-sharing-imag...
  - antranigv 3y ago
    OVAs are basically ZIP files with some XML. If you want, you can convert an OVA to RAW image or VMDK or whatever the latest fancy format is, and bhyve can boot that for you. Better to use Raw. bhyve, unlike other "famous" hypervisors is pretty stable, has good enough virtualized drivers (altho I'm sure Oxide has made it better) and can boot a VM with 1.5 TB of RAM and 240 vCPU[1]. Something I was not able to do with anything other than bhyve. I know this is HackerNews, so I have to say it, marketing != engineering. Just because the FreeBSD project's marketing suck, doesn't mean engineering is bad. usually it better than the mainstream ones. 1: https://antranigv.am/posts/2023/10/bhyve-cpu-allocation-256/ https://antranigv.am/posts/2023/10/bhyve-cpu-allocation-256/

mad_vill 3y ago

oh wow ... can't wait to waste a bunch of time trying to get this running in hyper-v.

rhinoceraptor 3y ago
I would recommend trying SmartOS or OmniOS instead, since the Oxide rack isn't filled with IBM compatible personal computers on sleds, and they have no BIOS or UEFI.

yjftsjthsd-h 3y ago

Sweet:) And a big thanks for writing what appears to be clear and straightforward documentation; IMO that's an area that the illumos community has historically struggled with. And seeing a new source release talking about consolidations gives me the warm fuzzies, even if this does seem to depart from the traditional gate paradigm unless I'm seriously misreading the repo organization here. Some (mostly tooling) questions: - Why gmake? Especially since dmake is needed later anyways? - Instructions say run rustup with bash explicitly; is that a defect in upstream, or is the local sh not completely posix compatible? - How is this developed internally? Do Oxide folks run illlmos workstations or is this all developed in Virtual machines or SSHed to servers? - Why MPL? GPL compatibility?

steveklabnik 3y ago
I can't answer all your questions, because I don't actually work on helios, but I do have an answer to some of them: > Do Oxide folks run illlmos workstations or is this all developed in Virtual machines or SSHed to servers? I wrote about this topic here: https://news.ycombinator.com/item?id=39181727 https://news.ycombinator.com/item?id=39181727 That said, some folks certainly run illumos on a workstation. > Why MPL? GPL compatibility? On MPL: https://news.ycombinator.com/item?id=39181844 https://news.ycombinator.com/item?id=39181844 That said in that comment I didn't really speak to the "why." We feel like it's a good compromise in the possibility space: more copyleft than BSD, but also less restrictive than the GPL.
antranigv 3y ago
> is that a defect in upstream, or is the local sh not completely posix compatible? AFAIK it's an issue with upstream. Just like most open-source projects, there is Linuxism/Bashism in there.
jclulow 3y ago
FWIW, though for historical reasons we use dmake to build the core operating system, I tend to recommend people use GNU make (gmake) for new Makefiles in other consolidations. It's broadly available (including on other platforms) and has more modern features.

deleted 3y ago

[deleted]

snvzz 3y ago

The name is, by unfortunate coincidence, also used by another operating system's microkernel[0]. 0. https://sr.ht/~sircmpwn/helios/ https://sr.ht/~sircmpwn/helios/

xer0x 3y ago

Oh the feels!

segmondy 3y ago

I really hope Oxide succeeds. I thought they were crazy when they announced what they were going to do. It's not the kind of play you see from a start up, but they were determined. Most folks that started the race at that time are out. I hope their computer is ready to deploy GPUs. Deploying multiple GPUs today is a freaking pain.