20 ms·

I get that Oxide has a lot of ex-Joyent folks, but I can't help but wonder how much the choice of a Solaris-derived OS will hobble adoption, support for devices

by cmdrk 5y ago

I get that Oxide has a lot of ex-Joyent folks, but I can't help but wonder how much the choice of a Solaris-derived OS will hobble adoption, support for devices, etc. In many ways this feels like SmartOS all over again - a killer concept that will be eclipsed by a technically inferior but more tractable (for contributors, for management) solution.

porker 5y ago
As someone who got massively excited by SmartOS, only to see adoption never reach even the minimal levels I hoped for - yes, I hear you. What would the hosting story look like now if 8(?) years ago 25% of servers had adopted SmartOS?
- EvanAnderson 5y ago
  I got massively excited by SmartOS, too. Coming from a vSphere and Hyper-V world I found it to be a joy to use. The way that Illumos zones leverage ZFS is really, really cool. The tooling in SmartOS was very nice to use, too. I never used it in production anywhere, admittedly. I also never got a chance to try out Triton. I'm on the fence about whether or not I keep my SmartOS hosts in my home network now that Illumos is a second-class citizen when it comes to OpenZFS.
  - NexRebular 5y ago
    We run Triton and individual SmartOS boxes in production. They have pretty much replaced all of our VMWare and linux-based hypervisors save some very specific use cases, mostly relating to GPU passthrough and NVIDIA. The case with OpenZFS does worry me as well. I fear the developers start slowly introducing linuxisms thus sacrificing portability and stability for the great penguin.
    - nwilkens 5y ago
      We also run Triton on our public cloud @mnx.io I would love to hear more about the replacement of VMware experience and any other Triton details good/bad.
      - NexRebular 5y ago
        Anything specific you'd like to know? Generally there hasn't been anything game breaking and our biggest issues have been with running out of logging space for the core services (went with too low capacity disks in the beginning, doh) and one failed upgrade which didn't even take down the whole cluster during fixing. Our clients can now also provision their own VMs with the Triton API unlike with VMware which required admins to do it. Bhyve like KVM before has also been rock solid in everything we run from simple web servers to kubernetes clusters. For issues we've found the Joyent/SmartOS IRC channels to be excellent and they have helped us tremendously in debugging and fixing things. It's the best support I've encountered for a FOSS product by far and one of the biggest reasons I'm such an illumos advocate now.
        nwilkens 5y ago
        This is great information, and we've had similar experiences. I'm also looking forward to further testing LinuxCN (https://github.com/joyent/linux-live/tree/linuxcn https://github.com/joyent/linux-live/tree/linuxcn) on Triton in the near future! Are you running Manta (https://github.com/joyent/manta https://github.com/joyent/manta) for anything? If so, is that meeting you needs for object storage?
        NexRebular 5y ago
        Not running Manta at the moment but it's on the menu. The demand for object storage hasn't been high enough to justify the spent time to set it up for now. Not sure about linuxCN either. I would rather run !linux on bare metal as much as possible, and having purged most of the penguins from infrastructure it would feel a bit weird to go immediately back ;) In addition, we also have SPARC hardware running on OpenBSD due great hardware support, including ldoms! Of course would be nice to run illumos on them too, but alas...
anaisbetts 5y ago
I thought this too, but if the goal of the control plane software and blade host OS is solely to create VMs and not to actually be a general-purpose OS, this probably doesn't matter as much?
- mise_en_place 5y ago
  Idk, KVM/libvirt/qemu has worked fine for me. It is very lightweight compared to say VMWare. If I don't want VMs I could use Docker/containerd. What problem does Oxide solve exactly?
  - rfoo 5y ago
    Sell you a few racks of servers, ready to use out of the box, I guess?
    - mise_en_place 5y ago
      Wouldn’t an enterprise go with what they’ve always used? Like Dell, EMC, HP, etc…and most startups will use a cloud provider instead of on prem
      - panick21_ 5y ago
        If the core idea of Oxide is true and putting together a bunch of PC comparable yourself with a whole bunch of complex and expensive software only to then have a huge maintenance and security nightmare then why wouldn't enterprise want to switch from Dell and co. The point of a startup is to disrupt the current market. And they are trying. Lets see how they do.
wmf 5y ago
If the Oxide stack is good someone could make a name for themselves by porting it to Linux to get wider hardware support.
- yjftsjthsd-h 5y ago
  Depends how deeply integrated it is; if nothing else, I suspect that bhyve and kvm have sufficiently different APIs that it would be at least quite annoying to paper over the differences.
- zozbot234 5y ago
  It might make even more sense to run a cutting-edge distributed OS on the actual Oxide hardware. With rack-scale platforms like this it could be feasible to do SSI with distributed memory across multiple "nodes". Current "cloud" platforms like Kubernetes are already planning on including support for automated checkpointing and migration, which is sort of the first step prior to going fully SSI.
  - bitbckt 5y ago
    I miss IRIX, too. :)
yjftsjthsd-h 5y ago
> but I can't help but wonder how much the choice of a Solaris-derived OS will hobble adoption, support for devices, etc. Does it matter? Oxide is building the whole stack - their own hardware with their own firmware to run their own OS with their own virtualization layer. They don't need support for arbitrary devices, because they control the hardware.
- qbasic_forever 5y ago
  I really worry about a startup taking a massive bet on their own custom hardware now in 2022. The world was much, much different in December 2019 when Oxide started than it is now. Let's hope the investment cash keeps flowing and the hardware gets to folks that purchased it.
  - zozbot234 5y ago
    Why? If anything, commodity/non-custom hardware is what's hurting right now. Fat margins on hardware imply a kind of inherent flexibility that can be used to weather even extreme shocks.
    - qbasic_forever 5y ago
      There are plenty of commodity chips that go into making a full server rack. If any little power regulator, etc. is backordered for months and years it's just more unexpected pain. And that's before we even get to the problems of entire factories shutting down, just look at what's happening to Apple & Foxconn of all companies in Shenzhen this week. If the big players are struggling the small fries are in for pain too.
      - bcantrill 5y ago
        The supply chain crisis is very, very real, but we are blessed with absolutely terrific operations folks coming from a wide range of industrial backgrounds (e.g., Apple, Lenovo, GE, P&G). They have pulled absolute supply chain miracles (knocking loudly on wood!) -- but we have also had the luxury of relatively small quantities (we're not buying millions of anything) and new design, where we can factor in lead times. tl;dr: Smaller players are able to do things that larger players can't -- which isn't to minimize how challenging it currently is!

by cmdrk 5y ago

porker 5y ago

As someone who got massively excited by SmartOS, only to see adoption never reach even the minimal levels I hoped for - yes, I hear you. What would the hosting story look like now if 8(?) years ago 25% of servers had adopted SmartOS?

EvanAnderson 5y ago
I got massively excited by SmartOS, too. Coming from a vSphere and Hyper-V world I found it to be a joy to use. The way that Illumos zones leverage ZFS is really, really cool. The tooling in SmartOS was very nice to use, too. I never used it in production anywhere, admittedly. I also never got a chance to try out Triton. I'm on the fence about whether or not I keep my SmartOS hosts in my home network now that Illumos is a second-class citizen when it comes to OpenZFS.
- NexRebular 5y ago
  We run Triton and individual SmartOS boxes in production. They have pretty much replaced all of our VMWare and linux-based hypervisors save some very specific use cases, mostly relating to GPU passthrough and NVIDIA. The case with OpenZFS does worry me as well. I fear the developers start slowly introducing linuxisms thus sacrificing portability and stability for the great penguin.
  - nwilkens 5y ago
    We also run Triton on our public cloud @mnx.io I would love to hear more about the replacement of VMware experience and any other Triton details good/bad.
    - NexRebular 5y ago
      Anything specific you'd like to know? Generally there hasn't been anything game breaking and our biggest issues have been with running out of logging space for the core services (went with too low capacity disks in the beginning, doh) and one failed upgrade which didn't even take down the whole cluster during fixing. Our clients can now also provision their own VMs with the Triton API unlike with VMware which required admins to do it. Bhyve like KVM before has also been rock solid in everything we run from simple web servers to kubernetes clusters. For issues we've found the Joyent/SmartOS IRC channels to be excellent and they have helped us tremendously in debugging and fixing things. It's the best support I've encountered for a FOSS product by far and one of the biggest reasons I'm such an illumos advocate now.
      - nwilkens 5y ago
        This is great information, and we've had similar experiences. I'm also looking forward to further testing LinuxCN (https://github.com/joyent/linux-live/tree/linuxcn https://github.com/joyent/linux-live/tree/linuxcn) on Triton in the near future! Are you running Manta (https://github.com/joyent/manta https://github.com/joyent/manta) for anything? If so, is that meeting you needs for object storage?
        NexRebular 5y ago
        Not running Manta at the moment but it's on the menu. The demand for object storage hasn't been high enough to justify the spent time to set it up for now. Not sure about linuxCN either. I would rather run !linux on bare metal as much as possible, and having purged most of the penguins from infrastructure it would feel a bit weird to go immediately back ;) In addition, we also have SPARC hardware running on OpenBSD due great hardware support, including ldoms! Of course would be nice to run illumos on them too, but alas...

anaisbetts 5y ago

I thought this too, but if the goal of the control plane software and blade host OS is solely to create VMs and not to actually be a general-purpose OS, this probably doesn't matter as much?

mise_en_place 5y ago
Idk, KVM/libvirt/qemu has worked fine for me. It is very lightweight compared to say VMWare. If I don't want VMs I could use Docker/containerd. What problem does Oxide solve exactly?
- rfoo 5y ago
  Sell you a few racks of servers, ready to use out of the box, I guess?
  - mise_en_place 5y ago
    Wouldn’t an enterprise go with what they’ve always used? Like Dell, EMC, HP, etc…and most startups will use a cloud provider instead of on prem
    - panick21_ 5y ago
      If the core idea of Oxide is true and putting together a bunch of PC comparable yourself with a whole bunch of complex and expensive software only to then have a huge maintenance and security nightmare then why wouldn't enterprise want to switch from Dell and co. The point of a startup is to disrupt the current market. And they are trying. Lets see how they do.

wmf 5y ago

If the Oxide stack is good someone could make a name for themselves by porting it to Linux to get wider hardware support.

yjftsjthsd-h 5y ago
Depends how deeply integrated it is; if nothing else, I suspect that bhyve and kvm have sufficiently different APIs that it would be at least quite annoying to paper over the differences.
zozbot234 5y ago
It might make even more sense to run a cutting-edge distributed OS on the actual Oxide hardware. With rack-scale platforms like this it could be feasible to do SSI with distributed memory across multiple "nodes". Current "cloud" platforms like Kubernetes are already planning on including support for automated checkpointing and migration, which is sort of the first step prior to going fully SSI.
- bitbckt 5y ago
  I miss IRIX, too. :)

yjftsjthsd-h 5y ago

> but I can't help but wonder how much the choice of a Solaris-derived OS will hobble adoption, support for devices, etc. Does it matter? Oxide is building the whole stack - their own hardware with their own firmware to run their own OS with their own virtualization layer. They don't need support for arbitrary devices, because they control the hardware.

qbasic_forever 5y ago
I really worry about a startup taking a massive bet on their own custom hardware now in 2022. The world was much, much different in December 2019 when Oxide started than it is now. Let's hope the investment cash keeps flowing and the hardware gets to folks that purchased it.
- zozbot234 5y ago
  Why? If anything, commodity/non-custom hardware is what's hurting right now. Fat margins on hardware imply a kind of inherent flexibility that can be used to weather even extreme shocks.
  - qbasic_forever 5y ago
    There are plenty of commodity chips that go into making a full server rack. If any little power regulator, etc. is backordered for months and years it's just more unexpected pain. And that's before we even get to the problems of entire factories shutting down, just look at what's happening to Apple & Foxconn of all companies in Shenzhen this week. If the big players are struggling the small fries are in for pain too.
    - bcantrill 5y ago
      The supply chain crisis is very, very real, but we are blessed with absolutely terrific operations folks coming from a wide range of industrial backgrounds (e.g., Apple, Lenovo, GE, P&G). They have pulled absolute supply chain miracles (knocking loudly on wood!) -- but we have also had the luxury of relatively small quantities (we're not buying millions of anything) and new design, where we can factor in lead times. tl;dr: Smaller players are able to do things that larger players can't -- which isn't to minimize how challenging it currently is!