12 ms·

I don't see it as a big deal - rather, I see it as a huge amount of venture cap spent on some very bright people to build something no one really wants, or, at

I don't see it as a big deal - rather, I see it as a huge amount of venture cap spent on some very bright people to build something no one really wants, or, at best, is niche. Also, it has little to do with the cloud; it is yet another hyperconverged infra. Weirdly, it is attached to something very few people want: Solaris. This relates to the people behind it who still can't figure out why Linux won and Solaris didn't.

sanderjd 3y ago
> something no one really wants, or, at best, is niche Could be! Seems too early to tell though, and remains to be seen whether it pencils. Which is the whole idea of starting a new venture, no?
Voultapher 3y ago
Right, who wants or benefits from open source firmware anyway. Also there are many situations where renting, for example a flat makes a lot of sense. And there are many situations where the financials and or enabled options of owning something make a lot of sense. Right now, the kind of experience you get with AWS and co. can only be rented, not bought. Some people want to buy houses instead of renting them.
- Always_Anon 3y ago
  >Right, who wants or benefits from open source firmware anyway. Their competition has open source firmware as well: https://www.dell.com/en-us/blog/enabling-open-embedded-systems-management-on-poweredge-servers/ https://www.dell.com/en-us/blog/enabling-open-embedded-syste...
  - bcantrill 3y ago
    So OpenBMC is fine (happy for them!), but having open firmware is much deeper and broader than that: yes, it's the service processor (in contrast to the BMC which is a closed part on Dell machines) -- but it's also the root-of-trust and (especially) the host CPU itself. We at Oxide have open source software from first instruction out of the AMD PSP; I elaborated more on our approach in my OSFC 2022 talk.[0] [0] https://www.osfc.io/2022/talks/i-have-come-to-bury-the-bios-not-to-open-it-the-need-for-holistic-systems/ https://www.osfc.io/2022/talks/i-have-come-to-bury-the-bios-...
    - Always_Anon 3y ago
      Dell now ships with OpenBMC iDRACs and such. How does what you mention differ from the RoT in Dells? https://www.dell.com/en-us/blog/hardware-root-trust/ https://www.dell.com/en-us/blog/hardware-root-trust/
      - c_o_n_v_e_x 3y ago
        Dell uses trusted platform modules (TPM). It's a separate chipset than the BMC chipset. For a mostly open source solution, not only would you need open source BMC firmware, you must have an open source UEFI/BIOS/boot firmware like CoreBoot, LinuxBoot, Oreboot, Uboot, etc.
- necovek 3y ago
  Well, you can buy your own hardware and set it up with OpenStack and use it as a private cloud. Companies like Canonical or Redhat make a lot of money by providing software (mostly open source) to support exactly that use case. And Canonical played with a cluster-in-a-box all the way back in 2013-2014: https://www.zdnet.com/article/canonicals-cloud-in-a-box-under-the-hood/ https://www.zdnet.com/article/canonicals-cloud-in-a-box-unde... You could turn it into an OpenStack cloud in ~20 mins with an automated Juju OpenStack install.
  - jjav 3y ago
    > Well, you can buy your own hardware and set it up with OpenStack and use it as a private cloud. Companies like Canonical or Redhat make a lot of money by providing software (mostly open source) to support exactly that use case. Sure you can, but then who will diagnose and fix your hardware/OS interaction problems when you have parts from five vendors in the mix? If you haven't lived through this, the answer is: nobody. Everyone points fingers at the other 4 and ignore your calls. Back in the day you could buy a fully integrated system (from CPU to hardware to OS) from Sun or SGI or HP and you had a single company to answer all the calls, so it was much better. Today you can't really get this level of integration and support anymore. (Actually, you probably can from IBM, which is why they're still around. But I have no experience in the IBM universe.) This is why Oxide is so exciting to me. I hope I can be in a company that becomes a customer at some point.
    - Always_Anon 3y ago
      >Sure you can, but then who will diagnose and fix your hardware/OS interaction problems when you have parts from five vendors in the mix? Dell is a single vendor that will diagnose and fix all of your hardware issues. With Oxide you're locked into what looks like a Solaris derivative OS running on the metal and you're only allowed to provision VMs which is a huge disadvantage. I run a fleet of over 30,000 nodes in three continents and the majority is Flatcar Linux running on bare metal. Also have a decent amount of RHEL running for specific apps. We can pick and choose our bare metal OS which is something you cannot do with Oxide. That's a tough pill to swallow.
NexRebular 3y ago
The fact that it's not on linux is one of the great things about it. There is too much linux on critical infrastructure already and the monoculture just keeps on growing. At least with Oxide there is a glimmer of hope for a better future in this regard.
eduction 3y ago
When you're deploying VMs, which is the use case here, the substrate OS becomes significantly less important. Those VMs will mostly just be linux. Yes they are using illumos/Solaris to host this but they don't sell on that, they sell on the functionality of this layer — allowing people to deploy to owned infra in a way that is similar to how they'd deploy to AWS or Azure. How much do you ever think about the system hosting your VM on those clouds? You think about your VMs, the API or web interface to deploy and configure, but not the host OS. With Oxide racks the customers are not maintaining the illumos substrate (as long as Oxide is around). You could be right about demand, there is risk in a venture like this. But presumably the team thought about this - I think folks who worked at Sun, Oracle, Joyent, and Samsung and made SmartOS probably developed a decent sense of market demand, enough to make a convincing case to their funders.
- yencabulator 3y ago
  > When you're deploying VMs, which is the use case here, the substrate OS becomes significantly less important. Those VMs will mostly just be linux. Now you need to know both the OS they chose and the OS you chose... (No, I don't believe it'll be 100% hands-off for the host. This is an early stage product, with a lot of custom parts, their own distributed block storage, hypervisor, and so on.)
  - sanderjd 3y ago
    I have no clue what OS runs my VMs on EC2.
  - tinco 3y ago
    This true for other hypervisors too. Enterprises are still paying hundreds of millions to VMware, who knows what's going on in there? I wouldn't have picked Opensolaris, but it's a lot better than other vendors that are either fully closed source, or thin proprietary wrappers over Linux with spotty coverage and you're not allowed to touch the underlying OS for risk of disrupting the managed product.
    - sgt 3y ago
cashsterling 3y ago
Back in the day... Sun Micro was a GOAT and pushed the envelope on Unix computing 20-30 years ago. Solaris was stable and high performing. I don't run on-prem clusters or clouds but know a couple people who do and, at large enough scale, it is a constant "fuck-shit-stack on top of itself" (to quote Reggie Watts). There is almost always something wrong and some people upset about it. The promise of a fully integrated system (compute HW, network HW, all firmware/drivers written by experts using Rust wherever possible) that pays attention to optimizing all your OpEx metrics is a big deal. It may take Oxide a couple more years to really break into the market in a big way, but if they can stick it out, they will do very well.
- icedchai 3y ago
  I used to love Sun and Solaris. Then the dot-com bubble burst, and Linux ate its lunch. I haven't seen a new Solaris system deployed in over 20 years.
steveklabnik 3y ago
Just to be clear, Illumos (it hasn't been Solaris in a very long time) is an implementation detail. It's not customer facing.
- yencabulator 3y ago
  It'll become customer facing the moment something doesn't work right.
  - ahl 3y ago
    It won't. In the same way that AWS customers aren't debugging hypervisor, or Dell customers aren't debugging the BIOS, or Samsung SSD customers aren't debugging the firmware. Products choose where to draw the line between customer-serviceable parts and those that require a support call. In this case, expect Oxide to fix it when something doesn't work right.
  - jjav 3y ago
    When Apple supports OSX for consumers, they don't exactly surface the fact that there's BSD semi-hidden in there somewhere. That's because they own the whole stack, from CPU to GUI and support it as a unit. That's the benefit of having a product where a single owner builds and supports it as a whole. My impression of Oxide is that that's the level of single source of truth they are bringing to enterprise in-house cloud. So, I strongly doubt the innards would ever become customer-facing (unless the customer specifically wants that, being open source after all).
    - yencabulator 3y ago
      Apple is a horrible example, with Apple when you have a problem, you often end up with an unfixable issue that Apple won't even acknowledge. You definitely don't want to taint Oxide's reputation with that association. As for why I think Helios will become customer facing: Oxide is a small startup. They have limited resources. Their computers expensive enough to be very much business critical. You'll get some support by Oxide logging in remotely to customer systems and digging around, but pretty soon the customer will want to do that themselves to monitor/troubleshoot the problems as they happen. Imagine you're observing a recurring but rare I/O slowdown that seems to trigger under some certain conditions, and tell me a competent sysadmin wouldn't want to log in on all the related boxes (client Helios, >=3 server Helioses for the block store) and look at the logs & stats.

sanderjd 3y ago

> something no one really wants, or, at best, is niche Could be! Seems too early to tell though, and remains to be seen whether it pencils. Which is the whole idea of starting a new venture, no?

Voultapher 3y ago

Right, who wants or benefits from open source firmware anyway. Also there are many situations where renting, for example a flat makes a lot of sense. And there are many situations where the financials and or enabled options of owning something make a lot of sense. Right now, the kind of experience you get with AWS and co. can only be rented, not bought. Some people want to buy houses instead of renting them.

Always_Anon 3y ago
>Right, who wants or benefits from open source firmware anyway. Their competition has open source firmware as well: https://www.dell.com/en-us/blog/enabling-open-embedded-systems-management-on-poweredge-servers/ https://www.dell.com/en-us/blog/enabling-open-embedded-syste...
- bcantrill 3y ago
  So OpenBMC is fine (happy for them!), but having open firmware is much deeper and broader than that: yes, it's the service processor (in contrast to the BMC which is a closed part on Dell machines) -- but it's also the root-of-trust and (especially) the host CPU itself. We at Oxide have open source software from first instruction out of the AMD PSP; I elaborated more on our approach in my OSFC 2022 talk.[0] [0] https://www.osfc.io/2022/talks/i-have-come-to-bury-the-bios-not-to-open-it-the-need-for-holistic-systems/ https://www.osfc.io/2022/talks/i-have-come-to-bury-the-bios-...
  - Always_Anon 3y ago
    Dell now ships with OpenBMC iDRACs and such. How does what you mention differ from the RoT in Dells? https://www.dell.com/en-us/blog/hardware-root-trust/ https://www.dell.com/en-us/blog/hardware-root-trust/
    - c_o_n_v_e_x 3y ago
      Dell uses trusted platform modules (TPM). It's a separate chipset than the BMC chipset. For a mostly open source solution, not only would you need open source BMC firmware, you must have an open source UEFI/BIOS/boot firmware like CoreBoot, LinuxBoot, Oreboot, Uboot, etc.
necovek 3y ago
Well, you can buy your own hardware and set it up with OpenStack and use it as a private cloud. Companies like Canonical or Redhat make a lot of money by providing software (mostly open source) to support exactly that use case. And Canonical played with a cluster-in-a-box all the way back in 2013-2014: https://www.zdnet.com/article/canonicals-cloud-in-a-box-under-the-hood/ https://www.zdnet.com/article/canonicals-cloud-in-a-box-unde... You could turn it into an OpenStack cloud in ~20 mins with an automated Juju OpenStack install.
- jjav 3y ago
  > Well, you can buy your own hardware and set it up with OpenStack and use it as a private cloud. Companies like Canonical or Redhat make a lot of money by providing software (mostly open source) to support exactly that use case. Sure you can, but then who will diagnose and fix your hardware/OS interaction problems when you have parts from five vendors in the mix? If you haven't lived through this, the answer is: nobody. Everyone points fingers at the other 4 and ignore your calls. Back in the day you could buy a fully integrated system (from CPU to hardware to OS) from Sun or SGI or HP and you had a single company to answer all the calls, so it was much better. Today you can't really get this level of integration and support anymore. (Actually, you probably can from IBM, which is why they're still around. But I have no experience in the IBM universe.) This is why Oxide is so exciting to me. I hope I can be in a company that becomes a customer at some point.
  - Always_Anon 3y ago
    >Sure you can, but then who will diagnose and fix your hardware/OS interaction problems when you have parts from five vendors in the mix? Dell is a single vendor that will diagnose and fix all of your hardware issues. With Oxide you're locked into what looks like a Solaris derivative OS running on the metal and you're only allowed to provision VMs which is a huge disadvantage. I run a fleet of over 30,000 nodes in three continents and the majority is Flatcar Linux running on bare metal. Also have a decent amount of RHEL running for specific apps. We can pick and choose our bare metal OS which is something you cannot do with Oxide. That's a tough pill to swallow.

NexRebular 3y ago

The fact that it's not on linux is one of the great things about it. There is too much linux on critical infrastructure already and the monoculture just keeps on growing. At least with Oxide there is a glimmer of hope for a better future in this regard.

eduction 3y ago

When you're deploying VMs, which is the use case here, the substrate OS becomes significantly less important. Those VMs will mostly just be linux. Yes they are using illumos/Solaris to host this but they don't sell on that, they sell on the functionality of this layer — allowing people to deploy to owned infra in a way that is similar to how they'd deploy to AWS or Azure. How much do you ever think about the system hosting your VM on those clouds? You think about your VMs, the API or web interface to deploy and configure, but not the host OS. With Oxide racks the customers are not maintaining the illumos substrate (as long as Oxide is around). You could be right about demand, there is risk in a venture like this. But presumably the team thought about this - I think folks who worked at Sun, Oracle, Joyent, and Samsung and made SmartOS probably developed a decent sense of market demand, enough to make a convincing case to their funders.

yencabulator 3y ago
> When you're deploying VMs, which is the use case here, the substrate OS becomes significantly less important. Those VMs will mostly just be linux. Now you need to know both the OS they chose and the OS you chose... (No, I don't believe it'll be 100% hands-off for the host. This is an early stage product, with a lot of custom parts, their own distributed block storage, hypervisor, and so on.)
- sanderjd 3y ago
  I have no clue what OS runs my VMs on EC2.
- tinco 3y ago
  This true for other hypervisors too. Enterprises are still paying hundreds of millions to VMware, who knows what's going on in there? I wouldn't have picked Opensolaris, but it's a lot better than other vendors that are either fully closed source, or thin proprietary wrappers over Linux with spotty coverage and you're not allowed to touch the underlying OS for risk of disrupting the managed product.
  - sgt 3y ago

cashsterling 3y ago

Back in the day... Sun Micro was a GOAT and pushed the envelope on Unix computing 20-30 years ago. Solaris was stable and high performing. I don't run on-prem clusters or clouds but know a couple people who do and, at large enough scale, it is a constant "fuck-shit-stack on top of itself" (to quote Reggie Watts). There is almost always something wrong and some people upset about it. The promise of a fully integrated system (compute HW, network HW, all firmware/drivers written by experts using Rust wherever possible) that pays attention to optimizing all your OpEx metrics is a big deal. It may take Oxide a couple more years to really break into the market in a big way, but if they can stick it out, they will do very well.

icedchai 3y ago
I used to love Sun and Solaris. Then the dot-com bubble burst, and Linux ate its lunch. I haven't seen a new Solaris system deployed in over 20 years.

steveklabnik 3y ago

Just to be clear, Illumos (it hasn't been Solaris in a very long time) is an implementation detail. It's not customer facing.

yencabulator 3y ago
It'll become customer facing the moment something doesn't work right.
- ahl 3y ago
  It won't. In the same way that AWS customers aren't debugging hypervisor, or Dell customers aren't debugging the BIOS, or Samsung SSD customers aren't debugging the firmware. Products choose where to draw the line between customer-serviceable parts and those that require a support call. In this case, expect Oxide to fix it when something doesn't work right.
- jjav 3y ago
  When Apple supports OSX for consumers, they don't exactly surface the fact that there's BSD semi-hidden in there somewhere. That's because they own the whole stack, from CPU to GUI and support it as a unit. That's the benefit of having a product where a single owner builds and supports it as a whole. My impression of Oxide is that that's the level of single source of truth they are bringing to enterprise in-house cloud. So, I strongly doubt the innards would ever become customer-facing (unless the customer specifically wants that, being open source after all).
  - yencabulator 3y ago
    Apple is a horrible example, with Apple when you have a problem, you often end up with an unfixable issue that Apple won't even acknowledge. You definitely don't want to taint Oxide's reputation with that association. As for why I think Helios will become customer facing: Oxide is a small startup. They have limited resources. Their computers expensive enough to be very much business critical. You'll get some support by Oxide logging in remotely to customer systems and digging around, but pretty soon the customer will want to do that themselves to monitor/troubleshoot the problems as they happen. Imagine you're observing a recurring but rare I/O slowdown that seems to trigger under some certain conditions, and tell me a competent sysadmin wouldn't want to log in on all the related boxes (client Helios, >=3 server Helioses for the block store) and look at the logs & stats.