5 ms·

Finally I actually understand what they're building. Now I must ask: Why? On-prem servers aren't a new invention. The market seems pretty saturated (and shrink

Finally I actually understand what they're building. Now I must ask: Why? On-prem servers aren't a new invention. The market seems pretty saturated (and shrinking). Virtualization isn't a new invention either. The market seems pretty well served, at least commercially. Can the integration of both be a convincing enough advantage? The management UI certainly looks nice; it's something I'd like to have on my KVM box at home (any good Proxmox alternatives?). I don't see why it'd have to be bound to an enormous server.

mhh__ 3y ago
YOLO - sometimes it's worth paying more money for something someone has actually sat down and designed properly.
- throwawaylinux 3y ago
  So what has been designed properly and what hasn't?
  - mhh__ 3y ago
    The random piles of servers in most offices above a certain headcount
faitswulff 3y ago
One benefit is that they're competing in an industry where the time to get a sled up and running for compute can be measured in weeks (they say they've heard up to 90 days) whereas their solution is basically plug and play - Bryan was trying to get Steve to admit set up took "hours" whereas Steve was hedging and saying customers could get started "within a week." They go into more details on their podcast, and this section in particular covers the bootstrap time: https://youtu.be/5P5Mk_IggE0?t=3381 https://youtu.be/5P5Mk_IggE0?t=3381 Pretty fascinating stuff.
- throwawaylinux 3y ago
  Do I have it right that they ship their own hypervisor (that's based on maybe Solaris, not KVM) as firmware? Let's assume a very small team can compete on a technical level, it still seems like it could cut out a lot of the potential market. I can't imagine that large cloud / "web scale" companies would want that. Most want a fair bit of control of their own hypervisor and management stacks based around KVM. And "enterprise" type companies are going to have issues with certification I would have thought -- will RedHat, Microsoft, SAP, Oracle, etc certify their supported products on top of this hypervisor? Seems like a difficult and expensive process. So what's left? Companies that support their own virtual machine software but don't support their own hypervisor and don't like what's available from vmware or Microsoft or RedHat. A small niche. Or are my assumptions wrong?
  - panick21_ 3y ago
    I think quite often when we assume 'most want to fair bit of control' is just not true. Enterprises want something that just works, they want control if they can't have something that just works. If you have a team that is struggling building an internal cloud with all this control (and problems) and all this commodity hardware (and its problems) then maybe they would be happy to switch to something that just works. > And "enterprise" type companies are going to have issues with certification I would have thought -- will RedHat, Microsoft, SAP, Oracle, etc certify their supported products on top of this hypervisor? Seems like a difficult and expensive process. If that was the case and nobody running any of these would run their rack, then I wouldn't think they would not have received any funding. But I don't know enough about these certification process to really comment. > Companies that support their own virtual machine software but don't support their own hypervisor and don't like what's available from vmware or Microsoft or RedHat Non of these come with a fully integrated rack. The competition would be somebody willing to buy a rack of Dell servers with VMWare software. Or somebody willing to buy a rack of Dell server and then use RedHat and set up all their own cloud style infrastructure.
    - throwawaylinux 3y ago
      > I think quite often when we assume 'most want to fair bit of control' is just not true. Enterprises want something that just works, they want control if they can't have something that just works. That's not what I'm assuming here. Read carefully, I divide the market into 3 categories. Those who support their own VM image software and hypervisors, those who support neither, and those who support VM image but not hypervisor. First is Amazon, Google, Facebook and the like (and it's not an assumption we can see their public contributions to KVM, QEMU, etc., and hear their talks about some of what they use internally). Second is "enterprise" who wants something that just works. Third is ? and would they want to support their software on a niche hypervisor? > If that was the case and nobody running any of these would run their rack, then I wouldn't think they would not have received any funding. But I don't know enough about these certification process to really comment. Well it is the case that enterprise (supported) software is not just supported on any hypervisor. https://access.redhat.com/articles/973163 https://access.redhat.com/articles/973163 RHEL runs on their own KVM as well as MS, VMware, some cloud vendors. Some application software also gets certified to hardware and hypervisors, not just operating system (e.g., SAP does this). > Non of these come with a fully integrated rack. It's not fully integrated if it doesn't come with the guest software though, is it? > The competition would be somebody willing to buy a rack of Dell servers with VMWare software. Or somebody willing to buy a rack of Dell server and then use RedHat and set up all their own cloud style infrastructure. Right. And the problem for Oxide is that the competition will have fully certified and supported operating system and application software for their virtual machines.
deleted 3y ago
[deleted]
chamakits 3y ago
I'm not affiliated with them, but I recall this being marketed at some point as giving you the flexibility and customization powers that the Googles and Facebooks of the world have with their on-prem infrastructure without needing to have as deep of a dedicated staff as they do to just this which allowed them to develop all their custom tooling in the first place. Basically if you are on-prem, and you are dissatisfied with what you are getting out of today's onprem sellers. Things like bad firmware with slow update cycles, issues with rack/power supplies/cabling/interconnecting systems. Closed down systems that don't allow much customization, etc. They are open sourcing a lot of their work along the way Again, I'm not affiliated with them, and my info may be outdated so take it with a grain of salt. But that's how I've seen them for some time.
- wmf 3y ago
  I would say Oxide is inflexible and non-customizable since they have exactly one hardware configuration and few software features at this point. Their claim is more that their rack works and everything else on the market is full of bugs.
  - throwaway2037 3y ago
    I'm not an infra engineer, but this claim "everything else on the market is full of bugs" might be the killer app. Of course, it needs to be true. What if they iterate to an insanely stable embedded code base (BIOS, etc.)? Then, continuously upgrade the hardware to use the latest CPU/RAM/NVME. I could see that being very valuable.
    - yencabulator 3y ago
      I fully expect there will be data-loss bugs and poor performance during recovery in their in-house distributed block storage solution. That's just in the nature of the problem domain, and this is all new code: https://github.com/oxidecomputer/crucible https://github.com/oxidecomputer/crucible
strangemonad 3y ago
A dumbed down interpretation. What most people can buy off the shelf for servers more or less looks like a pc shoved into an odd looking case (1u, 3U rack). To get anywhere near the cost / performance of the big players you need something that’s designed for the data center server workloads.
rtpg 3y ago
The short answer is that on-prem is used by a lot of companies for many reasons that go beyond "legacy/people don't know about cloud" (and even go beyond "regulatory environments"!), and these boxes are meant for people who have serious requirements. I'd recommend the O+F ep posted in sibling, but I think here the pitch is "well you need this hardware anyways right? How about buying one that's easy to use and doesn't take a month to get working?" All built by people who are so obsessed with root cause analysis that they've ended up writing their own firmware, running on an OS where these people are common contributors.
- solarkraft 3y ago
  Thanks, I think now I get the excitement (from the engineering perspective more than the business one)! "Fixing everything" is probably everyone's dream.
bpye 3y ago
It feels like they’re trying to give people the benefits that hyperscale cloud operators have, without being one?
- steveklabnik 3y ago
  This is the most succinct pitch for Oxide, the problem is that many people do not know what the hyperscalers even are, so it doesn't land with a lot of folks.
- superb_dev 3y ago
  That is exactly their goal. Hyperscale for the rest of us
jitl 3y ago
Public cloud like AWS is a premium product. When your web scale business wants to increase profits by cutting costs and you’ve already done a few passes making your software run faster, owning your own metal & renting colo space starts looking like a big avenue for savings. Especially if you’re doing something bandwidth intensive, where AWS makes you pay through the nose. I think we’ll see a fair number of companies move back towards owning metal, especially if the metal is super easy to manage. I think the Oxide pitch is making owning racks sensible for ~2000 eng companies instead of ~20,000 eng companies.
throwaway2037 3y ago
Stupid question (no trolling, I promise): Could this product be valuable to public cloud vendors, like AWS, Google, Oracle, IBM, etc? Or even medium sized ones, like Linode? My thinking: Could this be the cloud inversion moment akin to TSMC and outsourcing semi-conductor manuf?
- dbish 3y ago
  Probably not. These vendors have massive supply lines that are designed and tuned to their needs.
- littlestymaar 3y ago
  For big vendors, no. Because what oxide does is basically sell the kind of server that the hyperscalers have been building for themselves internally. But for smaller cloud provider, who aren't running custom hardware made for hyperscale and are instead using of the shelf servers with all their flaws, then it could make a lot of sense.
panick21_ 3y ago
Cars aren't a new invention either, and yet people build new ones. There is an existing market, it is large. Making a product for that market that is just simply 'really good' has potential to make money. Almost all great computer companies started into markets where things already existed that were comparable.
dasil003 3y ago
I am definitely not an infra engineer, but I see it as an attempt to fix problems that have accumulated over decades as standards calcified and no single vendor was able to improve due to interoperability issues. Some of these things can’t be tackled without a full end-to-end solution, hence the full server. It doesn’t mean components couldn’t be swappable in the future, but at the beginning it’s best to develop on a narrow set of hardware until a solid baseline of stability is reached. Sort of an Apple-like approach for large scale servers.

mhh__ 3y ago

YOLO - sometimes it's worth paying more money for something someone has actually sat down and designed properly.

throwawaylinux 3y ago
So what has been designed properly and what hasn't?
- mhh__ 3y ago
  The random piles of servers in most offices above a certain headcount

faitswulff 3y ago

One benefit is that they're competing in an industry where the time to get a sled up and running for compute can be measured in weeks (they say they've heard up to 90 days) whereas their solution is basically plug and play - Bryan was trying to get Steve to admit set up took "hours" whereas Steve was hedging and saying customers could get started "within a week." They go into more details on their podcast, and this section in particular covers the bootstrap time: https://youtu.be/5P5Mk_IggE0?t=3381 https://youtu.be/5P5Mk_IggE0?t=3381 Pretty fascinating stuff.

throwawaylinux 3y ago
Do I have it right that they ship their own hypervisor (that's based on maybe Solaris, not KVM) as firmware? Let's assume a very small team can compete on a technical level, it still seems like it could cut out a lot of the potential market. I can't imagine that large cloud / "web scale" companies would want that. Most want a fair bit of control of their own hypervisor and management stacks based around KVM. And "enterprise" type companies are going to have issues with certification I would have thought -- will RedHat, Microsoft, SAP, Oracle, etc certify their supported products on top of this hypervisor? Seems like a difficult and expensive process. So what's left? Companies that support their own virtual machine software but don't support their own hypervisor and don't like what's available from vmware or Microsoft or RedHat. A small niche. Or are my assumptions wrong?
- panick21_ 3y ago
  I think quite often when we assume 'most want to fair bit of control' is just not true. Enterprises want something that just works, they want control if they can't have something that just works. If you have a team that is struggling building an internal cloud with all this control (and problems) and all this commodity hardware (and its problems) then maybe they would be happy to switch to something that just works. > And "enterprise" type companies are going to have issues with certification I would have thought -- will RedHat, Microsoft, SAP, Oracle, etc certify their supported products on top of this hypervisor? Seems like a difficult and expensive process. If that was the case and nobody running any of these would run their rack, then I wouldn't think they would not have received any funding. But I don't know enough about these certification process to really comment. > Companies that support their own virtual machine software but don't support their own hypervisor and don't like what's available from vmware or Microsoft or RedHat Non of these come with a fully integrated rack. The competition would be somebody willing to buy a rack of Dell servers with VMWare software. Or somebody willing to buy a rack of Dell server and then use RedHat and set up all their own cloud style infrastructure.
  - throwawaylinux 3y ago
    > I think quite often when we assume 'most want to fair bit of control' is just not true. Enterprises want something that just works, they want control if they can't have something that just works. That's not what I'm assuming here. Read carefully, I divide the market into 3 categories. Those who support their own VM image software and hypervisors, those who support neither, and those who support VM image but not hypervisor. First is Amazon, Google, Facebook and the like (and it's not an assumption we can see their public contributions to KVM, QEMU, etc., and hear their talks about some of what they use internally). Second is "enterprise" who wants something that just works. Third is ? and would they want to support their software on a niche hypervisor? > If that was the case and nobody running any of these would run their rack, then I wouldn't think they would not have received any funding. But I don't know enough about these certification process to really comment. Well it is the case that enterprise (supported) software is not just supported on any hypervisor. https://access.redhat.com/articles/973163 https://access.redhat.com/articles/973163 RHEL runs on their own KVM as well as MS, VMware, some cloud vendors. Some application software also gets certified to hardware and hypervisors, not just operating system (e.g., SAP does this). > Non of these come with a fully integrated rack. It's not fully integrated if it doesn't come with the guest software though, is it? > The competition would be somebody willing to buy a rack of Dell servers with VMWare software. Or somebody willing to buy a rack of Dell server and then use RedHat and set up all their own cloud style infrastructure. Right. And the problem for Oxide is that the competition will have fully certified and supported operating system and application software for their virtual machines.

deleted 3y ago

[deleted]

chamakits 3y ago

I'm not affiliated with them, but I recall this being marketed at some point as giving you the flexibility and customization powers that the Googles and Facebooks of the world have with their on-prem infrastructure without needing to have as deep of a dedicated staff as they do to just this which allowed them to develop all their custom tooling in the first place. Basically if you are on-prem, and you are dissatisfied with what you are getting out of today's onprem sellers. Things like bad firmware with slow update cycles, issues with rack/power supplies/cabling/interconnecting systems. Closed down systems that don't allow much customization, etc. They are open sourcing a lot of their work along the way Again, I'm not affiliated with them, and my info may be outdated so take it with a grain of salt. But that's how I've seen them for some time.

wmf 3y ago
I would say Oxide is inflexible and non-customizable since they have exactly one hardware configuration and few software features at this point. Their claim is more that their rack works and everything else on the market is full of bugs.
- throwaway2037 3y ago
  I'm not an infra engineer, but this claim "everything else on the market is full of bugs" might be the killer app. Of course, it needs to be true. What if they iterate to an insanely stable embedded code base (BIOS, etc.)? Then, continuously upgrade the hardware to use the latest CPU/RAM/NVME. I could see that being very valuable.
  - yencabulator 3y ago
    I fully expect there will be data-loss bugs and poor performance during recovery in their in-house distributed block storage solution. That's just in the nature of the problem domain, and this is all new code: https://github.com/oxidecomputer/crucible https://github.com/oxidecomputer/crucible

strangemonad 3y ago

A dumbed down interpretation. What most people can buy off the shelf for servers more or less looks like a pc shoved into an odd looking case (1u, 3U rack). To get anywhere near the cost / performance of the big players you need something that’s designed for the data center server workloads.

rtpg 3y ago

The short answer is that on-prem is used by a lot of companies for many reasons that go beyond "legacy/people don't know about cloud" (and even go beyond "regulatory environments"!), and these boxes are meant for people who have serious requirements. I'd recommend the O+F ep posted in sibling, but I think here the pitch is "well you need this hardware anyways right? How about buying one that's easy to use and doesn't take a month to get working?" All built by people who are so obsessed with root cause analysis that they've ended up writing their own firmware, running on an OS where these people are common contributors.

solarkraft 3y ago
Thanks, I think now I get the excitement (from the engineering perspective more than the business one)! "Fixing everything" is probably everyone's dream.

bpye 3y ago

It feels like they’re trying to give people the benefits that hyperscale cloud operators have, without being one?

steveklabnik 3y ago
This is the most succinct pitch for Oxide, the problem is that many people do not know what the hyperscalers even are, so it doesn't land with a lot of folks.
superb_dev 3y ago
That is exactly their goal. Hyperscale for the rest of us

jitl 3y ago

Public cloud like AWS is a premium product. When your web scale business wants to increase profits by cutting costs and you’ve already done a few passes making your software run faster, owning your own metal & renting colo space starts looking like a big avenue for savings. Especially if you’re doing something bandwidth intensive, where AWS makes you pay through the nose. I think we’ll see a fair number of companies move back towards owning metal, especially if the metal is super easy to manage. I think the Oxide pitch is making owning racks sensible for ~2000 eng companies instead of ~20,000 eng companies.

throwaway2037 3y ago

Stupid question (no trolling, I promise): Could this product be valuable to public cloud vendors, like AWS, Google, Oracle, IBM, etc? Or even medium sized ones, like Linode? My thinking: Could this be the cloud inversion moment akin to TSMC and outsourcing semi-conductor manuf?

dbish 3y ago
Probably not. These vendors have massive supply lines that are designed and tuned to their needs.
littlestymaar 3y ago
For big vendors, no. Because what oxide does is basically sell the kind of server that the hyperscalers have been building for themselves internally. But for smaller cloud provider, who aren't running custom hardware made for hyperscale and are instead using of the shelf servers with all their flaws, then it could make a lot of sense.

panick21_ 3y ago

Cars aren't a new invention either, and yet people build new ones. There is an existing market, it is large. Making a product for that market that is just simply 'really good' has potential to make money. Almost all great computer companies started into markets where things already existed that were comparable.

dasil003 3y ago

I am definitely not an infra engineer, but I see it as an attempt to fix problems that have accumulated over decades as standards calcified and no single vendor was able to improve due to interoperability issues. Some of these things can’t be tackled without a full end-to-end solution, hence the full server. It doesn’t mean components couldn’t be swappable in the future, but at the beginning it’s best to develop on a narrow set of hardware until a solid baseline of stability is reached. Sort of an Apple-like approach for large scale servers.