4 ms·

Let's imagine a 42U rack full of Dell or HPE servers. Each one has a couple of power supplies, console ports, a video port and a GPU to drive it, USB ports for

by __d 3y ago

Let's imagine a 42U rack full of Dell or HPE servers. Each one has a couple of power supplies, console ports, a video port and a GPU to drive it, USB ports for keyboard, and mouse, etc. More to the point with Oxide, there's layer upon layer of software cruft as well: various levels of proprietary firmware blobs driving the boot process, and then the management stack that underlies a commodity PC platform OS. All of that is gone, replaced with a bottom-up redesign, of both the hardware and the software, with the specific purpose of running large scale modern application loads. This goes well beyond a blade server, and is a completely different beast than a rack of DL360s.

throwawaylinux 3y ago
I know it might seem like it, but there's just not that much in it. All video, USB, "console ports" can be driven from a small chip using a couple of watts. Some do all that on their BMC SoC, even. And if your requirements include redundant power supplies, then that's what you get, and if they don't you get something without them. And the boot and runtime firmware might be complicated and have its own issues, but there are not large runtime performance overheads in that either that you can just "rewrite it all" and get a big speedup. Proof of the pudding will be in the eating I guess. Does Oxide talk about performance advantages at all, or have numbers?
- __d 3y ago
  A couple of watts here, and a couple of watts there, and pretty soon you're talking about real money! If it didn't matter, then there'd be no interest in blades, or OCP, and all the cloud vendors would use standard rackmount whitebox servers. At the scale of one rack, the difference is fairly small. But once you're building out cages full of racks, maybe it matters more? And it's not just power and space (although they both matter), it's the attack surface, and the ability to actually have your hardware do what you want, and the benefits of a stack that's built for purpose, not cobbled together out of off-the-shelf parts. It's not for everyone, to be sure, but I suspect that for its target market, it'll be a very successful product/family. As you say, we will see. I wish them well.
  - throwawaylinux 3y ago
    > A couple of watts here, and a couple of watts there, and pretty soon you're talking about real money! > Not with the current state of the art. > If it didn't matter, then there'd be no interest in blades, or OCP, and all the cloud vendors would use standard rackmount whitebox servers. I'm talking about it mattering from the starting point of blades, OCP, "cloud" systems! The thread is about where the oxide niche is and what advantages it has over competition. The idea there is huge amount to be won on southbridge chipset and IO ports in large scale systems is simply not true. Quite amazing that people who don't understand this are posting in this thread as though they are experts in the matter.
    - __d 3y ago
      >> A couple of watts here, and a couple of watts there, and pretty soon you're talking about real money! > Not with the current state of the art. Perhaps you could educate me: what's the state of the art that makes watts free? >> If it didn't matter, then there'd be no interest in blades, or OCP, and all the cloud vendors would use standard rackmount whitebox servers. > I'm talking about it mattering from the starting point of blades, OCP, "cloud" systems! > The thread is about where the oxide niche is and what advantages it has over competition. The idea there is huge amount to be won on southbridge chipset and IO ports in large scale systems is simply not true. I don't think anyone has asserted that there's "huge amount"s to be won from the simplified hardware? It's my view that there is a small amount to be won (more if you're coming from DL380s, and a bit less if you're replacing blades, for instance). But those small amounts do add up, and contribute to the value proposition. I think there are other parts of the Oxide product that are more compelling, but that's the point: it's the total package that you're buying, not just the missing VGA socket on the front panel. Imagine that I'm an enterprise computing user. I currently have a few small (wrt cloud providers) datacenters, stuffed with racks of any tier 1 vendor's rack mount, blade or even OCP systems. And they're running some combination of plain VMs, perhaps some orchestration platform (k8s, or Mesos, or whatever). And it's time for my 3/5/8-yearly refresh of a bunch of those systems. A unified, coherent, targeted product (such as Oxide) might be a compelling offer. It promises to deliver cloud-type hardware and software efficiencies at a smaller scale. It gives me the savings of not renting from AWS/Azure/etc, while not dealing with the hodge-podge of the standard PC hardware and software stack. > Quite amazing that people who don't understand this are posting in this thread as though they are experts in the matter. Welcome to the Internet?
bcantrill 3y ago
Exactly, and the amount of cruft in there is indeed stunning! And broadly, not needed -- though there was a moment when we really wondered about KBRST_L, the ancient input to the CPU to indicate a keyboard reset! (We recounted this story on our "Tales from the Bringup Lab"[0], which is a must-listen if you haven't heard it.) [0] https://www.youtube.com/watch?v=lhji-kP3Lhk#t=1h12m56s https://www.youtube.com/watch?v=lhji-kP3Lhk#t=1h12m56s

by __d 3y ago

throwawaylinux 3y ago

I know it might seem like it, but there's just not that much in it. All video, USB, "console ports" can be driven from a small chip using a couple of watts. Some do all that on their BMC SoC, even. And if your requirements include redundant power supplies, then that's what you get, and if they don't you get something without them. And the boot and runtime firmware might be complicated and have its own issues, but there are not large runtime performance overheads in that either that you can just "rewrite it all" and get a big speedup. Proof of the pudding will be in the eating I guess. Does Oxide talk about performance advantages at all, or have numbers?

__d 3y ago
A couple of watts here, and a couple of watts there, and pretty soon you're talking about real money! If it didn't matter, then there'd be no interest in blades, or OCP, and all the cloud vendors would use standard rackmount whitebox servers. At the scale of one rack, the difference is fairly small. But once you're building out cages full of racks, maybe it matters more? And it's not just power and space (although they both matter), it's the attack surface, and the ability to actually have your hardware do what you want, and the benefits of a stack that's built for purpose, not cobbled together out of off-the-shelf parts. It's not for everyone, to be sure, but I suspect that for its target market, it'll be a very successful product/family. As you say, we will see. I wish them well.
- throwawaylinux 3y ago
  > A couple of watts here, and a couple of watts there, and pretty soon you're talking about real money! > Not with the current state of the art. > If it didn't matter, then there'd be no interest in blades, or OCP, and all the cloud vendors would use standard rackmount whitebox servers. I'm talking about it mattering from the starting point of blades, OCP, "cloud" systems! The thread is about where the oxide niche is and what advantages it has over competition. The idea there is huge amount to be won on southbridge chipset and IO ports in large scale systems is simply not true. Quite amazing that people who don't understand this are posting in this thread as though they are experts in the matter.
  - __d 3y ago
    >> A couple of watts here, and a couple of watts there, and pretty soon you're talking about real money! > Not with the current state of the art. Perhaps you could educate me: what's the state of the art that makes watts free? >> If it didn't matter, then there'd be no interest in blades, or OCP, and all the cloud vendors would use standard rackmount whitebox servers. > I'm talking about it mattering from the starting point of blades, OCP, "cloud" systems! > The thread is about where the oxide niche is and what advantages it has over competition. The idea there is huge amount to be won on southbridge chipset and IO ports in large scale systems is simply not true. I don't think anyone has asserted that there's "huge amount"s to be won from the simplified hardware? It's my view that there is a small amount to be won (more if you're coming from DL380s, and a bit less if you're replacing blades, for instance). But those small amounts do add up, and contribute to the value proposition. I think there are other parts of the Oxide product that are more compelling, but that's the point: it's the total package that you're buying, not just the missing VGA socket on the front panel. Imagine that I'm an enterprise computing user. I currently have a few small (wrt cloud providers) datacenters, stuffed with racks of any tier 1 vendor's rack mount, blade or even OCP systems. And they're running some combination of plain VMs, perhaps some orchestration platform (k8s, or Mesos, or whatever). And it's time for my 3/5/8-yearly refresh of a bunch of those systems. A unified, coherent, targeted product (such as Oxide) might be a compelling offer. It promises to deliver cloud-type hardware and software efficiencies at a smaller scale. It gives me the savings of not renting from AWS/Azure/etc, while not dealing with the hodge-podge of the standard PC hardware and software stack. > Quite amazing that people who don't understand this are posting in this thread as though they are experts in the matter. Welcome to the Internet?

bcantrill 3y ago

Exactly, and the amount of cruft in there is indeed stunning! And broadly, not needed -- though there was a moment when we really wondered about KBRST_L, the ancient input to the CPU to indicate a keyboard reset! (We recounted this story on our "Tales from the Bringup Lab"[0], which is a must-listen if you haven't heard it.) [0] https://www.youtube.com/watch?v=lhji-kP3Lhk#t=1h12m56s https://www.youtube.com/watch?v=lhji-kP3Lhk#t=1h12m56s