6 ms·

What is missing is a middle ground: an improvement over the IPMI broken-state-of-things that is aimed for automated deployments, without demanding a certain net

What is missing is a middle ground: an improvement over the IPMI broken-state-of-things that is aimed for automated deployments, without demanding a certain network environment for PXE booting and configuration of IP addresses for management. I haven't used RedFish but it seems it does not address the problems of IPMI in this regard: To be fully automatable there shouldn't be an assumption about the network environment - rather I would like that to have an L2-based protocol (one has to use the MAC address as identifier anyway) or if IPv4/v6 is used then at least demand mandatory link-local auto-addressing by default. However, RedFish seems to support emulating a CD-ROM drive for booting an installation media, this is a nice idea but what I really would like to see is directly writing an image to the hard disk. Then of coure the hardware manufacturers would need to write high-quality firmware and not the regular quirk- and bug-ridden stuff that we know from IPMI implementations.

bcantrill 4y ago
Your last sentence reveals what you probably already know to be true: there isn't a middle ground. We tried for years to get hyperscaler-class software running on commodity-class servers -- and came to the conclusion that the hyperscalers came to (and as Alan Kay famously observed): to be really serious about elastic infrastructure, one needs to do one's own hardware. Now, ~2.5 years into Oxide, I can substantiate that with more detail that I could have ever imagined: there is no middle ground -- and indeed, this is why the infrastructure world has become so acutely bifurcated!
- mwcampbell 4y ago
  Do you think there's at least a place for a holistic system that's not a whole rack? Or do you think that those of us with smaller computing needs will just have to either rent from a hyperscaler or put up with PC servers?
  - bcantrill 4y ago
    Definitely, but it's not a market that Oxide intends to go after (at least not for the foreseeable future). Part of the challenge is that it really does require diverging from the reference designs (as these have effectively enshrined a proprietary BMC and everything that comes with it) -- and once you are away from reference designs, you are talking about a very different economic proposition. This is part of why we are so transparent and open about everything we're doing: our hope is that we can enable others to pick up the baton and serve those markets that we do not intend to address.
deivid 4y ago
I have put our implementation of _exactly_ this in production last week. Boot a "CD-ROM" via IPMI into a custom installer, flash image (generated from a Dockerfile!) to disk, reboot. Install takes ~15 seconds, full process (going via POST once) takes ~4 minutes. This also allows us to have a "hardware validation" image (that one doesn't get persisted to disk). Not sure if there's any plans to make it public right now, but I'll ask around. Feel free to contact me via email (in profile)
- justinclift 4y ago
  > but I'll ask around. Please do. There's a need for it. :)
- yjftsjthsd-h 4y ago
  > flash image (generated from a Dockerfile!) Any hints about how to do this? I'd love to use something as nice to work with as a Dockerfile to build bare metal installs.
  - deivid 4y ago
    I wrote down the kernel of the idea quite a while ago here[0], however, the actual version we run nowadays is using UEFI and written in Go (installer) + Python (API that generates the ISO on demand) Fly.io[1] also does this (although they boot the result in a VM, the concepts are the same) [0]: https://blog.davidv.dev/docker-based-images-on-baremetal.html https://blog.davidv.dev/docker-based-images-on-baremetal.htm... [1]: https://fly.io/blog/docker-without-docker/ https://fly.io/blog/docker-without-docker/
- Klasiaster 4y ago
  Yes, automation with IPMI can be done (for a particular hardware due to quirks), and I recommend doing it (and I have done it) with a separate DHCP environment for the BMC NICs to be at least agnostic about the OS NIC's network environment. So, I share your excitement when it works but I'm not happy with the out-of-the box experience of IPMI. I wish there was something better but unfortunately this is just one case of a split/mismatch of what HW people provide and what software people need.

bcantrill 4y ago

Your last sentence reveals what you probably already know to be true: there isn't a middle ground. We tried for years to get hyperscaler-class software running on commodity-class servers -- and came to the conclusion that the hyperscalers came to (and as Alan Kay famously observed): to be really serious about elastic infrastructure, one needs to do one's own hardware. Now, ~2.5 years into Oxide, I can substantiate that with more detail that I could have ever imagined: there is no middle ground -- and indeed, this is why the infrastructure world has become so acutely bifurcated!

mwcampbell 4y ago
Do you think there's at least a place for a holistic system that's not a whole rack? Or do you think that those of us with smaller computing needs will just have to either rent from a hyperscaler or put up with PC servers?
- bcantrill 4y ago
  Definitely, but it's not a market that Oxide intends to go after (at least not for the foreseeable future). Part of the challenge is that it really does require diverging from the reference designs (as these have effectively enshrined a proprietary BMC and everything that comes with it) -- and once you are away from reference designs, you are talking about a very different economic proposition. This is part of why we are so transparent and open about everything we're doing: our hope is that we can enable others to pick up the baton and serve those markets that we do not intend to address.

deivid 4y ago

I have put our implementation of _exactly_ this in production last week. Boot a "CD-ROM" via IPMI into a custom installer, flash image (generated from a Dockerfile!) to disk, reboot. Install takes ~15 seconds, full process (going via POST once) takes ~4 minutes. This also allows us to have a "hardware validation" image (that one doesn't get persisted to disk). Not sure if there's any plans to make it public right now, but I'll ask around. Feel free to contact me via email (in profile)

justinclift 4y ago
> but I'll ask around. Please do. There's a need for it. :)
yjftsjthsd-h 4y ago
> flash image (generated from a Dockerfile!) Any hints about how to do this? I'd love to use something as nice to work with as a Dockerfile to build bare metal installs.
- deivid 4y ago
  I wrote down the kernel of the idea quite a while ago here[0], however, the actual version we run nowadays is using UEFI and written in Go (installer) + Python (API that generates the ISO on demand) Fly.io[1] also does this (although they boot the result in a VM, the concepts are the same) [0]: https://blog.davidv.dev/docker-based-images-on-baremetal.html https://blog.davidv.dev/docker-based-images-on-baremetal.htm... [1]: https://fly.io/blog/docker-without-docker/ https://fly.io/blog/docker-without-docker/
Klasiaster 4y ago
Yes, automation with IPMI can be done (for a particular hardware due to quirks), and I recommend doing it (and I have done it) with a separate DHCP environment for the BMC NICs to be at least agnostic about the OS NIC's network environment. So, I share your excitement when it works but I'm not happy with the out-of-the box experience of IPMI. I wish there was something better but unfortunately this is just one case of a split/mismatch of what HW people provide and what software people need.