3 ms·
I am wondering when would you use kubernetes on oxide vs running kubernetes with kubevirt on baremetal? At first look, it feels like oxide is equivalent to pro
by bitlad 2mo ago
I am wondering when would you use kubernetes on oxide vs running kubernetes with kubevirt on baremetal?
At first look, it feels like oxide is equivalent to proxmox or some virtualization tool, may be it uses qemu stuff underneath.
Just curious.
The reason I am asking this is that, we have lot onprem scenarios in our business. We are tightly coupled with k8s, to solve this we started building an internal project that is kubernetes API compatible [1] but runs containerd or WASM or our platform natively.
Just curious how oxides work in this scenario
[1] https://github.com/debarshibasak/superkube https://github.com/debarshibasak/superkube
- e12e 2mo agoWhen oxide provide your metal?
- bakies 2mo agoYeah me too. Like I kinda compare oxide to coreweave and coreweave is built on top of k8s afaik. I'm wondering what the underlying tech oxide leans on. If I was doing this I would be using talos and k8s on baremetal and building on top of that. Container first rather than VM first seems like a big advantage imo. Most workloads will be containers.
- bitlad 2mo agoCoreweave is a cloud provider right? I dont think it is apple to apple comparison.
- bakies 2mo agoWhat's a cloud? K8s is a cloud imo, so is what oxide is providing afaict
- fragmede 2mo agoCloud is an API you can hit to get access to hardware. Oxide sells hardware.
- bakies 2mo agoOxide has an API on top that gets you services. The whole pitch is AWS with purchased hardware, no? That's cloud
- fragmede 2mo agoYes but you buy the hardware and have to rack it, and then you have a cloud at home.
- steveklabnik 2mo agoExtreme nit: because you purchase a rack at a time, you don't actually rack it. You wheel it in, plug in network and power, and hit "on".
- fragmede 2mo agoOh man, genuinely thank you for that nit. Gives me different imagery to imagine for the Oxide node I'll never be able to afford.
- steveklabnik 2mo agoI'm glad, I meant it in semi-jest, because it's true, but also just a very funny thing to "well actually" about.
- esseph 2mo ago> The whole pitch is AWS with purchased hardware, no? HBOM SBOM Unparalleled support w/ vertical integration Tons of companies will see you a "cloud". The other parts seem pretty unique.
- esseph 2mo agoMost security minded orgs and also AWS deploy containers in VMs for better security oxide and coreweave? Oxide is in-house custom everything. Switches. Racks. Power bus. BMC. Firmware. OS. Software and APIs. Virtualization. Etc.
- wmf 2mo agoYou would use Oxide if you hate Dell/HPE/Lenovo/Supermicro. Also most people won't run k8s on bare metal because they want to dynamically provision a bunch of clusters.
- sudomateo 2mo agoGood question. We use our own hypervisor[0] that's not KVM/QEMU. We don't have nested virtualization today so we don't follow the KubeVirt model, though we are discussing what CRDs like OxideInstance would look like for those that want to operate solely in Kubernetes manifests. The core primitive on Oxide is the instance (virtual machine). We could support some container primitive, but that's a larger product direction discussion. Our host OS is Helios (Illumos) so there are details to iron out there regarding what abstractions we would build and expose to the users. Not impossible but not something we're currently pursuing either given that we have other immediate product asks. If you're at a scale where compute density, power efficiency, security, and rack-level API management matters then that's where Oxide makes sense for you. [0] https://github.com/oxidecomputer/propolis https://github.com/oxidecomputer/propolis
- moondev 2mo agoCould there theoretically be a swappable hypervisor for oxide some day? I'm imagining a oxide Linux distro of sorts running on the hosts, and using kvm for vms, but still oxide control plane bits downstream. Then you get nested virt and gpu support, and could maybe even virtualize a traditional oxide host. Like running multi-host vcenter clusters inside a single esxi, amazingly powerful lab scenarios, teardown is even fun
- sudomateo 1mo agoTheoretically, yes. Practically, highly unlikely. Switching to Linux and KVM would gain the ecosystem benefits but Oxide would lose control of the host OS which can impact our security stance. Better to tightly integrate at this layer and expose the primitives that are needed. It's not impossible, but we're already maintaining the existing Helios host OS. Adding another wouldn't be beneficial.
- p_l 2mo agoOxide is shipping the baremetal server as well, with integrated hypervisor - so running kubernetes with kubevirt on it makes less sense than integrating with the host hypervisor