4 ms·

Kairos: Immutable Distro for K8s at the Edge

46 points by porjo 1y ago

klooney 1y ago
https://kairos.io/docs/reference/faq/ https://kairos.io/docs/reference/faq/ I was wondering how it compares to Talos and now I'm confused. Is it basically a bootloader+host kernel shim and nothing else?
- TrueDuality 1y ago
  That is also what I came here to find out. Would love to hear from the creators of the project how it compares and contrasts to Talos. We've been running Talos for a few bare-metal and air-gapped cluster deployments with pretty good success but do have some pain-points.
  - udev4096 1y ago
    Is it better than k3s?
    - mdasen 1y ago
      It's very different from k3s. With k3s, you have a Linux installation like Debian or Fedora and you install k3s on it. You can SSH into the box, install any other Linux program not running in Kubernetes, etc. It also means that you need to run security updates and all the other stuff that goes along with administering a Linux box. With Talos, it's just Kubernetes running on the box. There's no SSH or anything. Yes, it's a Linux kernel running, but you don't have a way of running stuff on the box outside of Kubernetes. For me, Talos is great. If I'm setting up some boxes for K8s, I don't want to have to deal with admin'ing a Linux box. I don't want to login to the box and run some non-K8s service on it. I just want a K8s node and that's what Talos gives me. I think that's also the experience most people want. It's why people pay AWS, GCS, and Azure tons of money to get hosted K8s nodes rather than a Linux box they need to admin.
      - SR2Z 1y ago
        IDK about Talos but Kairos runs k3s.
  - evrflx 1y ago
    Would love to hear about the pain points: Please elaborate, as I am currently in the decision phase and Talos as of now the top contender.
    - dachrillz 1y ago
      Not OP, but when we tested it out it was painful to handle usb disks. The reason being that if you have two they get named sda/sdb randomly. We managed to overwrite the usb we were using to install talos since that one was named sda one boot and sdb the next. This lead ut to develop the “pullout technique” when installing… This mostly only happened because it was a test cluster where we used usb disks, probably not a problem when one properly provisions. Otherwise it was great! But it does feel akward not booting into an environment where you have a terminal at first
      - vizzier 1y ago
        This does sound like it could be solved with better installDiskSelectors[0]. Talos has done a fair bit of work in improving this and UserVolumeConfigs in the last couple of 1.x revisions. Alternatively, network booting in some fashion is an option. [1] [0] https://www.talos.dev/v1.11/reference/configuration/v1alpha1/config/#Config.machine.install.diskSelector https://www.talos.dev/v1.11/reference/configuration/v1alpha1... [1] https://www.talos.dev/v1.11/talos-guides/install/bare-metal-platforms/pxe/ https://www.talos.dev/v1.11/talos-guides/install/bare-metal-...
      - INTPenis 1y ago
        I recognize this from my bare-metal homelab setup. But at work we used VMs for Talos nodes so this was not an issue. And if I had to deploy it on bare-metal at work I'd most likely use PXE booting instead of USB.
        bakies 1y ago
        I use pxe boot for my homelab baremetal :)
    - askedrelic 1y ago
deleted 1y ago
[deleted]
tantalic 1y ago
I'm (still) very interested about this project. Two years ago, I came across this when setting up a Raspberry Pi based "home lab" Kubernetes cluster (https://github.com/tantalic/shaving-yaks https://github.com/tantalic/shaving-yaks). I ran into some issues (I believe all Raspberry Pi specific) and ended up going with k0s. While I am happy with k0s, I do wish Kairos worked out for me and I might have to give it a try again soon.
sixarm 1y ago
Been successful using Kairos to build and deploy an immutable RHEL9 to environments without internet access. Appreciate the flexibility to work with different distros and the maintainers provide great support.
- zobzu 1y ago
  RH also provides immutable OSes (Kairo uses some of their stuff) via Fedora Atomic and Fedora CoreOS, so for a single RHEL-like immutable instance, i'd just go CoreOS or Kinoite The main reasons to use Kairo are perhaps more around the P2P mesh, using Cloudinit, deb vs rpm etc.
candiddevmike 1y ago
We faced a similar need and wanted something more "familiar". We built EtchaOS as an in-memory, immutable variant of popular distributions like Debian, bundled with Docker and containerd, and powered by an imperative, stateful configuration management tool (Etcha): https://etcha.dev/etchaos/ https://etcha.dev/etchaos/
- thebruce87m 1y ago
  I know balena etcher is a different product, but balena do operate in your space too so it seems like an unfortunate name clash.
suralind 1y ago
Nice project. I will forever be spoiled by Talos' simplicity.
deleted 1y ago
[deleted]

46 points by porjo 1y ago

https://kairos.io/docs/reference/faq/ https://kairos.io/docs/reference/faq/ I was wondering how it compares to Talos and now I'm confused. Is it basically a bootloader+host kernel shim and nothing else?

TrueDuality 1y ago
That is also what I came here to find out. Would love to hear from the creators of the project how it compares and contrasts to Talos. We've been running Talos for a few bare-metal and air-gapped cluster deployments with pretty good success but do have some pain-points.
- udev4096 1y ago
  Is it better than k3s?
  - mdasen 1y ago
    It's very different from k3s. With k3s, you have a Linux installation like Debian or Fedora and you install k3s on it. You can SSH into the box, install any other Linux program not running in Kubernetes, etc. It also means that you need to run security updates and all the other stuff that goes along with administering a Linux box. With Talos, it's just Kubernetes running on the box. There's no SSH or anything. Yes, it's a Linux kernel running, but you don't have a way of running stuff on the box outside of Kubernetes. For me, Talos is great. If I'm setting up some boxes for K8s, I don't want to have to deal with admin'ing a Linux box. I don't want to login to the box and run some non-K8s service on it. I just want a K8s node and that's what Talos gives me. I think that's also the experience most people want. It's why people pay AWS, GCS, and Azure tons of money to get hosted K8s nodes rather than a Linux box they need to admin.
    - SR2Z 1y ago
      IDK about Talos but Kairos runs k3s.
- evrflx 1y ago
  Would love to hear about the pain points: Please elaborate, as I am currently in the decision phase and Talos as of now the top contender.
  - dachrillz 1y ago
    Not OP, but when we tested it out it was painful to handle usb disks. The reason being that if you have two they get named sda/sdb randomly. We managed to overwrite the usb we were using to install talos since that one was named sda one boot and sdb the next. This lead ut to develop the “pullout technique” when installing… This mostly only happened because it was a test cluster where we used usb disks, probably not a problem when one properly provisions. Otherwise it was great! But it does feel akward not booting into an environment where you have a terminal at first
    - vizzier 1y ago
      This does sound like it could be solved with better installDiskSelectors[0]. Talos has done a fair bit of work in improving this and UserVolumeConfigs in the last couple of 1.x revisions. Alternatively, network booting in some fashion is an option. [1] [0] https://www.talos.dev/v1.11/reference/configuration/v1alpha1/config/#Config.machine.install.diskSelector https://www.talos.dev/v1.11/reference/configuration/v1alpha1... [1] https://www.talos.dev/v1.11/talos-guides/install/bare-metal-platforms/pxe/ https://www.talos.dev/v1.11/talos-guides/install/bare-metal-...
    - INTPenis 1y ago
      I recognize this from my bare-metal homelab setup. But at work we used VMs for Talos nodes so this was not an issue. And if I had to deploy it on bare-metal at work I'd most likely use PXE booting instead of USB.
      - bakies 1y ago
        I use pxe boot for my homelab baremetal :)
  - askedrelic 1y ago

deleted 1y ago

[deleted]

tantalic 1y ago

I'm (still) very interested about this project. Two years ago, I came across this when setting up a Raspberry Pi based "home lab" Kubernetes cluster (https://github.com/tantalic/shaving-yaks https://github.com/tantalic/shaving-yaks). I ran into some issues (I believe all Raspberry Pi specific) and ended up going with k0s. While I am happy with k0s, I do wish Kairos worked out for me and I might have to give it a try again soon.

sixarm 1y ago

Been successful using Kairos to build and deploy an immutable RHEL9 to environments without internet access. Appreciate the flexibility to work with different distros and the maintainers provide great support.

zobzu 1y ago
RH also provides immutable OSes (Kairo uses some of their stuff) via Fedora Atomic and Fedora CoreOS, so for a single RHEL-like immutable instance, i'd just go CoreOS or Kinoite The main reasons to use Kairo are perhaps more around the P2P mesh, using Cloudinit, deb vs rpm etc.

candiddevmike 1y ago

We faced a similar need and wanted something more "familiar". We built EtchaOS as an in-memory, immutable variant of popular distributions like Debian, bundled with Docker and containerd, and powered by an imperative, stateful configuration management tool (Etcha): https://etcha.dev/etchaos/ https://etcha.dev/etchaos/

thebruce87m 1y ago
I know balena etcher is a different product, but balena do operate in your space too so it seems like an unfortunate name clash.

suralind 1y ago

Nice project. I will forever be spoiled by Talos' simplicity.

deleted 1y ago

[deleted]