7 ms·
Talos: Secure, immutable, and minimal Linux OS for running Kubernetes
- philips 2y agoThis team has a pretty active YouTube channel that is worth checking out too. https://youtube.com/@siderolabs https://youtube.com/@siderolabs
- PhilipJFry 2y agoPreviously on Hacker News: https://news.ycombinator.com/item?id=37846995 https://news.ycombinator.com/item?id=37846995 https://news.ycombinator.com/item?id=31798362 https://news.ycombinator.com/item?id=31798362 https://news.ycombinator.com/item?id=31486815 https://news.ycombinator.com/item?id=31486815
- breadwinner 2y agoIf you can't login to it then it is not good for development. If it is not good for development it is not good for production because ideally your dev and production environment should be the same.
- birdiesanders 2y agoYou shouldn’t be working on the node OS for k8s dev work.
- breadwinner 2y agoYou assume I was talking about the node OS, but I was talking about the container.
- breadwinner 2y agoI said ideally your dev and production environment should be the same. That should have been a hint I was talking about container.
- rstat1 2y agok8s is k8s, it doesn't really matter much what the host OS is.
- hadlock 2y agoYou should not be logging in to production unless something has gone seriously wrong. I've not seen a company where developers (minus a handful of "blessed" staff) even know how to access prod, let alone log in.
- breadwinner 2y agoDuring development you will need to login to the pod, to review settings, directory contents and so on. If the OS running in the pod does not allow you to do that - during development - then that's severely limiting.
- vbezhenar 2y agoTalos is OS running in the host. You can run in the pod whatever you need. Also you shouldn't really use pod OS to debug it. Kubernetes supports debug containers: you launch a separate container (presumably with convenient debug environment) and mounts selected container rootfs inside, so you can inspect it as needed. It also helps, when the target container does not work and you can't just exec into it. There's a recommendation to remove everything from the container that's not necessary for running a given program, that reduces attack surface.
- gotbeans 2y agoYou can still exec yourself into the pods. No one said you cannot. There is no shell or ssh on the hosts for you to login to, but still if you absolutely must you can create a privileged container and mount /. Whole point is you shouldn't.
- yjftsjthsd-h 2y ago> If it is not good for development it is not good for production because ideally your dev and production environment should be the same. Correct: Your dev environment should also not let you do stuff on the host machines. In an k8s environment, you run everything in pods. Don't compromise on security and operational concerns just because it's a dev environment. > If you can't login to it then it is not good for development. You develop inside pods, and you are more than welcome to install any shell and other programs you want inside containers. (Or for working at the k8s level it doesn't matter; you `kubectl apply` or run helm against the k8s API, it doesn't matter what's happening on the host.)
- breadwinner 2y agoIs Talos for running inside pods, or running on the node. It is not immediately clear from the website.
- sierra1011 2y ago"Talos: Secure, immutable, and minimal Linux OS for running Kubernetes" I guess we'll never know... But in truth it's for running on hosts.
- FooBarWidget 2y agoSome customers of an internal Kubernetes platform complain that their pods keep getting evicted because their nodes keep running out of disk space. The platform's maintainers' first instinct was that customers' pods should not write to ephemeral storage, e.g., should not write to log files on the filesystem unless mounted from external storage. But that turns out not to be the case: customers' pods did not write anything to disk at all. So why were nodes running out of disk space? Prometheus metrics show which partitions use how much disk space but cannot go into more detail. The team wanted to inspect the node's filesystem to figure out what exactly is using so much disk space. The first thing they tried is to run a management pod that contains standard tools such as 'df' and that mounts the host's filesystem. Unfortunately the act of scheduling a pod on that node causes it to experience disk pressure, and so the management pod gets evicted. So, being dogmatic about "the host should not have any tools installed" is good and all, but how do you debug this scenario without tools on the host? We eventually figured it out. By logging into the host OS and using the shell tools there.
- vbezhenar 2y agoIt's trivial to mount host root fs into container, so I don't see any issues "logging in" to it, as long as it works.
- antonvs 2y agoIt’s designed for running Kubernetes. You would log into the containers running on it if you need to, there’s no need to log into the underlying host. Managed Kubernetes clusters already work like this.
- cassianoleal 2y agoI’ve been writing and managing development platforms on Kubernetes for 6-7 years. In this time, I remember having to SSH into a host node exactly once. This was me, the platform engineer - not an application developer. Even then, having is a strong word. I could have just as well done with a privileged container with host access. Application developers have nothing to do on the host. As in, they gain nothing from it, and could potentially make everything worse for themselves and the other applications and teams on the platform.
- breadwinner 2y agoI was talking about logging into the container, not the node. Is Talos an OS for the node or the container?
- cassianoleal 2y agoFor the nodes. For the containers there's things like Google's distroless which are great for reducing attack surface as well.
- breadwinner 2y agoTheir website only says it is "for Kubernetes", that could mean containers, nodes or both.
- cassianoleal 2y ago> Supports cloud platforms, bare metal, and virtualization platforms > Production ready: supports some of the largest Kubernetes clusters in the world > It only takes 3 minutes to launch a Talos cluster on your laptop inside Docker. > delivers current stable Kubernetes Whilst you're not wrong, and the website could be clearer, there are plenty of clues.
- agilob 2y agoWe're long past the times when you would ssh into prod as root, run `nano /var/www/index.php` to fix a bug.
- deleted 2y ago[deleted]
- Veraticus 2y agoWe use Talos really extensively in production. It’s been an amazing solution for our Kubernetes clusters. Highly recommended for a really smart, really directed Linux distro.
- dtx1 2y agoIs this available and usable on a raspberry pi?
- sciencesama 2y agoYup !! If you want to save your sd card !!
- JustinGarrison 2y agoI’ll be doing a live stream setting it up on raspberry pi 4b tomorrow https://www.youtube.com/live/HsY8D9aO84Y?si=VL5LPG_M9GwfM7d_ https://www.youtube.com/live/HsY8D9aO84Y?si=VL5LPG_M9GwfM7d_ Talos doesn’t support older models (too slow) or the 5 yet (waiting for uboot support)
- dtx1 2y agonice, i'll be there.
- russfink 2y agoCan you install a shell? Does it work with MS Secure Boot? If not, what small distro might do that?
- hadlock 2y agoDepends on what you want to do with it. Alpine Linux is pretty popular for size optimized containers. I played around with Tiny Core Linux (TCL, used to just be tiny linux) back in the 5.0 days it is size optimized and supports desktop use in some forms.
- vbezhenar 2y agoIt's signed with their own keys, so you can use it with Secure Boot, but you need to enroll their keys on the first boot. They did't sign it with MS keys.
- amluto 2y agoI considered deploying Talos a few weeks ago, and I ran into this: https://github.com/siderolabs/talos/issues/8367 https://github.com/siderolabs/talos/issues/8367 Unless I’ve missed something, this isn’t a big deal in an AWS-style cloud where extra storage volumes (EBS, etc) have essentially no incremental cost, and maybe it’s okay on bare metal if the bare metal is explicitly designed with a completely separate boot disk (this includes Raspberry Pi using SD for boot and some other device for actual storage), but it seemed like a mostly showstopping issue for an average server that was specced with the intent to boot off a partition. I suppose one could fudge it with NVMe namespaces if the hardware cooperates. (I’ve never personally tried setting up a nontrivial namespace setup.) Has anyone set up Talos in a useful way on a server with a single disk or a single RAID array?
- rstat1 2y agoThis is my main issue with it right now, I'm wasting a whole physical disk in one of my home lab machines because Talos demands control over a full disk for its uses. This is one of the more annoying problems with any "immutable" distro (or at least the ones I've tried). They demand a very specific partition layout that basically forces you to devote a whole disk to them, which as you said in a cloud environment (or any other environment where VMs are used in place of physical machines) doesn't matter, but ends up mattering a lot when using physical machines.
- sroussey 2y agoI’m curious on your use case. I’ve always had boot disk or boot disk array separate from any data array. Are you referring to things like a home directory?
- amluto 2y agoI’m referring to a server without a dedicated boot disk. This is not especially rare. On an edge-style machine, you generally have a very limited number of M.2 slots, and dedicating one to a boot device uses up potentially 100% of your potential storage capacity, not to mention that an extra drive would be a non negligible fraction of total cost. On modern servers, there may not be a usable SATA controller, and NVMe usually costs 4 PCIe lanes. Those aren’t free. You also end up paying the absurd OEM premium for an extra disk if you go with a big name OEM. Sadly, there is no industry standard cheap, reliable boot device standard, at least as far as I’ve ever seen. Maybe someone should push USB3 for this use case — the price is certainly right, and performance is likely just fine.
- mrbluecoat 2y agoA related insightful read: https://www.siderolabs.com/blog/there-are-only-12-binaries-in-talos-linux/ https://www.siderolabs.com/blog/there-are-only-12-binaries-i...
- ksec 2y agoAround 90MB when downloaded compress. Not Sure am I the only one who is interested in its size when I see the word minimal.
- gotbeans 2y agoIt probably accounts for the k8s release unless they download it post-boot?
- JustinGarrison 2y agoK8s binaries are downloaded separately. Specifically they’re downloaded and run inside containers on the host.
- JustinGarrison 2y ago90MB is the size of the kernel, our init binary (written in go), the 12 binaries we include by default, and various required config files and drivers https://www.siderolabs.com/blog/there-are-only-12-binaries-in-talos-linux/ https://www.siderolabs.com/blog/there-are-only-12-binaries-i...
- rompledorph 2y agoThe documentation seems to be lacking. I am specifically interested in gvisor and kata support, but cannot find information on installing additional runtimes
- glitchcrab 2y agoThey are system extensions - https://www.talos.dev/v1.7/talos-guides/configuration/system-extensions/ https://www.talos.dev/v1.7/talos-guides/configuration/system... And there is a repo of them here - https://github.com/siderolabs/extensions https://github.com/siderolabs/extensions
- Szpadel 2y agoI'm surprised there is no support for cri-o
- glitchcrab 2y agoSame, I tried googling to see if there was any reason but I couldn't find anything helpful
- JustinGarrison 2y agoYou can include them and download an image from our image builder factory.talos.dev They’re even shareable images. Eg here’s the image with gvisor included https://factory.talos.dev/?arch=amd64&cmdline-set=true&extensions=-&extensions=siderolabs%2Fgvisor&platform=metal&target=metal&version=1.7.5 https://factory.talos.dev/?arch=amd64&cmdline-set=true&exten...
- Sparkyte 2y agoI'm comfortable with using alpine, is this another one of the many container solutions for tiny containers?
- JustinGarrison 2y agoTalos is for the host installation where the kubelet runs, not as a base OS for the base container image
- Sparkyte 2y agoOh I see what you mean! I think I'll keep with either centos or redhat due to the ability to receive updates on vulnerabilities then. FedRamp stuff does require CVEs to be handled in a timely fashion.
- JustinGarrison 2y agoCompliance and security are extremely important. Because of Talos’ single purpose nature and extreme small size it hasn’t needed patches for the recent “big” CVEs (xz utils, SSH, etc) because we don’t even have that software present. When you get sick of patching let us know
- JustinGarrison 2y agoFWIW Talos has DoD users from multiple countries. The areas that need a lot of security have repeatedly chosen Talos when they compare it to traditional Linux distros.
- deleted 2y ago[deleted]
- splix 2y agoWe've been using it for a while, and I'm absolutely happy with the project. Before that, we had a Kubespray based setup. It's a bunch of Ansible script and it allows to make any custom setup, like absolutely anything as you in control of the machines. But the other side of this is that it's extremely easy to break everything. Which we did a couple of times. And so any upgrade is a risk of loosing the whole cluster, so we decided it must be run in VM with full backup before each upgrade. Another problem that it takes about an hour to apply a change, because Ansible has to apply all the scripts each time. Then we migrated to Talos, and it's a day and night. The initial setup took like an hour, including reading the docs and a tutorial. Easy to setup, easy to maintain, easy to upgrade (and it takes minutes). Note that we run the nodes as VMs in Proxmox, so the disk and network setup are outside of Talos scope, as well as backups, and it's actually simplifies everything. So it "just works" and we can focus on your app not the cluster setup.
- JustinGarrison 2y agoThanks for the feedback and happy to hear you like Talos!
- apexalpha 2y agoI think a word is missing from the front page: Talos improves security further by mounting the root filesystem as read-only and removing any host-level such as a shell and SSH. After host-level, probably 'access'.
- JustinGarrison 2y agoThanks for the interest in Talos Linux! I work at Sidero (creators of Talos) and there are lots of “secure, immutable, and minimal” Linux distos out there. Something that Talos does differently is everything is an API. Machine configuration, upgrades, debugging…it’s all APIs. This helps with maintaining systems way beyond the usual cloud-init and systemd wrappers in other “minimal” distros. The second big change is Talos Linux is only designed for Kubernetes. It’s not a generic Linux kernel+container runtime. The init system was designed to run the kubelet and publish an API that feels like a Kubernetes native component. This drastically reduces the Linux knowledge required to run, scale, and maintain a complex system like Kubernetes. I’ve been doing a set of live streams called Talos Linux install fest walking new users through setting up their first cluster on Talos. Each install is in a new environment so please check it out. https://www.youtube.com/siderolabs/streams https://www.youtube.com/siderolabs/streams