8 ms·
There are only 12 binaries in Talos Linux
- cedws 3y ago>As opposed to systemd which is over 3000 lines of C code I’ll never comprehend. Well, technically true, but systemd is a whole lot more than 3000 lines... I can see another binary in the demo video called apid, does that one not count? Any comparison with Bottlerocket OS?
- JustinGarrison 3y agoMany of the binaries you see in the demo video are showing processes running from inside containers. There will be a lot more processes once you start pulling containers and starting containerized services. Notably the kubelet is also missing from the list because it's not built into the OS but pulled as needed from the correct version of Kubernetes requested. Bottlerocket runs systemd and also runs 2 versions of containerd. One for the system and one for workloads. This (in theory) hardens the OS more, but in practice makes things extremely annoying to manage because you have to get a shell on the host to access the API. disclaimer, I used to work at AWS on EKS and closely with the Bottlerocket team.
- bantunes 3y agoThe systemd hate is getting long in the tooth now. It's not like it doesn't do anything with its line count, or that the code is obfuscated.
- JustinGarrison 3y agoI’m actually a big fan of systemd. It’s an awesome, general purpose, and flexible init system. I don’t think the complexity it brings is required for Kubernetes.
- tehbeard 3y agoit could have been said without the x lines of code comment. "lines of code" is so often used as a "disparagement" about software rather than a metric for understandability. Something along the lines of "...not needing a general purpose init system that integrates with logging, network and mounting, when all we are running is Kubernetes."
- hosh 3y agoThere's work in a new kublet replacement that moves things that would normally go into daemonset into systemd (or something like systemd). There's also a neat feature of podman that runs pods as systemd units, which is a nice intermediate step between a more traditional pet server and a full kubernetes cluster.
- cpuguy83 3y agohttps://github.com/cpuguy83/containerd-shim-systemd-v1 https://github.com/cpuguy83/containerd-shim-systemd-v1 to do this with containerd.
- hosh 3y agoAlso this one: https://github.com/virtual-kubelet/systemk https://github.com/virtual-kubelet/systemk Although what I was thinking of was an article written somewhere and posted here in HN, and more a broad rethink on Kubernetes.
- hughesjj 3y agoThe second I learned how to write systems unit files was the second I evicted initv/rc.d scripts from my mind. Well, okay, from my search terms at least ;-) It can even kind of replace cron with timers, and no more mucking with grub. Also, true parallel init tasks. Love it.
- LAC-Tech 3y agoI'm not reading that as hate, I read that as criticising systemd in the context of a stripped down system designed to do one thing. I have systemd on the laptop I am typing on right now. Do I want it on some tiny embedded linux device? probably not.
- JustinGarrison 3y agoThanks for clarifying because that was the vibe I was going for (not hate). I'm using projectbluefin.io for all my laptops/desktops and love it. Wouldn't want the same on single-purpose, production servers.
- shrubble 3y agoSo there is a quantum of criticism/observations about systemd that can be made but after that, no more is acceptable?
- jchw 3y agoIn retrospect, it would've saved a lot of trouble and misunderstandings if systemd had called the init daemon "systemd-init" to make it clear that not literally everything that is under the umbrella is part of the init daemon.
- yjftsjthsd-h 3y agoEh... Most of the other components have a hard dependency on the init part; I'm not convinced that they're all that separate.
- jchw 3y agoIt's not the other way around though, which is a very important distinction. You don't need to use systemd-networkd or systemd-resolved or any manner of other things just to use the init daemon. The init daemon itself is extremely useful, and there are many machines that use the init daemon without most of the other services under the umbrella. It makes sense that a lot of the other services in systemd depend on the init daemon, it provides a lot of baseline services and features that are used for the rest of it. As a matter of fact, I don't even know what other init daemon I would choose if I wanted similar features around system daemon management, as there's a lot in the surface area that is genuinely useful. Honestly, there's a lot of useful stuff for handling secrets, handling UNIX domain sockets, temporary files, sandboxing apps, setting resource limits, managing unit lifecycles, etc. There are a few features I find somewhat more dubious (personally I'm not sold on DynamicUsers) but by and large I actually like a lot of the surface area systemd's init daemon provides and if I were to use something else I'd want something in a similar ballpark.
- Jumziey 3y agoSystemD bashing aside :p Talos is pretty awesome for setting up clusters. At home I just run talos with matchbox for PXE bootstrapping it works like a charm. Been really easy to maintain too. I normally just update matchbox and then reset a machine at a time with talos ctl for a clean install. It's something very reassuring with completely reset your machines so you know you could reinstall or replace them easily. Granted just used in a home setting running smaller workloads for backups, private projects, git etc.
- wmf 3y agoWhere is networking configured? I assume the system has to have an IP address before containerd can fetch images.
- JustinGarrison 3y agoEverything is API driven and static networking can be configured via kernel args https://www.talos.dev/latest/reference/configuration/v1alpha1/config/ https://www.talos.dev/latest/reference/configuration/v1alpha...
- burnte 3y agoDHCP I would assume.
- wmf 3y agoYeah but where is the DHCP client? In the kernel?
- andrewrynhard 3y agoIn machined (PID1 of Talos).
- funcDropShadow 3y agoAnd moving a network protocol implementation into PID1 is good why? So any security vulnerability in the DHCP implementation gives you root.
- birdiesanders 3y agoYou use machineConfigs that are used to provision the base OS and configure it, and the clusterConfig is used to bootstrap k8s on those machines. You can make subtypes and super types, you can have different networking setups, whatever you like, just apply and the OS is driven to state, then k8s is brought up from there. You are presented a kubeconfig after. Changes are done via application of updated machineConfig. Works great in practices and if you write an operator you can manage the config generation via k8s manifests and get wild with it.
- E39M5S62 3y agoIt's disingenuous to say that /sbin/init (machined/main.go) is less than 400 lines of code. Sure, that file is. What about all of the in-tree modules that are being imported? A super lazy summing of Go lines in the master branch of the repo: $ find . -name *.go | xargs wc -l | tail -1 354085 total Heck, there are almost 100k lines under internal/app! $ find internal/app -name *.go | xargs wc -l | tail -1 96885 total I'm curious what argument you are making here with regards to the number of lines in a single file.
- JustinGarrison 3y agoI know there are a lot more lines and I didn’t count any of the imports from systemd either. 300 loc (machined) vs 3000 loc (systemd) was the closest comparison I could think of without crawling all imports and deps. Would be happy to update with a different comparison you think is more fair.
- raziel2p 3y agoHow about we just don't compare lines of code at all, as if it's a useful metric of anything?
- Karellen 3y ago"Measuring programming progress by lines of code is like measuring aircraft building progress by weight." -- attributed to Bill Gates
- coldtea 3y agoFor measuring bloat however they're a good proxy.
- deleted 3y ago[deleted]
- JustinGarrison 3y agoIs there a metrics that can convey the complexity of a general purpose init system like systemd vs a single purpose init like Talos' machined? That is what I was trying to convey and couldn't find a reasonable metric.
- abound 3y agoBig fan of Talos, have used it in some homelab + cloud clusters over the years, currently powers all my self-hosting. The `talosctl` command is great, and any time you need to do node-level debugging, there's always something like node-shell [1]. [1] https://github.com/kvaps/kubectl-node-shell https://github.com/kvaps/kubectl-node-shell
- cperciva 3y agoThe /sbin/init binary is hard linked to /sbin/dashboard, /sbin/poweroff, /sbin/shutdown, and /sbin/wrapperd. While this technically is 5 files, it’s a single file hard linked 4 times to provide convenience commands. Err, that's definitely 1 file with 5 directory entries, not 5 files.
- titanomachy 3y agoWonder why they would use hard links instead of symlinks. Edit: interesting, seems like there's a mild performance benefit. https://unix.stackexchange.com/a/20716 https://unix.stackexchange.com/a/20716
- 0xbadcafebee 3y agoThere's a lot of benefits actually, as symbolic links often need to be handled as a completely different type of file with different semantics which often leads to bugs. Hard links are always better when you know you are dealing with a single static host & filesystem. Symbolic links in such a case are only better for indicating quickly to the user which files are linked to which others.
- vundercind 3y agoReverse question: why use symlinks when you can get away with hard links?
- AtlasBarfed 3y agoMain thing I've seen with hard links is that deletions delete the source file which about 90% of the time isn't what an end user wants
- whartung 3y agoOnly if its the last hard link. If nothing else, its a wee bit of insurance from deletion since no single link removal should remove the file. Anecdote, eons ago, we had a problem where the vendor needed to log in to the machine with the intent that they were going to upload some utilities, fix a problem, and then delete them. Before I let them in, I set up a script that constantly scanned the directory tree they were in, and hard linked everything so I could look at what they were using later.
- rwiggins 3y agoSuper cool. I always enjoy reading about systems that challenge, well, "ossified" assumptions. An OS not providing a shell, for example? Madness! ... or is it genius, if the OS has a specific purpose...? It's thought-provoking, if nothing else. I'm a bit skeptical of parts. For instance, the "init" binary being less than 400 lines of golang - wow! And sure, main.go [1] is less than 400 lines and very readable. Then you squint at the list of imported packages, or look to the left at the directory list and realize main.go isn't nearly the entire init binary. That `talosctl list` invocation [2] didn't escape my notice either. Sure, the base OS may have only a handful of binaries - how many of those traditional utilities have been stuffed into the API server? Not that I disagree with the approach! I think every company eventually replaces direct shell access with a daemon like this. It's just that "binary footprint" can get a bit funny if you have a really sophisticated API server sitting somewhere. [1]: https://github.com/siderolabs/talos/blob/main/internal/app/machined/main.go https://github.com/siderolabs/talos/blob/main/internal/app/m... [2]: https://www.talos.dev/v1.6/reference/cli/#talosctl-list https://www.talos.dev/v1.6/reference/cli/#talosctl-list
- sspiff 3y agoExactly this. I was thinking of making a similar comment but you made it far better than I could. Number of binaries is kind of a meaningless metric, especially for a system that historically follows the UNIX philosphy of each program doing one thing. Sure, a shell is complicated and a potential risk, and perhaps it's a good idea to exclude from the base system in this context. But I'd rather have ls, tr and wc on my system than some bespoke, all-encompassing API service that has been far less battle tested providing similar functionality. And like you rightly pointed out, these new binaries all contain their own list of dependencies which are pulled in at build time and need to be taken into scope as well. That's not to say Talos or its approach doesn't hold merit, but I think it's a little disengenious to simply point at the number of binaries.
- JustinGarrison 3y agoI agree number of binaries is an arbitrary metric but also an indicator that things work differently with Talos. You have to use the declarative API for management which some people could see as a bad thing. I’d also like to point out that the system API is designed to be extendable and adaptable to different operating systems. We’d love for more vendors to create adapters/shims to get the benefits of API managed Linux https://github.com/cosi-project/community https://github.com/cosi-project/community
- 0xbadcafebee 3y ago20 years ago, I used to make custom Linux distros for fun. Floppy distros, CDROM distros, RAM-resident distros, network-boot distros. In a few of them, I custom-made my own binary that was both the init system, and a few applications, stripped it down, and shipped just that as the distro (basically just a few files and my static binary). A lot of people downloaded them, and it was great fun - to start. Problem is when you want to do more things. You have to start finding workarounds to bolt-on additional tools, or maybe you just throw one or two extra tools in there by default. Over time you find more and more missing things or incompatibilities with other systems, which make it harder to cover more use cases. And finally you realize that "the tiniest system" is a lot more effort than it's worth, and what you really want is "a slim yet compatible system". The system you end up with is a lot fatter, but a lot less headache. (The security benefits of fewer files are overblown, too. If you audit and harden the system, it doesn't matter how many binaries you have, because the attack vectors they use will be mitigated)
- andrewrynhard 3y agoIn the case of Talos, Kubernetes can provide the flexibility you want from a more traditional Linux distribution.
- AtlasBarfed 3y agoThat sounds like Kool aid. Can you expound more on this?
- andrewrynhard 3y agoWith Kubernetes you can schedule workloads in a number of different ways. Let’s say you insisted on having a shell and package manager. Run your favorite distro’s container as a DaemonSet. With the proper mounts and permissions you can do a lot. In other words use Kubernetes to do the things need to do. Then what role does the OS really play? Well in the case of Talos it’s only there to run Kubernetes.
- birdiesanders 3y ago
- andPerSand 3y ago[dead]
- miki123211 3y agoI wonder how a really slimmed down distro like Alpine would compare here, particularly in terms of image size. It offers most of the standard Linux utilities we know and love, but most of them are actually just symlinks to Busybox, which is ~900K on my (ARM64) system. That's less than a hello world in Go, for a program that can replace most common Linux utilities in daily usage.
- JustinGarrison 3y agoBusybox is only 1 binary symlinked hundreds of times
- Brian_K_White 3y agoYes, and? (They and we all know that. They said as much theirself right in the comment.)
- JustinGarrison 3y agoThe benefit of Talos isn't low binary count but that _can_ reduce the amount of maintenance required. The benefit is declarative API driven management. You spend less time automating a system to a desired state in a similar vein Kubernetes provides a declarative API.
- Brian_K_White 3y agoWhat does any of this have to do with my question? You just restated a fact, that busybox is a binary with a lot of symlinks, but why? So what? Yes, also the sky is blue and water is wet. It's like you didn't read the comment you responded to. It wasn't even about the benefit of Talos.
- birdiesanders 3y agoWe operate talos and alpine based nodes, many thousands of them. The build chain for alpine is many orders of magnitude more complex than the build pipe for our talos image modifications. Alpine is really not made for doing a lot of “host” tasks and needs much coercion to get it to be capable of running something like k3s, and much more complex to get kubeadm clusters running on it. In the end the complexity is required for flexibility, alpine nodes can be modified on a whim, talos is R/O and ephemeral, but more secure.
- suralind 3y agoBig fan of Talos, I use it on Hetzner and it's a joy!
- ofrzeta 3y agoHow do you install it?
- xcdzvyn 3y agoI see Talos only supports XFS, what potential reasons could they have to prefer XFS to competitors? I've always struggled to compare filesystems fairly. My justification for ext4 is just that everybody else uses it :)
- cchance 3y agoxfs is technically more robust and performant than ext4, ext4's just the most widely supported, but most use xfs when available
- rickette 3y agoMongo and Elasticsearch also recommend using XFS. So Talos isn't unique in this. XFS is somewhat more often preferred in data intensive systems.