3 ms·

20 years ago, I used to make custom Linux distros for fun. Floppy distros, CDROM distros, RAM-resident distros, network-boot distros. In a few of them, I custom

20 years ago, I used to make custom Linux distros for fun. Floppy distros, CDROM distros, RAM-resident distros, network-boot distros. In a few of them, I custom-made my own binary that was both the init system, and a few applications, stripped it down, and shipped just that as the distro (basically just a few files and my static binary). A lot of people downloaded them, and it was great fun - to start. Problem is when you want to do more things. You have to start finding workarounds to bolt-on additional tools, or maybe you just throw one or two extra tools in there by default. Over time you find more and more missing things or incompatibilities with other systems, which make it harder to cover more use cases. And finally you realize that "the tiniest system" is a lot more effort than it's worth, and what you really want is "a slim yet compatible system". The system you end up with is a lot fatter, but a lot less headache. (The security benefits of fewer files are overblown, too. If you audit and harden the system, it doesn't matter how many binaries you have, because the attack vectors they use will be mitigated)

andrewrynhard 3y ago
In the case of Talos, Kubernetes can provide the flexibility you want from a more traditional Linux distribution.
- AtlasBarfed 3y ago
  That sounds like Kool aid. Can you expound more on this?
  - andrewrynhard 3y ago
    With Kubernetes you can schedule workloads in a number of different ways. Let’s say you insisted on having a shell and package manager. Run your favorite distro’s container as a DaemonSet. With the proper mounts and permissions you can do a lot. In other words use Kubernetes to do the things need to do. Then what role does the OS really play? Well in the case of Talos it’s only there to run Kubernetes.
  - birdiesanders 3y ago
    You literally do not need more than the most basic of host-OS facilities to run any Linux workload you want in a pod, the OS just needs to run containerd and have a kernel, talos adds the management plane to that mix to make it useable, but the userland provided by the oci image will not see any difference at all than if it were running on Ubuntu; it will see the kernel.
JustinGarrison 3y ago
We use “system extensions” to give you flexibility while keeping the base small. Want GPU drivers? Add the extension. Need Tailscale? Extension. https://www.talos.dev/v1.6/talos-guides/configuration/system-extensions/ https://www.talos.dev/v1.6/talos-guides/configuration/system...
- Brian_K_White 3y ago
  How is that different than adding binaries, in the context of this comment where the point was "In the end you end up just adding everything back in that you originally took out, because managing all the little weird different subsets is not worth the benefit." In the context of that assertion, adding or subtracting "extensions" and adding or subtracting binaries are equivalent. Both are adding or subtracting "piece of code that provides fuction".
  - JustinGarrison 3y ago
    They are similar but extensions don't have to be binaries. It can also be files you need to be available before Kubernetes starts. Talos is purpose built to run Kubernetes workloads and not general purpose Linux. Hopefully, you don't have to add _everything_ back to the OS, but we know some things cannot run as a container or Kubernetes workload. Extensions are required for specialized hardware (eg network, GPUs) and is the closest thing to a "package manager" available in Talos. Extensions can be binaries but don't have to be. We have a lot of common extensions provided and maintained by us but anyone can create extensions as needed. One nice thing about extensions is they get layered and you don't have to pre-build an artifact like you do with other Linux distros with something like packer. factory.talos.dev will let you pick you extensions and get an artifact no packer/bash/config management required.
    - 0xbadcafebee 3y ago
      Like the other commenter said, this is the kind of thing custom distros always have to do, that lead to either reinventing or just going back to a big fat distro. I've gone around this wheel a couple times. It's a nice idea to have something custom-tailored, but a tailored suit only fits one person, and is expensive. You can still build a system to manage Kubernetes nodes without making your own distro. Even a heavily modified stock distro gains you benefits from basing off someone else's work. You can reuse the solutions they've made, and contribute back your customizations for your specific use cases. That's how today's distro installers/package managers/etc came to include all the functionality they have. You couldn't eject a CDROM from a Busybox system, until one weird kid in high school decided to try to use Busybox to make a CD-bootable RAM-resident distro, found out it had no 'eject' command, and then sent a patch in to Busybox to add it. Now everyone can use that command, and that functionality is still there 20 years later. It's also a lot harder for users to use proprietary solutions than ones they're familiar with. Your OS has no shell or console, only an API? So if there's a problem, how do I drop in with gdb, strace, tcpdump, and the entire suite of Linux debugging tools, to try and quickly diagnose and then patch an issue? I'm sure you've created some way to do it, but now I need to go find out how to do it, and probably use whatever stock tools are there, which may have their own quirks or incompatibilities. But I get that a corporation's interest is mostly in "get something working now" as opposed to "get something working that will be better in the long run". DIY/NIH often becomes the engineering department's watchwords, and a custom distro is one of those eventualities.
Dwedit 3y ago
MX Linux would be that "slim-yet-compatible" system. It can load itself completely into RAM, and has many distro-specific components such as Frugal Install and Remastering, yet also lets you install standard Debian packages.

andrewrynhard 3y ago

In the case of Talos, Kubernetes can provide the flexibility you want from a more traditional Linux distribution.

AtlasBarfed 3y ago
That sounds like Kool aid. Can you expound more on this?
- andrewrynhard 3y ago
  With Kubernetes you can schedule workloads in a number of different ways. Let’s say you insisted on having a shell and package manager. Run your favorite distro’s container as a DaemonSet. With the proper mounts and permissions you can do a lot. In other words use Kubernetes to do the things need to do. Then what role does the OS really play? Well in the case of Talos it’s only there to run Kubernetes.
- birdiesanders 3y ago
  You literally do not need more than the most basic of host-OS facilities to run any Linux workload you want in a pod, the OS just needs to run containerd and have a kernel, talos adds the management plane to that mix to make it useable, but the userland provided by the oci image will not see any difference at all than if it were running on Ubuntu; it will see the kernel.

JustinGarrison 3y ago

We use “system extensions” to give you flexibility while keeping the base small. Want GPU drivers? Add the extension. Need Tailscale? Extension. https://www.talos.dev/v1.6/talos-guides/configuration/system-extensions/ https://www.talos.dev/v1.6/talos-guides/configuration/system...

Brian_K_White 3y ago
How is that different than adding binaries, in the context of this comment where the point was "In the end you end up just adding everything back in that you originally took out, because managing all the little weird different subsets is not worth the benefit." In the context of that assertion, adding or subtracting "extensions" and adding or subtracting binaries are equivalent. Both are adding or subtracting "piece of code that provides fuction".
- JustinGarrison 3y ago
  They are similar but extensions don't have to be binaries. It can also be files you need to be available before Kubernetes starts. Talos is purpose built to run Kubernetes workloads and not general purpose Linux. Hopefully, you don't have to add _everything_ back to the OS, but we know some things cannot run as a container or Kubernetes workload. Extensions are required for specialized hardware (eg network, GPUs) and is the closest thing to a "package manager" available in Talos. Extensions can be binaries but don't have to be. We have a lot of common extensions provided and maintained by us but anyone can create extensions as needed. One nice thing about extensions is they get layered and you don't have to pre-build an artifact like you do with other Linux distros with something like packer. factory.talos.dev will let you pick you extensions and get an artifact no packer/bash/config management required.
  - 0xbadcafebee 3y ago
    Like the other commenter said, this is the kind of thing custom distros always have to do, that lead to either reinventing or just going back to a big fat distro. I've gone around this wheel a couple times. It's a nice idea to have something custom-tailored, but a tailored suit only fits one person, and is expensive. You can still build a system to manage Kubernetes nodes without making your own distro. Even a heavily modified stock distro gains you benefits from basing off someone else's work. You can reuse the solutions they've made, and contribute back your customizations for your specific use cases. That's how today's distro installers/package managers/etc came to include all the functionality they have. You couldn't eject a CDROM from a Busybox system, until one weird kid in high school decided to try to use Busybox to make a CD-bootable RAM-resident distro, found out it had no 'eject' command, and then sent a patch in to Busybox to add it. Now everyone can use that command, and that functionality is still there 20 years later. It's also a lot harder for users to use proprietary solutions than ones they're familiar with. Your OS has no shell or console, only an API? So if there's a problem, how do I drop in with gdb, strace, tcpdump, and the entire suite of Linux debugging tools, to try and quickly diagnose and then patch an issue? I'm sure you've created some way to do it, but now I need to go find out how to do it, and probably use whatever stock tools are there, which may have their own quirks or incompatibilities. But I get that a corporation's interest is mostly in "get something working now" as opposed to "get something working that will be better in the long run". DIY/NIH often becomes the engineering department's watchwords, and a custom distro is one of those eventualities.

Dwedit 3y ago

MX Linux would be that "slim-yet-compatible" system. It can load itself completely into RAM, and has many distro-specific components such as Frugal Install and Remastering, yet also lets you install standard Debian packages.