5 ms·

I find kubernetes - systemd overlap angle as a very valid point. When I first started to learn kubernetes more often than not I was thinking 'hey, we can do tha

by betaby 4y ago

I find kubernetes - systemd overlap angle as a very valid point. When I first started to learn kubernetes more often than not I was thinking 'hey, we can do that with systemd as well!'

e1g 4y ago
Same - we now use systemd to sandbox our apps, set resource limits, hide the rest of the system, mount read-only FS, etc. Add a custom SELinux policy for your service, and you get a compelling alternative to containerization (at least at runtime).
- GordonS 4y ago
  Do you have a blog perchance? I'd love to read more about this, maybe with fully-featured examples.
  - e1g 4y ago
    No, but here is one of our service files for systemd to manage a Node.js backend instance. Could be a useful starting point to create your own https://gist.github.com/eugene1g/22aa2b2c8897a29a54fba876b0cf88dd https://gist.github.com/eugene1g/22aa2b2c8897a29a54fba876b0c... Many sandboxing features require a relatively modern systemd and will do nothing on older distros (we run RHEL9).
    - GordonS 4y ago
      This is great, thanks, and I appreciate that's it's commented too! Also like the "negative" commented out config, that's something I often do myself with footguns that feel "obvious".
- mike_hearn 4y ago
  Yes, I reached the same conclusion when setting up the (fairly simple) servers needed for my company. SystemD + standard scripting can take you a long way, perhaps with Ansible if you need more. I've only used Kubernetes a bit, but I used Borg for quite some years. A fantastic system if you need an incredibly regularized prod environment with lots of replicas for everything i.e. if you're trying to replicate Google's internal setup. Extremely painful the moment you need to step outside that world. I had to do that once or twice at Google and quickly started wishing I could just grab some normal Linux machines and install some packages that depended on Apache and a DB. Actually said company makes packaging tools and Linux packages are fairly simple so we started with the ability to make server packages that use systemd. That support is still there and makes it fairly easy to build a signed apt repository from build system outputs (e.g. jars, native binaries), or even by re-packaging an existing server download [1]. What you get out is debs that start up the service on install, ensure it starts automatically at boot, restarts it across upgrades and shuts it down cleanly if the machine shuts down. Such packages can be built from any machine including Windows and macOS. The nice thing about doing packaging this way is you can specify dependencies (both install and service startup) on things like postgresql, and you get a template nginx/apache2 reverse proxy config as well. Unfortunately there doesn't seem to be a way to make those templates 100% usable out of the box due to limits in their config languages, but it's still nice to have. There's also pre-canned config snippets for sandboxing, setting up cron jobs and other useful systemd features [2]. We package and run all our servers this way. One big rented colo machine is sufficient right now. If there was a need for clustering we'd just point the machines at the same apt repo and use some shell scripting or Ansible, I think. There's lots of tools for making Docker containers out there already, but I never really clicked with Docker for various reasons. Conveyor is mostly focused on desktop apps but perhaps there's more demand for easily making systemd using packages than I thought. The server side support is there and we dogfood it, but isn't really advertised and it lacks a tutorial. [1] https://conveyor.hydraulic.dev/2.1/tasks/server/ https://conveyor.hydraulic.dev/2.1/tasks/server/ [2] https://conveyor.hydraulic.dev/2.1/stdlib/systemd/ https://conveyor.hydraulic.dev/2.1/stdlib/systemd/
- xani_ 4y ago
  More than that, you can automatically mount whole block device (or file-as-device), even run integrity checks on it. If someone is building say an embedded system that has to run different services (say a NAS, or a router), it's pretty compelling alternative to a bit more heavy weight containers like docker.
- sass_muffin 4y ago
  How do you scale across multiple computers reliably?
- doublerebel 4y ago
  This sounds like SMF on Solaris (and derivatives). I used that pattern successfully on Illumos/SmartOS for several years.
politelemon 4y ago
I noticed the same but came away with a different conclusion, "hey, k8s is reimplementing the OS.". In other words, concepts which have existed for years, battle tested and well known, are new and untested in the k8s world.
- patrickthebold 4y ago
  But k8s is distributed across several computers. If you could have Linux manage resources and run processes across several computers I think people would do that.
  - pjmlp 4y ago
    As in, https://en.wikipedia.org/wiki/Beowulf_cluster https://en.wikipedia.org/wiki/Beowulf_cluster https://en.wikipedia.org/wiki/MOSIX https://en.wikipedia.org/wiki/MOSIX That is the thing with repeating history.
- andrewstuart2 4y ago
  They're not even really new or untested, just being done at a different plane of abstraction. Running a system efficiently across multiple machines requires different implementations of familiar abstractions than running on a single machine. Even the single machine approach has changed dramatically in the last 20+ years. We started single program, went to single user, to multi-user, to multi-processor, to multi-core, to NUMA, etc (not necessarily in that order), and a significant subset of systems are now multi-machine, even if you ignore the internet. All of these new paradigms tend to have similar base requirements/optimizations that would kind of be pointless to reinvent versus simply adapting to the new infrastructure style.
  - andrewstuart 4y ago
    Hey! We're both here in the same thread.....
- goodpoint 4y ago
  > concepts which have existed for years, battle tested and well known, are new and untested in the k8s world ...and they also create lock-in wrt k8s.
JanMa 4y ago
I often noticed this as well. After reading the article I wonder: Would it be feasible to write a kubernetes-shim for systemd to reduce the overlap instead of reinventing everything? This would not fix any of the valid criticism the author has of systemd but instead make it do even more things. Nevertheless I somehow find the idea very intriguing
- rckclmbr 4y ago
  CoreOS did pretty much this (minus k8s, but same abstraction), and I loved it. Much easier for simple deployments, harder for anything else
pram 4y ago
This is how fleet on CoreOS was designed. It just managed systemd containers across your nodes (IIRC its been a while)
- pengaru 4y ago
  More like it managed systemd services/units across the cluster, like a distributed systemd. That was still early days for systemd-nspawn, folks would be using docker containers under systemd services. And that caused heaps of other problems because docker was aspiring to do PID 1's jobs as well.
arinlen 4y ago
> 'hey, we can do that with systemd as well!' But does systemd do these things in a user-friendly way? Or at least as friendly as Kubernetes? Unfortunately, no. Don't underestimate the importance of UX.
- master_crab 4y ago
  Systemd do things in as friendly a way as kubernetes? No one should ever confuse kubernetes for user-friendly.
  - arinlen 4y ago
    > No one should ever confuse kubernetes for user-friendly. Kubernetes is user-friendly, specially when compared to each and any of its alternatives. And moreso when compared with systemd. We live in a day and age where it's possible to get a whole web app up and running in a Kubernetes cluster from a fresh Ubuntu install with a single snap installation and a kubectl apply -k <kustomize dir>. How long would it take to get systemd to containerize a single app?
    - tadfisher 4y ago
      systemd-nspawn -x -a [command] [args]
    - mike_hearn 4y ago
      Isn't that skipping the work involved in writing the kustomize directory? Seems like Kustomize [1] was explicitly created to make Kubernetes easier to use, also. Anyway, you're right in the general case, but specific cases really depend on how complicated your app actually is. If we're allowed to introduce extra tools then gosh darn it I'm going to promote Conveyor again because in that case it gets easier to use systemd too. Here's what a server config looks like: https://gist.github.com/mikehearn/5485a7343d9fe838d33d0b028161ae35 https://gist.github.com/mikehearn/5485a7343d9fe838d33d0b0281... All of ~25 lines, some of which is just optional demo stuff. To use it you'd compile your app (a JVM app in this case), run "conveyor make debian-package", upload the resulting package and install it with "apt install ./whatever.deb". Or alternatively upload it to a static file server (s3 bucket or whatever) and then run "apt-get update && apt-get upgrade" on each host. The server will start/restart automatically. It's not containerized in the Docker sense but it does run in a lightweight sandbox using the DynamicUser feature, and you can lock it down further if you want by setting the right systemd keys. Now, you're going to say that Kubernetes does a lot more for you, that it can deploy many kinds of apps simultaneously, configure networking, let replicas find each other etc and so it's easier to use for 'real' apps. Granted, all true. The above workflow isn't optimized, does less and would need more work to be competitive. Also the resulting packages assume there's an apt repository somewhere, which takes a bit of work to set up (this isn't strictly needed in the server case and we'll fix it at some point). Still, whilst maybe it's better for everyone to just learn Kubernetes at some point, but a lot of us have learned Linux/UNIX in the past and have needs that can be met cheaply with that toolset. [1] https://kustomize.io/ https://kustomize.io/

by betaby 4y ago

I find kubernetes - systemd overlap angle as a very valid point. When I first started to learn kubernetes more often than not I was thinking 'hey, we can do that with systemd as well!'

e1g 4y ago

Same - we now use systemd to sandbox our apps, set resource limits, hide the rest of the system, mount read-only FS, etc. Add a custom SELinux policy for your service, and you get a compelling alternative to containerization (at least at runtime).

GordonS 4y ago
Do you have a blog perchance? I'd love to read more about this, maybe with fully-featured examples.
- e1g 4y ago
  No, but here is one of our service files for systemd to manage a Node.js backend instance. Could be a useful starting point to create your own https://gist.github.com/eugene1g/22aa2b2c8897a29a54fba876b0cf88dd https://gist.github.com/eugene1g/22aa2b2c8897a29a54fba876b0c... Many sandboxing features require a relatively modern systemd and will do nothing on older distros (we run RHEL9).
  - GordonS 4y ago
    This is great, thanks, and I appreciate that's it's commented too! Also like the "negative" commented out config, that's something I often do myself with footguns that feel "obvious".
mike_hearn 4y ago
Yes, I reached the same conclusion when setting up the (fairly simple) servers needed for my company. SystemD + standard scripting can take you a long way, perhaps with Ansible if you need more. I've only used Kubernetes a bit, but I used Borg for quite some years. A fantastic system if you need an incredibly regularized prod environment with lots of replicas for everything i.e. if you're trying to replicate Google's internal setup. Extremely painful the moment you need to step outside that world. I had to do that once or twice at Google and quickly started wishing I could just grab some normal Linux machines and install some packages that depended on Apache and a DB. Actually said company makes packaging tools and Linux packages are fairly simple so we started with the ability to make server packages that use systemd. That support is still there and makes it fairly easy to build a signed apt repository from build system outputs (e.g. jars, native binaries), or even by re-packaging an existing server download [1]. What you get out is debs that start up the service on install, ensure it starts automatically at boot, restarts it across upgrades and shuts it down cleanly if the machine shuts down. Such packages can be built from any machine including Windows and macOS. The nice thing about doing packaging this way is you can specify dependencies (both install and service startup) on things like postgresql, and you get a template nginx/apache2 reverse proxy config as well. Unfortunately there doesn't seem to be a way to make those templates 100% usable out of the box due to limits in their config languages, but it's still nice to have. There's also pre-canned config snippets for sandboxing, setting up cron jobs and other useful systemd features [2]. We package and run all our servers this way. One big rented colo machine is sufficient right now. If there was a need for clustering we'd just point the machines at the same apt repo and use some shell scripting or Ansible, I think. There's lots of tools for making Docker containers out there already, but I never really clicked with Docker for various reasons. Conveyor is mostly focused on desktop apps but perhaps there's more demand for easily making systemd using packages than I thought. The server side support is there and we dogfood it, but isn't really advertised and it lacks a tutorial. [1] https://conveyor.hydraulic.dev/2.1/tasks/server/ https://conveyor.hydraulic.dev/2.1/tasks/server/ [2] https://conveyor.hydraulic.dev/2.1/stdlib/systemd/ https://conveyor.hydraulic.dev/2.1/stdlib/systemd/
xani_ 4y ago
More than that, you can automatically mount whole block device (or file-as-device), even run integrity checks on it. If someone is building say an embedded system that has to run different services (say a NAS, or a router), it's pretty compelling alternative to a bit more heavy weight containers like docker.
sass_muffin 4y ago
How do you scale across multiple computers reliably?
doublerebel 4y ago
This sounds like SMF on Solaris (and derivatives). I used that pattern successfully on Illumos/SmartOS for several years.

politelemon 4y ago

I noticed the same but came away with a different conclusion, "hey, k8s is reimplementing the OS.". In other words, concepts which have existed for years, battle tested and well known, are new and untested in the k8s world.

patrickthebold 4y ago
But k8s is distributed across several computers. If you could have Linux manage resources and run processes across several computers I think people would do that.
- pjmlp 4y ago
  As in, https://en.wikipedia.org/wiki/Beowulf_cluster https://en.wikipedia.org/wiki/Beowulf_cluster https://en.wikipedia.org/wiki/MOSIX https://en.wikipedia.org/wiki/MOSIX That is the thing with repeating history.
andrewstuart2 4y ago
They're not even really new or untested, just being done at a different plane of abstraction. Running a system efficiently across multiple machines requires different implementations of familiar abstractions than running on a single machine. Even the single machine approach has changed dramatically in the last 20+ years. We started single program, went to single user, to multi-user, to multi-processor, to multi-core, to NUMA, etc (not necessarily in that order), and a significant subset of systems are now multi-machine, even if you ignore the internet. All of these new paradigms tend to have similar base requirements/optimizations that would kind of be pointless to reinvent versus simply adapting to the new infrastructure style.
- andrewstuart 4y ago
  Hey! We're both here in the same thread.....
goodpoint 4y ago
> concepts which have existed for years, battle tested and well known, are new and untested in the k8s world ...and they also create lock-in wrt k8s.

JanMa 4y ago

I often noticed this as well. After reading the article I wonder: Would it be feasible to write a kubernetes-shim for systemd to reduce the overlap instead of reinventing everything? This would not fix any of the valid criticism the author has of systemd but instead make it do even more things. Nevertheless I somehow find the idea very intriguing

rckclmbr 4y ago
CoreOS did pretty much this (minus k8s, but same abstraction), and I loved it. Much easier for simple deployments, harder for anything else

pram 4y ago

This is how fleet on CoreOS was designed. It just managed systemd containers across your nodes (IIRC its been a while)

pengaru 4y ago
More like it managed systemd services/units across the cluster, like a distributed systemd. That was still early days for systemd-nspawn, folks would be using docker containers under systemd services. And that caused heaps of other problems because docker was aspiring to do PID 1's jobs as well.

arinlen 4y ago

> 'hey, we can do that with systemd as well!' But does systemd do these things in a user-friendly way? Or at least as friendly as Kubernetes? Unfortunately, no. Don't underestimate the importance of UX.

master_crab 4y ago
Systemd do things in as friendly a way as kubernetes? No one should ever confuse kubernetes for user-friendly.
- arinlen 4y ago
  > No one should ever confuse kubernetes for user-friendly. Kubernetes is user-friendly, specially when compared to each and any of its alternatives. And moreso when compared with systemd. We live in a day and age where it's possible to get a whole web app up and running in a Kubernetes cluster from a fresh Ubuntu install with a single snap installation and a kubectl apply -k <kustomize dir>. How long would it take to get systemd to containerize a single app?
  - tadfisher 4y ago
    systemd-nspawn -x -a [command] [args]
  - mike_hearn 4y ago
    Isn't that skipping the work involved in writing the kustomize directory? Seems like Kustomize [1] was explicitly created to make Kubernetes easier to use, also. Anyway, you're right in the general case, but specific cases really depend on how complicated your app actually is. If we're allowed to introduce extra tools then gosh darn it I'm going to promote Conveyor again because in that case it gets easier to use systemd too. Here's what a server config looks like: https://gist.github.com/mikehearn/5485a7343d9fe838d33d0b028161ae35 https://gist.github.com/mikehearn/5485a7343d9fe838d33d0b0281... All of ~25 lines, some of which is just optional demo stuff. To use it you'd compile your app (a JVM app in this case), run "conveyor make debian-package", upload the resulting package and install it with "apt install ./whatever.deb". Or alternatively upload it to a static file server (s3 bucket or whatever) and then run "apt-get update && apt-get upgrade" on each host. The server will start/restart automatically. It's not containerized in the Docker sense but it does run in a lightweight sandbox using the DynamicUser feature, and you can lock it down further if you want by setting the right systemd keys. Now, you're going to say that Kubernetes does a lot more for you, that it can deploy many kinds of apps simultaneously, configure networking, let replicas find each other etc and so it's easier to use for 'real' apps. Granted, all true. The above workflow isn't optimized, does less and would need more work to be competitive. Also the resulting packages assume there's an apt repository somewhere, which takes a bit of work to set up (this isn't strictly needed in the server case and we'll fix it at some point). Still, whilst maybe it's better for everyone to just learn Kubernetes at some point, but a lot of us have learned Linux/UNIX in the past and have needs that can be met cheaply with that toolset. [1] https://kustomize.io/ https://kustomize.io/