4 ms·

Not sure what you’re asking about. If Systemd didn’t start it, Systemd doesn’t manage it. Nothing strange about that…? Systemd does not start all the Kubernetes

by fuzzy2 5y ago

Not sure what you’re asking about. If Systemd didn’t start it, Systemd doesn’t manage it. Nothing strange about that…? Systemd does not start all the Kubernetes stuff. k3s could stop everything, but it doesn’t. It’s a design decision.

mongol 5y ago
It did start it. For example the k3s process did start a container registry, but it did not stop it. I thought one of the points of systemd was that it would reap parentless processes
- c0l0 5y ago
  Which processes systemd allows to live on after stopping an associated service depends on the kind of KillMode= (see systemd.kill(5)) that was specified for the service unit. The default ('control-group') is usually what you want, esp. if you are looking to not recreate the kind of chaos improper process management SysV-style init scripts frequently caused. Chances are the k3s service unit specifies something else (like 'none') - whether or not that is actually wise would probably require some research.
  - mongol 5y ago
    Thanks for explaining that. My comment was not a criticism against systemd, more a surprise that k3s did something that was not expected by me when I interacted with it through systemd. For example, httpd does not supply a kill-all-httpd.sh script as far as I know. Perhaps there are systemctl verbs that can be added to support what k3s devs apparently thought neccessary to implement in a separate script.
- justinsaccount 5y ago
  It does exactly what the configuration tells it to do. At one point some distributions accidentally configured the sshd or cron unit to do that. People were a bit surprised when restarting sshd or cron killed every single background child process. See https://www.freedesktop.org/software/systemd/man/systemd.kill.html#KillMode= https://www.freedesktop.org/software/systemd/man/systemd.kil... Not to mention if k3s is just managing containers running under docker or containerd, those processes are not parentless just because k3s stopped.
  - mongol 5y ago
    My point is, this feels like an area where there is more work to do. Not surprising, the container landscape is a relatively new and rapidly evolving field that hardly existed when systemd was invented. But the fact that a piece of software supplies a killall.sh script reminds me of the old situation before systemd, when every software provided their own lifecycle scripts.
- smarterclayton 5y ago
  Depending on the container runtime in use (containerd, crio), a separate process is managing and running those container processes (so k3s is telling that process what to do, but doesn’t “own” the containers). Also, modern container runtimes interface directly with systemd to launch transient units (think systemd services but without on disk config), so restarting a cri-o for example won’t restart those processes. That allows systemd to manage the process and for k3s/kubelet/cri-o to be upgraded / crash and recover. Under crio to stop all containers you can (well, should but YMMV depending on any changes we’ve made recently) systemctl stop kubepods.slice Note that doing that a subsequent start shouldn’t bring the containers back unless k3s and crio were both running (the transient units don’t survive restart). Note: there’s tons of complexity here that has changed over the last five or so years as container runtimes evolve, so if anything I said is wrong now it was right at one point and might actually be a bug. Kube for instance didn’t use to gracefully shut down containers on node reboot, and over last few years we’ve gradually fixed all of the complexities so that you do get the correct shutdown behavior between systemd (which takes the “power down” input) to the unit (which encodes how long graceful shutdown should take based on the pod termination grace period seconds) to cri-o (which has to set the correct behavior in systemd) to kubelet (which has to stay running to keep updating the API) to the other components on the node (like container networking, which needs to stay up until all pod network pods are done). systemd makes a lot of that a lot less painful than it would without.
  - mongol 5y ago
    Thank you for enlightening me on this topic!
- fuzzy2 5y ago
  Well, yes and no. Systemd starts the k3s master process (or whatever it is called). It also stops this process. At this point, Systemd’s work is done and it is working properly. Now k3s of course starts tons of stuff, mostly in containers. Therein lies the problem: By default, Systemd indeed tries to get rid of the entire process tree. It does this using control groups (cgroups). Unfortunately, it is easy for a process to “escape” its control group, at least for root. Stopping the master process could absolutely be made to also stop everything. It would be great if k3s did properly integrate with Systemd. It doesn’t, but Systemd is not to blame for that. That’s my point.

by fuzzy2 5y ago

mongol 5y ago

It did start it. For example the k3s process did start a container registry, but it did not stop it. I thought one of the points of systemd was that it would reap parentless processes

c0l0 5y ago
Which processes systemd allows to live on after stopping an associated service depends on the kind of KillMode= (see systemd.kill(5)) that was specified for the service unit. The default ('control-group') is usually what you want, esp. if you are looking to not recreate the kind of chaos improper process management SysV-style init scripts frequently caused. Chances are the k3s service unit specifies something else (like 'none') - whether or not that is actually wise would probably require some research.
- mongol 5y ago
  Thanks for explaining that. My comment was not a criticism against systemd, more a surprise that k3s did something that was not expected by me when I interacted with it through systemd. For example, httpd does not supply a kill-all-httpd.sh script as far as I know. Perhaps there are systemctl verbs that can be added to support what k3s devs apparently thought neccessary to implement in a separate script.
justinsaccount 5y ago
It does exactly what the configuration tells it to do. At one point some distributions accidentally configured the sshd or cron unit to do that. People were a bit surprised when restarting sshd or cron killed every single background child process. See https://www.freedesktop.org/software/systemd/man/systemd.kill.html#KillMode= https://www.freedesktop.org/software/systemd/man/systemd.kil... Not to mention if k3s is just managing containers running under docker or containerd, those processes are not parentless just because k3s stopped.
- mongol 5y ago
  My point is, this feels like an area where there is more work to do. Not surprising, the container landscape is a relatively new and rapidly evolving field that hardly existed when systemd was invented. But the fact that a piece of software supplies a killall.sh script reminds me of the old situation before systemd, when every software provided their own lifecycle scripts.
smarterclayton 5y ago
Depending on the container runtime in use (containerd, crio), a separate process is managing and running those container processes (so k3s is telling that process what to do, but doesn’t “own” the containers). Also, modern container runtimes interface directly with systemd to launch transient units (think systemd services but without on disk config), so restarting a cri-o for example won’t restart those processes. That allows systemd to manage the process and for k3s/kubelet/cri-o to be upgraded / crash and recover. Under crio to stop all containers you can (well, should but YMMV depending on any changes we’ve made recently) systemctl stop kubepods.slice Note that doing that a subsequent start shouldn’t bring the containers back unless k3s and crio were both running (the transient units don’t survive restart). Note: there’s tons of complexity here that has changed over the last five or so years as container runtimes evolve, so if anything I said is wrong now it was right at one point and might actually be a bug. Kube for instance didn’t use to gracefully shut down containers on node reboot, and over last few years we’ve gradually fixed all of the complexities so that you do get the correct shutdown behavior between systemd (which takes the “power down” input) to the unit (which encodes how long graceful shutdown should take based on the pod termination grace period seconds) to cri-o (which has to set the correct behavior in systemd) to kubelet (which has to stay running to keep updating the API) to the other components on the node (like container networking, which needs to stay up until all pod network pods are done). systemd makes a lot of that a lot less painful than it would without.
- mongol 5y ago
  Thank you for enlightening me on this topic!
fuzzy2 5y ago
Well, yes and no. Systemd starts the k3s master process (or whatever it is called). It also stops this process. At this point, Systemd’s work is done and it is working properly. Now k3s of course starts tons of stuff, mostly in containers. Therein lies the problem: By default, Systemd indeed tries to get rid of the entire process tree. It does this using control groups (cgroups). Unfortunately, it is easy for a process to “escape” its control group, at least for root. Stopping the master process could absolutely be made to also stop everything. It would be great if k3s did properly integrate with Systemd. It doesn’t, but Systemd is not to blame for that. That’s my point.