18 ms·
Back in 2016, systemd started killing user processes on logout (rather than send them the SIGHUP signal, as POSIX says should happen). This caused problems for
by jimrandomh 8y ago
Back in 2016, systemd started killing user processes on logout (rather than send them the SIGHUP signal, as POSIX says should happen). This caused problems for programs like nohup, screen and tmux, which deliberately keep running. Systemd's response was to say that they should incorporate systemd's library, and use systemd's new daemonization API. As far as I know, none of them did.
Two years later, you can find hundreds of support requests across the internet, from frustrated users who are having their sessions killed by systemd.
Bugs are annoying, but that's life. On the other hand, when you're an impacted user who's lost work, and researching the bug leads you to a years-old discussion in which someone is actively denying that the bug exists and refusing to fix it, that's infuriating. I don't think systemd's developers deserve the trust that maintaining a core piece of infrastructure requires; they don't seem to care enough about whether they've broken things.
- MereInterest 8y agoAbsolutely. I can understand implementing this feature for some special cases, like containers that should clear all hint of a user away on log off. It should never have been the default, and breaks an entire category of software. In my standard .bashrc file, I have the following snippet to warn me if I am on a system with that stupid setting enabled. if which loginctl > /dev/null && loginctl >& /dev/null; then if loginctl show-user | grep KillUserProcesses | grep -q yes; then echo "systemd is set to kill user processes on logoff" echo "This will break screen, tmux, emacs --daemon, nohup, etc" echo "Tell the sysadmin to set KillUserProcesses=no in /etc/systemd/login.conf" fi fi
- m45t3r 8y agoThanks, now I know why Emacs daemon keeps delaying my restarts in the system (just discovered that NixOS defaults KillUserProcesses to false). Turning this on to true, for me it does no make sense to a user service (yeah, I run emacs as a user's systemd service) to keep running after I logout of my system. P.S.: And the fact that for some people this behavior makes sense is why I think Lenart decision to put this as an option makes sense.
- MereInterest 8y agoI'm glad that it helped resolve your issue, though I still don't think it was an appropriate choice for a default. I tend to do most of my work on a remote server, using tmux and emacs daemon to pick up right where I left off in the case of a dropped connection. That systemd would terminate my process when I explicitly requested it not to be is very abnormal.
- pas 8y agoYou haven't requested systemd, you started a user scope, and haven't started a service for what you need. POSIX is nice, but rather lacking in certain aspects, such as security anf administration-friendliness. cgroups help with both, but people have to understand them and use them well.
- MereInterest 8y agoHandling and ignoring SIGHUP is the explicit way to indicate that a program should not be terminated. That systemd invented a new category and then ex post facto declared that everybody else was wrong for not using it is ridiculous. Systemd changing behavior such that I must "Simon says nohup" is completely asinine.
- newnewpdro 8y agoThere is no NOHUP signal, you're referring to SIGHUP. See the enable-linger option for loginctl and KillUserProcesses for logind.conf. KillUserProcesses was set to default enabled on 4/9/2016, prior to that it didn't happen, but was configurable if desired. So you were always able to change the config to restore the previous behavior from the moment the default turned it on. Edit: Here is the commit where it happened https://github.com/systemd/systemd/commit/97e5530cf2076a2b4fc55755917262607aaa6338 https://github.com/systemd/systemd/commit/97e5530cf2076a2b4f...
- jimrandomh 8y agoI meant SIGHUP. Edited.
- mirimir 8y agoI just checked a few Debian stretch boxes that I setup, and "KillUserProcesses=no" is set on them all. And until a few minutes ago, I didn't even know to check. So how can it be the default?
- coryrc 8y agoDebian maintainers make many improvements to upstream and only rarely mess up (ssh key generation).
- deleted 8y ago[deleted]
- shittyadmin 8y agoIf you comment out that line it'll be on by default - Debian fixed it for you with their own default configuration file, because 99% of their users would only be annoyed by it. This is why we have distro vendors, to build a system that works in the real world with software from developers with opinions that... differ to say the least.
- JdeBP 8y ago> So you were always able to change the config to restore the previous behavior from the moment the default turned it on. No, you were not. The thing that people are missing here is that neither of the systemd-logind behaviours, with KillUserProcesses=yes or KillUserProcesses=no, is the long-standing behaviour of kernel login sessions all of the way back to 7th Edition that nohup, tmux, screen, emacs --daemon, mosh-server, deluged, and more all interoperate with. The behaviour of kernel login sessions is that end of login session is a HUP signal to the session leader, and that termination of the entire TTY login service (such as at system shutdown) is a TERM signal to everything followed by a KILL signal to everything then remaining. The systemd-logind session behaviour with KillUserProcesses=no is no signals at all at the end of the login session, and at termination of the TTY login service both HUP and TERM signals together then KILL signals, to everything. The systemd-logind session behaviour with KillUserProcesses=yes is both HUP and TERM signals together then KILL signals, to everything, both at login session termination and at TTY login service stop. As I pointed out years ago, the fix is to make systemd-logind use KillUnit at hangup and StopUnit at service termination, actually providing the conventional behaviour which it currently does not in any mode and addressing the original problems (with some background GNOME utilities in a login session that were never being sent a HUP signal at logout and would have exited had they been) that motivated this whole mechanism in the first place. * https://news.ycombinator.com/item?id=12335128 https://news.ycombinator.com/item?id=12335128 * https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394#221 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394#221 * https://news.ycombinator.com/item?id=11798604 https://news.ycombinator.com/item?id=11798604
- xyzzyz 8y agoSystemd's response was to say that they should incorporate systemd's library, and use systemd's new daemonization API. By "use systemd's new demonization API" you mean, instead of $ screen systemd asks you to write $ systemd-run --scope --user screen instead. Annoying to have to learn a new thing, but hardly the unbearable burden. On the other hand, when you're an impacted user who's lost work, and researching the bug leads you to a years-old discussion in which someone is actively denying that the bug exists and refusing to fix it, that's infuriating. Because it's a bug for some, and intended behavior for others. Look, you make it as if they introduced a bug on purpose to screw with some people. It's clearly not the case, there was a specific tradeoff involved.
- nerdponx 8y agoIt's a bug because it violates the expectations of an uninformed user. You aren't given a warning about it, it's not documented in big bold letters anywhere, and it's also not POSIX compliant.
- zorpner 8y agoAnnoying to have to learn a new thing, but hardly the unbearable burden. Rather, a breaking change to everyone's scripts and processes for zero benefit.
- xyzzyz 8y agoThere's a benefit, you're just not seeing it. Again, do you think that the systemd developers decided to implement it just to screw with people? As I said, there's a specific trade-off involved here. I agree that it might not be the most desirable default, but if that's the case, then the guilt also falls on the distribution maintainers, who either ignored the big bold letters in the changelog, or didn't bother to test the everyone's standard workflows before pushing to stable.
- zorpner 8y agoThere's a benefit, you're just not seeing it. Not to appeal to self-authority, but I have been maintaining production Linux systems in large-scale environments since the late 90s. If there were a benefit that outweighed the unnecessary breaking changes, I would see it, even if I didn't appreciate it. There isn't. You should stop and think before you assume that other people are incompetent, both because it would make you a better interlocutor, and as a bonus it wouldn't violate HN's principle of charity.
- MertsA 8y agoThat comes down entirely to how systemd is configured. If you don't like what your chosen distro has picked as the default then complain to them. systemd didn't force anyone's hand on the subject, they just added the feature. It's a pretty natural design choice IMHO. When I want to log out, I don't want to let some hung up daemon keep running just because it wasn't able to process the SIGHUP sent to it. How else do you propose to make sure that when I log off my ssh-agent is really terminated and not just locked up with my keys still in memory? The POSIX approach is insufficient, there's no way to know if a process received a signal and chose to ignore it and keep running or if it received a signal but it was deadlocked and kept running.
- inferiorhuman 8y ago> How else do you propose to make sure that when I log off my ssh-agent is really terminated and not just locked up with my keys still in memory? Perhaps with a signal handler?
- pas 8y agoThat was the nice and friendly POSIX way, turns out it's really convenient for malware to stick around that way. Now user session isolation and termination works (cgroups), but it of course breaks backward comp.
- semi-extrinsic 8y agoIf only there was a way for the system init program to identify and keep a list of processes it has spawned, you could imagine like a unique numerical Process ID, and then if there was a program that could check the Process Status, and another that could kill the process identified by this... PID with increasing levels of aggressiveness...
- wbl 8y agoPIDs get reused so this doesn't work well.
- jimrandomh 8y agoSystemd developers, if you're reading this: this isn't the sort of bug where people grumble for awhile and then get over it, because things are still broken, and the workaround being circulated (KillUserProcesses=no) doesn't fully work. (https://github.com/systemd/systemd/issues/8486 https://github.com/systemd/systemd/issues/8486) As long as people continue to encounter this issue anew--and they still are--people will be angry at the systemd maintainers.
- avar 8y agoThe bug you've linked to was closed[1] by the reporter with "Thanks for the clarification guys. Much appreciated!", after it was pointed out to them that something they were trying ho do with "KillUserProcesses=no" was better done in another way. 1. Edit: Not literally closed by the reporter. Lennart Poettering closed it, "closed by the reporter" as in "the issue was resolved to the reporter's satisfaction".
- dvfjsdhgfv 8y ago> The bug you've linked to was closed by the reporter Are we reading the same bug report? The one I'm looking at was closed by the creator of Systemd.
- jesuslop 8y agoIs there a daemonization API as such? I think there was only the "way of doing" shown in man 7 daemon.
- JdeBP 8y agoThe systemd people have their own version of that manual page. * https://freedesktop.org/software/systemd/man/daemon.html https://freedesktop.org/software/systemd/man/daemon.html IBM was explaining what to do back in 1995. * http://jdebp.eu./FGA/unix-daemon-design-mistakes-to-avoid.html http://jdebp.eu./FGA/unix-daemon-design-mistakes-to-avoid.ht...
- masonic 8y agokilling user processes on logout By "killing", do you mean some other signal than (or in addition to) SIGHUP? Does it send SIGKILL?
- seba_dos1 8y agoThat's the whole issue here. It does.
- poettering 8y agoYou know, because we knew this would be controversial we made sure it was both a compile-time option and a runtime option. Yes the upstream default of both defaults to on, but that's just upstream. We made it very easy and supported for downstream distros to switch between opt-out and opt-in of this option for their users. We have encouraged distributions to leave it on, but we were fully aware that for compatibility reasons this is something downstreams likely wanted to turn off, and most compat-minded distros did, as we expected. Now I am used to taking blame for apparently everything that every went wrong on Linux, but you might as well blame your downstream distros for this as much you want to blame us upstream about this, as it's up to them to pick the right compile-time options matching their userbase and requirements in compatibility, and if they didn't do that to your liking, then maybe you should complain to them first. (And yes, I still consider it a weakness of UNIX that "logout" doesn't really mean "logout", but just "maybe, please, if you'd be so kind, i'd like to exit, but not quite". I mean, that's not how you build a secure system. We fixed that really, fully knowing it would depart from UNIX tradition, but that's why we made it both compile-time and runtime configurable) (Also, nobody has to "incorporate" systemd's library to avoid the automatic clean-up. In fact, there's no library we provide that could do that. What was requested though is to either run things as child of systemd --user or just register a separate PAM session, neither of which requires any systemd-specific library.) Lennart
- tasuki 8y agoLennart, thanks for the information. Mind explaining why you chose to kill user processes on logout as the default?
- poettering 8y agoI think my comment above explained that already.
- m4r35n357 8y agoThey show that you do not understand what "log out" is for in Unix.
- majewsky 8y ago> killing user processes on logout (rather than send them the SIGHUP signal, as POSIX says should happen) TIL what nohup(1) is for.
- zbentley 8y agoSort of. While it's debatable when SIGHUP should be sent as part of controlled system logout/whatever, the signal itself was originally used upon abrupt disconnection (hang up) of the controlling terminal of a program.
- cout 8y agoI agree that on Linux-based systems, SIGHUP is a reasonable mechanism for killing processes when a user closes an ssh session, and that ignoring SIGHUP is a reasonable way to avoid getting terminated. I disagree that POSIX says that processes should expect a SIGHUP when a user logs out (SIGHUP means the controlling terminal was closed). I am not at all a POSIX expert, so please correct me if I misunderstand, but afaict POSIX explicitly does not specify what happens to the controlling terminal when a user logs out (http://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd_chap03.html http://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd_...): > POSIX.1 does not specify how controlling terminal access is affected by a user logging out (that is, by a controlling process terminating). 4.2 BSD uses the vhangup() function to prevent any access to the controlling terminal through file descriptors opened prior to logout. System V does not prevent controlling terminal access through file descriptors opened prior to logout (except for the case of the special file, /dev/tty). Some implementations choose to make processes immune from job control after logout (that is, such processes are always treated as if in the foreground); other implementations continue to enforce foreground/background checks after logout. Therefore, a Conforming POSIX.1 Application should not attempt to access the controlling terminal after logout since such access is unreliable. If an implementation chooses to deny access to a controlling terminal after its controlling process exits, POSIX.1 requires a certain type of behavior (see Controlling Terminal ).
- cryptonector 8y agoEleven years earlier when SMF was added to what would eventually be Solaris 10, we had this same problem. Some of us had to drop everything to fix "bugs" in cron, sshd, ... introduced by SMF. Systemd is basically SMF, done poorly, because NIH.
- JdeBP 8y agoAlso two years ago, I explained how one could make this work, by having logind use KillUnit at hangup and StopUnit at shutdown. * https://news.ycombinator.com/item?id=12335128 https://news.ycombinator.com/item?id=12335128 * https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394#221 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394#221