13 ms·

I wish systemd logged information about the source of “transactions”

127 points by zdw 5y ago

AndyMcConachie 5y ago
systemd is why I love OpenBSD.
- Datagenerator 5y ago
  The acronym POLA translates to peace where systemd creates havoc and burns many hours globally having people investigate what the heck is going on in this almost binary blob of spaghetti.
- chasil 5y ago
  Guess what? https://www.itsfoss.net/initware-as-a-systemd-fork-on-openbsd/amp/ https://www.itsfoss.net/initware-as-a-systemd-fork-on-openbs...
  - guilhas 5y ago
    A very slim down fork would possibly not be that bad
- gunapologist99 5y ago
  Don't forget Artix, Void, MX Linux, FreeBSD, Alpine, Gentoo/Funtoo..
- ufo 5y ago
  Does OpenBSD log what source caused the machine to reboot?
usr1106 5y ago
While generally the systemd documentation is pretty good (Start with reading the original blog series as a primer, later search for "systemd directives" using your favourite search engine) I have always found the transactions concept underdocumented. Does anyone have a good link? Edit: Are transactions and jobs the same thing? Both are mentioned in the documentation here and there, without having an own page or chapter AFAIK.
- egberts1 5y ago
  “Documentation” is “Pretty Good? “Start with” the “blog”? Clearly, the two does not connect well … for systemd.
  - jeppesen-io 5y ago
    > ... as a primer It's worked very well for me. Not seeing your disconnect here. Blog for concepts, man pages for specifics
    - noobermin 5y ago
      I wish more developers wrote enough into the man page so you don't need to google things. More of bash's man page and less of i3's.
      - throw0101a 5y ago
        Don't forget all the stuff in GNU's info pages.
        badsectoracula 5y ago
        I'm of the opinion that "info" is for long form guides and "man" is for short references. (replace "info" with some better reader than the default one though)
    - egberts1 5y ago
      For instance, I wanted to know why my custom systemd.conf unit file is causing my custom daemon to restart whenever the Ethernet cord gets unplugged or its netdev goes offline.
  - usr1106 5y ago
    It's open source, you get what you pay for. Actually, you get much more, but you need to fill any gaps with your own efforts. Accessing the spread out documentation is one of them. While they are not comparable entities I'd say systemd documentation is in better shape than Linux kernel documentation. (Not to negate the efforts of those who do work with kernel documentation, but to stress the huge areas of no or pretty obsolete documentation). Of course you can buy Linux (incl. user space) from the commercial players. I have only worked in one small project in my career that did, but I did not notice that better documentation was worth paying for.
Ono-Sendai 5y ago
Ran into a similar problem, trying to work out why systemd was stopping my service.
- avian 5y ago
  systemd just has so many reasons to kill processes, and invents new ones with new releases. Timeouts for things it thinks it should be short-lived, resource limits, service isolation and sandboxing settings, etc. They are mostly documented, but you need to know where to look. While I came to like some aspects of systemd, debugging why things die for apparently no reason after systemd was upgraded has eaten many of my workdays.
  - buserror 5y ago
    New one as of this morning is to kill STOP processes it doesn't like. I have no idea why, or how to revent it. All I did was update my 'sid' debian and here goes
    - zokier 5y ago
      > All I did was update my 'sid' debian and here goes As it turns out it's called "unstable" for a reason
      - buserror 5y ago
        Hate to tell you, but I've been using sid for well over 20 years, and it is usually more 'stable' than most distro out there.
TimWolla 5y ago
A peer of mine was exiting their SSH sessions with 'exit'. One time apparently they already typed 'systemctl', probably in an attempt to check the status of a service, changed their mind and then later wanted to close the session using 'exit', actually executing 'systemctl exit'. This translated into a shutdown of the machine in question. After being able to piece together what happened with the machine's logs and the bash history I recommended to simply exit all programs/sessions with Ctrl+D. It works almost everywhere and would have prevented this exact issue.
- Denvercoder9 5y ago
  I recommend not using root privileges unnecessarily. `systemctl status` and most other querying commands don't need root, while the dangerous things like `systemctl exit` do.
  - usr1106 5y ago
    Sure, but some systems do poweroff without being root. Has happened to me, don't remember the details. Maybe a polkit thing because users are supposed to shut down their own laptop?
    - CraigJPerry 5y ago
      Yeah true, if the user is logged on via a physical tty or local X session (i.e. the policykit subject.local attribute == true) then in some distros they will get permission to shutdown or reboot. They won’t have the permission if connected remotely though.
  - TimWolla 5y ago
    I can't say what they had in mind when typing `systemctl`. Even they couldn't. Because of the delay in the shutdown to cleanly stop the services, they had already forgotten that they just exited a SSH session and thus the connection between the machine being dead and typing `exit` was not obvious. Maybe it was `systemctl status`. Maybe it was intended to be a `reload` (which would require elevated privileges).
ploek 5y ago
Similarly I discovered yesterday that the systemd service definition for auditd includes the `RefuseManualStop` option for this exact reason. When stopping (and thus also when restarting) the service via systemd, auditd is unable to log who shut it down, so it just disallows being stopped. (https://linux-audit.redhat.narkive.com/3weoVaZE/rational-behind-refusemanualstop-yes-in-auditd-service#post2 https://linux-audit.redhat.narkive.com/3weoVaZE/rational-beh...) The workaround is to use the service command instead. Manually I usually do that anyway, muscle memory etc. But Ansible's service module will default to systemctl if it finds systemd. So there I had to add a "use: service".
blueflow 5y ago
Related incident: "Systemd killing processes each minute at second 27" (German): https://blog.uberspace.de/systemd-anzuenden/ https://blog.uberspace.de/systemd-anzuenden/
OJFord 5y ago
It seems odd to me to go to the effort to tweet & blog about 'wishing' for something like this, but not to open a GitHub issue requesting it: https://github.com/systemd/systemd/issues https://github.com/systemd/systemd/issues (At least, I can't find it searching there, and OP doesn't link one, so I assume there isn't one.)
- dorianmariefr 5y ago
  Good point, created one https://github.com/systemd/systemd/issues/21497 https://github.com/systemd/systemd/issues/21497 (I'm not the author though)
- dijit 5y ago
  I'm probably going to get a lot of hate for saying this; because anything anti-systemd tends to attract weird people who claim that you want bash scripts back but: Making issues on systemd's github page is often met with stoicism and reluctance on the part of the systemd developers. There's countless examples of the systemd maintainers refusing to fix bugs or acknowledge that bugs exist. Making a public statement could be more effective.
  - OJFord 5y ago
    Maybe, personally though (if that were my experience and I was OP) I'd try an issue first, and then if it was shot down I could tweet & blog 'unfortunately, doesn't look like it's going to happen' or whatever, referencing my attempt. It just seems like they went to about the same amount of effort, but on something with a much lower (IMO) probability of being actioned. (Not least because if it were it'd probably go though roughly the same process anyway, such as through the issue now opened by a sibling comment to yours. Maybe PR without issue, but which OP could (skills permitting) also have done.)
gunapologist99 5y ago
Aside from the maintenance and uptime issues, OP is actually raising a very real security concern; often an attacker will reboot the machine to restart all of the logging processes or to load in a LKM. Knowing why, and when (and which process) just forced a reboot is a very real requirement. Systemd also seemingly randomly attacks my processes, and it's almost impossible to actually figure out why. (At least the kernel OOM killer actually logs "Out of memory, killed this process."[0]) 0. https://linuxwheel.com/oom-killer-explained/ https://linuxwheel.com/oom-killer-explained/

127 points by zdw 5y ago

AndyMcConachie 5y ago

systemd is why I love OpenBSD.

Datagenerator 5y ago
The acronym POLA translates to peace where systemd creates havoc and burns many hours globally having people investigate what the heck is going on in this almost binary blob of spaghetti.
chasil 5y ago
Guess what? https://www.itsfoss.net/initware-as-a-systemd-fork-on-openbsd/amp/ https://www.itsfoss.net/initware-as-a-systemd-fork-on-openbs...
- guilhas 5y ago
  A very slim down fork would possibly not be that bad
gunapologist99 5y ago
Don't forget Artix, Void, MX Linux, FreeBSD, Alpine, Gentoo/Funtoo..
ufo 5y ago
Does OpenBSD log what source caused the machine to reboot?

usr1106 5y ago

While generally the systemd documentation is pretty good (Start with reading the original blog series as a primer, later search for "systemd directives" using your favourite search engine) I have always found the transactions concept underdocumented. Does anyone have a good link? Edit: Are transactions and jobs the same thing? Both are mentioned in the documentation here and there, without having an own page or chapter AFAIK.

egberts1 5y ago
“Documentation” is “Pretty Good? “Start with” the “blog”? Clearly, the two does not connect well … for systemd.
- jeppesen-io 5y ago
  > ... as a primer It's worked very well for me. Not seeing your disconnect here. Blog for concepts, man pages for specifics
  - noobermin 5y ago
    I wish more developers wrote enough into the man page so you don't need to google things. More of bash's man page and less of i3's.
    - throw0101a 5y ago
      Don't forget all the stuff in GNU's info pages.
      - badsectoracula 5y ago
        I'm of the opinion that "info" is for long form guides and "man" is for short references. (replace "info" with some better reader than the default one though)
  - egberts1 5y ago
    For instance, I wanted to know why my custom systemd.conf unit file is causing my custom daemon to restart whenever the Ethernet cord gets unplugged or its netdev goes offline.
- usr1106 5y ago
  It's open source, you get what you pay for. Actually, you get much more, but you need to fill any gaps with your own efforts. Accessing the spread out documentation is one of them. While they are not comparable entities I'd say systemd documentation is in better shape than Linux kernel documentation. (Not to negate the efforts of those who do work with kernel documentation, but to stress the huge areas of no or pretty obsolete documentation). Of course you can buy Linux (incl. user space) from the commercial players. I have only worked in one small project in my career that did, but I did not notice that better documentation was worth paying for.

Ono-Sendai 5y ago

Ran into a similar problem, trying to work out why systemd was stopping my service.

avian 5y ago
systemd just has so many reasons to kill processes, and invents new ones with new releases. Timeouts for things it thinks it should be short-lived, resource limits, service isolation and sandboxing settings, etc. They are mostly documented, but you need to know where to look. While I came to like some aspects of systemd, debugging why things die for apparently no reason after systemd was upgraded has eaten many of my workdays.
- buserror 5y ago
  New one as of this morning is to kill STOP processes it doesn't like. I have no idea why, or how to revent it. All I did was update my 'sid' debian and here goes
  - zokier 5y ago
    > All I did was update my 'sid' debian and here goes As it turns out it's called "unstable" for a reason
    - buserror 5y ago
      Hate to tell you, but I've been using sid for well over 20 years, and it is usually more 'stable' than most distro out there.

TimWolla 5y ago

A peer of mine was exiting their SSH sessions with 'exit'. One time apparently they already typed 'systemctl', probably in an attempt to check the status of a service, changed their mind and then later wanted to close the session using 'exit', actually executing 'systemctl exit'. This translated into a shutdown of the machine in question. After being able to piece together what happened with the machine's logs and the bash history I recommended to simply exit all programs/sessions with Ctrl+D. It works almost everywhere and would have prevented this exact issue.

Denvercoder9 5y ago
I recommend not using root privileges unnecessarily. `systemctl status` and most other querying commands don't need root, while the dangerous things like `systemctl exit` do.
- usr1106 5y ago
  Sure, but some systems do poweroff without being root. Has happened to me, don't remember the details. Maybe a polkit thing because users are supposed to shut down their own laptop?
  - CraigJPerry 5y ago
    Yeah true, if the user is logged on via a physical tty or local X session (i.e. the policykit subject.local attribute == true) then in some distros they will get permission to shutdown or reboot. They won’t have the permission if connected remotely though.
- TimWolla 5y ago
  I can't say what they had in mind when typing `systemctl`. Even they couldn't. Because of the delay in the shutdown to cleanly stop the services, they had already forgotten that they just exited a SSH session and thus the connection between the machine being dead and typing `exit` was not obvious. Maybe it was `systemctl status`. Maybe it was intended to be a `reload` (which would require elevated privileges).

ploek 5y ago

Similarly I discovered yesterday that the systemd service definition for auditd includes the `RefuseManualStop` option for this exact reason. When stopping (and thus also when restarting) the service via systemd, auditd is unable to log who shut it down, so it just disallows being stopped. (https://linux-audit.redhat.narkive.com/3weoVaZE/rational-behind-refusemanualstop-yes-in-auditd-service#post2 https://linux-audit.redhat.narkive.com/3weoVaZE/rational-beh...) The workaround is to use the service command instead. Manually I usually do that anyway, muscle memory etc. But Ansible's service module will default to systemctl if it finds systemd. So there I had to add a "use: service".

blueflow 5y ago

Related incident: "Systemd killing processes each minute at second 27" (German): https://blog.uberspace.de/systemd-anzuenden/ https://blog.uberspace.de/systemd-anzuenden/

OJFord 5y ago

It seems odd to me to go to the effort to tweet & blog about 'wishing' for something like this, but not to open a GitHub issue requesting it: https://github.com/systemd/systemd/issues https://github.com/systemd/systemd/issues (At least, I can't find it searching there, and OP doesn't link one, so I assume there isn't one.)

dorianmariefr 5y ago
Good point, created one https://github.com/systemd/systemd/issues/21497 https://github.com/systemd/systemd/issues/21497 (I'm not the author though)
dijit 5y ago
I'm probably going to get a lot of hate for saying this; because anything anti-systemd tends to attract weird people who claim that you want bash scripts back but: Making issues on systemd's github page is often met with stoicism and reluctance on the part of the systemd developers. There's countless examples of the systemd maintainers refusing to fix bugs or acknowledge that bugs exist. Making a public statement could be more effective.
- OJFord 5y ago
  Maybe, personally though (if that were my experience and I was OP) I'd try an issue first, and then if it was shot down I could tweet & blog 'unfortunately, doesn't look like it's going to happen' or whatever, referencing my attempt. It just seems like they went to about the same amount of effort, but on something with a much lower (IMO) probability of being actioned. (Not least because if it were it'd probably go though roughly the same process anyway, such as through the issue now opened by a sibling comment to yours. Maybe PR without issue, but which OP could (skills permitting) also have done.)

gunapologist99 5y ago

Aside from the maintenance and uptime issues, OP is actually raising a very real security concern; often an attacker will reboot the machine to restart all of the logging processes or to load in a LKM. Knowing why, and when (and which process) just forced a reboot is a very real requirement. Systemd also seemingly randomly attacks my processes, and it's almost impossible to actually figure out why. (At least the kernel OOM killer actually logs "Out of memory, killed this process."[0]) 0. https://linuxwheel.com/oom-killer-explained/ https://linuxwheel.com/oom-killer-explained/