12 ms·

> I have used init scripts, I have written init scripts. I hated it. Every program needed it's own little way of doing things and writing an init script for a d

by ilyt 3y ago

> I have used init scripts, I have written init scripts. I hated it. Every program needed it's own little way of doing things and writing an init script for a daemon was a hassle of bash scripting and copy pasting. We had, no joking, thousands lines of init scripts that were just ripped out of distro (mostly CentOS/RHEL... Debian had some pre-made wrappers that mostly did the right thing) and fixed by us because people saying "it's so easy" to write init script ARE LYING. And some apps, like Pacemaker (app for making other things HA), are really sensitive on scripts not being done fully correctly Some tidbits: * script reports app is stopped right after you run start... because it left creating PID to application, app took some time to boot (Java), and the script didn't wait for PID creation. So running "start, then run status to check whether it worked" returned that app did not work. * script reports app is running after reboot. The app is not running. Script read off PID file off /var/run (that at the time wasn't in tmpfs like now /run is), saw PID exists and didn't try to start it at all. The pid was of a different application. That was MySQL, not some random obscure app... * "stop" action doesn't actually return when app is stopped, it just sends signal. sysv goes ahead and shuts down system before app manages to write all of the data. Or in case of pacemaker it does weird stuff because "status returning running after sending stop" should not happen. * other various ways "status" was broken randomly

bandrami 3y ago
OTOH "systemctl start httpd.service" exits with 0 even if apache doesn't start because for some reason "a misconfigured apache didn't start" is considered correct behavior. I don't know that any init has actually solved that problem.
- Arnavion 3y ago
  This requires the application to tell the service manager to wait for some signal that it has successfully "finished starting" beyond just "the process was created". There's no standard way to do it. In systemd's case this is done by opting in to inherit a socket that can be used to communicate that information (`Type=notify`, then calling `sd_notify()`).
  - bandrami 3y ago
    Exactly: like I said below the pain point was never "generate a textual representation of the startup logic appropriate to the init system" but always "concretely define what you mean by a service being started or stopped and how to determine that".
  - ilyt 3y ago
    there is one other way If you set Type=forking and the app exits with non-zero code on start then that will get propagated back to systemctl start but notify is far better option even if requiring extra app support.
- ilyt 3y ago
  Neither does some init scripts we had to fix so it's 0:0 on this particular issue. That issue is actually systemd considering success if config and dependencies are okay and service is started; it doesn't wait to return so it doesn't know whether service "works okay now", because it might exit with error 20ms after start. Type=notify fixes that issue but obviously needs support from app themself to signal "I'm up and running now" Less robust way is to set Type=forking then * if app exited with okay, it is considered running * if app exited with not okay, you get error on start but obviously then app need to be of self-forking type. It's an iffy problem to solve; I guess systemd could have some wait interval to wait for service that might crash some milliseconds after start but that would delay every start unnecesarily
  - bandrami 3y ago
    Right, like I said I don't think any init system has actually figured this out but that is what as an admin I actually care about rather than startup times or the aesthetic properties of the configuration files. You would need some kind of semantics that define what it means for a system to be "ready". In practice we tended to lean towards some kind of polling of the service itself (which is at least slightly better than "sleep 3" or whatever). Now, it's true that if applications would adopt systemd's Notify system this could really help, but this is equivalent to saying "if software were different it might be better".
    - ilyt 3y ago
      Yeah its PITA problem. Like, service can fork, return okay then die after a second coz someone set memory size wrong. No real sensible way for init system to prevent it. Or takes 5 minutes to load then hits some bad data and crashes. We just set services on auto-restart, monitor it and flag alert if service is restarting too often.
- atomicnumber3 3y ago
  Fwiw nginx does do this correctly. I'm not sure what httpd does differently that it doesn't mimic this behavior.
- throw0101a 3y ago
  > OTOH "systemctl start httpd.service" I've also had issues with My/MariaDB not starting and the output capture by systemd and journald be absolutely useless in debugging the issue (i.e., nothing shows up). Generally resort to running mysqld manually at the CLI to either get some kind of stdout/err output or maybe sent to the error log file. Of course if there's an error really early in the program/daemon initialization you used to be able to run strace against the start-up script, but that's not possible with systemctl. I've run into that situation when (e.g.) /var/lib/mysql was a link to a mount point (secondary drive) and SELinux or AppArmour kicked in.
  - ilyt 3y ago
    > Of course if there's an error really early in the program/daemon initialization you used to be able to run strace against the start-up script, but that's not possible with systemctl. I mean it's PITA but you can just add strace command to start of ExecStart... > I've run into that situation when (e.g.) /var/lib/mysql was a link to a mount point (secondary drive) and SELinux or AppArmour kicked in. Yeah we hit that with some units that had maintainers preemptively setting security policy so distro-blessed paths were only writable ones for a given unit. But on plus side all could be fixed in few lines of overrides vs putting whole init script into the repo > I've also had issues with My/MariaDB not starting and the output capture by systemd and journald be absolutely useless in debugging the issue (i.e., nothing shows up). Generally resort to running mysqld manually at the CLI to either get some kind of stdout/err output or maybe sent to the error log file. Do you have systemd unit for that mysql problem ? There is option to redirect stdout/err to logs directly but systemd is too stupid to extract that logs back and show it in the status command. I definitely saw that issue you mentioned (it almost looked like it wasn't logged anywhere if app exited too quickly). But I don't see it on recent debians at the very least. ● mariadb.service - MariaDB 10.5.15 database server Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2023-05-08 12:41:26 CEST; 1 months 21 days ago Docs: man:mariadbd(8) https://mariadb.com/kb/en/library/systemd/ Process: 2913787 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS) Process: 2913788 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS) Process: 2913790 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] && systemct> Process: 2913887 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS) Process: 2913889 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS) Main PID: 2913870 (mariadbd) Status: "Taking your SQL requests now..." Tasks: 18 (limit: 2340) Memory: 303.5M CPU: 10h 32min 54.873s CGroup: /system.slice/mariadb.service └─2913870 /usr/sbin/mariadbd May 08 12:41:26 hostname systemd[1]: Starting MariaDB 10.5.15 database server... May 08 12:41:26 hostname mariadbd[2913870]: 2023-05-08 12:41:26 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-0+deb11u1) starting as process 2913870 ... May 08 12:41:26 hostname systemd[1]: Started MariaDB 10.5.15 database server. May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: Looking for 'mysql' as: /usr/bin/mysql May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: This installation of MariaDB is already upgraded to 10.5.15-MariaDB. May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: There is no need to run mysql_upgrade again for 10.5.15-MariaDB. May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: You can use --force if you still want to run mysql_upgrade

by ilyt 3y ago

bandrami 3y ago

OTOH "systemctl start httpd.service" exits with 0 even if apache doesn't start because for some reason "a misconfigured apache didn't start" is considered correct behavior. I don't know that any init has actually solved that problem.

Arnavion 3y ago
This requires the application to tell the service manager to wait for some signal that it has successfully "finished starting" beyond just "the process was created". There's no standard way to do it. In systemd's case this is done by opting in to inherit a socket that can be used to communicate that information (`Type=notify`, then calling `sd_notify()`).
- bandrami 3y ago
  Exactly: like I said below the pain point was never "generate a textual representation of the startup logic appropriate to the init system" but always "concretely define what you mean by a service being started or stopped and how to determine that".
- ilyt 3y ago
  there is one other way If you set Type=forking and the app exits with non-zero code on start then that will get propagated back to systemctl start but notify is far better option even if requiring extra app support.
ilyt 3y ago
Neither does some init scripts we had to fix so it's 0:0 on this particular issue. That issue is actually systemd considering success if config and dependencies are okay and service is started; it doesn't wait to return so it doesn't know whether service "works okay now", because it might exit with error 20ms after start. Type=notify fixes that issue but obviously needs support from app themself to signal "I'm up and running now" Less robust way is to set Type=forking then * if app exited with okay, it is considered running * if app exited with not okay, you get error on start but obviously then app need to be of self-forking type. It's an iffy problem to solve; I guess systemd could have some wait interval to wait for service that might crash some milliseconds after start but that would delay every start unnecesarily
- bandrami 3y ago
  Right, like I said I don't think any init system has actually figured this out but that is what as an admin I actually care about rather than startup times or the aesthetic properties of the configuration files. You would need some kind of semantics that define what it means for a system to be "ready". In practice we tended to lean towards some kind of polling of the service itself (which is at least slightly better than "sleep 3" or whatever). Now, it's true that if applications would adopt systemd's Notify system this could really help, but this is equivalent to saying "if software were different it might be better".
  - ilyt 3y ago
    Yeah its PITA problem. Like, service can fork, return okay then die after a second coz someone set memory size wrong. No real sensible way for init system to prevent it. Or takes 5 minutes to load then hits some bad data and crashes. We just set services on auto-restart, monitor it and flag alert if service is restarting too often.
atomicnumber3 3y ago
Fwiw nginx does do this correctly. I'm not sure what httpd does differently that it doesn't mimic this behavior.
throw0101a 3y ago
> OTOH "systemctl start httpd.service" I've also had issues with My/MariaDB not starting and the output capture by systemd and journald be absolutely useless in debugging the issue (i.e., nothing shows up). Generally resort to running mysqld manually at the CLI to either get some kind of stdout/err output or maybe sent to the error log file. Of course if there's an error really early in the program/daemon initialization you used to be able to run strace against the start-up script, but that's not possible with systemctl. I've run into that situation when (e.g.) /var/lib/mysql was a link to a mount point (secondary drive) and SELinux or AppArmour kicked in.
- ilyt 3y ago
  > Of course if there's an error really early in the program/daemon initialization you used to be able to run strace against the start-up script, but that's not possible with systemctl. I mean it's PITA but you can just add strace command to start of ExecStart... > I've run into that situation when (e.g.) /var/lib/mysql was a link to a mount point (secondary drive) and SELinux or AppArmour kicked in. Yeah we hit that with some units that had maintainers preemptively setting security policy so distro-blessed paths were only writable ones for a given unit. But on plus side all could be fixed in few lines of overrides vs putting whole init script into the repo > I've also had issues with My/MariaDB not starting and the output capture by systemd and journald be absolutely useless in debugging the issue (i.e., nothing shows up). Generally resort to running mysqld manually at the CLI to either get some kind of stdout/err output or maybe sent to the error log file. Do you have systemd unit for that mysql problem ? There is option to redirect stdout/err to logs directly but systemd is too stupid to extract that logs back and show it in the status command. I definitely saw that issue you mentioned (it almost looked like it wasn't logged anywhere if app exited too quickly). But I don't see it on recent debians at the very least. ● mariadb.service - MariaDB 10.5.15 database server Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2023-05-08 12:41:26 CEST; 1 months 21 days ago Docs: man:mariadbd(8) https://mariadb.com/kb/en/library/systemd/ Process: 2913787 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS) Process: 2913788 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS) Process: 2913790 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] && systemct> Process: 2913887 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS) Process: 2913889 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS) Main PID: 2913870 (mariadbd) Status: "Taking your SQL requests now..." Tasks: 18 (limit: 2340) Memory: 303.5M CPU: 10h 32min 54.873s CGroup: /system.slice/mariadb.service └─2913870 /usr/sbin/mariadbd May 08 12:41:26 hostname systemd[1]: Starting MariaDB 10.5.15 database server... May 08 12:41:26 hostname mariadbd[2913870]: 2023-05-08 12:41:26 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-0+deb11u1) starting as process 2913870 ... May 08 12:41:26 hostname systemd[1]: Started MariaDB 10.5.15 database server. May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: Looking for 'mysql' as: /usr/bin/mysql May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: This installation of MariaDB is already upgraded to 10.5.15-MariaDB. May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: There is no need to run mysql_upgrade again for 10.5.15-MariaDB. May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: You can use --force if you still want to run mysql_upgrade