3 ms·

Let's say that you're starting Apache Tomcat. You have to dig into the tomcat startup scripts and figure out where, if anywhere, it writes the pid file to disk

Let's say that you're starting Apache Tomcat. You have to dig into the tomcat startup scripts and figure out where, if anywhere, it writes the pid file to disk so that you can later use it to determine if the process is running or not. If java happened to crash, though, this pid file is stale and might not point to a valid process. There's a chance that the pid has been reused, and it could have been reused by anything -- even another java process! This is important, because this pid file is used to determine which process receives the kill signal. If you get it wrong, and have the right permissions, you can accidentally kill something that you did not intend to kill. This is further complicated if you want to run multiple instances of Tomcat because now you need to have a unique path for this pid file per tomcat instance. If the thing that you're trying to run doesn't fork, you then have to execute it in the background and then store the result of $! somewhere so that you know how to kill the process later on. It's all very error prone and the process for each daemon is often different.

herpderperator 5y ago
> Let's say that you're starting Apache Tomcat. You have to dig into the tomcat startup scripts and figure out where, if anywhere, it writes the pid file to disk so that you can later use it to determine if the process is running or not. https://tomcat.apache.org/tomcat-8.5-doc/windows-service-howto.html https://tomcat.apache.org/tomcat-8.5-doc/windows-service-how... The commandline argument is --PidFile and --LogPath. Most, if not all, programs allow you to customise this. It should never be a guessing game, especially when you are the one creating the init file, therefore you are the one in control of the running program.
- fullstop 5y ago
  Those arguments are for the Windows service and don't appear to have a corresponding Linux option. On the Linux side, if things are still done the same way as they were in the past, you had to set CATALINA_PID before starting the java process. It's still a guessing game, though, even with CATALINA_PID. It is entirely possible for Java to crash (something which RC scripts do not handle, at all) and another java process starting up which happens to be assigned the same process id as the dead java process. This can not happen with systemd units because each service unit is its own Linux cgroup and it can tell which processes belong to the service.
  - toast0 5y ago
    You could just run every daemon in its own jail. You don't need to chroot (unless you want to), and don't need to do any jail specific network config (unless you want to), but you could use the jail name to ensure no more than one tomcat (or tomcat-config) process, and you could use the jail name/id to capture process tree to kill it without ambiguity. With respect to servers that crash, approaches differ, but you could use an off the shelf, single purpose daemon respawner, or you could change the whole system, or you could endeavor to not have crashing servers, such that if it crashes, it's worth a human taking a look.
    - throwawaysysd 5y ago
      Sure, you can do all that and set it up manually. I'd love to be corrected here but last I checked this was not done automatically in any BSD. Systemd recognizes this is so common that it does it automatically for every service using the Linux equivalent (cgroups). IMO now that we have these tools, every sysadmin is always going to want to use cgroups/jails for every service all the time and never want to use pidfiles because pidfiles are tremendously broken, error-prone and racy.
      - _0w8t 5y ago
        Even with systemd one may need to deal with pid files for services with more than one process. Systemd has heuristics to detect the main process to send the kill signal or detect a crash, but it can guess wrong. Telling it the pid file location makes things reliable. Plus creation of a pid file serves as a synchronization point that systemd uses to know that the process is ready. The latter is useful when the service does not support systemd native API for state transition notifications. And the best thing is that even if systemd has to be told about pid files, one never needs to deal with stalled files or pid reuse. Systemd automatically removes them on process shutdown.
        throwawaysysd 5y ago
        A pid file is never actually reliable though. Since the supervised process has control over it, it can write any number it wants in there and trick the service manager into signaling the wrong process. As long as the process is not root and can't escape its own cgroup, systemd's pid detection is going to be less error-prone in basically every case. I can't stress this enough. Pid files are really bad. The fact that we used to have to use them is a flaw in the OS design. Using them for startup notification is also a hack in and of itself. The kernel has enough information to fully track lifecycle of these processes without writing information into files that are inherently prone to race conditions, and we now have better APIs to expose this information, so we shouldn't need to use these hacks anymore. I don't think there is any Unix-like system left that considers the old forking daemon style to be a good idea, and systemd's handling of pidfiles is really just a legacy compatibility thing.
- hsbauauvhabzb 5y ago
  How about Apache, Nginx, pulseaudio or ldapd?
trasz 5y ago
Your description is fine, but it’s missing one crucial detail: it’s specific to Linux. In FreeBSD this is already taken care of by rc infrastructure; there is no need for the user or sysadmin to mess with it.
- fullstop 5y ago
  I fully agree that it is a Linux only solution, but that's kind of the topic here. This still doesn't handle restarting crashed services, and it still is true if you need to make init scripts for your own services outside of the ports tree. It took me a long time to come to terms with systemd, but I am very glad that I did. For me it makes defining services both easy and reliable.
  - trasz 5y ago
    Not the solution - the problem is specific to Linux. In FreeBSD pidfiles are something taken care of a decade ago, you don’t need to think about them.
    - fullstop 5y ago
      We're almost in agreement -- FreeBSD does handle pidfiles better and more uniformly than Linux does. It doesn't help you if you want to run two instances of the same service. I looked at one of my FreeBSD servers to see how it was handled in the case of ZNC, an IRC bouncer that I use. ZNC doesn't produce a pid file on startup, so the FreeBSD RC framework tries to find a matching process in the output of 'ps'. [1] As soon as you attempt to run multiple instances of a given service, this falls over completely. [1] daemon(8) helps -- it handles restarting of processes if they crash, and it can manage pid files and prevent them from getting stale. Nothing on my FreeBSD system uses it. The unbound (dns cache) port uses the pidfile option for rc.subr(8). Looking at how check_pidfile is implemented, it attempts to verify that the process represented by the pidfile matches the process name. Pids also wrap on FreeBSD, so you have a chance of a false positive match if you run multiple instances of a given daemon. I could, of course, change unbound's rc scripts to use daemon, but that feels like a lot of thinking about pidfiles for something that was taken care of a decade ago. I do like FreeBSD, don't get me wrong, and I use it in my every day life. systemd solves problems for me, and I really like the way it manages process groups by utilizing kernel features. 1. https://docs.freebsd.org/en/articles/rc-scripting/#rcng-daemon 2. https://unix.stackexchange.com/questions/503150/rc-scripts-for-multiple-zope-instances-in-freebsd

herpderperator 5y ago

> Let's say that you're starting Apache Tomcat. You have to dig into the tomcat startup scripts and figure out where, if anywhere, it writes the pid file to disk so that you can later use it to determine if the process is running or not. https://tomcat.apache.org/tomcat-8.5-doc/windows-service-howto.html https://tomcat.apache.org/tomcat-8.5-doc/windows-service-how... The commandline argument is --PidFile and --LogPath. Most, if not all, programs allow you to customise this. It should never be a guessing game, especially when you are the one creating the init file, therefore you are the one in control of the running program.

fullstop 5y ago
Those arguments are for the Windows service and don't appear to have a corresponding Linux option. On the Linux side, if things are still done the same way as they were in the past, you had to set CATALINA_PID before starting the java process. It's still a guessing game, though, even with CATALINA_PID. It is entirely possible for Java to crash (something which RC scripts do not handle, at all) and another java process starting up which happens to be assigned the same process id as the dead java process. This can not happen with systemd units because each service unit is its own Linux cgroup and it can tell which processes belong to the service.
- toast0 5y ago
  You could just run every daemon in its own jail. You don't need to chroot (unless you want to), and don't need to do any jail specific network config (unless you want to), but you could use the jail name to ensure no more than one tomcat (or tomcat-config) process, and you could use the jail name/id to capture process tree to kill it without ambiguity. With respect to servers that crash, approaches differ, but you could use an off the shelf, single purpose daemon respawner, or you could change the whole system, or you could endeavor to not have crashing servers, such that if it crashes, it's worth a human taking a look.
  - throwawaysysd 5y ago
    Sure, you can do all that and set it up manually. I'd love to be corrected here but last I checked this was not done automatically in any BSD. Systemd recognizes this is so common that it does it automatically for every service using the Linux equivalent (cgroups). IMO now that we have these tools, every sysadmin is always going to want to use cgroups/jails for every service all the time and never want to use pidfiles because pidfiles are tremendously broken, error-prone and racy.
    - _0w8t 5y ago
      Even with systemd one may need to deal with pid files for services with more than one process. Systemd has heuristics to detect the main process to send the kill signal or detect a crash, but it can guess wrong. Telling it the pid file location makes things reliable. Plus creation of a pid file serves as a synchronization point that systemd uses to know that the process is ready. The latter is useful when the service does not support systemd native API for state transition notifications. And the best thing is that even if systemd has to be told about pid files, one never needs to deal with stalled files or pid reuse. Systemd automatically removes them on process shutdown.
      - throwawaysysd 5y ago
        A pid file is never actually reliable though. Since the supervised process has control over it, it can write any number it wants in there and trick the service manager into signaling the wrong process. As long as the process is not root and can't escape its own cgroup, systemd's pid detection is going to be less error-prone in basically every case. I can't stress this enough. Pid files are really bad. The fact that we used to have to use them is a flaw in the OS design. Using them for startup notification is also a hack in and of itself. The kernel has enough information to fully track lifecycle of these processes without writing information into files that are inherently prone to race conditions, and we now have better APIs to expose this information, so we shouldn't need to use these hacks anymore. I don't think there is any Unix-like system left that considers the old forking daemon style to be a good idea, and systemd's handling of pidfiles is really just a legacy compatibility thing.
hsbauauvhabzb 5y ago
How about Apache, Nginx, pulseaudio or ldapd?

trasz 5y ago

Your description is fine, but it’s missing one crucial detail: it’s specific to Linux. In FreeBSD this is already taken care of by rc infrastructure; there is no need for the user or sysadmin to mess with it.

fullstop 5y ago
I fully agree that it is a Linux only solution, but that's kind of the topic here. This still doesn't handle restarting crashed services, and it still is true if you need to make init scripts for your own services outside of the ports tree. It took me a long time to come to terms with systemd, but I am very glad that I did. For me it makes defining services both easy and reliable.
- trasz 5y ago
  Not the solution - the problem is specific to Linux. In FreeBSD pidfiles are something taken care of a decade ago, you don’t need to think about them.
  - fullstop 5y ago
    We're almost in agreement -- FreeBSD does handle pidfiles better and more uniformly than Linux does. It doesn't help you if you want to run two instances of the same service. I looked at one of my FreeBSD servers to see how it was handled in the case of ZNC, an IRC bouncer that I use. ZNC doesn't produce a pid file on startup, so the FreeBSD RC framework tries to find a matching process in the output of 'ps'. [1] As soon as you attempt to run multiple instances of a given service, this falls over completely. [1] daemon(8) helps -- it handles restarting of processes if they crash, and it can manage pid files and prevent them from getting stale. Nothing on my FreeBSD system uses it. The unbound (dns cache) port uses the pidfile option for rc.subr(8). Looking at how check_pidfile is implemented, it attempts to verify that the process represented by the pidfile matches the process name. Pids also wrap on FreeBSD, so you have a chance of a false positive match if you run multiple instances of a given daemon. I could, of course, change unbound's rc scripts to use daemon, but that feels like a lot of thinking about pidfiles for something that was taken care of a decade ago. I do like FreeBSD, don't get me wrong, and I use it in my every day life. systemd solves problems for me, and I really like the way it manages process groups by utilizing kernel features. 1. https://docs.freebsd.org/en/articles/rc-scripting/#rcng-daemon 2. https://unix.stackexchange.com/questions/503150/rc-scripts-for-multiple-zope-instances-in-freebsd