11 ms·

You don't need to daemonize

115 points by vezzy-fnord 11y ago

ah- 11y ago
Seeing then FQDN with the trailing dot I wonder how much software breaks when you use it. I bet the duplicate check here doesn't treat homepage.ntlworld.com. and homepage.ntlworld.com as equal.
- vezzy-fnord 11y ago
  JdeBP is famous for his ungodly pedantry, so it'd be blasphemous for me not to honor RFC 1738 [1]. [1] http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/web-fully-qualified-domain-name.html http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/we...
  - nly 11y ago
    Wrt hostnames RFC 1738 just refers back to 1034/35, which came 7 years earlier. It'd be pretty sad if we couldn't get this right after 28+ years
  - bshimmin 11y ago
    He's also famous enough that whenever I see ntlworld.com [1] I immediately think, "Ah, must be JdBP - that'll be worth reading." [1] It's actually slightly astonishing his hosting still works at that URL given that NTL was consumed by Virgin nearly a decade ago.
    - darkr 11y ago
      I must admit, I felt a pang of nostalgia seeing that domain name. Must have been nearly 15 years ago that we got our super-fast 64k connection with free ntlworld.com web hosting, and we could finally have always-on internet at home. Oh how things have changed..
- batou 11y ago
  Actually none from experience. It's there for a really good reason. Say you want to visit www.hahaha.com but your network admin is a rotten bastard, and this does happen a lot in corporate networks, well what happens is they set up a DNS record of www.hahaha.com.myevilcorporation.local and set your default domain suffix to myevilcorporation.local. When you hit www.hahaha.com you don't go to www.hahaha.com; you go to some internal site. The browser tends not to give you any clues about this so you just carry on without knowing. If you visit www.hahaha.com. (note the trailing dot) then you go to www.hahaha.com. as that dot is the root of the DNS heirarchy and myevilcorporation.local never ever resolves. Of course this throws all sorts of warnings if you are using SSL but for the average user and site, that isn't necessarily true. Hell we did it years ago and deployed an internal CA cert to all our workstations so it looks like a ton of our corporate sites are actually internet-based and SSL covered but are on internal networks only with private certs. I suspect that this is less of an issue these days and browsers have protection against this but I wouldn't be 100% sure of it. So in conclusion, it's really important.
  - amalcon 11y ago
    Public service announcement: It's best to use "example.com" (or .net, .org, etc) for this sort of thing, because there's no real server there to accidentally send traffic to or alter the search rankings of. It's not usually that big a deal, but every so often someone will paste the wrong thing or an automated process will misinterpret something. It's better if this happens to a nonexistent domain.
    - vacri 11y ago
      There is a real server at example.com - check out http://example.com/ http://example.com/ - or, if you want to be secure, it's also on https :) example.com is the standard FQDN to use for examples, though. http://example.com/ http://example.com/ links to the documentation...
jebblue 11y ago
I wrote a Java server that blocks spam based on my criteria and set it to run like this: http://manpages.ubuntu.com/manpages/trusty/man8/update-rc.d.8.html http://manpages.ubuntu.com/manpages/trusty/man8/update-rc.d.... I didn't do any daemonzing like we used to do in the C days so I'm not sure that the systemd approach mentioned in the article does it better (not against the idea, maybe it's needed on non-Ubuntu systems?). I just needed to write a simple shell script with the basics and this: ... case "$1" in start) startup ;; stop) shutdown ;; restart) restartit ;; esac ...
- deleted 11y ago
  [deleted]
- fbuilesv 11y ago
  If all you're doing is start/stop/restart on Ubuntu you should take a look at Upstart[0]. See a simple example here: http://askubuntu.com/a/251581 http://askubuntu.com/a/251581 [0] http://upstart.ubuntu.com/ http://upstart.ubuntu.com/
  - hactually 11y ago
    upstart is deprecated.
    - vezzy-fnord 11y ago
      Not that I know of. ChromeOS still uses it and though Ubuntu is phasing it out, they haven't completely gotten rid of it as a session manager yet AFAIK. They need to support it for a few more years anyway due to 14.04 LTS.
      - hobarrera 11y ago
        That's exactly the difference between deprecated an obsolete. The former means it's on it's way to the latter. Putting work into upstart just means you'll have work migrating to systemd in less than a year or two.
snorkel 11y ago
I don't understand. If I write a server app that listens for client connections, why not daemonize that? Is systemd doing all of the listening then invoking the server app if a client connects?
- ninkendo 11y ago
  Run in the foreground and systemd will do the right thing. Your app is still the one listening for connections, systemd just calls wait() until you exit, then launches you again. There's no need for you to do a fork() or setsid() or whatever, that was a relic of the "just run all these init files as scripts" nature of sysv init.
  - eropple 11y ago
    Definitely. Also, when you're in non-systemd environments, runit or supervisord will happily manage your undaemons.
    - cgag 11y ago
      I just started using runit and really like how simple it is. I feel like nows a good time to plug void linux http://www.voidlinux.eu/ http://www.voidlinux.eu/ which I think is pretty cool, it's the only OS/distro I know that uses runit as the init system.
  - joevandyk 11y ago
    How does that work with nginx or haproxy, which support zero-downtime restarts?
    - otterley 11y ago
      If nginx and haproxy switch over to SO_REUSEPORT (and the SO_REUSEPORT semantics are fixed in Linux to work like FreeBSD's) it should no longer be an issue: its own worker supervisor need not fork itself, and it can simply manage the children.
    - 11y ago
pjungwir 11y ago
For Rails apps I've been using god to manage unicorn and delayed_job/resque processes in recent years. I've been mildly skeptical of systemd---not really out of true knowledge but because keeping things simple matters to me, and that seems to be the gist of the criticisms. But if systemd would let me do away with god, that would be nice! I agree that pidfiles are a hack and often cause problems. It seems like I'm going to need to learn systemd soon anyway, so I'll be looking into whether it can do what I need for Rails apps.
kazinator 11y ago
You can run a daemon-like process with its stdin/out firmly attached to a TTY: a virtual TTY inside a screen or tmux session that can be detached and reattached.
- Animats 11y ago
  As was pointed out a few days ago, if the output stream exerts backpressure, write calls will stall. Stopping scrolling at the output end is enough to do that.
  - __david__ 11y ago
    But that's the whole point of screen and tmux—It's always got a terminal to suck up stdout.
  - deleted 11y ago
    [deleted]
lealanko 11y ago
The problem with automatic daemonization is that there is no way to indicate startup failure: you just fire and forget the process in the background. If the process dies, there's no clear way of knowing whether it was due to an error in startup conditions (possibly as simple as a configuration error) or a runtime failure at a later time. In the first case, restarting makes no sense. In the latter case it might be a reasonable option. With manual daemonization, the process can delay forking until it has ensured that its configuration is sound and it has the resources it needs. This way the parent process can indicate with its exit status whether the service was successfully launched.
- icebraining 11y ago
  Usually these daemon managers detect when the process is dying right after starting up and indicate that with a message like "[process] is respawning too fast". Besides, there are always logs.
  - eropple 11y ago
    But that's not a way to detect service failure. That's a way to detect potential service failure. The only unambiguous failure notification is the return code.
    - icebraining 11y ago
      Yes, which systemd will also display, and you can configure it to handle it differently (e.g. don't auto-restart on certain exit codes).
      - deleted 11y ago
        [deleted]
- yrro 11y ago
  If your daemon is of Type=forking then failure to initialize should be indicated by the initial process (the one launched by systemd) outputting an error message and then failing. Only once the grandchild (not the child, you must double-fork in order to start up correctly) has indicated to the initial process that it is ready to provide the service, should the initial process exit, cleanly, thereby notifying systemd that startup was successful. See daemon(7) for the full details.
scurvy 11y ago
We still run daemontools even in systemd systems. The OS packaged cruft runs in systemd. Everything important runs in daemontools. Still the best "services manager" out there.
- otterley 11y ago
  Why? It's no longer necessary. Also, daemontools suffers from the same problem as other non-systemd service managers; it can't manage processes that daemonize themselves for good reason (like nginx or Unicorn). systemd can handle it because it creates a unique cgroup for every service; if the service needs to be stopped, all processes in the cgroup are terminated.
  - vezzy-fnord 11y ago
    There's plenty of successors to daemontools following the same principles, but more advanced: s6, nosh, perp, runit, etc. That said, from what I know systemd doesn't necessarily handle double-forking daemons any better. That is to say, if you set the service to Type=forking, it'll still need to use a PID file (with its inherent race conditions) or employ a PID guessing heuristic, which can fail. This is a universal problem that no hack can truly solve. The bottom line is if you're running under a service manager, you don't daemonize. You delegate to the service manager to daemonize for you. Any deviation from this will be finicky on most Unix-likes. The purpose of cgroups here is instead process tracking, i.e. reliably killing all children. But for some services this is exactly what you don't want, and then there's nothing stopping you from allocating cgroups yourself if needed, or talk to a cgroup hierarchy management daemon like cgmanager. Ultimately, you just want some unit of isolation here, so you can use whatever your platform has, i.e. jails or contracts.
    - thwarted 11y ago
      One little known feature added in Linux 3.4 is the PR_SET_CHILD_SUBREAPER prctl(2) [0] that lets you designate the process as being a "subreaper" and it will be notified of any descending subprocess exits, just like init would be. [0] http://man7.org/linux/man-pages/man2/prctl.2.html http://man7.org/linux/man-pages/man2/prctl.2.html
quicksilver03 11y ago
I tend to always use daemonize to start a Java program because typically I need to pass several startup options (like Xms, Xmx, etc.) and systemd unit files cannot perform even the simplest variable interpolation needed [1]. Someday systemd will go beyond simple key-value substitution, and on that day I'll stop using daemonize. [1]: http://www.opsfordevelopers.com/2015/05/startup-scripts-for-java-based-services/ http://www.opsfordevelopers.com/2015/05/startup-scripts-for-...
- hactually 11y ago
  What's wrong with doing something like the following: ... [Service] Type=forking EnvironmentFile=-/usr/local/myJavaConfig ExecStart=/bin/sh -c '${JAVA_HOME} ${JAVA_OPTS}' Or if you're after passing things in, use the other technique that lets you use systemctl myservice@your_sub.service
  - JdeBP 11y ago
    Two things are wrong, neither of which, however, make daemonize appropriate. First, and trivially, there's a missing /bin/java (-: Second, that's a potential readiness protocol and a main PID mismatch. See http://unix.stackexchange.com/a/200365/5132 http://unix.stackexchange.com/a/200365/5132 for details. And then http://askubuntu.com/a/624871/43344 http://askubuntu.com/a/624871/43344 . Remembering to exec is another lesson to bring along from the daemontools world.
- yrro 11y ago
  That's a case for creating a wrapper shell script that you launch through ExecStart=. Nothing to do with whether your process does the daemon(7) dance or not.
  - quicksilver03 11y ago
    Correct, I should've been clearer that I use daemonize mostly out of frustration with systemd limitations and inability to tell me precisely what's wrong when there are errors launching the service.
__david__ 11y ago
I've been using daemon-manager for the past couple years—its most interesting point to me is that you can start and stop user daemons without require root privileges, which make it nice for non-root scripts/bots (for the paranoid age) and for multi-user systems (admittedly decreasingly common in the AWS and Docker age).
- icebraining 11y ago
  "systemd offers users the ability to manage services under the user's control with a per-user systemd instance, enabling users to start, stop, enable, and disable their own units." https://wiki.archlinux.org/index.php/Systemd/User https://wiki.archlinux.org/index.php/Systemd/User
joshribakoff 11y ago
I've used the python package supervisord with good success. Out of the box it manages stdout & stderr, rotates log files, restarts crashed processes, and has plugins for things like REST APIs to restart the process (for example, during an automated deploy). Another plugin can monitor memory usage & take some action when thresholds are reached. I find it very easy to use. You can commit your configuration file for supervisor into your app's repo, and symlink it in to supervisor's conf.d directory. Other plugins add stuff like web based dash boards. Its great for everything from managing a single process on a single machine, or a large cluster of worker processes on multiple servers.
- kzahel 11y ago
  I like supervisord for these reasons too. It's really handy.

115 points by vezzy-fnord 11y ago

ah- 11y ago

Seeing then FQDN with the trailing dot I wonder how much software breaks when you use it. I bet the duplicate check here doesn't treat homepage.ntlworld.com. and homepage.ntlworld.com as equal.

vezzy-fnord 11y ago
JdeBP is famous for his ungodly pedantry, so it'd be blasphemous for me not to honor RFC 1738 [1]. [1] http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/web-fully-qualified-domain-name.html http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/we...
- nly 11y ago
  Wrt hostnames RFC 1738 just refers back to 1034/35, which came 7 years earlier. It'd be pretty sad if we couldn't get this right after 28+ years
- bshimmin 11y ago
  He's also famous enough that whenever I see ntlworld.com [1] I immediately think, "Ah, must be JdBP - that'll be worth reading." [1] It's actually slightly astonishing his hosting still works at that URL given that NTL was consumed by Virgin nearly a decade ago.
  - darkr 11y ago
    I must admit, I felt a pang of nostalgia seeing that domain name. Must have been nearly 15 years ago that we got our super-fast 64k connection with free ntlworld.com web hosting, and we could finally have always-on internet at home. Oh how things have changed..
batou 11y ago
Actually none from experience. It's there for a really good reason. Say you want to visit www.hahaha.com but your network admin is a rotten bastard, and this does happen a lot in corporate networks, well what happens is they set up a DNS record of www.hahaha.com.myevilcorporation.local and set your default domain suffix to myevilcorporation.local. When you hit www.hahaha.com you don't go to www.hahaha.com; you go to some internal site. The browser tends not to give you any clues about this so you just carry on without knowing. If you visit www.hahaha.com. (note the trailing dot) then you go to www.hahaha.com. as that dot is the root of the DNS heirarchy and myevilcorporation.local never ever resolves. Of course this throws all sorts of warnings if you are using SSL but for the average user and site, that isn't necessarily true. Hell we did it years ago and deployed an internal CA cert to all our workstations so it looks like a ton of our corporate sites are actually internet-based and SSL covered but are on internal networks only with private certs. I suspect that this is less of an issue these days and browsers have protection against this but I wouldn't be 100% sure of it. So in conclusion, it's really important.
- amalcon 11y ago
  Public service announcement: It's best to use "example.com" (or .net, .org, etc) for this sort of thing, because there's no real server there to accidentally send traffic to or alter the search rankings of. It's not usually that big a deal, but every so often someone will paste the wrong thing or an automated process will misinterpret something. It's better if this happens to a nonexistent domain.
  - vacri 11y ago
    There is a real server at example.com - check out http://example.com/ http://example.com/ - or, if you want to be secure, it's also on https :) example.com is the standard FQDN to use for examples, though. http://example.com/ http://example.com/ links to the documentation...

jebblue 11y ago

I wrote a Java server that blocks spam based on my criteria and set it to run like this: http://manpages.ubuntu.com/manpages/trusty/man8/update-rc.d.8.html http://manpages.ubuntu.com/manpages/trusty/man8/update-rc.d.... I didn't do any daemonzing like we used to do in the C days so I'm not sure that the systemd approach mentioned in the article does it better (not against the idea, maybe it's needed on non-Ubuntu systems?). I just needed to write a simple shell script with the basics and this: ... case "$1" in start) startup ;; stop) shutdown ;; restart) restartit ;; esac ...

deleted 11y ago
[deleted]
fbuilesv 11y ago
If all you're doing is start/stop/restart on Ubuntu you should take a look at Upstart[0]. See a simple example here: http://askubuntu.com/a/251581 http://askubuntu.com/a/251581 [0] http://upstart.ubuntu.com/ http://upstart.ubuntu.com/
- hactually 11y ago
  upstart is deprecated.
  - vezzy-fnord 11y ago
    Not that I know of. ChromeOS still uses it and though Ubuntu is phasing it out, they haven't completely gotten rid of it as a session manager yet AFAIK. They need to support it for a few more years anyway due to 14.04 LTS.
    - hobarrera 11y ago
      That's exactly the difference between deprecated an obsolete. The former means it's on it's way to the latter. Putting work into upstart just means you'll have work migrating to systemd in less than a year or two.

snorkel 11y ago

I don't understand. If I write a server app that listens for client connections, why not daemonize that? Is systemd doing all of the listening then invoking the server app if a client connects?

ninkendo 11y ago
Run in the foreground and systemd will do the right thing. Your app is still the one listening for connections, systemd just calls wait() until you exit, then launches you again. There's no need for you to do a fork() or setsid() or whatever, that was a relic of the "just run all these init files as scripts" nature of sysv init.
- eropple 11y ago
  Definitely. Also, when you're in non-systemd environments, runit or supervisord will happily manage your undaemons.
  - cgag 11y ago
    I just started using runit and really like how simple it is. I feel like nows a good time to plug void linux http://www.voidlinux.eu/ http://www.voidlinux.eu/ which I think is pretty cool, it's the only OS/distro I know that uses runit as the init system.
- joevandyk 11y ago
  How does that work with nginx or haproxy, which support zero-downtime restarts?
  - otterley 11y ago
    If nginx and haproxy switch over to SO_REUSEPORT (and the SO_REUSEPORT semantics are fixed in Linux to work like FreeBSD's) it should no longer be an issue: its own worker supervisor need not fork itself, and it can simply manage the children.
  - 11y ago

pjungwir 11y ago

For Rails apps I've been using god to manage unicorn and delayed_job/resque processes in recent years. I've been mildly skeptical of systemd---not really out of true knowledge but because keeping things simple matters to me, and that seems to be the gist of the criticisms. But if systemd would let me do away with god, that would be nice! I agree that pidfiles are a hack and often cause problems. It seems like I'm going to need to learn systemd soon anyway, so I'll be looking into whether it can do what I need for Rails apps.

kazinator 11y ago

You can run a daemon-like process with its stdin/out firmly attached to a TTY: a virtual TTY inside a screen or tmux session that can be detached and reattached.

Animats 11y ago
As was pointed out a few days ago, if the output stream exerts backpressure, write calls will stall. Stopping scrolling at the output end is enough to do that.
- __david__ 11y ago
  But that's the whole point of screen and tmux—It's always got a terminal to suck up stdout.
- deleted 11y ago
  [deleted]

lealanko 11y ago

The problem with automatic daemonization is that there is no way to indicate startup failure: you just fire and forget the process in the background. If the process dies, there's no clear way of knowing whether it was due to an error in startup conditions (possibly as simple as a configuration error) or a runtime failure at a later time. In the first case, restarting makes no sense. In the latter case it might be a reasonable option. With manual daemonization, the process can delay forking until it has ensured that its configuration is sound and it has the resources it needs. This way the parent process can indicate with its exit status whether the service was successfully launched.

icebraining 11y ago
Usually these daemon managers detect when the process is dying right after starting up and indicate that with a message like "[process] is respawning too fast". Besides, there are always logs.
- eropple 11y ago
  But that's not a way to detect service failure. That's a way to detect potential service failure. The only unambiguous failure notification is the return code.
  - icebraining 11y ago
    Yes, which systemd will also display, and you can configure it to handle it differently (e.g. don't auto-restart on certain exit codes).
    - deleted 11y ago
      [deleted]
yrro 11y ago
If your daemon is of Type=forking then failure to initialize should be indicated by the initial process (the one launched by systemd) outputting an error message and then failing. Only once the grandchild (not the child, you must double-fork in order to start up correctly) has indicated to the initial process that it is ready to provide the service, should the initial process exit, cleanly, thereby notifying systemd that startup was successful. See daemon(7) for the full details.

scurvy 11y ago

We still run daemontools even in systemd systems. The OS packaged cruft runs in systemd. Everything important runs in daemontools. Still the best "services manager" out there.

otterley 11y ago
Why? It's no longer necessary. Also, daemontools suffers from the same problem as other non-systemd service managers; it can't manage processes that daemonize themselves for good reason (like nginx or Unicorn). systemd can handle it because it creates a unique cgroup for every service; if the service needs to be stopped, all processes in the cgroup are terminated.
- vezzy-fnord 11y ago
  There's plenty of successors to daemontools following the same principles, but more advanced: s6, nosh, perp, runit, etc. That said, from what I know systemd doesn't necessarily handle double-forking daemons any better. That is to say, if you set the service to Type=forking, it'll still need to use a PID file (with its inherent race conditions) or employ a PID guessing heuristic, which can fail. This is a universal problem that no hack can truly solve. The bottom line is if you're running under a service manager, you don't daemonize. You delegate to the service manager to daemonize for you. Any deviation from this will be finicky on most Unix-likes. The purpose of cgroups here is instead process tracking, i.e. reliably killing all children. But for some services this is exactly what you don't want, and then there's nothing stopping you from allocating cgroups yourself if needed, or talk to a cgroup hierarchy management daemon like cgmanager. Ultimately, you just want some unit of isolation here, so you can use whatever your platform has, i.e. jails or contracts.
  - thwarted 11y ago
    One little known feature added in Linux 3.4 is the PR_SET_CHILD_SUBREAPER prctl(2) [0] that lets you designate the process as being a "subreaper" and it will be notified of any descending subprocess exits, just like init would be. [0] http://man7.org/linux/man-pages/man2/prctl.2.html http://man7.org/linux/man-pages/man2/prctl.2.html

quicksilver03 11y ago

I tend to always use daemonize to start a Java program because typically I need to pass several startup options (like Xms, Xmx, etc.) and systemd unit files cannot perform even the simplest variable interpolation needed [1]. Someday systemd will go beyond simple key-value substitution, and on that day I'll stop using daemonize. [1]: http://www.opsfordevelopers.com/2015/05/startup-scripts-for-java-based-services/ http://www.opsfordevelopers.com/2015/05/startup-scripts-for-...

hactually 11y ago
What's wrong with doing something like the following: ... [Service] Type=forking EnvironmentFile=-/usr/local/myJavaConfig ExecStart=/bin/sh -c '${JAVA_HOME} ${JAVA_OPTS}' Or if you're after passing things in, use the other technique that lets you use systemctl myservice@your_sub.service
- JdeBP 11y ago
  Two things are wrong, neither of which, however, make daemonize appropriate. First, and trivially, there's a missing /bin/java (-: Second, that's a potential readiness protocol and a main PID mismatch. See http://unix.stackexchange.com/a/200365/5132 http://unix.stackexchange.com/a/200365/5132 for details. And then http://askubuntu.com/a/624871/43344 http://askubuntu.com/a/624871/43344 . Remembering to exec is another lesson to bring along from the daemontools world.
yrro 11y ago
That's a case for creating a wrapper shell script that you launch through ExecStart=. Nothing to do with whether your process does the daemon(7) dance or not.
- quicksilver03 11y ago
  Correct, I should've been clearer that I use daemonize mostly out of frustration with systemd limitations and inability to tell me precisely what's wrong when there are errors launching the service.

__david__ 11y ago

I've been using daemon-manager for the past couple years—its most interesting point to me is that you can start and stop user daemons without require root privileges, which make it nice for non-root scripts/bots (for the paranoid age) and for multi-user systems (admittedly decreasingly common in the AWS and Docker age).

icebraining 11y ago
"systemd offers users the ability to manage services under the user's control with a per-user systemd instance, enabling users to start, stop, enable, and disable their own units." https://wiki.archlinux.org/index.php/Systemd/User https://wiki.archlinux.org/index.php/Systemd/User

joshribakoff 11y ago

I've used the python package supervisord with good success. Out of the box it manages stdout & stderr, rotates log files, restarts crashed processes, and has plugins for things like REST APIs to restart the process (for example, during an automated deploy). Another plugin can monitor memory usage & take some action when thresholds are reached. I find it very easy to use. You can commit your configuration file for supervisor into your app's repo, and symlink it in to supervisor's conf.d directory. Other plugins add stuff like web based dash boards. Its great for everything from managing a single process on a single machine, or a large cluster of worker processes on multiple servers.

kzahel 11y ago
I like supervisord for these reasons too. It's really handy.