5 ms·

Process Managers: The Good, the Bad, and the Ugly

53 points by mattlong 13y ago

jrochkind1 13y ago
This is a super helpful writeup on a topic I needed an intro to, thanks so much!
- mattlong 13y ago
  You're welcome! I'm glad it was helpful. A lot of these devops and sysadmin related topics can be difficult to break into...especially with the rise of PaaS.
  - jrochkind1 13y ago
    Any experience with 'God', written in ruby? That's the one tool in this class that I've heard of a lot, that wasn't covered. (My guess is you're a pythonista over a rubyist)
cmwright 13y ago
Was surprised not to see God (http://godrb.com/ http://godrb.com/) here, has it fallen from grace?
- mattlong 13y ago
  There are several other systems (including God) I still really want to evaluate, unfortunately I just haven't had the time to do so yet. I'll add a note to the post saying this isn't meant to be an exhaustive list yet!
  - ocharles 13y ago
    Don't forget Angel and systemd :)
obviouslygreen 13y ago
Thanks very much for this! It's great to have someone else's insight into several of these.
- mattlong 13y ago
  You're welcome! Would love to hear your thoughts and insights on the topic. :)
bradleybuda 13y ago
I've always used (and loved) daemontools / runit (http://smarden.org/runit/ http://smarden.org/runit/) - it's a little bit minimal, but it's well-documented, does what it claims to do, and it's rock-solid.
- mattlong 13y ago
  daemontools is high on the list of other options I want to evaluate, I just haven't gotten around to it yet!
jkmcf 13y ago
FWIW, from a few months ago: http://jtimberman.housepub.org/blog/2012/12/29/process-supervision-solved-problem/ http://jtimberman.housepub.org/blog/2012/12/29/process-super... tl;dr use runit
jdnier 13y ago
Another recent comparison: http://tech.cueup.com/blog/2013/03/08/running-daemons/ http://tech.cueup.com/blog/2013/03/08/running-daemons/
gyepi 13y ago
For years, I used daemontools (it still runs a bunch of machines) then moved to runit, which follows the same philosophy. Both are excellent tools and have worked extremely well under many different circumstances. Simple tools that do one thing extremely well. Every managed service requires a short, 1 to 3 line, script to actually start the target program, which may, initially, seem off-putting, but actually provides the often missing opportunity to easily customize the program environment. BTW, runit also comes with svlogd and chpst, both of which have frequently proved invaluable. The latter is useful in its own right. Most of the tools discussed either lack flexibility, try to do too much, or simply don't offer any benefits over runit and company.
- mattlong 13y ago
  I agree with pretty much all of what you said about how a system like this should work. daemontools was next on my list of solutions to try out. I wish I had discovered runit before doing this rundown...my google-fu is apparently lacking a bit :)
simonw 13y ago
I swear this is one of the hardest problems in computer science, which is crazy - how hard should it really be to say "run this process, and restart it if it crashes"?
- otterley 13y ago
  It's not a hard problem at all; it's a rather simple loop: while true: pid = fork() if pid: # parent wait(pid) else: # child exec("/path/to/supervised/process") The key here is not to let the supervised process daemonize itself. If it does, the parent can no longer wait() on the child.
duggan 13y ago
Timely, since I spent some time over the weekend digging into the internals of Supervisor. One of the most interesting features, for me, is that Supervisor has an XML-RPC interface[1], allowing for some creative interaction with supervisor and the processes it manages. [1] http://supervisord.org/api.html http://supervisord.org/api.html
ionelm 13y ago
The article is a bit wrong about supervisord. You can load environment variables from a defaults file - just use a wrapper script. Eg: #!/bin/sh source /etc/myservice/defaults exec celeryd $@ And start that from supervisord. Also, another nice thing, very useful for daemons that spawn their own subprocesses is killasgroup option. This is worth mentioning given the celery examples - celery can and will leave orphan processes around between restarts ...
sciurus 13y ago
"The convenience of custom commands like /etc/init.d/nginx configtest to check my nginx configuration syntax without affecting the running nginx service is useful enough to keep me from migrating my nginx daemonization to Upstart." Needing to run "nginx -t" instead of "/etc/init.d/nginx configtest" seems like a minor tradeoff for the benefits Upstart provides. It's even shorter to type!
cmccabe 13y ago
systemd is the way to go. Don't waste time debugging someone's half-baked pile of {shell,Perl,Python} scripts

53 points by mattlong 13y ago

jrochkind1 13y ago

This is a super helpful writeup on a topic I needed an intro to, thanks so much!

mattlong 13y ago
You're welcome! I'm glad it was helpful. A lot of these devops and sysadmin related topics can be difficult to break into...especially with the rise of PaaS.
- jrochkind1 13y ago
  Any experience with 'God', written in ruby? That's the one tool in this class that I've heard of a lot, that wasn't covered. (My guess is you're a pythonista over a rubyist)

cmwright 13y ago

Was surprised not to see God (http://godrb.com/ http://godrb.com/) here, has it fallen from grace?

mattlong 13y ago
There are several other systems (including God) I still really want to evaluate, unfortunately I just haven't had the time to do so yet. I'll add a note to the post saying this isn't meant to be an exhaustive list yet!
- ocharles 13y ago
  Don't forget Angel and systemd :)

obviouslygreen 13y ago

Thanks very much for this! It's great to have someone else's insight into several of these.

mattlong 13y ago
You're welcome! Would love to hear your thoughts and insights on the topic. :)

bradleybuda 13y ago

I've always used (and loved) daemontools / runit (http://smarden.org/runit/ http://smarden.org/runit/) - it's a little bit minimal, but it's well-documented, does what it claims to do, and it's rock-solid.

mattlong 13y ago
daemontools is high on the list of other options I want to evaluate, I just haven't gotten around to it yet!

jkmcf 13y ago

FWIW, from a few months ago: http://jtimberman.housepub.org/blog/2012/12/29/process-supervision-solved-problem/ http://jtimberman.housepub.org/blog/2012/12/29/process-super... tl;dr use runit

jdnier 13y ago

Another recent comparison: http://tech.cueup.com/blog/2013/03/08/running-daemons/ http://tech.cueup.com/blog/2013/03/08/running-daemons/

gyepi 13y ago

For years, I used daemontools (it still runs a bunch of machines) then moved to runit, which follows the same philosophy. Both are excellent tools and have worked extremely well under many different circumstances. Simple tools that do one thing extremely well. Every managed service requires a short, 1 to 3 line, script to actually start the target program, which may, initially, seem off-putting, but actually provides the often missing opportunity to easily customize the program environment. BTW, runit also comes with svlogd and chpst, both of which have frequently proved invaluable. The latter is useful in its own right. Most of the tools discussed either lack flexibility, try to do too much, or simply don't offer any benefits over runit and company.

mattlong 13y ago
I agree with pretty much all of what you said about how a system like this should work. daemontools was next on my list of solutions to try out. I wish I had discovered runit before doing this rundown...my google-fu is apparently lacking a bit :)

simonw 13y ago

I swear this is one of the hardest problems in computer science, which is crazy - how hard should it really be to say "run this process, and restart it if it crashes"?

otterley 13y ago
It's not a hard problem at all; it's a rather simple loop: while true: pid = fork() if pid: # parent wait(pid) else: # child exec("/path/to/supervised/process") The key here is not to let the supervised process daemonize itself. If it does, the parent can no longer wait() on the child.

duggan 13y ago

Timely, since I spent some time over the weekend digging into the internals of Supervisor. One of the most interesting features, for me, is that Supervisor has an XML-RPC interface[1], allowing for some creative interaction with supervisor and the processes it manages. [1] http://supervisord.org/api.html http://supervisord.org/api.html

ionelm 13y ago

The article is a bit wrong about supervisord. You can load environment variables from a defaults file - just use a wrapper script. Eg: #!/bin/sh source /etc/myservice/defaults exec celeryd $@ And start that from supervisord. Also, another nice thing, very useful for daemons that spawn their own subprocesses is killasgroup option. This is worth mentioning given the celery examples - celery can and will leave orphan processes around between restarts ...

sciurus 13y ago

"The convenience of custom commands like /etc/init.d/nginx configtest to check my nginx configuration syntax without affecting the running nginx service is useful enough to keep me from migrating my nginx daemonization to Upstart." Needing to run "nginx -t" instead of "/etc/init.d/nginx configtest" seems like a minor tradeoff for the benefits Upstart provides. It's even shorter to type!

cmccabe 13y ago

systemd is the way to go. Don't waste time debugging someone's half-baked pile of {shell,Perl,Python} scripts