4 ms·

We run both nginx and unicorn through daemontools. Both have options to prevent daemonization. Why run something terrible (systemd) when we already have someth

by scurvy 11y ago

We run both nginx and unicorn through daemontools. Both have options to prevent daemonization. Why run something terrible (systemd) when we already have something that works really well (daemontools)?

KirinDave 11y ago
> Why run something terrible (systemd) It is terrible only because you say it is. It's been a massive help for us in adopting it, and accepting the help it offers makes custom software systems a lot easier to write in a robust way. It's only "terrible" if you have axe to grind about the "right" way and then define that as daemontools.
- cgag 11y ago
  I use systemd to start runit which is like daemontools. I consider daemontools/runit it right because they gives me all the benefits I'd want out of systemd (not doing anything special to daemonize my process, supervision, log handling) while being much simpler and more well tested.
  - KirinDave 11y ago
    > while being much simpler The sum of the init functional that can deliver what systemd is doing (and it is good to do those things) is more complex than a suite of tools for launching daemons in an unhelpful environment. Sure. I submit the universal configuration file tooling is fantastic in a day and age where we provision and configure machines with template-based tools like salt and ansible rather than synchronized command tools like puppet.
    - cgag 11y ago
      I'm not sure I understand what you're saying. Is it that it's nice to be able to copy over a systemd service template file with ansible vs running commands to start services? If so, how is that different from copying runit service files instead? If not, can you elaborate / reword?
      - KirinDave 11y ago
        Sorry, that post was on mobile. Honestly, I do not relish posting on this terrible corner of the internet. But I'll respond, and it's better late than never. There are 2 things: 1. Systemd's boot process does more (and shows better performance and in some cases better resilience). So while many people complain it is more complicated and this is true; it is still very simple for what it is doing (which is much more than daemontools). 2. Runit and monit and stuff all eventually appeal to launcher shell scripts. You do not 'copy' these over with ansible in a real operational deployment, you 'generate' them with profile-specific variables (e.g., beta vs production, datacenter-specific values, etc). Template generation of shell scripts is MUCH more subtly error prone than generating configuration files with simpler grammar and static correctness checks. Systemd makes more functionality available as a configuration option as opposed to functionality available via shell scripting. I think that makes it much better.
- scurvy 11y ago
  I have an axe to grind about the way it was introduced. Fedora and Red Hat massively mis-managed it. Previously, they introduced various things in tech preview releases, and it either stayed or went with subsequent releases. systemd was thrust upon us by an axe-grinding developer.
  - icebraining 11y ago
    Sounds like your problem is with Red Hat, not systemd.
  - hactually 11y ago
    What axe did they grind? I've got only good things to say about systemd so perhaps I can't understand the issue?
  - rodgerd 11y ago
    So, stupid politics rather than technical merit.
- jamiesonbecker 11y ago
  > It's only "terrible" if you have axe to grind about the "right" way and then define that as daemontools. Conversely, It's only "terrible" if you have axe to grind about the "right" way and then define that as systemd.
viraptor 11y ago
While I'm not a fan of systemd (same thing can be achieved with upstart/supervisord for example), there's just no point to run daemontools on top of the new inits. deamontools provides: logging stdout, restart, keeping things running. That's pretty much all. The exact thing is already provided by systemd/upstart/supervisord, with additional benefits: resource limits, namespaces, different behaviour on multiple failures (daemontools will just keep restarting your process as fast as possible, taking 100% CPU if possible and flooding the logs), support for syslog/journal rather than just local files. By putting daemontools on top of systemd, you're just adding simple system on top of a complex one. You're losing features, but not gaining anything instead.
- pmahoney 11y ago
  You've got some of the daemontools details incorrect. > resource limits Daemontools includes the softlimit [1] helper. > namespaces It's not clear to me why this need be built in to a supervisor instead of being applied through generic helper programs such as unshare(1). > daemontools will just keep restarting your process as fast as possible, taking 100% CPU if possible supervise [2]: "It restarts ./run if ./run exits. It pauses for a second after starting ./run, so that it does not loop too quickly if ./run exits immediately." Simple and not configurable, sure, but should not hammer the CPU. > support for syslog/journal rather than just local files Daemontools svscan [3] "optionally starts a pair of supervise processes, one for a subdirectory s, one for s/log, with a pipe between them". The "s/log" script can do anything you wish, reading logs on stdin. If you use the included "multilog" program, then it's certainly geared toward writing local files, though it does include the ability to run arbitrary post-processor during rotation (which might, for example, fire off a job to copy the logs to an aggregator). Or you can not run multilog and just send the logs to syslog or whatever. What I do myself is send save the logs locally using s6's multilog analog, s6-log, and also pipe them to a local syslog that forwards them to an aggregator outside my control. And, in spite of newer supervisors adding more and more features beyond what daemontools provides, daemontools is still an awesome system that was 14 years ahead of its time. [1] http://cr.yp.to/daemontools/softlimit.html http://cr.yp.to/daemontools/softlimit.html [2] http://cr.yp.to/daemontools/supervise.html http://cr.yp.to/daemontools/supervise.html [3] http://cr.yp.to/daemontools/svscan.html http://cr.yp.to/daemontools/svscan.html
otterley 11y ago
> We run both nginx and unicorn through daemontools. Both have options to prevent daemonization. How are you able to do hot-reloads without daemonizing the master process? Or have you simply chosen to sacrifice this feature?
- scurvy 11y ago
  Hot reload? Like a config reload? Just send the master process a HUP.
  - otterley 11y ago
    That can't possibly work correctly with runsv, given the way in which Unicorn performs configuration reloads. Eventually the original process that runsv forked will die, and runsv will try to restart it and fail (because the original process launched a new generation that continues to listen to the socket). Then 'sv stop' will fail to work because the new generation of master and workers isn't under runsv's supervision anymore.
    - scurvy 11y ago
      nginx doesn't die. It just reloads the config file and applies the changes. I can't remember what Unicorn does because we almost never change its config. Also, we're using daemontools not runit.

by scurvy 11y ago

KirinDave 11y ago

> Why run something terrible (systemd) It is terrible only because you say it is. It's been a massive help for us in adopting it, and accepting the help it offers makes custom software systems a lot easier to write in a robust way. It's only "terrible" if you have axe to grind about the "right" way and then define that as daemontools.

cgag 11y ago
I use systemd to start runit which is like daemontools. I consider daemontools/runit it right because they gives me all the benefits I'd want out of systemd (not doing anything special to daemonize my process, supervision, log handling) while being much simpler and more well tested.
- KirinDave 11y ago
  > while being much simpler The sum of the init functional that can deliver what systemd is doing (and it is good to do those things) is more complex than a suite of tools for launching daemons in an unhelpful environment. Sure. I submit the universal configuration file tooling is fantastic in a day and age where we provision and configure machines with template-based tools like salt and ansible rather than synchronized command tools like puppet.
  - cgag 11y ago
    I'm not sure I understand what you're saying. Is it that it's nice to be able to copy over a systemd service template file with ansible vs running commands to start services? If so, how is that different from copying runit service files instead? If not, can you elaborate / reword?
    - KirinDave 11y ago
      Sorry, that post was on mobile. Honestly, I do not relish posting on this terrible corner of the internet. But I'll respond, and it's better late than never. There are 2 things: 1. Systemd's boot process does more (and shows better performance and in some cases better resilience). So while many people complain it is more complicated and this is true; it is still very simple for what it is doing (which is much more than daemontools). 2. Runit and monit and stuff all eventually appeal to launcher shell scripts. You do not 'copy' these over with ansible in a real operational deployment, you 'generate' them with profile-specific variables (e.g., beta vs production, datacenter-specific values, etc). Template generation of shell scripts is MUCH more subtly error prone than generating configuration files with simpler grammar and static correctness checks. Systemd makes more functionality available as a configuration option as opposed to functionality available via shell scripting. I think that makes it much better.
scurvy 11y ago
I have an axe to grind about the way it was introduced. Fedora and Red Hat massively mis-managed it. Previously, they introduced various things in tech preview releases, and it either stayed or went with subsequent releases. systemd was thrust upon us by an axe-grinding developer.
- icebraining 11y ago
  Sounds like your problem is with Red Hat, not systemd.
- hactually 11y ago
  What axe did they grind? I've got only good things to say about systemd so perhaps I can't understand the issue?
- rodgerd 11y ago
  So, stupid politics rather than technical merit.
jamiesonbecker 11y ago
> It's only "terrible" if you have axe to grind about the "right" way and then define that as daemontools. Conversely, It's only "terrible" if you have axe to grind about the "right" way and then define that as systemd.

viraptor 11y ago

While I'm not a fan of systemd (same thing can be achieved with upstart/supervisord for example), there's just no point to run daemontools on top of the new inits. deamontools provides: logging stdout, restart, keeping things running. That's pretty much all. The exact thing is already provided by systemd/upstart/supervisord, with additional benefits: resource limits, namespaces, different behaviour on multiple failures (daemontools will just keep restarting your process as fast as possible, taking 100% CPU if possible and flooding the logs), support for syslog/journal rather than just local files. By putting daemontools on top of systemd, you're just adding simple system on top of a complex one. You're losing features, but not gaining anything instead.

pmahoney 11y ago
You've got some of the daemontools details incorrect. > resource limits Daemontools includes the softlimit [1] helper. > namespaces It's not clear to me why this need be built in to a supervisor instead of being applied through generic helper programs such as unshare(1). > daemontools will just keep restarting your process as fast as possible, taking 100% CPU if possible supervise [2]: "It restarts ./run if ./run exits. It pauses for a second after starting ./run, so that it does not loop too quickly if ./run exits immediately." Simple and not configurable, sure, but should not hammer the CPU. > support for syslog/journal rather than just local files Daemontools svscan [3] "optionally starts a pair of supervise processes, one for a subdirectory s, one for s/log, with a pipe between them". The "s/log" script can do anything you wish, reading logs on stdin. If you use the included "multilog" program, then it's certainly geared toward writing local files, though it does include the ability to run arbitrary post-processor during rotation (which might, for example, fire off a job to copy the logs to an aggregator). Or you can not run multilog and just send the logs to syslog or whatever. What I do myself is send save the logs locally using s6's multilog analog, s6-log, and also pipe them to a local syslog that forwards them to an aggregator outside my control. And, in spite of newer supervisors adding more and more features beyond what daemontools provides, daemontools is still an awesome system that was 14 years ahead of its time. [1] http://cr.yp.to/daemontools/softlimit.html http://cr.yp.to/daemontools/softlimit.html [2] http://cr.yp.to/daemontools/supervise.html http://cr.yp.to/daemontools/supervise.html [3] http://cr.yp.to/daemontools/svscan.html http://cr.yp.to/daemontools/svscan.html

otterley 11y ago

> We run both nginx and unicorn through daemontools. Both have options to prevent daemonization. How are you able to do hot-reloads without daemonizing the master process? Or have you simply chosen to sacrifice this feature?

scurvy 11y ago
Hot reload? Like a config reload? Just send the master process a HUP.
- otterley 11y ago
  That can't possibly work correctly with runsv, given the way in which Unicorn performs configuration reloads. Eventually the original process that runsv forked will die, and runsv will try to restart it and fail (because the original process launched a new generation that continues to listen to the socket). Then 'sv stop' will fail to work because the new generation of master and workers isn't under runsv's supervision anymore.
  - scurvy 11y ago
    nginx doesn't die. It just reloads the config file and applies the changes. I can't remember what Unicorn does because we almost never change its config. Also, we're using daemontools not runit.