6 ms·

Some Reflections on Writing Unix Daemon

97 points by ltratt 3y ago

bluetomcat 3y ago
It's Unix parlance for a forked child process that detaches from its interactive terminal with the setsid() syscall, becomes a child of init, possibly closes or redirects the 0/1/2 file descriptors to logging facilities, possibly changes its CWD, and no longer receives terminal-originating signals like SIGHUP, SIGTTIN, SIGTTOU, etc. Apart from that, a robust non-daemon program should use the same defensive techniques. In a short-running command-line utility, it might seem "acceptable" to leak memory, not close resources and not do adequate error-checking, but that's just sloppy programming.
- ksherlock 3y ago
  Well... "Since the missile will explode when it hits it's target or at the end of it's flight, the ultimate in garbage collection is performed without programmer intervention." https://groups.google.com/g/comp.lang.ada/c/E9bNCvDQ12k/m/1tezW24ZxdAJ https://groups.google.com/g/comp.lang.ada/c/E9bNCvDQ12k/m/1t... "to my surprise the cp command didn't exit. Looking at the source again, I found that cp disassembles its hash table data structures nicely after copying (the forget_all call). Since the virtual size of the cp process was now more than 17 GB and the server only had 10 GB of RAM, it did a lot of swapping." (I believe GNU cp was updated to not free up structures at exiting due after this). https://lists.gnu.org/archive/html/coreutils/2014-08/msg00012.html https://lists.gnu.org/archive/html/coreutils/2014-08/msg0001...
Joker_vD 3y ago
> Rather than having processes detach themselves from the terminal, these managers run daemons as if they were normal (albeit long-running) programs. This slightly simplifies the daemons themselves and provides a more homogeneous experience for the user. It tremendously simplifies the daemons themselves and indeed does provide a more homogeneous experience for the user. Remember: do one thing only, and do it well. The "daemonizing" part is is a second thing and it belongs in a separate utility. If the user wants to run your daemon "interactively" (e.g. being able to press Ctrl-C to stop it), they should be able to do so by simply running your daemon from the shell. If the user wants to run your daemon "in the background", whatever it means to the user, they can arrange so by themselves. Why is it such a difficult idea for the "daemon" writers to accept is beyond me: there is negative value in a program forcefully detaching itself from every single way to supervise it (double- and triple-forking defeats supervising via wait(), closing FDs defeats watching for the closing of an injected pipe, etc).
- ninkendo 3y ago
  Not to mention that in order to properly start and stop such double-forked daemons, you need everyone to agree on how pidfiles work, where to store them, how to garbage collect them, where to write logs, their permissions, how to rotate them, etc etc etc. With the systemd/“just stay in the foreground” approach you just stay in the foreground and the process monitor can just wait() on you. And for logging you can just write to stdout and let the process monitor unify all the logging. Do one thing and do it well indeed.
- matheusmoreira 3y ago
  > Why is it such a difficult idea for the "daemon" writers to accept is beyond me There's plenty of old documentation out there saying things like "a proper daemon completely detaches from its controlling terminal, opens log files, etc."
  - Joker_vD 3y ago
    There is also plenty of old documentation that strongly advises against doing that, see [0] for references (e.g. the aside with an excerpt from the 1995-era AIX docs). [0] https://jdebp.uk/FGA/unix-daemon-design-mistakes-to-avoid.html https://jdebp.uk/FGA/unix-daemon-design-mistakes-to-avoid.ht...
- bluetomcat 3y ago
  > Why is it such a difficult idea for the "daemon" writers to accept is beyond me Because the basic OS interface is a 50-year-old design with "terminals" being the central I/O entity. Differentiating between "interactive" and "daemon" mode becomes tricky for a daemon writer. In interactive mode, SIGHUP must terminate your process. In daemon mode, you may interpret it any way you like. Same for SIGINT, SIGQUIT, SIGPIPE, etc.
  - zare_st 3y ago
    Tricky? Checking if you're bound to TTY is trivial, checking whether you have a parent is trivial, checking whether interactive shell is somewhere in the tree is trivial, having some if's in signal handlers is trivial, etc. Besides "daemon writers" could just default to foreground, because if it's ran as a daemon it is ran from a script hence nobody cares for extra option typing such as -d --daemon, it's written once in a file, as opposed to doing "program --foreground" every time you want it to run in front of you
Mave83 3y ago
SystemD took away the pain. You no longer have to think and reinvent the demonize, logging and so on. Just just start what ever you want in a while(1) loop and write to stdout and stderr. No log rotate nightmare, etc. This makes daemons easier to debug, as they run just in foreground so that you can start them from the cli to thinker with them. SystemD is highly discussed and some love and some hate it. But such improvements are unmatched and extremely helpful.
- simoncion 3y ago
  OpenRC, daemontools, and many other service managers provide daemonizing wrappers, and all substantially predate systemd. The Systemd Cabal are far from the first to have noticed and attempted to resolve most of the problems they tackled... they do have the most effective "developer relations" team of all of the projects in this space, though. (I do continue to love their old "It's so much better than System V Init!" talking point... as well as their "You'll never have to write a shell script to start or manage a service again! Declarative, INI-formatted Unit Files For Everyone!" one.)
  - WesolyKubeczek 3y ago
    The systemd cabal one-upped all other efforts that pre-dated them by leveraging cgroups such that what rlimits your services run with is dictated by the unit file alone, and not by whatever your user session had when you issued the start or restart command.
    - stonogo 3y ago
      And they did so by sweeping the entire UNIX landscape into the dustbin, only supporting linux, and in return giving us a management system that can't be sure anything is running[1]. systemd is leaps and bounds better than most of the init systems that came before it, but it's not a good daemon supervisor, and the developers have basically declared that it's "good enough" so improvements are unwelcome. 1 - http://ewontfix.com/15/ http://ewontfix.com/15/
matheusmoreira 3y ago
Related: https://mywiki.wooledge.org/ProcessManagement https://mywiki.wooledge.org/ProcessManagement
habibur 3y ago
> There are use-cases for async/await, particularly for single-threaded languages, or for people writing network servers that have to have to deal with vast numbers of queries. Rust is multi-threaded – indeed, its type system forbids most classic multi-threading errors – and very few of us write servers that deal with vast numbers of queries. This is the portion that I was seeking his feedback on. Rust originally was designed to work with OS native threads. Later Nginx like software emulated threads were proven far more efficient and fast. Here a single thread jumps from connection to connection, instead of waiting for the other side to respond. In C you can do it but you need the ability for a function to resume from the mid of its body where it left off. That asks for "co-routines" which even though possible makes the code complex. Rust solution is async/await. But now they have two solutions, the more integrated multi threads and then this newly introduced single threaded async/await. It's better to get feedback from people that have worked on it and their good or bad experience.
- zokier 3y ago
  The most common Rust async runtime, tokio, is multithreaded by default.
asa400 3y ago
> My experience with snare and pizauth is that Rust is a viable language for writing daemons in. Rust isn’t a perfect language (e.g. unsafe Rust currently has no meaningful semantics so any code which uses unsafe comes with fewer guarantees than C) What exactly does the author mean when they say that unsafe Rust has "no meaningful semantics"? Is this a term of art in language analysis or is the author just saying "it's weird"?
- zokier 3y ago
  Rust as a language is in practice defined as whatever rustc does, there is no authorative specification like ISO C standard
- ltratt 3y ago
  As an example, I like to point people at https://doc.rust-lang.org/std/cell/struct.UnsafeCell.html https://doc.rust-lang.org/std/cell/struct.UnsafeCell.html which for many years now has contained this line: > The precise Rust aliasing rules are somewhat in flux, but the main points are not contentious I've sometimes found myself in situations where the only way I've been able to deal with this is to check the compiler's output and trawl forums for hints by Rust's developers about what they think/hope the semantics are/will be. Historically speaking, this situation isn't uncommon: working out exactly what a language's semantics should be is hard, particularly when it has many novel aspects. Most major languages go through this sort of sequence. Some sooner or later than others --- and some end up addressing it more thoroughly than others). Eventually I expect Rust to develop something similar to the modern C spec, but we're not there yet.
  - asa400 3y ago
    Excellent - thank you for the example and the clarification. This is exactly what I was looking for.
- viraptor 3y ago
  It's a weird claim. See https://doc.rust-lang.org/book/ch19-01-unsafe-rust.html https://doc.rust-lang.org/book/ch19-01-unsafe-rust.html > It’s important to understand that unsafe doesn’t turn off the borrow checker or disable any other of Rust’s safety checks: if you use a reference in unsafe code, it will still be checked. The unsafe keyword only gives you access to these five features that are then not checked by the compiler for memory safety. Fully defined rust or not, unsafe code blocks definitely do not have fewer guarantees than C.
eadmund 3y ago
> Ever since I have tried to think “can I solve this problem in a way that reuses known conventions or do I really, really have to do something different?” The answer, once I can control my ego, is nearly always “I don’t need to do something different”. This really is the way of wisdom. Not that there’s not room for improvement — there is — but it is normally more effective to attack the problem at hand than try to solve ancillary problems as well.
tmoertel 3y ago
To make daemons easier to write, more uniform, and secure, djb created https://cr.yp.to/daemontools.html https://cr.yp.to/daemontools.html and used it for many of his projects In the corresponding FAQ list, he says this about daemons that detach themselves: > How can I supervise a daemon that puts itself into the background? When I run inetd, my shell script exits immediately, so supervise keeps trying to restart it. > Answer: The best answer is to fix the daemon. Having every daemon put itself into the background is bad software design.

97 points by ltratt 3y ago

It's Unix parlance for a forked child process that detaches from its interactive terminal with the setsid() syscall, becomes a child of init, possibly closes or redirects the 0/1/2 file descriptors to logging facilities, possibly changes its CWD, and no longer receives terminal-originating signals like SIGHUP, SIGTTIN, SIGTTOU, etc. Apart from that, a robust non-daemon program should use the same defensive techniques. In a short-running command-line utility, it might seem "acceptable" to leak memory, not close resources and not do adequate error-checking, but that's just sloppy programming.

ksherlock 3y ago
Well... "Since the missile will explode when it hits it's target or at the end of it's flight, the ultimate in garbage collection is performed without programmer intervention." https://groups.google.com/g/comp.lang.ada/c/E9bNCvDQ12k/m/1tezW24ZxdAJ https://groups.google.com/g/comp.lang.ada/c/E9bNCvDQ12k/m/1t... "to my surprise the cp command didn't exit. Looking at the source again, I found that cp disassembles its hash table data structures nicely after copying (the forget_all call). Since the virtual size of the cp process was now more than 17 GB and the server only had 10 GB of RAM, it did a lot of swapping." (I believe GNU cp was updated to not free up structures at exiting due after this). https://lists.gnu.org/archive/html/coreutils/2014-08/msg00012.html https://lists.gnu.org/archive/html/coreutils/2014-08/msg0001...

Joker_vD 3y ago

> Rather than having processes detach themselves from the terminal, these managers run daemons as if they were normal (albeit long-running) programs. This slightly simplifies the daemons themselves and provides a more homogeneous experience for the user. It tremendously simplifies the daemons themselves and indeed does provide a more homogeneous experience for the user. Remember: do one thing only, and do it well. The "daemonizing" part is is a second thing and it belongs in a separate utility. If the user wants to run your daemon "interactively" (e.g. being able to press Ctrl-C to stop it), they should be able to do so by simply running your daemon from the shell. If the user wants to run your daemon "in the background", whatever it means to the user, they can arrange so by themselves. Why is it such a difficult idea for the "daemon" writers to accept is beyond me: there is negative value in a program forcefully detaching itself from every single way to supervise it (double- and triple-forking defeats supervising via wait(), closing FDs defeats watching for the closing of an injected pipe, etc).

ninkendo 3y ago
Not to mention that in order to properly start and stop such double-forked daemons, you need everyone to agree on how pidfiles work, where to store them, how to garbage collect them, where to write logs, their permissions, how to rotate them, etc etc etc. With the systemd/“just stay in the foreground” approach you just stay in the foreground and the process monitor can just wait() on you. And for logging you can just write to stdout and let the process monitor unify all the logging. Do one thing and do it well indeed.
matheusmoreira 3y ago
> Why is it such a difficult idea for the "daemon" writers to accept is beyond me There's plenty of old documentation out there saying things like "a proper daemon completely detaches from its controlling terminal, opens log files, etc."
- Joker_vD 3y ago
  There is also plenty of old documentation that strongly advises against doing that, see [0] for references (e.g. the aside with an excerpt from the 1995-era AIX docs). [0] https://jdebp.uk/FGA/unix-daemon-design-mistakes-to-avoid.html https://jdebp.uk/FGA/unix-daemon-design-mistakes-to-avoid.ht...
bluetomcat 3y ago
> Why is it such a difficult idea for the "daemon" writers to accept is beyond me Because the basic OS interface is a 50-year-old design with "terminals" being the central I/O entity. Differentiating between "interactive" and "daemon" mode becomes tricky for a daemon writer. In interactive mode, SIGHUP must terminate your process. In daemon mode, you may interpret it any way you like. Same for SIGINT, SIGQUIT, SIGPIPE, etc.
- zare_st 3y ago
  Tricky? Checking if you're bound to TTY is trivial, checking whether you have a parent is trivial, checking whether interactive shell is somewhere in the tree is trivial, having some if's in signal handlers is trivial, etc. Besides "daemon writers" could just default to foreground, because if it's ran as a daemon it is ran from a script hence nobody cares for extra option typing such as -d --daemon, it's written once in a file, as opposed to doing "program --foreground" every time you want it to run in front of you

Mave83 3y ago

SystemD took away the pain. You no longer have to think and reinvent the demonize, logging and so on. Just just start what ever you want in a while(1) loop and write to stdout and stderr. No log rotate nightmare, etc. This makes daemons easier to debug, as they run just in foreground so that you can start them from the cli to thinker with them. SystemD is highly discussed and some love and some hate it. But such improvements are unmatched and extremely helpful.

simoncion 3y ago
OpenRC, daemontools, and many other service managers provide daemonizing wrappers, and all substantially predate systemd. The Systemd Cabal are far from the first to have noticed and attempted to resolve most of the problems they tackled... they do have the most effective "developer relations" team of all of the projects in this space, though. (I do continue to love their old "It's so much better than System V Init!" talking point... as well as their "You'll never have to write a shell script to start or manage a service again! Declarative, INI-formatted Unit Files For Everyone!" one.)
- WesolyKubeczek 3y ago
  The systemd cabal one-upped all other efforts that pre-dated them by leveraging cgroups such that what rlimits your services run with is dictated by the unit file alone, and not by whatever your user session had when you issued the start or restart command.
  - stonogo 3y ago
    And they did so by sweeping the entire UNIX landscape into the dustbin, only supporting linux, and in return giving us a management system that can't be sure anything is running[1]. systemd is leaps and bounds better than most of the init systems that came before it, but it's not a good daemon supervisor, and the developers have basically declared that it's "good enough" so improvements are unwelcome. 1 - http://ewontfix.com/15/ http://ewontfix.com/15/

matheusmoreira 3y ago

Related: https://mywiki.wooledge.org/ProcessManagement https://mywiki.wooledge.org/ProcessManagement

habibur 3y ago

> There are use-cases for async/await, particularly for single-threaded languages, or for people writing network servers that have to have to deal with vast numbers of queries. Rust is multi-threaded – indeed, its type system forbids most classic multi-threading errors – and very few of us write servers that deal with vast numbers of queries. This is the portion that I was seeking his feedback on. Rust originally was designed to work with OS native threads. Later Nginx like software emulated threads were proven far more efficient and fast. Here a single thread jumps from connection to connection, instead of waiting for the other side to respond. In C you can do it but you need the ability for a function to resume from the mid of its body where it left off. That asks for "co-routines" which even though possible makes the code complex. Rust solution is async/await. But now they have two solutions, the more integrated multi threads and then this newly introduced single threaded async/await. It's better to get feedback from people that have worked on it and their good or bad experience.

zokier 3y ago
The most common Rust async runtime, tokio, is multithreaded by default.

asa400 3y ago

> My experience with snare and pizauth is that Rust is a viable language for writing daemons in. Rust isn’t a perfect language (e.g. unsafe Rust currently has no meaningful semantics so any code which uses unsafe comes with fewer guarantees than C) What exactly does the author mean when they say that unsafe Rust has "no meaningful semantics"? Is this a term of art in language analysis or is the author just saying "it's weird"?

zokier 3y ago
Rust as a language is in practice defined as whatever rustc does, there is no authorative specification like ISO C standard
ltratt 3y ago
As an example, I like to point people at https://doc.rust-lang.org/std/cell/struct.UnsafeCell.html https://doc.rust-lang.org/std/cell/struct.UnsafeCell.html which for many years now has contained this line: > The precise Rust aliasing rules are somewhat in flux, but the main points are not contentious I've sometimes found myself in situations where the only way I've been able to deal with this is to check the compiler's output and trawl forums for hints by Rust's developers about what they think/hope the semantics are/will be. Historically speaking, this situation isn't uncommon: working out exactly what a language's semantics should be is hard, particularly when it has many novel aspects. Most major languages go through this sort of sequence. Some sooner or later than others --- and some end up addressing it more thoroughly than others). Eventually I expect Rust to develop something similar to the modern C spec, but we're not there yet.
- asa400 3y ago
  Excellent - thank you for the example and the clarification. This is exactly what I was looking for.
viraptor 3y ago
It's a weird claim. See https://doc.rust-lang.org/book/ch19-01-unsafe-rust.html https://doc.rust-lang.org/book/ch19-01-unsafe-rust.html > It’s important to understand that unsafe doesn’t turn off the borrow checker or disable any other of Rust’s safety checks: if you use a reference in unsafe code, it will still be checked. The unsafe keyword only gives you access to these five features that are then not checked by the compiler for memory safety. Fully defined rust or not, unsafe code blocks definitely do not have fewer guarantees than C.

eadmund 3y ago

> Ever since I have tried to think “can I solve this problem in a way that reuses known conventions or do I really, really have to do something different?” The answer, once I can control my ego, is nearly always “I don’t need to do something different”. This really is the way of wisdom. Not that there’s not room for improvement — there is — but it is normally more effective to attack the problem at hand than try to solve ancillary problems as well.

tmoertel 3y ago

To make daemons easier to write, more uniform, and secure, djb created https://cr.yp.to/daemontools.html https://cr.yp.to/daemontools.html and used it for many of his projects In the corresponding FAQ list, he says this about daemons that detach themselves: > How can I supervise a daemon that puts itself into the background? When I run inetd, my shell script exits immediately, so supervise keeps trying to restart it. > Answer: The best answer is to fix the daemon. Having every daemon put itself into the background is bad software design.