3 ms·

It's odd to me that systemd didn't implement a "run X number of processes" feature. I've also found it difficult to find a well-written tiny application written

It's odd to me that systemd didn't implement a "run X number of processes" feature. I've also found it difficult to find a well-written tiny application written in C/Rust/Go whose only job it is to start and supervise X number of processes. supervisord is the well-known workhorse here but it's really old and quite buggy.

lantry 4y ago
Have you seen overmind? https://github.com/DarthSim/overmind https://github.com/DarthSim/overmind
- raziel2p 4y ago
  Unfortunately Procfile doesn't support declaratively setting the number of processes, Heroku designed it so you have to specify that with their CLI.
ilyt 4y ago
From my experience there are very little use cases where you want to start same app exactly multiple times, usually you always need to vary at least one parameter (usually config file or workdir), and that's pretty app-specific. ...that being said make template unit `myapp@` then just systemctl start myapp@1 systemctl start myapp@2 systemctl start myapp@3 can also tie them to start/stop together via dependencies or make wrapper script that starts all and use man systemd.kill modes to send signal to all of them.
- raziel2p 4y ago
  Queue workers is a classic example. It's not necessarily about wanting an exact multiple of processes, but more about wanting to scale them up and down with a declarative configuration format. In the past I did actually use the templated solution, e.g. systemctl enable myapp-worker@{1..4} systemctl start/restart/reload/stop myapp-worker@* However, in order to scale the number of processes down, you have to know how many are running already. Not that difficult to write a wrapper script around of course, but why isn't it just built-in?
  - ilyt 4y ago
    Okay, so we add an option to specify number of processed spawned within service You increase it, it spawns more, you decrease it, what next ? How systemd knows which process to kill ? You want to kill the idle one, not the one currently doing the work. The working process MIGHT BE IDLE coz it is waiting on network. Systemd have no way of knowing. There is no way to find out which process should be killed without processes itself cooperating. And frankly because it's rare use case. Majority of apps that use have workers follow pattern of having a process that accepts requests and spawns or sends them to the workers and just accept number of workers as a parameter. Precisely because of the above, you do want to have logic on stopping the worker. And it's cheaper on memory to spawn threads instead of fully fledged processes anyway. Systemd doesn't know when you need more processes so you have to script it anyway even if it had a feature to modify number of spawned daemons within single service. If you have that already, might as well write the the worker spawner there. Personally the feature I miss is ability to have watchdog. Auto-restart is nice but if there was an option to "run this command, if it fails then restart main ExecStart app" it would've been perfect, easy way to add healthcheck that checks whether app actually works vs being unresponsive.
    - 20after4 4y ago
      There is a watchdog feature where (if I remember correctly) systemd listens on a socket and the service must write to that socket periodically or it gets automatically killed and restarted. Unfortunately, the feature is rather obscure and not well documented. I found that the sd-notify[1] package on NPM works well. If your app happens to use NodeJS then it's pretty simple to set up. If not, looking at the way that package is implemented is probably a good way to figure out how the feature works so that you can do the same thing in your own application. Maybe the documentation for the feature has improved, however, the last time I checked (only a few months ago) everything I found was bad. I did stop looking once I found that the NPM package works well. [1] https://www.npmjs.com/package/sd-notify https://www.npmjs.com/package/sd-notify
  - ratorx 4y ago
    Bit OT: Are dynamically scalable process workers on a single machine the right choice? It seems a bit niche to be in a situation where you need the flexibility of dynamic scheduling whilst only having a single machine. Once you have more than 1 machine, systemd isn’t designed for it anyway, so you’d be looking at a cluster scheduler like Nomad or Kubernetes anyway. In the single machine case, would it not be better to either: a) Just spin up a static number n of workers (where n is the number of machine cores) and let them idle if not in use. b) Use a thread pool (or green threads) inside a single process with shared memory (if e.g. idle memory use by workers is a concern). I can’t immediately see the benefit of being able to have dynamic control over a number of workers AND keep them as separate processes that need to be adjusted manually. I guess there may not be a choice if the software is not designed to be run inside threads AND memory use is a concern.

lantry 4y ago

Have you seen overmind? https://github.com/DarthSim/overmind https://github.com/DarthSim/overmind

raziel2p 4y ago
Unfortunately Procfile doesn't support declaratively setting the number of processes, Heroku designed it so you have to specify that with their CLI.

ilyt 4y ago

From my experience there are very little use cases where you want to start same app exactly multiple times, usually you always need to vary at least one parameter (usually config file or workdir), and that's pretty app-specific. ...that being said make template unit `myapp@` then just systemctl start myapp@1 systemctl start myapp@2 systemctl start myapp@3 can also tie them to start/stop together via dependencies or make wrapper script that starts all and use man systemd.kill modes to send signal to all of them.

raziel2p 4y ago
Queue workers is a classic example. It's not necessarily about wanting an exact multiple of processes, but more about wanting to scale them up and down with a declarative configuration format. In the past I did actually use the templated solution, e.g. systemctl enable myapp-worker@{1..4} systemctl start/restart/reload/stop myapp-worker@* However, in order to scale the number of processes down, you have to know how many are running already. Not that difficult to write a wrapper script around of course, but why isn't it just built-in?
- ilyt 4y ago
  Okay, so we add an option to specify number of processed spawned within service You increase it, it spawns more, you decrease it, what next ? How systemd knows which process to kill ? You want to kill the idle one, not the one currently doing the work. The working process MIGHT BE IDLE coz it is waiting on network. Systemd have no way of knowing. There is no way to find out which process should be killed without processes itself cooperating. And frankly because it's rare use case. Majority of apps that use have workers follow pattern of having a process that accepts requests and spawns or sends them to the workers and just accept number of workers as a parameter. Precisely because of the above, you do want to have logic on stopping the worker. And it's cheaper on memory to spawn threads instead of fully fledged processes anyway. Systemd doesn't know when you need more processes so you have to script it anyway even if it had a feature to modify number of spawned daemons within single service. If you have that already, might as well write the the worker spawner there. Personally the feature I miss is ability to have watchdog. Auto-restart is nice but if there was an option to "run this command, if it fails then restart main ExecStart app" it would've been perfect, easy way to add healthcheck that checks whether app actually works vs being unresponsive.
  - 20after4 4y ago
    There is a watchdog feature where (if I remember correctly) systemd listens on a socket and the service must write to that socket periodically or it gets automatically killed and restarted. Unfortunately, the feature is rather obscure and not well documented. I found that the sd-notify[1] package on NPM works well. If your app happens to use NodeJS then it's pretty simple to set up. If not, looking at the way that package is implemented is probably a good way to figure out how the feature works so that you can do the same thing in your own application. Maybe the documentation for the feature has improved, however, the last time I checked (only a few months ago) everything I found was bad. I did stop looking once I found that the NPM package works well. [1] https://www.npmjs.com/package/sd-notify https://www.npmjs.com/package/sd-notify
- ratorx 4y ago
  Bit OT: Are dynamically scalable process workers on a single machine the right choice? It seems a bit niche to be in a situation where you need the flexibility of dynamic scheduling whilst only having a single machine. Once you have more than 1 machine, systemd isn’t designed for it anyway, so you’d be looking at a cluster scheduler like Nomad or Kubernetes anyway. In the single machine case, would it not be better to either: a) Just spin up a static number n of workers (where n is the number of machine cores) and let them idle if not in use. b) Use a thread pool (or green threads) inside a single process with shared memory (if e.g. idle memory use by workers is a concern). I can’t immediately see the benefit of being able to have dynamic control over a number of workers AND keep them as separate processes that need to be adjusted manually. I guess there may not be a choice if the software is not designed to be run inside threads AND memory use is a concern.