4 ms·

Queue workers is a classic example. It's not necessarily about wanting an exact multiple of processes, but more about wanting to scale them up and down with a d

Queue workers is a classic example. It's not necessarily about wanting an exact multiple of processes, but more about wanting to scale them up and down with a declarative configuration format. In the past I did actually use the templated solution, e.g. systemctl enable myapp-worker@{1..4} systemctl start/restart/reload/stop myapp-worker@* However, in order to scale the number of processes down, you have to know how many are running already. Not that difficult to write a wrapper script around of course, but why isn't it just built-in?

ilyt 4y ago
Okay, so we add an option to specify number of processed spawned within service You increase it, it spawns more, you decrease it, what next ? How systemd knows which process to kill ? You want to kill the idle one, not the one currently doing the work. The working process MIGHT BE IDLE coz it is waiting on network. Systemd have no way of knowing. There is no way to find out which process should be killed without processes itself cooperating. And frankly because it's rare use case. Majority of apps that use have workers follow pattern of having a process that accepts requests and spawns or sends them to the workers and just accept number of workers as a parameter. Precisely because of the above, you do want to have logic on stopping the worker. And it's cheaper on memory to spawn threads instead of fully fledged processes anyway. Systemd doesn't know when you need more processes so you have to script it anyway even if it had a feature to modify number of spawned daemons within single service. If you have that already, might as well write the the worker spawner there. Personally the feature I miss is ability to have watchdog. Auto-restart is nice but if there was an option to "run this command, if it fails then restart main ExecStart app" it would've been perfect, easy way to add healthcheck that checks whether app actually works vs being unresponsive.
- 20after4 4y ago
  There is a watchdog feature where (if I remember correctly) systemd listens on a socket and the service must write to that socket periodically or it gets automatically killed and restarted. Unfortunately, the feature is rather obscure and not well documented. I found that the sd-notify[1] package on NPM works well. If your app happens to use NodeJS then it's pretty simple to set up. If not, looking at the way that package is implemented is probably a good way to figure out how the feature works so that you can do the same thing in your own application. Maybe the documentation for the feature has improved, however, the last time I checked (only a few months ago) everything I found was bad. I did stop looking once I found that the NPM package works well. [1] https://www.npmjs.com/package/sd-notify https://www.npmjs.com/package/sd-notify
ratorx 4y ago
Bit OT: Are dynamically scalable process workers on a single machine the right choice? It seems a bit niche to be in a situation where you need the flexibility of dynamic scheduling whilst only having a single machine. Once you have more than 1 machine, systemd isn’t designed for it anyway, so you’d be looking at a cluster scheduler like Nomad or Kubernetes anyway. In the single machine case, would it not be better to either: a) Just spin up a static number n of workers (where n is the number of machine cores) and let them idle if not in use. b) Use a thread pool (or green threads) inside a single process with shared memory (if e.g. idle memory use by workers is a concern). I can’t immediately see the benefit of being able to have dynamic control over a number of workers AND keep them as separate processes that need to be adjusted manually. I guess there may not be a choice if the software is not designed to be run inside threads AND memory use is a concern.

ilyt 4y ago

Okay, so we add an option to specify number of processed spawned within service You increase it, it spawns more, you decrease it, what next ? How systemd knows which process to kill ? You want to kill the idle one, not the one currently doing the work. The working process MIGHT BE IDLE coz it is waiting on network. Systemd have no way of knowing. There is no way to find out which process should be killed without processes itself cooperating. And frankly because it's rare use case. Majority of apps that use have workers follow pattern of having a process that accepts requests and spawns or sends them to the workers and just accept number of workers as a parameter. Precisely because of the above, you do want to have logic on stopping the worker. And it's cheaper on memory to spawn threads instead of fully fledged processes anyway. Systemd doesn't know when you need more processes so you have to script it anyway even if it had a feature to modify number of spawned daemons within single service. If you have that already, might as well write the the worker spawner there. Personally the feature I miss is ability to have watchdog. Auto-restart is nice but if there was an option to "run this command, if it fails then restart main ExecStart app" it would've been perfect, easy way to add healthcheck that checks whether app actually works vs being unresponsive.

20after4 4y ago
There is a watchdog feature where (if I remember correctly) systemd listens on a socket and the service must write to that socket periodically or it gets automatically killed and restarted. Unfortunately, the feature is rather obscure and not well documented. I found that the sd-notify[1] package on NPM works well. If your app happens to use NodeJS then it's pretty simple to set up. If not, looking at the way that package is implemented is probably a good way to figure out how the feature works so that you can do the same thing in your own application. Maybe the documentation for the feature has improved, however, the last time I checked (only a few months ago) everything I found was bad. I did stop looking once I found that the NPM package works well. [1] https://www.npmjs.com/package/sd-notify https://www.npmjs.com/package/sd-notify

ratorx 4y ago

Bit OT: Are dynamically scalable process workers on a single machine the right choice? It seems a bit niche to be in a situation where you need the flexibility of dynamic scheduling whilst only having a single machine. Once you have more than 1 machine, systemd isn’t designed for it anyway, so you’d be looking at a cluster scheduler like Nomad or Kubernetes anyway. In the single machine case, would it not be better to either: a) Just spin up a static number n of workers (where n is the number of machine cores) and let them idle if not in use. b) Use a thread pool (or green threads) inside a single process with shared memory (if e.g. idle memory use by workers is a concern). I can’t immediately see the benefit of being able to have dynamic control over a number of workers AND keep them as separate processes that need to be adjusted manually. I guess there may not be a choice if the software is not designed to be run inside threads AND memory use is a concern.