5 ms·

If you know python, then why bother with shell scripts in the first place?

laumars 5y ago
Because if the main bulk of your problem is getting solved with external executables, like the ffmpeg example given, then it makes more sense to call them from a scripting language that is designed around forking other processes. Python be far more powerful than your average shell but it sucks for writing shell scripts.
- Joker_vD 5y ago
  The thing is, bash kinda sucks at managing subprocesses the moment you start doing anything even slightly more interesting than "wait for the single launched subprocess/the last process in the pipe" (in fact, all UNIXes suck at it because of their fragile PID management but let's talk about the shell in particular). For example, imagine you want to run two processes in parallel and wait until both of them end; and you also want to be able to press Ctrl+C and interrupt them both. The cleanest way I could've find after much googling is ( trap 'kill 0' SIGINT worker A & worker B & wait ) It's a very delicate pattern because, for example, the use of subshell here is critical: without it SIGINT doesn't get delivered to the "worker" processes. One would think such a useful workflow ought to have a better built-in support but apparently not: people reinvent it all the time with $! and manual PID files and those things are very unreliable.
  - laumars 5y ago
    That’s a very specific use case you have there though. I can’t think of any occasion when I’ve actually wanted to do that in the 20 years I’ve been a sysadmin for *nix. The few occasions I have had multiple daemons I’ve wanted to start and have the ability to terminate it made more sense to create an init file (of varying formats over the years) or Docker container. Managing multiple processes with a single signal is a bit of a UNIX anti-pattern and thus creating a sub-shell that is parent to both processes does feel more more idiomatic to how UNIX (never mind shells) should operate. But even that feels wrong in terms of how processes should be managed. Hence the init/ Docker suggestions. As for PIDs files, I have no love for them either. But in fairness, their role isn’t to manage a persistent shell but rather to manage a persistent service being queried from a non-persistent shell session. In a way, they’re like a RESTful API before REST was a thing. So they are not designed to fit the role you’re describing of a persistent shell session managing two long running but not persistent processes.
    - Joker_vD 5y ago
      > That’s a very specific use case you have there though Upload two large files to two different machines in parallel, starting at roughly the same time to compare the throughput. Or "run in parallel and measure the difference" scenario. Or parallelizing any, esp. network/distributed, work a-la make/xargs. Heck, init used to do exactly this: run a getty for each tty in parallel, indefinitely restarting them. Sure, you can do that from two/three/four/... different xterms, as well as stopping them manually and that's what I usually do but... that's tedious and trivial stuff, perfect for automating if you can automate it, that is. As for services/demons I agree completely; in fact my other gripe about the UNIX process management is how easy it is to break out of a process group. Thankfully, docker gives you confidence that when you stop your service, no runaway (great-great-...-grand-)child process will survive. In olden days, however, lots of things insisted on daemonizing themselves and fighting against any ways to control or even observe them: breaking from the process groups, de-attaching from the terminal sessions, double- and triple-forking, closing all file descriptors to defeat the self-pipe trick, etc. Ugh. If you really need some child process to outlive you (do you really? please consider again), then the only way to do that ought to be "service start" or "at now" or some other kind of "ask someone above you in the process hierarchy to launch that process".

Because if the main bulk of your problem is getting solved with external executables, like the ffmpeg example given, then it makes more sense to call them from a scripting language that is designed around forking other processes. Python be far more powerful than your average shell but it sucks for writing shell scripts.

Joker_vD 5y ago
The thing is, bash kinda sucks at managing subprocesses the moment you start doing anything even slightly more interesting than "wait for the single launched subprocess/the last process in the pipe" (in fact, all UNIXes suck at it because of their fragile PID management but let's talk about the shell in particular). For example, imagine you want to run two processes in parallel and wait until both of them end; and you also want to be able to press Ctrl+C and interrupt them both. The cleanest way I could've find after much googling is ( trap 'kill 0' SIGINT worker A & worker B & wait ) It's a very delicate pattern because, for example, the use of subshell here is critical: without it SIGINT doesn't get delivered to the "worker" processes. One would think such a useful workflow ought to have a better built-in support but apparently not: people reinvent it all the time with $! and manual PID files and those things are very unreliable.
- laumars 5y ago
  That’s a very specific use case you have there though. I can’t think of any occasion when I’ve actually wanted to do that in the 20 years I’ve been a sysadmin for *nix. The few occasions I have had multiple daemons I’ve wanted to start and have the ability to terminate it made more sense to create an init file (of varying formats over the years) or Docker container. Managing multiple processes with a single signal is a bit of a UNIX anti-pattern and thus creating a sub-shell that is parent to both processes does feel more more idiomatic to how UNIX (never mind shells) should operate. But even that feels wrong in terms of how processes should be managed. Hence the init/ Docker suggestions. As for PIDs files, I have no love for them either. But in fairness, their role isn’t to manage a persistent shell but rather to manage a persistent service being queried from a non-persistent shell session. In a way, they’re like a RESTful API before REST was a thing. So they are not designed to fit the role you’re describing of a persistent shell session managing two long running but not persistent processes.
  - Joker_vD 5y ago
    > That’s a very specific use case you have there though Upload two large files to two different machines in parallel, starting at roughly the same time to compare the throughput. Or "run in parallel and measure the difference" scenario. Or parallelizing any, esp. network/distributed, work a-la make/xargs. Heck, init used to do exactly this: run a getty for each tty in parallel, indefinitely restarting them. Sure, you can do that from two/three/four/... different xterms, as well as stopping them manually and that's what I usually do but... that's tedious and trivial stuff, perfect for automating if you can automate it, that is. As for services/demons I agree completely; in fact my other gripe about the UNIX process management is how easy it is to break out of a process group. Thankfully, docker gives you confidence that when you stop your service, no runaway (great-great-...-grand-)child process will survive. In olden days, however, lots of things insisted on daemonizing themselves and fighting against any ways to control or even observe them: breaking from the process groups, de-attaching from the terminal sessions, double- and triple-forking, closing all file descriptors to defeat the self-pipe trick, etc. Ugh. If you really need some child process to outlive you (do you really? please consider again), then the only way to do that ought to be "service start" or "at now" or some other kind of "ask someone above you in the process hierarchy to launch that process".