4 ms·

I highly recommend that people learn how to use https://docs.python.org/3/library/subprocess.html https://docs.python.org/3/library/subprocess.html as a replace

I highly recommend that people learn how to use https://docs.python.org/3/library/subprocess.html https://docs.python.org/3/library/subprocess.html as a replacement for Bash. It's a little more work upfront, but it's vastly more maintainable.

ben509 7y ago
I disagree. Here's a warning from subprocess[1]: > Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process. The trouble with `communicate()` is it only handles very simple cases, basically, your output has to fit in memory. Same problem exists in asyncio.[2] Yes, you can usually work around this. That doesn't mean it's a good replacement; whereas complex pipelines are so trivial in bash that any user-defined function can be used in a pipeline, subprocess generally forces you to do the dumb thing and create a mess of temporary files. And that's not even considering you're writing 10 times as much code than you would to accomplish the same task. [1]: https://docs.python.org/3/library/subprocess.html#subprocess.Popen.stderr https://docs.python.org/3/library/subprocess.html#subprocess... [2]: https://docs.python.org/3/library/asyncio-subprocess.html#asyncio.asyncio.subprocess.Process.stderr https://docs.python.org/3/library/asyncio-subprocess.html#as...
- kortex 7y ago
  Why is this still such a problem? People have been talking about replacing bash with python for years, and yet trying to actually do just that, is a pain. Plumbum doesn't quite cut it. I think in part it's because the POSIX-style shell is so tightly wound with the OS, it's extremely proficient at spawning and forking processes, working with files, and communicating, and that experience is seamless. Python feels like a different world, and doing any subproc, pipes, or file i/o always feels like crossing some boundary and back.
  - detaro 7y ago
    I feel like this could be a situation parallel to JS on the web: You can change the language that's available everywhere only slowly and in limited ways, so people build compilers targeting it as the output language from "nicer" languages.
- lalaland1125 7y ago
  Yes, stdin.write, stdout.read and stderr.read can cause deadlocks if not used carefully. However, those operations are rarely needed and are complicated in Bash as well (usually requiring sub-shells, etc). 99% of Bash scripts can be written without using those operations (you simply pipe input/output from one program to the next rather than trying to manually forward it in your program). This is actually one of the main advantages of using Python. You can solve all those deadlock issues through careful use of threads, which are much better than the Bash sub-shell equivalent. If anyone has an example script that requires the use of those functions and is cumbersome in Python, please share it. We can always improve the Python standard library if necessary.
kortex 7y ago
I want to. I really do. I've been using python for a decade and I still have to always look up how to call subprocesses correctly and communicate with multiple processes. Plumbum is cool, but now I have an extra [""] for every command, and I need python and a lib (which usually means an env as well) to bring an app up. On the other hand, I try to migrate as much business logic into the app as possible, and increasingly kv as well, so bash is pretty much pointing at config files and commands.

ben509 7y ago

I disagree. Here's a warning from subprocess[1]: > Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process. The trouble with `communicate()` is it only handles very simple cases, basically, your output has to fit in memory. Same problem exists in asyncio.[2] Yes, you can usually work around this. That doesn't mean it's a good replacement; whereas complex pipelines are so trivial in bash that any user-defined function can be used in a pipeline, subprocess generally forces you to do the dumb thing and create a mess of temporary files. And that's not even considering you're writing 10 times as much code than you would to accomplish the same task. [1]: https://docs.python.org/3/library/subprocess.html#subprocess.Popen.stderr https://docs.python.org/3/library/subprocess.html#subprocess... [2]: https://docs.python.org/3/library/asyncio-subprocess.html#asyncio.asyncio.subprocess.Process.stderr https://docs.python.org/3/library/asyncio-subprocess.html#as...

kortex 7y ago
Why is this still such a problem? People have been talking about replacing bash with python for years, and yet trying to actually do just that, is a pain. Plumbum doesn't quite cut it. I think in part it's because the POSIX-style shell is so tightly wound with the OS, it's extremely proficient at spawning and forking processes, working with files, and communicating, and that experience is seamless. Python feels like a different world, and doing any subproc, pipes, or file i/o always feels like crossing some boundary and back.
- detaro 7y ago
  I feel like this could be a situation parallel to JS on the web: You can change the language that's available everywhere only slowly and in limited ways, so people build compilers targeting it as the output language from "nicer" languages.
lalaland1125 7y ago
Yes, stdin.write, stdout.read and stderr.read can cause deadlocks if not used carefully. However, those operations are rarely needed and are complicated in Bash as well (usually requiring sub-shells, etc). 99% of Bash scripts can be written without using those operations (you simply pipe input/output from one program to the next rather than trying to manually forward it in your program). This is actually one of the main advantages of using Python. You can solve all those deadlock issues through careful use of threads, which are much better than the Bash sub-shell equivalent. If anyone has an example script that requires the use of those functions and is cumbersome in Python, please share it. We can always improve the Python standard library if necessary.

kortex 7y ago

I want to. I really do. I've been using python for a decade and I still have to always look up how to call subprocesses correctly and communicate with multiple processes. Plumbum is cool, but now I have an extra [""] for every command, and I need python and a lib (which usually means an env as well) to bring an app up. On the other hand, I try to migrate as much business logic into the app as possible, and increasingly kv as well, so bash is pretty much pointing at config files and commands.