4 ms·

The author addresses this argument: > However, Python and Ruby aren't good shell replacements in general. Shell is a domain-specific language for dealing with

by jonahx 6y ago

The author addresses this argument: > However, Python and Ruby aren't good shell replacements in general. Shell is a domain-specific language for dealing with concurrent processes and the file system. But Python and Ruby have too much abstraction over these concepts, sometimes in the name of portability (e.g. to Windows). They hide what's really going on. From: http://www.oilshell.org/blog/2018/01/28.html#i-dont-understand-why-not-use-a-different-a-programming-language http://www.oilshell.org/blog/2018/01/28.html#i-dont-understa...

dmix 6y ago
Sounds like we need a new standard!
mikepurvis 6y ago
Even something that should be as simple as piping output from one process to another in Python is a nightmare, and fraught with opportunities to shoot yourself in the foot with buffering deadlocks and other nonsense. The consequence of this is that many of my projects which require these overlapping domains (think stuff like build automation) ends up being the two interleaved— either an outer shell script that calls small Python programs (for parsing, URL fetching, and basically any error prone stuff where sane recovery/cleanup may be necessary), or it's a Python program that renders little templates of bash into strings and passes them to bash stdin. Or in some cases, it's CMake that's the outer orchestration piece, using configure_file to fill templates and then invoke them either at configure time (with execute_process) or at build time (with add_custom_command).
- bscphil 6y ago
  > Even something that should be as simple as piping output from one process to another in Python is a nightmare, and fraught with opportunities to shoot yourself in the foot with buffering deadlocks and other nonsense. This hits the nail on the head for me. What I'd like is something bash-like to handle running programs, pipes, tab-complete for file names, and so on, and something like Python syntax for control flow and strings (quoting / escaping), and access to something powerful like Python's standard math libraries (and ability to import other stuff like requests). I don't know how you'd roll all that into a single shell, but as close as you can get is what I'd like to see in a new shell. (I've briefly looked at Oil before, but it has seemed a bit complicated to merit trying to switch full time at this point. I'm definitely following it for future developments though.) Edit: just found Xonsh from another user's comment, which seems almost exactly what I'm looking for, if a bit hacked together at first glance. Going to try that out.
- m45t3r 6y ago
  > Even something that should be as simple as piping output from one process to another in Python is a nightmare, and fraught with opportunities to shoot yourself in the foot with buffering deadlocks and other nonsense. This is true, but when I am writing a script in Python I generally don't need to pipe as often as I do in bash. Python offer many replacements for things that in Bash you need to call another process. That is, of course, if performance isn't necessary. When it is bash is still unbeatable because of the pipes.
  - mikepurvis 6y ago
    > Python offer many replacements for things that in Bash you need to call another process. True, you're never going to pipe something to grep in Python when you can just capture the output and use the inbuilt regex capability. That said, it can be surprising how much worse performance-wise Python can be than shelling out to external utilities. Consider the simple case of downloading and extracting a tarball. I found that check_call("curl path/to/thing.tar | tar -x", shell=True) was significantly faster than anything I could figure out how to do with requests streaming, httpx (async), and the inbuilt tarfile module.
- d0mine 6y ago
  Here's how to pipe two commands in Python: import subprocess subprocess.check_call("command1 | command2", shell=True) Depending on your use case there are other options e.g., fabric http://fabfile.org http://fabfile.org plumbum Use shell for what is really good for: concise DSL for running commands (one-liners). Leave complex logic (branches, explicit loops) for sane general purpose languages such as Python.
  - mikepurvis 6y ago
    Right, and I do that, but it's an obvious hack; it causes a separate sh process to spawn, you open yourself to shell injection issues if you're not careful, teeing the pipe is trickier than it should be, etc etc.
    - d0mine 6y ago
      you might have known this but the practice shows that It has to be repeated: there is no shell injection issues with literal string in your own code. YMMV but for those rare cases when I care about the shell injection, I just don't use shell and run the command directly in Python. Combination of shell one-liners and the main logic in Python works well in practice.
      - mikepurvis 6y ago
        Oh yeah, for sure. I think it's just frustrating having that one more piece of mental overhead. Like, it should be that there's one sane way to do all this stuff in Python. Instead it's literal blobs of shell in some cases, and subprocess.STDXX pipes linked up in other cases, and maybe sometimes you use shlex to sanitize your arguments, or you give up on streaming and just use communicate() to get the whole result in memory at once. Blah.
        d0mine 6y ago
        I don’t see it as an overhead, I see using shell syntax as just another DSL like regex. There are many different use cases related to running processes—it is natural that different solutions may be preferred for different cases (check_{call,output}, with/out shell, run, Popen, pty/PIPE, threads/Asunción—all may be useful. And it is just stdlib ).
acdha 6y ago
That’s not a very convincing post because it’s a bit too shallow: > I encountered a nice blog post, Replacing Shell Scripts with Python, which, in my opinion, inadvertently proves the opposite point. The Python version is longer and has more dependencies. In other words, it's more difficult to write and maintain. The Python version uses only the standard library, so it doesn’t make sense to worry about dependencies if you aren’t counting every program your shell scripts call similarly. Having just one thing providing a consistent baseline beats learning that your script needs to upgrade RHEL to get a feature which would make something safer or easier. Similarly, the Python script is longer because it does better work and consistent, useful error handling. That adds lines of code but it’s usability and correctness, not overhead. It’s similarly incorrect to treat line count as a proxy for difficulty, as anyone who’s ever had to deal with quoting or data structures in shell scripts knows. The use of pathlib or os.path sometimes gets grumbles for extra characters but I’ve found it nearly inevitable that the grumblers will have something fail or destroy data because they hit a filename or argument with a space or special character in it. On one notable occasion, that resulted in an `rm -rf something something /`. (If you’re writing a shell script, shellcheck is mandatory) The best criticism I’d make is that it could be easier to replace a shell pipeline. Python’s subprocess makes that pretty easy but you still need a minimum of 3 calls per command plus whatever you need to actually do with the output. Since subprocess.{run,check_output} handles most of my needs this isn’t a big deal and there are excellent modules if you don’t mind dependencies.
- cle 6y ago
  This. The Bash script actually has more dependencies, it relies on a number of external programs (ps, kill, mkdir, sort, ls, cp, echo). What versions are on your system? What versions on the systems of the people running the script? Do they support the same features? If you're running on a Mac are you using the Mac-shipped programs or the GNU coreutils from homebrew? Also, that Bash script that the author is defending has a subtle bug in it, because Bash is subtle. The bug is here: $APP $filename >"${output_dir}/summary_${name}.txt" if [ $? != 0 ]; then echo "Error $? in app" fi The return value of [ ... ] overwrites the $?, so the inner $? is the result of the test, not $APP. Which is exactly the kind of crap that everyone's talking about when saying that we should avoid Bash scripts. It's full of landmines like this. Bash is great for interactive work on the terminal. It is not great for writing correct, maintainable programs.
saghm 6y ago
> However, Python and Ruby aren't good shell replacements in general. Shell is a domain-specific language for dealing with concurrent processes and the file system. But Python and Ruby have too much abstraction over these concepts, sometimes in the name of portability (e.g. to Windows). Ruby has pretty good thing wrappers around running shell commands (e.g. using backticks to run a command and get the output as a string or calling system() on a string to execute as a command and get back the error code), and my understanding is that it doesn't really have great Windows support. My impression here is that the author of this FAQ is probably not super familiar with Ruby based on the way they seem to equate the abstractions in both languages.

by jonahx 6y ago

dmix 6y ago

Sounds like we need a new standard!

mikepurvis 6y ago

Even something that should be as simple as piping output from one process to another in Python is a nightmare, and fraught with opportunities to shoot yourself in the foot with buffering deadlocks and other nonsense. The consequence of this is that many of my projects which require these overlapping domains (think stuff like build automation) ends up being the two interleaved— either an outer shell script that calls small Python programs (for parsing, URL fetching, and basically any error prone stuff where sane recovery/cleanup may be necessary), or it's a Python program that renders little templates of bash into strings and passes them to bash stdin. Or in some cases, it's CMake that's the outer orchestration piece, using configure_file to fill templates and then invoke them either at configure time (with execute_process) or at build time (with add_custom_command).

bscphil 6y ago
> Even something that should be as simple as piping output from one process to another in Python is a nightmare, and fraught with opportunities to shoot yourself in the foot with buffering deadlocks and other nonsense. This hits the nail on the head for me. What I'd like is something bash-like to handle running programs, pipes, tab-complete for file names, and so on, and something like Python syntax for control flow and strings (quoting / escaping), and access to something powerful like Python's standard math libraries (and ability to import other stuff like requests). I don't know how you'd roll all that into a single shell, but as close as you can get is what I'd like to see in a new shell. (I've briefly looked at Oil before, but it has seemed a bit complicated to merit trying to switch full time at this point. I'm definitely following it for future developments though.) Edit: just found Xonsh from another user's comment, which seems almost exactly what I'm looking for, if a bit hacked together at first glance. Going to try that out.
m45t3r 6y ago
> Even something that should be as simple as piping output from one process to another in Python is a nightmare, and fraught with opportunities to shoot yourself in the foot with buffering deadlocks and other nonsense. This is true, but when I am writing a script in Python I generally don't need to pipe as often as I do in bash. Python offer many replacements for things that in Bash you need to call another process. That is, of course, if performance isn't necessary. When it is bash is still unbeatable because of the pipes.
- mikepurvis 6y ago
  > Python offer many replacements for things that in Bash you need to call another process. True, you're never going to pipe something to grep in Python when you can just capture the output and use the inbuilt regex capability. That said, it can be surprising how much worse performance-wise Python can be than shelling out to external utilities. Consider the simple case of downloading and extracting a tarball. I found that check_call("curl path/to/thing.tar | tar -x", shell=True) was significantly faster than anything I could figure out how to do with requests streaming, httpx (async), and the inbuilt tarfile module.
d0mine 6y ago
Here's how to pipe two commands in Python: import subprocess subprocess.check_call("command1 | command2", shell=True) Depending on your use case there are other options e.g., fabric http://fabfile.org http://fabfile.org plumbum Use shell for what is really good for: concise DSL for running commands (one-liners). Leave complex logic (branches, explicit loops) for sane general purpose languages such as Python.
- mikepurvis 6y ago
  Right, and I do that, but it's an obvious hack; it causes a separate sh process to spawn, you open yourself to shell injection issues if you're not careful, teeing the pipe is trickier than it should be, etc etc.
  - d0mine 6y ago
    you might have known this but the practice shows that It has to be repeated: there is no shell injection issues with literal string in your own code. YMMV but for those rare cases when I care about the shell injection, I just don't use shell and run the command directly in Python. Combination of shell one-liners and the main logic in Python works well in practice.
    - mikepurvis 6y ago
      Oh yeah, for sure. I think it's just frustrating having that one more piece of mental overhead. Like, it should be that there's one sane way to do all this stuff in Python. Instead it's literal blobs of shell in some cases, and subprocess.STDXX pipes linked up in other cases, and maybe sometimes you use shlex to sanitize your arguments, or you give up on streaming and just use communicate() to get the whole result in memory at once. Blah.
      - d0mine 6y ago
        I don’t see it as an overhead, I see using shell syntax as just another DSL like regex. There are many different use cases related to running processes—it is natural that different solutions may be preferred for different cases (check_{call,output}, with/out shell, run, Popen, pty/PIPE, threads/Asunción—all may be useful. And it is just stdlib ).

acdha 6y ago

That’s not a very convincing post because it’s a bit too shallow: > I encountered a nice blog post, Replacing Shell Scripts with Python, which, in my opinion, inadvertently proves the opposite point. The Python version is longer and has more dependencies. In other words, it's more difficult to write and maintain. The Python version uses only the standard library, so it doesn’t make sense to worry about dependencies if you aren’t counting every program your shell scripts call similarly. Having just one thing providing a consistent baseline beats learning that your script needs to upgrade RHEL to get a feature which would make something safer or easier. Similarly, the Python script is longer because it does better work and consistent, useful error handling. That adds lines of code but it’s usability and correctness, not overhead. It’s similarly incorrect to treat line count as a proxy for difficulty, as anyone who’s ever had to deal with quoting or data structures in shell scripts knows. The use of pathlib or os.path sometimes gets grumbles for extra characters but I’ve found it nearly inevitable that the grumblers will have something fail or destroy data because they hit a filename or argument with a space or special character in it. On one notable occasion, that resulted in an `rm -rf something something /`. (If you’re writing a shell script, shellcheck is mandatory) The best criticism I’d make is that it could be easier to replace a shell pipeline. Python’s subprocess makes that pretty easy but you still need a minimum of 3 calls per command plus whatever you need to actually do with the output. Since subprocess.{run,check_output} handles most of my needs this isn’t a big deal and there are excellent modules if you don’t mind dependencies.

cle 6y ago
This. The Bash script actually has more dependencies, it relies on a number of external programs (ps, kill, mkdir, sort, ls, cp, echo). What versions are on your system? What versions on the systems of the people running the script? Do they support the same features? If you're running on a Mac are you using the Mac-shipped programs or the GNU coreutils from homebrew? Also, that Bash script that the author is defending has a subtle bug in it, because Bash is subtle. The bug is here: $APP $filename >"${output_dir}/summary_${name}.txt" if [ $? != 0 ]; then echo "Error $? in app" fi The return value of [ ... ] overwrites the $?, so the inner $? is the result of the test, not $APP. Which is exactly the kind of crap that everyone's talking about when saying that we should avoid Bash scripts. It's full of landmines like this. Bash is great for interactive work on the terminal. It is not great for writing correct, maintainable programs.

saghm 6y ago

> However, Python and Ruby aren't good shell replacements in general. Shell is a domain-specific language for dealing with concurrent processes and the file system. But Python and Ruby have too much abstraction over these concepts, sometimes in the name of portability (e.g. to Windows). Ruby has pretty good thing wrappers around running shell commands (e.g. using backticks to run a command and get the output as a string or calling system() on a string to execute as a command and get back the error code), and my understanding is that it doesn't really have great Windows support. My impression here is that the author of this FAQ is probably not super familiar with Ruby based on the way they seem to equate the abstractions in both languages.