9 ms·

> The opposite of "it's like riding a bike" is "it's like programming in bash". > A phrase which means that no matter how many times you do something, you will

by yarinr 6y ago

> The opposite of "it's like riding a bike" is "it's like programming in bash". > A phrase which means that no matter how many times you do something, you will have to re-learn it every single time. Couldn't phrase it better myself. After wasting way too much time, I tend to go with Python for anything more than a couple of lines, even trivial stuff. That's, of course, unless there's a very compelling reason to use bash.

hedora 6y ago
I’m the opposite. Every time a python script throws an exception at me (pretty much every time I invoke a python script for the first time), my first debugging step is to read it and attempt to rewrite it in bash. 9 times out of 10, the result is 10-100x times shorter, with zero external dependencies, and is also easier to debug. In the other case, the python thing is some non-trivial monstrosity that should have been implemented in a compiled language with static type checking.
- busterarm 6y ago
  Give me $1 for every time I see a whole python script with loads of dependencies for the simplest sed/awk/jq task with pipes.
  - rmrfstar 6y ago
    Python is readable, awk is not. I'd rather spend an extra 5 minutes writing something that I don't have to burn 20 minutes understanding in just a few months time. I think the popularity of Python for scripting is pretty good evidence that PG was dead wrong on the importance of brevity in programming languages. Further evidence can be found in Python's development of type hints.
    - sillysaurusx 6y ago
      Python isn't that readable when piping. From https://docs.python.org/3/library/subprocess.html#replacing-shell-pipeline https://docs.python.org/3/library/subprocess.html#replacing-... output=$(dmesg | grep hda) becomes p1 = Popen(["dmesg"], stdout=PIPE) p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. output = p2.communicate()[0] This isn't a contrived example. One reason I like using macros in Python is because it simplifies exactly this boilerplate. You can write a simplifying function, but programs rarely do.
      - powersnail 6y ago
        I agree. Subprocess isn't really a shell-ish pipe. And it's sometimes slow (at least it used to be). This is why I use the shell overall, but when I need to process something that's better expressed in python, I use pypyp.
        sillysaurusx 6y ago
        Thanks for mentioning pypyp! It looks sweet.
      - d0mine 6y ago
        Why would you reimplement piping? Just use the shell: output = check_output(“dmesg | grep hda”, shell=True) Or regular Python string methods could be used instead of piping to grep.
        deleted 6y ago
        [deleted]
      - deleted 6y ago
        [deleted]
    - pbar 6y ago
      Another counterpoint to readability: the benefit is diluted if the reader isn’t familiar with the dependencies. I know this goes both ways, though in some circles (say, devops) there’s more familiarity with Unix tools than Python. So the cognitive overhead of getting familiar with something you’re not IMO outweighs readability in either case.
    - dig1 6y ago
      I believe it is a matter of taste and experience. Some people prefer to read & edit 3-4 concise lines, and some prefer ten pages of prose. > I think the popularity of Python for scripting is pretty good evidence that PG was dead wrong on the importance of brevity in programming languages. IMO Python has a fair number of brevity constructs, which was it's one of the selling points, besides keeping other things readable. If you look at the published code from the research community, many of it resembles Matlab or Mathematica's style.
    - czynskee 6y ago
      If you find awk unreadable, maybe check out this link? https://ferd.ca/awk-in-20-minutes.html https://ferd.ca/awk-in-20-minutes.html I actually find awk very readable. It's a far simpler language than Python really. The exception is perhaps if you need to do something complex, like parse a csv.
      - busterarm 6y ago
        Honestly I think if someone finds awk unreadable, they're probably cutting corners elsewhere and probably shouldn't be doing unix systems programming. It's like all of the bash scripts that we see that don't handle failures and traps or print a usage block.
        rmrfstar 6y ago
        Out of curiosity, I grepped the Qubes github repos for awk. Here's the most complicated use I could find: awk -F : '/\/home/ { print $1":"$3":"$4":"$6 } '
      - munificent 6y ago
        > It's a far simpler language than Python really. This is true, but if you already know Python, then "awk + Python" has greater total complexity than "just Python". So the question is does awk add enough value to be worth the incremental cost of learning it in addition to Python? I think for many, the answer is "no".
    - coward8675309 6y ago
      Das Tolle an Deutsch ist, dass es lesbar ist, während Englisch nicht.
      - busterarm 6y ago
        Ich stimme zu.
    - fomine3 6y ago
      In this context (vs Bash), I argue that awk is pretty readable and perl is somewhat readable.
  - rualca 6y ago
    > Give me $1 for every time I see a whole python script with loads of dependencies for the simplest sed/awk/jq task with pipes. I'll add good old Make to the list. It boggles the mind how some people prefer to reinvent the wheel instead of pulling a standard tool from a standard tool belt.
    - busterarm 6y ago
      I pretty much always agreed with this until Bazel became a thing. It's the first tool that I truly feel like it might be superior to Make.
  - aldanor 6y ago
    Read it as "give me ${1}", paused to think for a bit.
    - busterarm 6y ago
      Well played. :D
  - cle 6y ago
    You can do sed/awk/jq trivially in Python with no dependencies, other than Python itself. Which is a fewer number of dependencies than a script gluing together sed+awk+jq, each of those being a dependency managed implicitly by your distribution's package manager.
    - busterarm 6y ago
      except with sed, awk and jq, I don't have to worry about it being present on a production system and someone leveraging it to execute arbitrary, sophisticated code. You can't just assume that it's okay to have your preferred scripting language everywhere.
      - cle 6y ago
        I don't understand what you're arguing here. You don't run sed, awk, and jq directly, you run them via Bash, your preferred scripting language that can execute arbitrary, sophisticated code. Personally I don't use Python much for these use cases. For anything beyond trivial scripts I write Go programs, compiled into statically-linked executables that I can scp onto a host and run, the only dependency required at that point being libc. Or ideally the remote host is running inside a container, which would allow me to reproduce the dependencies locally (but also has a bunch of other complexity). But saying "just use Bash" and then patting ourselves on the back is not really solving the problem, it's just kicking the can down the road, possibly not even very far.
        busterarm 6y ago
        Bash can't be made to bind a socket that listens and receives instructions from the internet without also having something like socat or netcat or inetd and there's no privilege-free packaging system to rapidly get there. Python and other powerful scripting languages can. Having production servers that don't have Python/Ruby/Perl/etc unless they absolutely must is decades-old advice at this point. The smart ones take that one step further: no production server should have any binaries that aren't absolutely necessary. This is why we use containers.
- deleted 6y ago
  [deleted]
- pierrebai 6y ago
  Your point seemed like confirmation bias to me. I've been given to use just as many broken bash script as Python. (I'll admit I'm saying that in a false attempt at being fair. In my experience, Python script tend to work better. Then again, I tend to work in mixed env which includes Windows OS.) You're mentioning exception errors as if having an explicit error message on failure is a problem, not a plus. What happens when a bash script fails? Or worse, silently fails and leave you with a half-working setup at best and a seriously broken one due to some intermediary step failure being ignored? If you happen to notice the error that is... like the OP pointed out, almost all bash shell script ignore silently errors and figuring which of the hundred of lines fails on your particular setup is a fun exercise. And then... which bash debugger to you use to fix problem in bash shell script you been handed? Ah no, it's long reading and echo and trial and errors.
  - jlgaddis 6y ago
    > If you happen to notice the error that is... like the OP pointed out, almost all bash shell script ignore silently errors and figuring which of the hundred of lines fails on your particular setup is a fun exercise. > And then... which bash debugger to you use to fix problem in bash shell script you been handed? Ah no, it's long reading and echo and trial and errors. Or you could just add "-x" to the infamous "set -Eeuo pipefail" at the top of the script before running it, in which case it'll let you know exactly which of the hundreds of lines failed. Alternatively, if you're using the "script template" in the submission, you can just pass in the "-v" option when you run the script and it'll take care of setting it for you.
- 1vuio0pswjnm7 6y ago
  Same here except I use the default scripting shell on the UNIX-like OS, instead of bash. On Debian Linux and derivatives, it is known as dash: "Debian Almquist shell. A small POSIX-compliant shell that is faster than bash."
- cle 6y ago
  > with zero external dependencies I am very skeptical of this claim...in practice I've found it to rarely be the case. Usually shell scripts have more dependencies, and even worse, they are implicit since Bash provides no mechanism for managing dependencies. Programs like sed, awk, grep, cut, paste, curl, jq, gzip, tar, etc. These are all separate dependencies, usually in separate packages (some are grouped together into coreutils), supporting different feature sets between distributions and OSes. All of those can be replaced with the Python standard library. Bash itself is pretty anemic and isn't very useful without gluing together external programs, which is its raison d'etre after all.
  - Alupis 6y ago
    Are there any systems that run Bash but do not include CoreUtils? I think not... A shell script is literally intended to "glue together" external programs, including all of CoreUtils. If you are doing more than just "gluing together" other programs, then you are programming, not scripting, and you'd be far better off reaching for an actual programming language instead of a shell script of any kind. Python has plenty of warts too. You can't depend on it being installed on any given system, can't depend on the version even (Python 2.x or 3.x?), which libraries are installed, environment, venv support, etc.
    - cle 6y ago
      What version of coreutils? Try comparing the behavior of cp on macOS vs. a modern Linux distribution. They are very different. I'm not saying Python isn't without its problems too. I don't even like using Python that much for scripting. But we should be clear about what tradeoffs we're making when we write Bash scripts, and not pretending like we have "zero dependencies".
      - Alupis 6y ago
        > Try comparing the behavior of cp on macOS vs. a modern Linux distribution Not many people are running macOS servers... and servers tend to be where most shell scripts are being used. macOS has notoriously out of date versions of all system programs, CoreUtils and Bash among them, due to licensing issues with Apple's closed-source proprietary OS. BSD and Linux based systems, generally don't have this issue. That being said, a regular Shell Script (not a Bash Script) will run unmodified on macOS. When people say "shell script", they usually mean an actual `sh` script instead of a Bash script.
- m463 6y ago
  I think bash is marginally more portable, yet python has some wonderful things in the standard library that make it a way more generalized language that will go further. os.path, argparse and os.walk I use a lot.
  - hipjiveguy 6y ago
    agreed - but what's the equivalent of this boiler plate? One that traps errors, and sets up basics with no dependencies?
- stjohnswarts 6y ago
  I'm the opposite. If a shell script grows over say 100 lines, it's time to pull out the python.
- permalac 6y ago
  Do you have a python template that you could share?
digitalsushi 6y ago
I am on dev team that isn't fast to upskill, and have been rewarded for writing my build-test-release solutions in bash because my team is not generally able to self-manage their developer python environments. Without needing to install anything, it is very lucrative for us to use such an unpopular language.
hwc 6y ago
Python is nice in that it works fairly well in Windows too.
vietvu 6y ago
That's my way to do things now. I still have my 500 lines bash script file to backup multiple databases, with dynamic filter, on multiple remote servers... to remind me that I can do that much easier in Python.

by yarinr 6y ago

hedora 6y ago

I’m the opposite. Every time a python script throws an exception at me (pretty much every time I invoke a python script for the first time), my first debugging step is to read it and attempt to rewrite it in bash. 9 times out of 10, the result is 10-100x times shorter, with zero external dependencies, and is also easier to debug. In the other case, the python thing is some non-trivial monstrosity that should have been implemented in a compiled language with static type checking.

busterarm 6y ago
Give me $1 for every time I see a whole python script with loads of dependencies for the simplest sed/awk/jq task with pipes.
- rmrfstar 6y ago
  Python is readable, awk is not. I'd rather spend an extra 5 minutes writing something that I don't have to burn 20 minutes understanding in just a few months time. I think the popularity of Python for scripting is pretty good evidence that PG was dead wrong on the importance of brevity in programming languages. Further evidence can be found in Python's development of type hints.
  - sillysaurusx 6y ago
    Python isn't that readable when piping. From https://docs.python.org/3/library/subprocess.html#replacing-shell-pipeline https://docs.python.org/3/library/subprocess.html#replacing-... output=$(dmesg | grep hda) becomes p1 = Popen(["dmesg"], stdout=PIPE) p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. output = p2.communicate()[0] This isn't a contrived example. One reason I like using macros in Python is because it simplifies exactly this boilerplate. You can write a simplifying function, but programs rarely do.
    - powersnail 6y ago
      I agree. Subprocess isn't really a shell-ish pipe. And it's sometimes slow (at least it used to be). This is why I use the shell overall, but when I need to process something that's better expressed in python, I use pypyp.
      - sillysaurusx 6y ago
        Thanks for mentioning pypyp! It looks sweet.
    - d0mine 6y ago
      Why would you reimplement piping? Just use the shell: output = check_output(“dmesg | grep hda”, shell=True) Or regular Python string methods could be used instead of piping to grep.
      - deleted 6y ago
        [deleted]
    - deleted 6y ago
      [deleted]
  - pbar 6y ago
    Another counterpoint to readability: the benefit is diluted if the reader isn’t familiar with the dependencies. I know this goes both ways, though in some circles (say, devops) there’s more familiarity with Unix tools than Python. So the cognitive overhead of getting familiar with something you’re not IMO outweighs readability in either case.
  - dig1 6y ago
    I believe it is a matter of taste and experience. Some people prefer to read & edit 3-4 concise lines, and some prefer ten pages of prose. > I think the popularity of Python for scripting is pretty good evidence that PG was dead wrong on the importance of brevity in programming languages. IMO Python has a fair number of brevity constructs, which was it's one of the selling points, besides keeping other things readable. If you look at the published code from the research community, many of it resembles Matlab or Mathematica's style.
  - czynskee 6y ago
    If you find awk unreadable, maybe check out this link? https://ferd.ca/awk-in-20-minutes.html https://ferd.ca/awk-in-20-minutes.html I actually find awk very readable. It's a far simpler language than Python really. The exception is perhaps if you need to do something complex, like parse a csv.
    - busterarm 6y ago
      Honestly I think if someone finds awk unreadable, they're probably cutting corners elsewhere and probably shouldn't be doing unix systems programming. It's like all of the bash scripts that we see that don't handle failures and traps or print a usage block.
      - rmrfstar 6y ago
        Out of curiosity, I grepped the Qubes github repos for awk. Here's the most complicated use I could find: awk -F : '/\/home/ { print $1":"$3":"$4":"$6 } '
    - munificent 6y ago
      > It's a far simpler language than Python really. This is true, but if you already know Python, then "awk + Python" has greater total complexity than "just Python". So the question is does awk add enough value to be worth the incremental cost of learning it in addition to Python? I think for many, the answer is "no".
  - coward8675309 6y ago
    Das Tolle an Deutsch ist, dass es lesbar ist, während Englisch nicht.
    - busterarm 6y ago
      Ich stimme zu.
  - fomine3 6y ago
    In this context (vs Bash), I argue that awk is pretty readable and perl is somewhat readable.
- rualca 6y ago
  > Give me $1 for every time I see a whole python script with loads of dependencies for the simplest sed/awk/jq task with pipes. I'll add good old Make to the list. It boggles the mind how some people prefer to reinvent the wheel instead of pulling a standard tool from a standard tool belt.
  - busterarm 6y ago
    I pretty much always agreed with this until Bazel became a thing. It's the first tool that I truly feel like it might be superior to Make.
- aldanor 6y ago
  Read it as "give me ${1}", paused to think for a bit.
  - busterarm 6y ago
    Well played. :D
- cle 6y ago
  You can do sed/awk/jq trivially in Python with no dependencies, other than Python itself. Which is a fewer number of dependencies than a script gluing together sed+awk+jq, each of those being a dependency managed implicitly by your distribution's package manager.
  - busterarm 6y ago
    except with sed, awk and jq, I don't have to worry about it being present on a production system and someone leveraging it to execute arbitrary, sophisticated code. You can't just assume that it's okay to have your preferred scripting language everywhere.
    - cle 6y ago
      I don't understand what you're arguing here. You don't run sed, awk, and jq directly, you run them via Bash, your preferred scripting language that can execute arbitrary, sophisticated code. Personally I don't use Python much for these use cases. For anything beyond trivial scripts I write Go programs, compiled into statically-linked executables that I can scp onto a host and run, the only dependency required at that point being libc. Or ideally the remote host is running inside a container, which would allow me to reproduce the dependencies locally (but also has a bunch of other complexity). But saying "just use Bash" and then patting ourselves on the back is not really solving the problem, it's just kicking the can down the road, possibly not even very far.
      - busterarm 6y ago
        Bash can't be made to bind a socket that listens and receives instructions from the internet without also having something like socat or netcat or inetd and there's no privilege-free packaging system to rapidly get there. Python and other powerful scripting languages can. Having production servers that don't have Python/Ruby/Perl/etc unless they absolutely must is decades-old advice at this point. The smart ones take that one step further: no production server should have any binaries that aren't absolutely necessary. This is why we use containers.
deleted 6y ago
[deleted]
pierrebai 6y ago
Your point seemed like confirmation bias to me. I've been given to use just as many broken bash script as Python. (I'll admit I'm saying that in a false attempt at being fair. In my experience, Python script tend to work better. Then again, I tend to work in mixed env which includes Windows OS.) You're mentioning exception errors as if having an explicit error message on failure is a problem, not a plus. What happens when a bash script fails? Or worse, silently fails and leave you with a half-working setup at best and a seriously broken one due to some intermediary step failure being ignored? If you happen to notice the error that is... like the OP pointed out, almost all bash shell script ignore silently errors and figuring which of the hundred of lines fails on your particular setup is a fun exercise. And then... which bash debugger to you use to fix problem in bash shell script you been handed? Ah no, it's long reading and echo and trial and errors.
- jlgaddis 6y ago
  > If you happen to notice the error that is... like the OP pointed out, almost all bash shell script ignore silently errors and figuring which of the hundred of lines fails on your particular setup is a fun exercise. > And then... which bash debugger to you use to fix problem in bash shell script you been handed? Ah no, it's long reading and echo and trial and errors. Or you could just add "-x" to the infamous "set -Eeuo pipefail" at the top of the script before running it, in which case it'll let you know exactly which of the hundreds of lines failed. Alternatively, if you're using the "script template" in the submission, you can just pass in the "-v" option when you run the script and it'll take care of setting it for you.
1vuio0pswjnm7 6y ago
Same here except I use the default scripting shell on the UNIX-like OS, instead of bash. On Debian Linux and derivatives, it is known as dash: "Debian Almquist shell. A small POSIX-compliant shell that is faster than bash."
cle 6y ago
> with zero external dependencies I am very skeptical of this claim...in practice I've found it to rarely be the case. Usually shell scripts have more dependencies, and even worse, they are implicit since Bash provides no mechanism for managing dependencies. Programs like sed, awk, grep, cut, paste, curl, jq, gzip, tar, etc. These are all separate dependencies, usually in separate packages (some are grouped together into coreutils), supporting different feature sets between distributions and OSes. All of those can be replaced with the Python standard library. Bash itself is pretty anemic and isn't very useful without gluing together external programs, which is its raison d'etre after all.
- Alupis 6y ago
  Are there any systems that run Bash but do not include CoreUtils? I think not... A shell script is literally intended to "glue together" external programs, including all of CoreUtils. If you are doing more than just "gluing together" other programs, then you are programming, not scripting, and you'd be far better off reaching for an actual programming language instead of a shell script of any kind. Python has plenty of warts too. You can't depend on it being installed on any given system, can't depend on the version even (Python 2.x or 3.x?), which libraries are installed, environment, venv support, etc.
  - cle 6y ago
    What version of coreutils? Try comparing the behavior of cp on macOS vs. a modern Linux distribution. They are very different. I'm not saying Python isn't without its problems too. I don't even like using Python that much for scripting. But we should be clear about what tradeoffs we're making when we write Bash scripts, and not pretending like we have "zero dependencies".
    - Alupis 6y ago
      > Try comparing the behavior of cp on macOS vs. a modern Linux distribution Not many people are running macOS servers... and servers tend to be where most shell scripts are being used. macOS has notoriously out of date versions of all system programs, CoreUtils and Bash among them, due to licensing issues with Apple's closed-source proprietary OS. BSD and Linux based systems, generally don't have this issue. That being said, a regular Shell Script (not a Bash Script) will run unmodified on macOS. When people say "shell script", they usually mean an actual `sh` script instead of a Bash script.
m463 6y ago
I think bash is marginally more portable, yet python has some wonderful things in the standard library that make it a way more generalized language that will go further. os.path, argparse and os.walk I use a lot.
- hipjiveguy 6y ago
  agreed - but what's the equivalent of this boiler plate? One that traps errors, and sets up basics with no dependencies?
stjohnswarts 6y ago
I'm the opposite. If a shell script grows over say 100 lines, it's time to pull out the python.
permalac 6y ago
Do you have a python template that you could share?

digitalsushi 6y ago

I am on dev team that isn't fast to upskill, and have been rewarded for writing my build-test-release solutions in bash because my team is not generally able to self-manage their developer python environments. Without needing to install anything, it is very lucrative for us to use such an unpopular language.

hwc 6y ago

Python is nice in that it works fairly well in Windows too.

vietvu 6y ago

That's my way to do things now. I still have my 500 lines bash script file to backup multiple databases, with dynamic filter, on multiple remote servers... to remind me that I can do that much easier in Python.