5 ms·

This is terrible advice. Senior people say don’t do this because they’ve seen it done and it’s an absolute nightmare (I’ve seen it happen multiple times). It’s

by ra1231963 4y ago

This is terrible advice. Senior people say don’t do this because they’ve seen it done and it’s an absolute nightmare (I’ve seen it happen multiple times). It’s untyped, untestable, arcane, and unmaintabale. Even the link says don’t do this. Just don’t. Congrats on having the chops (and neckbeard) to pull this off. If you want to do it in a personal project, more power to you. But it doesn’t scale at organizations.

jmmv 4y ago
Anyone writing code in a language they don't know will write nightmarish code. No matter if it is shell, Python, Go, or what have you. I have seen it happen multiple times as well, with all of these languages. And /that/'s the problem. So, it depends on the context. "Organization" is a very broad term. If the team you are in is familiar with the shell, and the shell is the right tool for the job, there isn't that much of a problem. I'm not arguing for writing large shell scripts when there are better alternatives. But, sometimes, it's the right choice and it can be written properly. And even if it isn't... 500 lines of poorly written shell aren't that many in any case, and they aren't that bad compared to 5000 lines of poorly written Python for example.
- kaba0 4y ago
  I fail to see why you think it would take more lines to do in Python. And the problem with bash is similar to C: many people think they can write correct C code, but as it has been shown plenty of times, that doesn’t really work out in the end. Languages have different tradeoffs and bash really has no redeeming qualities, imo.
  - mananaysiempre 4y ago
    > I fail to see why you think it would take more lines to do in Python. Because Bourne shell is not a Python-class language like Ruby or perhaps even Scheme or AWK are. Things that are easy in shell can be hard in Python and vice versa: awkward wrangling of subprocess.Popen in Python can be a single line of shell; simple tree munging with lxml in Python can easily be a hundred or more lines of wasteful xmlstarlet invocations in shell. Parallel processing, even without the heavy artillery of make or GNU parallel, is easy in shell but hard in Python. Structured data is hard in shell but easy in Python. (Similarly, I wouldn’t use ML or Haskell to write a web scraper, and I wouldn’t use Python to write a compiler.)
    - patrakov 4y ago
      Regarding parallel processing, I have to disagree. It is bad in shell, because it makes best practices such as "set -e" impractical. The usual failure mode is a script that does not stop correctly on SIGTERM, and its children have to be hunted down and killed separately.
      - mananaysiempre 4y ago
        That’s fair. Perhaps a better way to put it would be that sloppy concurrency is easy. This doesn’t sound all that good, but it’s easy enough that I find myself parallelizing one-off things in shell that would be such a hassle to do nonsequentially in Python (compared to the magnitude of the problem) that the option wouldn’t even surface in my brain. (If I rewrite the shell script in Python later, the rewritten version is often sequential.) The shell error handling story for “run stream processing steps in lockstep” parallelism is much better than for “spawn a bunch of identical children” parallelism, though, and that’s often enough.
  - hedora 4y ago
    I’ll bite. Show me the equivalent of this in python, including error handling, parallelism, disk footprint, and real-time status output: #!/bin/bash -eux -o pipefail TMPDIR=$(mktemp -d tempdirXXXXX) trap “rm -rf $TMPDIR” SIGINT SIGTERM ERR EXIT ( cd $TMPDIR foo | grep x > foo.out & bar > bar.out & baz $TMP | tee baz.out & ) wait tar -cf - -C $TEMPDIR | zstd -6 -z - | pv | ssh server.com bash -c “cat - > out.tag.zstd”
    - EuAndreh 4y ago
      You can forego the final "bash -c" by leveraging dd(1) to avoid the redirection problem: ... | ssh server.com dd of=out.tag.zstd
      - EuAndreh 4y ago
        dd(1) reads from STDIN by default unless given an "if=" parameter, and writes to STDOUT by default umless given an "of=" parameter.
      - hedora 4y ago
        Thanks! Will definitely use that moving forward.
    - _dain_ 4y ago
      $ shellcheck myscript Line 1: #!/bin/bash -eux -o pipefail ^-- SC2096 (error): On most OS, shebangs can only specify a single parameter. Line 4: trap "rm -rf $TMPDIR" SIGINT SIGTERM ERR EXIT ^-- SC2064 (warning): Use single quotes, otherwise this expands now rather than when signalled. Line 6: cd $TMPDIR ^-- SC2086 (info): Double quote to prevent globbing and word splitting. Did you mean: (apply this, apply all SC2086) cd "$TMPDIR" Line 9: baz $TMP | tee baz.out & ^-- SC2086 (info): Double quote to prevent globbing and word splitting. Did you mean: (apply this, apply all SC2086) baz "$TMP" | tee baz.out & Line 13: tar -cf - -C $TEMPDIR | zstd -6 -z - | pv | ssh server.com bash -c "cat - > out.tag.zstd" ^-- SC2086 (info): Double quote to prevent globbing and word splitting. ^-- SC2153 (info): Possible misspelling: TEMPDIR may not be assigned. Did you mean TMPDIR? Did you mean: (apply this, apply all SC2086) tar -cf - -C "$TEMPDIR" | zstd -6 -z - | pv | ssh server.com bash -c "cat - > out.tag.zstd"
- mattpallissard 4y ago
  > Anyone writing code in a language they don't know will write nightmarish code. No matter if it is shell, Python, Go, or what have you. I have seen it happen multiple times as well, with all of these languages. And /that/'s the problem. Came here to say this.
- IshKebab 4y ago
  You're implying that all languages are equally likely to produce buggy programs. That's clearly false. There's a spectrum. At one end you have things like Haskell, OCaml, Rust, (or if you go really extreme Dafny etc.). At the other end you have Bash. There's no way a 500 line Bash script would require 5000 lines of Python.
hedora 4y ago
Python is untyped and untestable. The only way I’ve seen people use it is by importing whatever libraries were popular in whatever year they wrote it, so it’s also arcane. Also, to understand someone else’s python, you have to read at least 10x more lines of line noise, so it somehow manages to be less maintainable than shell. Concrete example: Every python code base I’ve encountered has some function named exec that poorly reimplements shell job control and argument passing. As the python code bases grow, they end up with more than one of these, all of which have different bugs. I usually discover this about 6 hours after the script has been sent to me, since someone who has heard the words “supply chain vulnerability” locked down whatever machine I need to run the script on. The last ten times this has happened (at multiple companies, with multiple teams) it ended up being faster to figure out what shell commands the python script wanted to invoke, port a few things to jq and then reimplement the rest in a mix of shell and perl. The result is always 1-10% the lines of code of the python, and actually runs without downloading an entire python distribution, dependency manager, and hundreds of broken packages at runtime.
- _dain_ 4y ago
  >Python is untyped and untestable. Completely untrue. You're conflating static typing with strong typing. And pytest is a great test framework. And you're recommending shell for better typing and testability? The language where everything is a string, there are no proper arrays, and functions don't have local variables? You have to be trolling.
  - oweiler 4y ago
    On top of that, fn in Bash can only return their status code. Yes, command substitution exists but is full of footguns.
  - hedora 4y ago
    I’ve probably spent two weeks (80+ active hours) debugging stuff like ‘“” is not None’. At this point, I’d prefer a language with only string types to one that only checks types in production. pytest is terrible, at least from how I’ve seen people use it. It hides the diagnostic error explaining why the test failed in a random place. Its output is not parseable. If you have a long running test that times out, then it buffers the stdout and stderr so you have to wait for the fallback test time out every. single. time.

jmmv 4y ago

Anyone writing code in a language they don't know will write nightmarish code. No matter if it is shell, Python, Go, or what have you. I have seen it happen multiple times as well, with all of these languages. And /that/'s the problem. So, it depends on the context. "Organization" is a very broad term. If the team you are in is familiar with the shell, and the shell is the right tool for the job, there isn't that much of a problem. I'm not arguing for writing large shell scripts when there are better alternatives. But, sometimes, it's the right choice and it can be written properly. And even if it isn't... 500 lines of poorly written shell aren't that many in any case, and they aren't that bad compared to 5000 lines of poorly written Python for example.

kaba0 4y ago
I fail to see why you think it would take more lines to do in Python. And the problem with bash is similar to C: many people think they can write correct C code, but as it has been shown plenty of times, that doesn’t really work out in the end. Languages have different tradeoffs and bash really has no redeeming qualities, imo.
- mananaysiempre 4y ago
  > I fail to see why you think it would take more lines to do in Python. Because Bourne shell is not a Python-class language like Ruby or perhaps even Scheme or AWK are. Things that are easy in shell can be hard in Python and vice versa: awkward wrangling of subprocess.Popen in Python can be a single line of shell; simple tree munging with lxml in Python can easily be a hundred or more lines of wasteful xmlstarlet invocations in shell. Parallel processing, even without the heavy artillery of make or GNU parallel, is easy in shell but hard in Python. Structured data is hard in shell but easy in Python. (Similarly, I wouldn’t use ML or Haskell to write a web scraper, and I wouldn’t use Python to write a compiler.)
  - patrakov 4y ago
    Regarding parallel processing, I have to disagree. It is bad in shell, because it makes best practices such as "set -e" impractical. The usual failure mode is a script that does not stop correctly on SIGTERM, and its children have to be hunted down and killed separately.
    - mananaysiempre 4y ago
      That’s fair. Perhaps a better way to put it would be that sloppy concurrency is easy. This doesn’t sound all that good, but it’s easy enough that I find myself parallelizing one-off things in shell that would be such a hassle to do nonsequentially in Python (compared to the magnitude of the problem) that the option wouldn’t even surface in my brain. (If I rewrite the shell script in Python later, the rewritten version is often sequential.) The shell error handling story for “run stream processing steps in lockstep” parallelism is much better than for “spawn a bunch of identical children” parallelism, though, and that’s often enough.
- hedora 4y ago
  I’ll bite. Show me the equivalent of this in python, including error handling, parallelism, disk footprint, and real-time status output: #!/bin/bash -eux -o pipefail TMPDIR=$(mktemp -d tempdirXXXXX) trap “rm -rf $TMPDIR” SIGINT SIGTERM ERR EXIT ( cd $TMPDIR foo | grep x > foo.out & bar > bar.out & baz $TMP | tee baz.out & ) wait tar -cf - -C $TEMPDIR | zstd -6 -z - | pv | ssh server.com bash -c “cat - > out.tag.zstd”
  - EuAndreh 4y ago
    You can forego the final "bash -c" by leveraging dd(1) to avoid the redirection problem: ... | ssh server.com dd of=out.tag.zstd
    - EuAndreh 4y ago
      dd(1) reads from STDIN by default unless given an "if=" parameter, and writes to STDOUT by default umless given an "of=" parameter.
    - hedora 4y ago
      Thanks! Will definitely use that moving forward.
  - _dain_ 4y ago
    $ shellcheck myscript Line 1: #!/bin/bash -eux -o pipefail ^-- SC2096 (error): On most OS, shebangs can only specify a single parameter. Line 4: trap "rm -rf $TMPDIR" SIGINT SIGTERM ERR EXIT ^-- SC2064 (warning): Use single quotes, otherwise this expands now rather than when signalled. Line 6: cd $TMPDIR ^-- SC2086 (info): Double quote to prevent globbing and word splitting. Did you mean: (apply this, apply all SC2086) cd "$TMPDIR" Line 9: baz $TMP | tee baz.out & ^-- SC2086 (info): Double quote to prevent globbing and word splitting. Did you mean: (apply this, apply all SC2086) baz "$TMP" | tee baz.out & Line 13: tar -cf - -C $TEMPDIR | zstd -6 -z - | pv | ssh server.com bash -c "cat - > out.tag.zstd" ^-- SC2086 (info): Double quote to prevent globbing and word splitting. ^-- SC2153 (info): Possible misspelling: TEMPDIR may not be assigned. Did you mean TMPDIR? Did you mean: (apply this, apply all SC2086) tar -cf - -C "$TEMPDIR" | zstd -6 -z - | pv | ssh server.com bash -c "cat - > out.tag.zstd"
mattpallissard 4y ago
> Anyone writing code in a language they don't know will write nightmarish code. No matter if it is shell, Python, Go, or what have you. I have seen it happen multiple times as well, with all of these languages. And /that/'s the problem. Came here to say this.
IshKebab 4y ago
You're implying that all languages are equally likely to produce buggy programs. That's clearly false. There's a spectrum. At one end you have things like Haskell, OCaml, Rust, (or if you go really extreme Dafny etc.). At the other end you have Bash. There's no way a 500 line Bash script would require 5000 lines of Python.

hedora 4y ago

Python is untyped and untestable. The only way I’ve seen people use it is by importing whatever libraries were popular in whatever year they wrote it, so it’s also arcane. Also, to understand someone else’s python, you have to read at least 10x more lines of line noise, so it somehow manages to be less maintainable than shell. Concrete example: Every python code base I’ve encountered has some function named exec that poorly reimplements shell job control and argument passing. As the python code bases grow, they end up with more than one of these, all of which have different bugs. I usually discover this about 6 hours after the script has been sent to me, since someone who has heard the words “supply chain vulnerability” locked down whatever machine I need to run the script on. The last ten times this has happened (at multiple companies, with multiple teams) it ended up being faster to figure out what shell commands the python script wanted to invoke, port a few things to jq and then reimplement the rest in a mix of shell and perl. The result is always 1-10% the lines of code of the python, and actually runs without downloading an entire python distribution, dependency manager, and hundreds of broken packages at runtime.

_dain_ 4y ago
>Python is untyped and untestable. Completely untrue. You're conflating static typing with strong typing. And pytest is a great test framework. And you're recommending shell for better typing and testability? The language where everything is a string, there are no proper arrays, and functions don't have local variables? You have to be trolling.
- oweiler 4y ago
  On top of that, fn in Bash can only return their status code. Yes, command substitution exists but is full of footguns.
- hedora 4y ago
  I’ve probably spent two weeks (80+ active hours) debugging stuff like ‘“” is not None’. At this point, I’d prefer a language with only string types to one that only checks types in production. pytest is terrible, at least from how I’ve seen people use it. It hides the diagnostic error explaining why the test failed in a random place. Its output is not parseable. If you have a long running test that times out, then it buffers the stdout and stderr so you have to wait for the fallback test time out every. single. time.