5 ms·

Reinventing escaping over and over again (which bash scripts in particular seem to encourage) is a suckers game. It's difficult to get right and if you're cons

Reinventing escaping over and over again (which bash scripts in particular seem to encourage) is a suckers game. It's difficult to get right and if you're constantly redoing it you're eventually going to make a mistake. I've worked in web security and it's sad to see how likely it is for people with good intentions to mess this up. I'm glad the author basically came to this conclusion. The winning strategy is to use a library/framework/whatever for embedding user-provided content into HTML. Sane HTML template libraries will do this. That library has had more time to get it right. Furthermore a well designed API will clearly indicate what is trusted vs. untrusted data and all untrusted data is properly encoded before being embedded. See the "Security Model" section of golangs HTML templates below. An alternative to using the git tools which is appropriate for serious work (shell pipelines are great for prototyping) is libgit2. It has bindings for many languages. It's very easy to use (sometimes (not always) easier than the CLI) and often much higher performance vs. big shell pipelines (operating on text gets slow pretty fast, and often you end up using xargs...) An example set of tools: https://golang.org/pkg/html/template/ https://golang.org/pkg/html/template/ + https://github.com/libgit2/git2go https://github.com/libgit2/git2go . It's not as succinct as a bash script but it's easier to build something that's correct. Use the shell to prototype, build it right in a saner environment.

falsedan 9y ago
> An alternative to using the git tools which is appropriate for serious work This belittles their and anyone else's bash scripts for git as 'toys'. That's unfair, and completely unnecessary for your point. > big shell pipelines (operating on text gets slow pretty fast, and often you end up using xargs...) Big shell pipelines are actually blazingly fast, since each command in the pipeline runs in parallel. > It's not as succinct as a bash script but it's easier to build something that's correct. Use the shell to prototype, build it right in a saner environment. I would say the ease depends on the relative difficulty and functionality of the languages. Casting bash as the insane bad language doesn't win you any points: we all know bash is a hot mess of a garbage fire, now let's use it to do useful good work. I would struggle to justify the work I'd need to do to write the equivalent in C (or python, where it would be slower) for one command that's only run in the release process.
- vog 9y ago
  > This belittles their and anyone else's bash scripts for git as 'toys'. That's unfair, and completely unnecessary for your point. I don't think so. It is a very common mistake to miss the point where you should switch from a shell script to Python. (... or Ruby, Haskell, OCaml, or any other language with good libraries, sane error handling and sane data structures. And yes, I'm saying this being aware of, and always using, "set -eu -o pipefail".) In other words: A complex collection of shell code is a very good warning indicator that somebody missed that point.
  - yorwba 9y ago
    Using a thin layer of bash to feed the output of a command into a short Python script is an excellent use of shell in my opinion.
    - Karunamon 9y ago
      But why would you, when you can just do subprocess.popen on the command and then slice and dice the result in a language that can do regexes, splitting, format strings, and so forth? I mean, if you're going to be using Python anyways, you might as well just use the standard library and skip the extra bash step entirely. The code might be a bit longer, but infinitely safer and more maintainable.
      - yorwba 9y ago
        If all you're doing is rewriting the pipelines you'd have written in Bash using subprocess in Python, you're not making your code safer or more maintainable. Unless I'm missing something, subprocess.Popen requires you to explicitly wire up the inputs and outputs, whereas Bash has clear and readable syntax for the most common idioms. For a program that mostly just calls other programs with the right parameters, Python only obfuscates the important bits with details that a shell abstracts away. For a program that has to do some non-trivial processing, pure Bash+coreutils definitely loses to a general-purpose programming language with good standard library. But that doesn't mean that you have to completely switch over. You can leave the parts in Bash that are just pipelines of other programs (again, Python can't improve anything there) and do your further processing in Python. Whether that's implemented as a Bash script calling Python in a pipeline or Python delegating part of it's processing to a Bash script just depends on the most natural entry point. It might feel "impure" to mix two different languages in a program, but I think using each language in the domain where it's strengths shine is a much better idea than attempting to emulate the core features of one language in the other.
mbrock 9y ago
Very recently I used libgit2 in an application. It was quite nice. Then I noticed my build was failing because I make static binaries and my setup couldn't handle libgit2 via Haskell for some obscure reason. So I rewrote my Git layer to use the Git porcelain shell tools (git mktree, git hash-object, git commit-tree, etc) as a quick and easy way to get it working again. I find it quite typical that using native libraries via FFI instead of shelling out to programs leads to tedious problems. And that's part of why I cherish the Unix tradition of combining different programs; shell scripting is one manifestation of that.
- chubot 9y ago
  (author here) Yup, that is exactly my experience. You're "solving" one problem but just creating another one: versions and dependencies. Shell is more reliable in that sense. Less software is better. Git already exists, let's use it. This link is in the article, I think a lot of people don't get it: http://www.catb.org/esr/writings/unix-koans/ten-thousand.html http://www.catb.org/esr/writings/unix-koans/ten-thousand.htm...
zeveb 9y ago
> The winning strategy is to use a library/framework/whatever for embedding user-provided content into HTML. I think I agree. Shell is very convenient, but it's just too easy to make mistakes. I think something like eshell[0] or a shell based on sweet expressions[1] could go a long way towards providing a powerful shell in a real programming language — with safety against foot-guns like one finds in bash & zsh (the latter is still my default shell, despite its problems). [0] https://www.gnu.org/software/emacs/manual/html_mono/eshell.html https://www.gnu.org/software/emacs/manual/html_mono/eshell.h... [1] http://readable.sourceforge.net/ http://readable.sourceforge.net/
chubot 9y ago
Yes, I addressed this point in the article -- you are advocating "the pedantic solution". The point of the article is that we can "have our cake and eat it too". (FWIW back in 2009, I designed the original template language that Go html/template was based on, JSON Template, and later Mike Samuel added auto-escaping.)

falsedan 9y ago

> An alternative to using the git tools which is appropriate for serious work This belittles their and anyone else's bash scripts for git as 'toys'. That's unfair, and completely unnecessary for your point. > big shell pipelines (operating on text gets slow pretty fast, and often you end up using xargs...) Big shell pipelines are actually blazingly fast, since each command in the pipeline runs in parallel. > It's not as succinct as a bash script but it's easier to build something that's correct. Use the shell to prototype, build it right in a saner environment. I would say the ease depends on the relative difficulty and functionality of the languages. Casting bash as the insane bad language doesn't win you any points: we all know bash is a hot mess of a garbage fire, now let's use it to do useful good work. I would struggle to justify the work I'd need to do to write the equivalent in C (or python, where it would be slower) for one command that's only run in the release process.

vog 9y ago
> This belittles their and anyone else's bash scripts for git as 'toys'. That's unfair, and completely unnecessary for your point. I don't think so. It is a very common mistake to miss the point where you should switch from a shell script to Python. (... or Ruby, Haskell, OCaml, or any other language with good libraries, sane error handling and sane data structures. And yes, I'm saying this being aware of, and always using, "set -eu -o pipefail".) In other words: A complex collection of shell code is a very good warning indicator that somebody missed that point.
- yorwba 9y ago
  Using a thin layer of bash to feed the output of a command into a short Python script is an excellent use of shell in my opinion.
  - Karunamon 9y ago
    But why would you, when you can just do subprocess.popen on the command and then slice and dice the result in a language that can do regexes, splitting, format strings, and so forth? I mean, if you're going to be using Python anyways, you might as well just use the standard library and skip the extra bash step entirely. The code might be a bit longer, but infinitely safer and more maintainable.
    - yorwba 9y ago
      If all you're doing is rewriting the pipelines you'd have written in Bash using subprocess in Python, you're not making your code safer or more maintainable. Unless I'm missing something, subprocess.Popen requires you to explicitly wire up the inputs and outputs, whereas Bash has clear and readable syntax for the most common idioms. For a program that mostly just calls other programs with the right parameters, Python only obfuscates the important bits with details that a shell abstracts away. For a program that has to do some non-trivial processing, pure Bash+coreutils definitely loses to a general-purpose programming language with good standard library. But that doesn't mean that you have to completely switch over. You can leave the parts in Bash that are just pipelines of other programs (again, Python can't improve anything there) and do your further processing in Python. Whether that's implemented as a Bash script calling Python in a pipeline or Python delegating part of it's processing to a Bash script just depends on the most natural entry point. It might feel "impure" to mix two different languages in a program, but I think using each language in the domain where it's strengths shine is a much better idea than attempting to emulate the core features of one language in the other.

mbrock 9y ago

Very recently I used libgit2 in an application. It was quite nice. Then I noticed my build was failing because I make static binaries and my setup couldn't handle libgit2 via Haskell for some obscure reason. So I rewrote my Git layer to use the Git porcelain shell tools (git mktree, git hash-object, git commit-tree, etc) as a quick and easy way to get it working again. I find it quite typical that using native libraries via FFI instead of shelling out to programs leads to tedious problems. And that's part of why I cherish the Unix tradition of combining different programs; shell scripting is one manifestation of that.

chubot 9y ago
(author here) Yup, that is exactly my experience. You're "solving" one problem but just creating another one: versions and dependencies. Shell is more reliable in that sense. Less software is better. Git already exists, let's use it. This link is in the article, I think a lot of people don't get it: http://www.catb.org/esr/writings/unix-koans/ten-thousand.html http://www.catb.org/esr/writings/unix-koans/ten-thousand.htm...

zeveb 9y ago

> The winning strategy is to use a library/framework/whatever for embedding user-provided content into HTML. I think I agree. Shell is very convenient, but it's just too easy to make mistakes. I think something like eshell[0] or a shell based on sweet expressions[1] could go a long way towards providing a powerful shell in a real programming language — with safety against foot-guns like one finds in bash & zsh (the latter is still my default shell, despite its problems). [0] https://www.gnu.org/software/emacs/manual/html_mono/eshell.html https://www.gnu.org/software/emacs/manual/html_mono/eshell.h... [1] http://readable.sourceforge.net/ http://readable.sourceforge.net/

chubot 9y ago

Yes, I addressed this point in the article -- you are advocating "the pedantic solution". The point of the article is that we can "have our cake and eat it too". (FWIW back in 2009, I designed the original template language that Go html/template was based on, JSON Template, and later Mike Samuel added auto-escaping.)