16 ms·

Maybe I'm missing something, but it has always seemed to be that people who complain about shell scripts do not approach shell scripting the way they approach a

Maybe I'm missing something, but it has always seemed to be that people who complain about shell scripts do not approach shell scripting the way they approach any other programming language. By this I mean that they seem to actively resist the idea that they need to learn the language to get the most out of it. One of my early jobs involved maintaining a pile of shell scripts that were part of automating a publishing system pipeline. So I picked up a couple of suggested books and learned everything I could about shell scripting. I quickly grew to enjoy working with shell scripts for automation. Yes, there are quirks and oddities in shell scripting but the language is very well document and good practices and style guides have been around for ages. Would you expect to understand any language without studying it first?

teknopaul 4y ago
This is so true. Torvalds wrote git in shell. You can do great things. Provided... a) agreed, RTFM, (man bash is not long) b) you dont try to do things shells shouldn't do. c) for things you shouldn't do, you write a clean stable cli interface in some other language. Often you can do c) in bash temporarily and port only if/when it gets complicated. (That's what git did)
- masklinn 4y ago
  > Torvalds wrote git in shell. Did he? Because that’s definitely not what the initial revision of git that’s in the repo shows. It has 1kloc of C, a readme, and a makefile. I struggle to think how you’d even get to the wonky tree format if you used the shell as your implementation language. The first shell scripts landed a few weeks later, in 839a7a06f35bf8cd563a41d6db97f453ab108129 and were convenience wrappers to facilitate interacting with contributors (a wrapper around `merge` to merge a single file, a trivial wrapper around `fsck-cache` to prune, and finally a somewhat more involved original `git pull` which used `rsync` to retrieve all of the remote’s object store then would merge its HEAD with yours). The next two scripts were literally called “example scripts” in their commit messages, and respectively for creating a signed tag and applying a patch.
  - rbonvall 4y ago
    If I remember correctly, git was originally distributed as several small programs like git-checkout, git-reset, etc., some written in C and others in shell.
  - rascul 4y ago
    The first commit of the git command that I found was a shell script. It called other scripts which used the C stuff behind the scenes. https://git.kernel.org/pub/scm/git/git.git/commit/git?id=e764b8e8b3c50b131be825532ba26fa346d6586e https://git.kernel.org/pub/scm/git/git.git/commit/git?id=e76...
    - masklinn 4y ago
      That’s not “git was originally written in shell” by any honest interpretation of the term.
      - theamk 4y ago
        here is "git commit" in shell: https://git.kernel.org/pub/scm/git/git.git/tree/git-commit-script?id=e764b8e8b3c50b131be825532ba26fa346d6586e https://git.kernel.org/pub/scm/git/git.git/tree/git-commit-s... here is "git status", with all its formatting glory: https://git.kernel.org/pub/scm/git/git.git/tree/git-status-script?id=e764b8e8b3c50b131be825532ba26fa346d6586e https://git.kernel.org/pub/scm/git/git.git/tree/git-status-s... but yes, I would not call it "git was written in shell". But some of its significant pieces were.
      - rascul 4y ago
        It's pretty much the only interpretation I can come up with that results in git being written in shell. I do think it's a bit of a stretch though.
    - tylersmith 4y ago
      And in only 3 lines of code!
- H4ZB7 4y ago
  > man bash is not long it's literally longer than reading the Go spec and even after reading it several times i still can't code bash
cb321 4y ago
I think what you may be missing is the article's first point: shells are also a high-frequency REPL. The programming language is mostly popular because you can capture/record interactively/incrementally developed logic into "a script" (from typescript(1), script(1), command history, or whatever). What leads people to think they understand the language without detailed study is their many high freq success cases of running commands. Success repetition breeds (over-)confidence. An alternate system design would be a "translator" or "translation assistant" from the high-frequency REPL/interactive/incremental domain into a more static domain (probably with human edits to correct automated mistakes). Part of what even this design would miss (and is also not elaborated upon in the article) is that the nature of interactive/incremental work is very much that of "prototyping". Prototyping often involves many simplifying assumptions, not just about newlines in filenames (a bug in the article's xargs usage) or permissions, but really a constellation of them -- whatever simplifies things. The biggest problems with scripts as a final product are more A) identifying and B) walking back all the various simplifying assumptions of the prototype.
throwawaaarrgh 4y ago
Every complaint on HN about Bash is from one of two people: 1) someone who has never read the man page, 2) someone who wishes Bash were Python. Bash is my favorite programming language, because it's not really a programming language. Rather than being some academic philosophical high-minded solution, it was written by systems practitioners for backwards compatibility and usability. Once you learn it, it replaces 85% of other languages for systems work.
- AtlasBarfed 4y ago
  Yes whitespace sensitivity, magic characters (WITH whitespace sensitivity), no standard library, no unicode, horrible math expressions, awful string, mishmash of acceptable good practices and outmoded stuff, printlns to debug, subpar error messages, no particularly good IDE, dynamic typing but really none of the convenience. I honestly have no idea how anyone got anything done before Stackoverflow with bash. It's all copy paste and half the things you need to do in modern scripting isn't in typical UNIX command set (curl, etc) and there's no dependency resolution mechanism. But yeah, read the manpage to fix everything. Sure dude. The article is right though. Command line and pipes is a flow type model, but programming is a different ordered thing in the natural sense: Command flow: cat file | grep values | sort values Programming flow: sort(grep(cat(file))
- martopix 4y ago
  I am admittedly someone who wishes Bash were Python (ish). What's wrong with that though? Of course I want the REPL features of Bash, its terseness, its builtin commands for all sorts of OS and filesystem operations. But is it wrong to ask for a more modern syntax on the language side of things (i.e. the for loop for example)?
  - graymatters 4y ago
    Ah, that’s kind of like refugees fleeing all kinds of failing states to better places, that have rule or law, democracy, human rights, prosperity and then actively trying to install the norms of the old country in the new place they fled to.
  - deleted 4y ago
efxhoy 4y ago
Do you have books to recommend about shell scripting?
- latexr 4y ago
  Not a book (and I’m also not the person you asked), but take a look at https://mywiki.wooledge.org/BashGuide https://mywiki.wooledge.org/BashGuide
- woodrowbarlow 4y ago
  for minutes spent versus knowledge gained, the best bang for buck in my experience is google's shell style guide. i don't agree with all of their rules, but their explanations for each give a ton of insight about the language. https://google.github.io/styleguide/shellguide.html https://google.github.io/styleguide/shellguide.html
- chasil 4y ago
  If you search for "POSIX shell," you will arrive here: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V... https://pubs.opengroup.org/onlinepubs/9699919799/utilities/ https://pubs.opengroup.org/onlinepubs/9699919799/utilities/
znpy 4y ago
I agree, but would like to add that people seem to forget all of the decency when writing shell scripts. They don’t do input checking and validation, they don’t check for errors, return codes and stuff… No wonder their scripts fail or do something completely unexpected. Can you blame it on the shell? No this is not a rhetorical question: you cannot blame the shell for your negligence and incompetence.
- bravetraveler 4y ago
  Not trying to be a pedant, I tend to agree, but isn't posing the question to give the statement/answer explicitly rhetorical? RE: > Can you blame it on the shell? > No this is not a rhetorical question: you cannot blame the shell for your negligence and incompetence. Asking largely out of confusion, I haven't completely woken up yet
- chasil 4y ago
  How can you check that argv[1] is numeric and greater than zero? I've settled on this, recently, for my own use: i=$((${1} + 0)); [ "$i" -le 0 ] && { echo nope; exit; }
  - znpy 4y ago
    i don't see the problem with what you wrote. also, you could use expr if you don't like fiddling with parentheses when dealing with numeric values.
    - chasil 4y ago
      A fork() will be imposed with expr; using the shell's native math is faster. In any case, I have never seen "best practices" for (numeric) type coercion and confirmation. Everybody seems to come up with something on their own.
      - znpy 4y ago
        If the overhead of a fork is too much for you you probably shouldn’t be writing a shell script anyway, so?
coldtea 4y ago
>By this I mean that they seem to actively resist the idea that they need to learn the language to get the most out of it. That wouldn't be the case if shell was actively resistant to learning it. First due to the varieties (different shells, POSIX vs additions, GNU vs UNIX/BSD versions, etc). Second due to the huge non-consistency between app flags, switches, and other aspects, which are essential part of the shell experience. The whole set (POSIX command line userland + shell) was never designed as a coherent whole. Third due to incredibly bad design of certain shell language aspects and historical cruft, that could very well be improved upon, but will never for compatibility reasons (or will, but in a new shell). Add to that, that unlike "any other programming language" shell is mostly useful for quick one-offs and any of the myriads of syntax/gotcha/option you've learned from writing a quick command line or script you'll need to lookup time and again (because you've forgotten it after weeks or months later when you need it again).
- getoffmyyawn 4y ago
  From what I can see, everything you listed applies to most long lived programming languages. I can certainly say the same things about javascript and python. Your last paragraph really highlights it. Why take this approach? When I need to use something time and again, I take a little time to write a clean, clear, and robust shell script to automate that task. Instead of the one liner in my history, it will usually be around 20 lines in a file. A shell script should be written in a verbose manner and include in comments whatever notes and links to docs are useful. A good one will serve you well for a long time. I have some I'm still using unmodified for >20 years.
  - coldtea 4y ago
    >From what I can see, everything you listed applies to most long lived programming languages. Yes, in the sense that both St. Louis, MI and Tokyo, Japan can be said to have crime.
    - deleted 4y ago
      [deleted]
oh_sigh 4y ago
I think that viewpoint comes from how shell scripts are used - many people write shell scripts, but once they reach a certain level of complexity, they reach for python or perl or some other "more real" language. So, people don't want to create complex shell scripts. So they don't put in the time to learn the complexities of the language.
getoffmyyawn 4y ago
For those looking for learning resources, here are some I've found useful: The Advanced Bash Scripting Guide[0] Classic Shell Scripting[1] Google Shell Style Guide[2] Wicked Cool Shell Scripts[3] GNU Bash Manual[4] 0. https://tldp.org/LDP/abs/html/ https://tldp.org/LDP/abs/html/ 1. https://www.goodreads.com/book/show/299533.Classic_Shell_Scripting https://www.goodreads.com/book/show/299533.Classic_Shell_Scr... 2. https://google.github.io/styleguide/shellguide.html https://google.github.io/styleguide/shellguide.html 3. https://nostarch.com/wcss2 https://nostarch.com/wcss2 4. https://www.gnu.org/software/bash/manual/ https://www.gnu.org/software/bash/manual/

teknopaul 4y ago

This is so true. Torvalds wrote git in shell. You can do great things. Provided... a) agreed, RTFM, (man bash is not long) b) you dont try to do things shells shouldn't do. c) for things you shouldn't do, you write a clean stable cli interface in some other language. Often you can do c) in bash temporarily and port only if/when it gets complicated. (That's what git did)

masklinn 4y ago
> Torvalds wrote git in shell. Did he? Because that’s definitely not what the initial revision of git that’s in the repo shows. It has 1kloc of C, a readme, and a makefile. I struggle to think how you’d even get to the wonky tree format if you used the shell as your implementation language. The first shell scripts landed a few weeks later, in 839a7a06f35bf8cd563a41d6db97f453ab108129 and were convenience wrappers to facilitate interacting with contributors (a wrapper around `merge` to merge a single file, a trivial wrapper around `fsck-cache` to prune, and finally a somewhat more involved original `git pull` which used `rsync` to retrieve all of the remote’s object store then would merge its HEAD with yours). The next two scripts were literally called “example scripts” in their commit messages, and respectively for creating a signed tag and applying a patch.
- rbonvall 4y ago
  If I remember correctly, git was originally distributed as several small programs like git-checkout, git-reset, etc., some written in C and others in shell.
- rascul 4y ago
  The first commit of the git command that I found was a shell script. It called other scripts which used the C stuff behind the scenes. https://git.kernel.org/pub/scm/git/git.git/commit/git?id=e764b8e8b3c50b131be825532ba26fa346d6586e https://git.kernel.org/pub/scm/git/git.git/commit/git?id=e76...
  - masklinn 4y ago
    That’s not “git was originally written in shell” by any honest interpretation of the term.
    - theamk 4y ago
      here is "git commit" in shell: https://git.kernel.org/pub/scm/git/git.git/tree/git-commit-script?id=e764b8e8b3c50b131be825532ba26fa346d6586e https://git.kernel.org/pub/scm/git/git.git/tree/git-commit-s... here is "git status", with all its formatting glory: https://git.kernel.org/pub/scm/git/git.git/tree/git-status-script?id=e764b8e8b3c50b131be825532ba26fa346d6586e https://git.kernel.org/pub/scm/git/git.git/tree/git-status-s... but yes, I would not call it "git was written in shell". But some of its significant pieces were.
    - rascul 4y ago
      It's pretty much the only interpretation I can come up with that results in git being written in shell. I do think it's a bit of a stretch though.
  - tylersmith 4y ago
    And in only 3 lines of code!
H4ZB7 4y ago
> man bash is not long it's literally longer than reading the Go spec and even after reading it several times i still can't code bash

cb321 4y ago

I think what you may be missing is the article's first point: shells are also a high-frequency REPL. The programming language is mostly popular because you can capture/record interactively/incrementally developed logic into "a script" (from typescript(1), script(1), command history, or whatever). What leads people to think they understand the language without detailed study is their many high freq success cases of running commands. Success repetition breeds (over-)confidence. An alternate system design would be a "translator" or "translation assistant" from the high-frequency REPL/interactive/incremental domain into a more static domain (probably with human edits to correct automated mistakes). Part of what even this design would miss (and is also not elaborated upon in the article) is that the nature of interactive/incremental work is very much that of "prototyping". Prototyping often involves many simplifying assumptions, not just about newlines in filenames (a bug in the article's xargs usage) or permissions, but really a constellation of them -- whatever simplifies things. The biggest problems with scripts as a final product are more A) identifying and B) walking back all the various simplifying assumptions of the prototype.

throwawaaarrgh 4y ago

Every complaint on HN about Bash is from one of two people: 1) someone who has never read the man page, 2) someone who wishes Bash were Python. Bash is my favorite programming language, because it's not really a programming language. Rather than being some academic philosophical high-minded solution, it was written by systems practitioners for backwards compatibility and usability. Once you learn it, it replaces 85% of other languages for systems work.

AtlasBarfed 4y ago
Yes whitespace sensitivity, magic characters (WITH whitespace sensitivity), no standard library, no unicode, horrible math expressions, awful string, mishmash of acceptable good practices and outmoded stuff, printlns to debug, subpar error messages, no particularly good IDE, dynamic typing but really none of the convenience. I honestly have no idea how anyone got anything done before Stackoverflow with bash. It's all copy paste and half the things you need to do in modern scripting isn't in typical UNIX command set (curl, etc) and there's no dependency resolution mechanism. But yeah, read the manpage to fix everything. Sure dude. The article is right though. Command line and pipes is a flow type model, but programming is a different ordered thing in the natural sense: Command flow: cat file | grep values | sort values Programming flow: sort(grep(cat(file))
martopix 4y ago
I am admittedly someone who wishes Bash were Python (ish). What's wrong with that though? Of course I want the REPL features of Bash, its terseness, its builtin commands for all sorts of OS and filesystem operations. But is it wrong to ask for a more modern syntax on the language side of things (i.e. the for loop for example)?
- graymatters 4y ago
  Ah, that’s kind of like refugees fleeing all kinds of failing states to better places, that have rule or law, democracy, human rights, prosperity and then actively trying to install the norms of the old country in the new place they fled to.
- deleted 4y ago

efxhoy 4y ago

Do you have books to recommend about shell scripting?

latexr 4y ago
Not a book (and I’m also not the person you asked), but take a look at https://mywiki.wooledge.org/BashGuide https://mywiki.wooledge.org/BashGuide
woodrowbarlow 4y ago
for minutes spent versus knowledge gained, the best bang for buck in my experience is google's shell style guide. i don't agree with all of their rules, but their explanations for each give a ton of insight about the language. https://google.github.io/styleguide/shellguide.html https://google.github.io/styleguide/shellguide.html
chasil 4y ago
If you search for "POSIX shell," you will arrive here: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V... https://pubs.opengroup.org/onlinepubs/9699919799/utilities/ https://pubs.opengroup.org/onlinepubs/9699919799/utilities/

znpy 4y ago

I agree, but would like to add that people seem to forget all of the decency when writing shell scripts. They don’t do input checking and validation, they don’t check for errors, return codes and stuff… No wonder their scripts fail or do something completely unexpected. Can you blame it on the shell? No this is not a rhetorical question: you cannot blame the shell for your negligence and incompetence.

bravetraveler 4y ago
Not trying to be a pedant, I tend to agree, but isn't posing the question to give the statement/answer explicitly rhetorical? RE: > Can you blame it on the shell? > No this is not a rhetorical question: you cannot blame the shell for your negligence and incompetence. Asking largely out of confusion, I haven't completely woken up yet
chasil 4y ago
How can you check that argv[1] is numeric and greater than zero? I've settled on this, recently, for my own use: i=$((${1} + 0)); [ "$i" -le 0 ] && { echo nope; exit; }
- znpy 4y ago
  i don't see the problem with what you wrote. also, you could use expr if you don't like fiddling with parentheses when dealing with numeric values.
  - chasil 4y ago
    A fork() will be imposed with expr; using the shell's native math is faster. In any case, I have never seen "best practices" for (numeric) type coercion and confirmation. Everybody seems to come up with something on their own.
    - znpy 4y ago
      If the overhead of a fork is too much for you you probably shouldn’t be writing a shell script anyway, so?

coldtea 4y ago

>By this I mean that they seem to actively resist the idea that they need to learn the language to get the most out of it. That wouldn't be the case if shell was actively resistant to learning it. First due to the varieties (different shells, POSIX vs additions, GNU vs UNIX/BSD versions, etc). Second due to the huge non-consistency between app flags, switches, and other aspects, which are essential part of the shell experience. The whole set (POSIX command line userland + shell) was never designed as a coherent whole. Third due to incredibly bad design of certain shell language aspects and historical cruft, that could very well be improved upon, but will never for compatibility reasons (or will, but in a new shell). Add to that, that unlike "any other programming language" shell is mostly useful for quick one-offs and any of the myriads of syntax/gotcha/option you've learned from writing a quick command line or script you'll need to lookup time and again (because you've forgotten it after weeks or months later when you need it again).

getoffmyyawn 4y ago
From what I can see, everything you listed applies to most long lived programming languages. I can certainly say the same things about javascript and python. Your last paragraph really highlights it. Why take this approach? When I need to use something time and again, I take a little time to write a clean, clear, and robust shell script to automate that task. Instead of the one liner in my history, it will usually be around 20 lines in a file. A shell script should be written in a verbose manner and include in comments whatever notes and links to docs are useful. A good one will serve you well for a long time. I have some I'm still using unmodified for >20 years.
- coldtea 4y ago
  >From what I can see, everything you listed applies to most long lived programming languages. Yes, in the sense that both St. Louis, MI and Tokyo, Japan can be said to have crime.
  - deleted 4y ago
    [deleted]

oh_sigh 4y ago

I think that viewpoint comes from how shell scripts are used - many people write shell scripts, but once they reach a certain level of complexity, they reach for python or perl or some other "more real" language. So, people don't want to create complex shell scripts. So they don't put in the time to learn the complexities of the language.

getoffmyyawn 4y ago

For those looking for learning resources, here are some I've found useful: The Advanced Bash Scripting Guide[0] Classic Shell Scripting[1] Google Shell Style Guide[2] Wicked Cool Shell Scripts[3] GNU Bash Manual[4] 0. https://tldp.org/LDP/abs/html/ https://tldp.org/LDP/abs/html/ 1. https://www.goodreads.com/book/show/299533.Classic_Shell_Scripting https://www.goodreads.com/book/show/299533.Classic_Shell_Scr... 2. https://google.github.io/styleguide/shellguide.html https://google.github.io/styleguide/shellguide.html 3. https://nostarch.com/wcss2 https://nostarch.com/wcss2 4. https://www.gnu.org/software/bash/manual/ https://www.gnu.org/software/bash/manual/