10 ms·

Stronger Shell

105 points by mattbowen 11y ago

bryanlarsen 11y ago
How many people learned shell scripting while engaged in morally ambiguous activities? For me it was scraping for porn ~25 years ago. I imagine lots of people learned scripting while "penetration testing".
enibundo 11y ago
actually [ is same as test and the last ] is just the last (useless) argument of [, iirc
- leni536 11y ago
  Actually you most probably have /bin/[ on your filesystem as a standalone 'test' executable or a link to 'test'. [ is a shell builtin in bash though.
wereHamster 11y ago
> You must have spaces inside your test expressions (so [[-z $FOO]] won't work; [[ -z $FOO ]] is correct). That's because '[[' is a special bash extension which started out as a better '[' which is actually a binary (/usr/bin/[). And, well, you have to have a space between an executable and its first argument. (nowadays '[' and '[[' are builtins in most shells, but the external binary still exists).
- vodik 11y ago
  The core difference is that: - `[` is treated as a statement - `[[` is treated as an expression Which, for example, is why you can't use `>` in `[` based expressions. Bash thinks you want to redirect the output of `[` to somewhere else. This restriction is lifted for `[[` and makes for much more natural looking code.
roneesh 11y ago
I'd be remiss if I didn't use the comments to mention the amazing and totally free, "Unix for the Beginning Mage". The author of the article also didn't mention this great resource. It's an amazing book and a short read. It took me about 3 hours to work through completely. www.unixmages.com If you don't know how to use your shell, block yourself off an evening with this book and change how you forever use you computer.
- faitswulff 11y ago
  I, too, was looking for this in his list. It's a really great resource.
realharo 11y ago
I personally can't stand bash for writing scripts (it's fine for typing one-liners into terminal though). This question and its top answer capture the insanity of bash quite nicely http://stackoverflow.com/questions/3601515/how-to-check-if-a-variable-is-set-in-bash http://stackoverflow.com/questions/3601515/how-to-check-if-a... .
- david-given 11y ago
  Is there a better shell? By which I mean more flexible, more consistent, and simpler. (Anyone who says 'zsh' is disqualified.) I've briefly looked at rc, but while it's significantly simpler and more orthogonal than sh it's got it's own weirdnesses; but the 'Design Principles' section here is worth reading: http://plan9.bell-labs.com/sys/doc/rc.html http://plan9.bell-labs.com/sys/doc/rc.html Surely there must be a usable shell wrapped around an actual modern language?
  - traverseda 11y ago
    I like xonsh. http://xonsh.org/ http://xonsh.org/ It has two modes, which can be a bit confusing. A python mode and a bash-ish mode. testdata = $(ls -l) #Takes thes bash command 'ls -l' and saves it to the python variable testdata echo ${testdata.upper()} | grep XLIO $(ls).split("\n") It uses some magic to figure out whether you're in a python context or a bash context, defaulting to python. If you save some stuff to a variable called "ls", you need to del(ls) before you can use ls as a command again.
  - toothbrush 11y ago
    > Anyone who says 'zsh' is disqualified. I'm curious as to why you say that? This is a naive question, i have no skin in the game.
    - david-given 11y ago
      zsh is really powerful and flexible, but it does so by being really, really complicated --- it's basically taking bash and turning all the weirdness and bizarity up to 11. It's an uber-bash, which is very much the opposite of what I'm looking for.
_yy 11y ago
If you're writing shell scripts, you might as well write Ansible playbooks. You can use Ansible playbooks locally without the SSH layer. Bash/sh is a horrible programming language.
- deleted 11y ago
  [deleted]
TazeTSchnitzel 11y ago
You don't need to learn Bash to write shell scripts. Many other languages support executing commands with special syntax (PHP and Perl have backticks for example). I write my scripts in PHP if they're moderately complex, and only use Bash for dumb lists of commands.
crimsonalucard 11y ago
interviewers tend not to test for shell skill even though your shell skills are usually more relevant to daily tasks. I'm debating whether to improve on my shell skills or algorithm skills.
- gtk40 11y ago
  My interview for the job I have now actually had me do some hands-on shell activities and had very little algorithms, but that was partly because they were wanting me to work with code deployment and other tasks where it would come in handy.
alpsgolden 11y ago
I've found that as soon as bash script goes to more than a couple lines, or as soon as it needs anything modestly complex, like "if" statements or functions, then it is almost always more efficient to write it in ipython. If you know python, then ipython is really superior way to do any ops and administration tasks. I've also found it easier to use for server setup than alternatives like ansible or puppet. I can just use standard, reusable classes and functions to do all my various activities.
- falcolas 11y ago
  Perhaps it's just me, but flow control statements and functions in Bash have yet to scare me off. It's still faster, and readable, to write even moderately complex scripts in bash. This is doubly true if you have to chain together external scripts - the "easy" ways to do it in Python can have some very major limitations. I liken Bash scripting to Perl scripting. You can write safe, beautiful, and readable code in both. However, you can also create a summoning circle to the 5th circle of hell if you don't spend the effort to make it safe, beautiful and readable.
  - alpsgolden 11y ago
    Part of this was that I became an expert in python before learning much bash, so it was just much easier to write in the language I already knew, and I found the bash syntax strange with a bunch of gotchas. I'm sure though, that if I did a ton of bash scripting I would not even notice the gotchas. You are correct that the main weakness of ipython is that it doesn't have pipes, and that chaining is very limited. But usually this does not impact me. And if all your scripting is in python, then you just import functions and run them, rather than chaining scripts together.
danharaj 11y ago
Confession: The horrors of bash set back my programming hobby until high school. Even then, I didn't actually like interacting with my computer programmatically until I found a comfortable set of tools sometime in college. I'm lucky that my day job lets me use the tools I like and avoid systems like the bash shell.
- ParadigmBlender 11y ago
  Interestingly my experience has been opposite. My first programming adventure was automating some file validation using bash. I agree though that the silly and inconsistent syntax made me really appreciate python when I started learning it.
heydonovan 11y ago
Just about everything I've learned about bash has been from the #bash IRC channel on Freenode. You'll see the same repeated warnings of not learning from the public web, due to the fact that misinformation spreads like a wild fire, and some articles out there are just flat out wrong (sort of like w3schools in the #css circle). There is just one thing I will never understand. The tired argument of "...but it isn't portable". Features that are new, and make programming bash easier (such as "[" vs "[["), are looked down upon in some circles because it isn't portable. If I'm programming for bash, then upon deployment, I'll be using bash. To use another shell, and hope it just works is a bit insane. I'd also like to second using the Fish shell. Far better than bash/zsh for everyday use, and I just can't go back to other shells.
- jlgaddis 11y ago
  Depends what you're going for, I guess. If it's something you're developing for public use and/or something you want to run on !Linux, portability might be a concern (FreeBSD, for example, doesn't ship with bash and, if you install it, it lands in /usr/local/bin instead of /bin).
- dmd 11y ago
  > upon deployment, I'll be using bash Unless you're on Ubuntu, in which case you will be using /bin/dash at occasionally entirely unexpected moments.
  - lfowles 11y ago
    For the want of a shebang, the core was dumped.
- StavrosK 11y ago
  Seconded. I can't live without fish, and I hear this "but what about my bash scripts?!" argument a lot. Unless you're actually sourcing the script, it has a magic "#!/bin/bash" line on top that makes it work correctly!
  - baghira 11y ago
chriswarbo 11y ago
Was worth reading if only to find out about ShellCheck. My current project has a lot of bash. The program itself is just `foo | bar | baz | ...`, but the associated test script has grown quite long. I've just added an extra test which calls shellcheck on each script, and it's spotted a bunch of redundant code for me :)
anh79 11y ago
Bash is good as a system shell. For interactive shell, zsh/fish/... wins. That's why I only learn Bash. One lesson for everything for both purposes ;)
- qznc 11y ago
  fish would be better than bash as a system shell as well. More consistent, for example. However, bash is available and usually preinstalled everywhere, in contrast to fish.
  - anh79 11y ago
    I agree. But "fish" is quite strange to me, especially its way to recall history. In "Bash", I can type "^ R" and browser the history very fast. In "fish", I have to type and select with up/down array; it's not easy to browser the history randomly (FIXME). My #Bash" history has > 96k entries (woh, believe it or not; because of this too big number, "xterm" + "screen" always stuck when exitting; but "urxvt" + "tmux" work perfectly thanks to "urxvt" daemon mode.) Porting this huge history database from my daily "bash" to "fish" is just a nightmare ... :D
    - anh79 11y ago
      (Talking about interactive "fish" shell; but that's the point: I need to be familiar with its interative mode first.)
    - ridiculous_fish 11y ago
      Actually fish will import your ~/.bash_history automatically!
- setpatchaddress 11y ago
  Programmer time is better spent learning almost any language other than bash. My god, the centuries wasted on minutiae like the stupid "[ is a program, that's why you have to put spaces around it." And system administrators should use Ruby if given a choice.
tachion 11y ago
What suprises me is how often Bash is being used as 'the shell' and software projects use it, even when they could simply call /bin/sh instead. This is particularly bad for people who port software to other platforms (where BSD systems dont come with Bash, for example) and have to deal with pure shell scripts calling Bash. One recent example is CoreOS/etcd that's currently dropping Bash in favor of Sh, because they simply didnt need it. Bash is a shell, but Bash is not the shell!
- philh 11y ago
  It may be better for those people to call /bin/bash, and fail early and obviously for anyone who doesn't have that; than to call /bin/sh, attempt to write portably without really knowing the quirks of this variant of the language, and possibly fail subtly in obscure situations.
- lisivka 11y ago
  Bash is just yet another dependency for a script. Perl script needs perl. Ruby script needs ruby. Python script needs python. Bash script needs bash. What wrong with that?
estrabd 11y ago
I generally avoid aliases. I prefer to rote learn complex commands; if it's fitting for an actual shell script then I'll just bang one of those little guys out in no time.
deleted 11y ago
[deleted]
IshKebab 11y ago
Please don't write shell code that you ever intend to distribute to anyone. The future will thank you.
CJefferson 11y ago
One this which greatly helped me was understanding that [ is a program, like any other, which takes arguments. It's a small thing, but made the shall less magic.
jchomali 11y ago
Nice! Thanks for sharing!

105 points by mattbowen 11y ago

How many people learned shell scripting while engaged in morally ambiguous activities? For me it was scraping for porn ~25 years ago. I imagine lots of people learned scripting while "penetration testing".

enibundo 11y ago

actually [ is same as test and the last ] is just the last (useless) argument of [, iirc

leni536 11y ago
Actually you most probably have /bin/[ on your filesystem as a standalone 'test' executable or a link to 'test'. [ is a shell builtin in bash though.

wereHamster 11y ago

> You must have spaces inside your test expressions (so [[-z $FOO]] won't work; [[ -z $FOO ]] is correct). That's because '[[' is a special bash extension which started out as a better '[' which is actually a binary (/usr/bin/[). And, well, you have to have a space between an executable and its first argument. (nowadays '[' and '[[' are builtins in most shells, but the external binary still exists).

vodik 11y ago
The core difference is that: - `[` is treated as a statement - `[[` is treated as an expression Which, for example, is why you can't use `>` in `[` based expressions. Bash thinks you want to redirect the output of `[` to somewhere else. This restriction is lifted for `[[` and makes for much more natural looking code.

roneesh 11y ago

I'd be remiss if I didn't use the comments to mention the amazing and totally free, "Unix for the Beginning Mage". The author of the article also didn't mention this great resource. It's an amazing book and a short read. It took me about 3 hours to work through completely. www.unixmages.com If you don't know how to use your shell, block yourself off an evening with this book and change how you forever use you computer.

faitswulff 11y ago
I, too, was looking for this in his list. It's a really great resource.

realharo 11y ago

I personally can't stand bash for writing scripts (it's fine for typing one-liners into terminal though). This question and its top answer capture the insanity of bash quite nicely http://stackoverflow.com/questions/3601515/how-to-check-if-a-variable-is-set-in-bash http://stackoverflow.com/questions/3601515/how-to-check-if-a... .

david-given 11y ago
Is there a better shell? By which I mean more flexible, more consistent, and simpler. (Anyone who says 'zsh' is disqualified.) I've briefly looked at rc, but while it's significantly simpler and more orthogonal than sh it's got it's own weirdnesses; but the 'Design Principles' section here is worth reading: http://plan9.bell-labs.com/sys/doc/rc.html http://plan9.bell-labs.com/sys/doc/rc.html Surely there must be a usable shell wrapped around an actual modern language?
- traverseda 11y ago
  I like xonsh. http://xonsh.org/ http://xonsh.org/ It has two modes, which can be a bit confusing. A python mode and a bash-ish mode. testdata = $(ls -l) #Takes thes bash command 'ls -l' and saves it to the python variable testdata echo ${testdata.upper()} | grep XLIO $(ls).split("\n") It uses some magic to figure out whether you're in a python context or a bash context, defaulting to python. If you save some stuff to a variable called "ls", you need to del(ls) before you can use ls as a command again.
- toothbrush 11y ago
  > Anyone who says 'zsh' is disqualified. I'm curious as to why you say that? This is a naive question, i have no skin in the game.
  - david-given 11y ago
    zsh is really powerful and flexible, but it does so by being really, really complicated --- it's basically taking bash and turning all the weirdness and bizarity up to 11. It's an uber-bash, which is very much the opposite of what I'm looking for.

_yy 11y ago

If you're writing shell scripts, you might as well write Ansible playbooks. You can use Ansible playbooks locally without the SSH layer. Bash/sh is a horrible programming language.

deleted 11y ago
[deleted]

TazeTSchnitzel 11y ago

You don't need to learn Bash to write shell scripts. Many other languages support executing commands with special syntax (PHP and Perl have backticks for example). I write my scripts in PHP if they're moderately complex, and only use Bash for dumb lists of commands.

crimsonalucard 11y ago

interviewers tend not to test for shell skill even though your shell skills are usually more relevant to daily tasks. I'm debating whether to improve on my shell skills or algorithm skills.

gtk40 11y ago
My interview for the job I have now actually had me do some hands-on shell activities and had very little algorithms, but that was partly because they were wanting me to work with code deployment and other tasks where it would come in handy.

alpsgolden 11y ago

I've found that as soon as bash script goes to more than a couple lines, or as soon as it needs anything modestly complex, like "if" statements or functions, then it is almost always more efficient to write it in ipython. If you know python, then ipython is really superior way to do any ops and administration tasks. I've also found it easier to use for server setup than alternatives like ansible or puppet. I can just use standard, reusable classes and functions to do all my various activities.

falcolas 11y ago
Perhaps it's just me, but flow control statements and functions in Bash have yet to scare me off. It's still faster, and readable, to write even moderately complex scripts in bash. This is doubly true if you have to chain together external scripts - the "easy" ways to do it in Python can have some very major limitations. I liken Bash scripting to Perl scripting. You can write safe, beautiful, and readable code in both. However, you can also create a summoning circle to the 5th circle of hell if you don't spend the effort to make it safe, beautiful and readable.
- alpsgolden 11y ago
  Part of this was that I became an expert in python before learning much bash, so it was just much easier to write in the language I already knew, and I found the bash syntax strange with a bunch of gotchas. I'm sure though, that if I did a ton of bash scripting I would not even notice the gotchas. You are correct that the main weakness of ipython is that it doesn't have pipes, and that chaining is very limited. But usually this does not impact me. And if all your scripting is in python, then you just import functions and run them, rather than chaining scripts together.

danharaj 11y ago

Confession: The horrors of bash set back my programming hobby until high school. Even then, I didn't actually like interacting with my computer programmatically until I found a comfortable set of tools sometime in college. I'm lucky that my day job lets me use the tools I like and avoid systems like the bash shell.

ParadigmBlender 11y ago
Interestingly my experience has been opposite. My first programming adventure was automating some file validation using bash. I agree though that the silly and inconsistent syntax made me really appreciate python when I started learning it.

heydonovan 11y ago

Just about everything I've learned about bash has been from the #bash IRC channel on Freenode. You'll see the same repeated warnings of not learning from the public web, due to the fact that misinformation spreads like a wild fire, and some articles out there are just flat out wrong (sort of like w3schools in the #css circle). There is just one thing I will never understand. The tired argument of "...but it isn't portable". Features that are new, and make programming bash easier (such as "[" vs "[["), are looked down upon in some circles because it isn't portable. If I'm programming for bash, then upon deployment, I'll be using bash. To use another shell, and hope it just works is a bit insane. I'd also like to second using the Fish shell. Far better than bash/zsh for everyday use, and I just can't go back to other shells.

jlgaddis 11y ago
Depends what you're going for, I guess. If it's something you're developing for public use and/or something you want to run on !Linux, portability might be a concern (FreeBSD, for example, doesn't ship with bash and, if you install it, it lands in /usr/local/bin instead of /bin).
dmd 11y ago
> upon deployment, I'll be using bash Unless you're on Ubuntu, in which case you will be using /bin/dash at occasionally entirely unexpected moments.
- lfowles 11y ago
  For the want of a shebang, the core was dumped.
StavrosK 11y ago
Seconded. I can't live without fish, and I hear this "but what about my bash scripts?!" argument a lot. Unless you're actually sourcing the script, it has a magic "#!/bin/bash" line on top that makes it work correctly!
- baghira 11y ago

chriswarbo 11y ago

Was worth reading if only to find out about ShellCheck. My current project has a lot of bash. The program itself is just `foo | bar | baz | ...`, but the associated test script has grown quite long. I've just added an extra test which calls shellcheck on each script, and it's spotted a bunch of redundant code for me :)

anh79 11y ago

Bash is good as a system shell. For interactive shell, zsh/fish/... wins. That's why I only learn Bash. One lesson for everything for both purposes ;)

qznc 11y ago
fish would be better than bash as a system shell as well. More consistent, for example. However, bash is available and usually preinstalled everywhere, in contrast to fish.
- anh79 11y ago
  I agree. But "fish" is quite strange to me, especially its way to recall history. In "Bash", I can type "^ R" and browser the history very fast. In "fish", I have to type and select with up/down array; it's not easy to browser the history randomly (FIXME). My #Bash" history has > 96k entries (woh, believe it or not; because of this too big number, "xterm" + "screen" always stuck when exitting; but "urxvt" + "tmux" work perfectly thanks to "urxvt" daemon mode.) Porting this huge history database from my daily "bash" to "fish" is just a nightmare ... :D
  - anh79 11y ago
    (Talking about interactive "fish" shell; but that's the point: I need to be familiar with its interative mode first.)
  - ridiculous_fish 11y ago
    Actually fish will import your ~/.bash_history automatically!
setpatchaddress 11y ago
Programmer time is better spent learning almost any language other than bash. My god, the centuries wasted on minutiae like the stupid "[ is a program, that's why you have to put spaces around it." And system administrators should use Ruby if given a choice.

tachion 11y ago

What suprises me is how often Bash is being used as 'the shell' and software projects use it, even when they could simply call /bin/sh instead. This is particularly bad for people who port software to other platforms (where BSD systems dont come with Bash, for example) and have to deal with pure shell scripts calling Bash. One recent example is CoreOS/etcd that's currently dropping Bash in favor of Sh, because they simply didnt need it. Bash is a shell, but Bash is not the shell!

philh 11y ago
It may be better for those people to call /bin/bash, and fail early and obviously for anyone who doesn't have that; than to call /bin/sh, attempt to write portably without really knowing the quirks of this variant of the language, and possibly fail subtly in obscure situations.
lisivka 11y ago
Bash is just yet another dependency for a script. Perl script needs perl. Ruby script needs ruby. Python script needs python. Bash script needs bash. What wrong with that?

estrabd 11y ago

I generally avoid aliases. I prefer to rote learn complex commands; if it's fitting for an actual shell script then I'll just bang one of those little guys out in no time.

deleted 11y ago

[deleted]

IshKebab 11y ago

Please don't write shell code that you ever intend to distribute to anyone. The future will thank you.

CJefferson 11y ago

One this which greatly helped me was understanding that [ is a program, like any other, which takes arguments. It's a small thing, but made the shall less magic.

jchomali 11y ago

Nice! Thanks for sharing!