3 ms·

Is there a greater principle behind all of these pitfalls, a mental model that I can learn and use to avoid these kinds of mistakes? Or is it sort of like CSS w

Is there a greater principle behind all of these pitfalls, a mental model that I can learn and use to avoid these kinds of mistakes? Or is it sort of like CSS where years and years of ad-hoc development make it an exercise in memorizing edge-cases?

Spivak 6y ago
Some off the top of my head. * On Linux file and directory names can be anything except NUL. Because of how much software already assumes they’re UTF-8 you can probably safely assume it too and not run into much trouble. But since bash basically just does some text substitutions and then calls eval you have to be defensive. * Know how bash parses front and back. Almost all the pitfalls are unexpected consequences of the parser not behaving how you expect. https://web.archive.org/web/20200124085320/http://mywiki.wooledge.org/BashParser https://web.archive.org/web/20200124085320/http://mywiki.woo...
- mfontani 6y ago
  IIRC directory (and file) names can contain anything but slash and null, not just null.
- bewuethr 6y ago
  Pretty much this, and as a subset of it: quoting, word splitting, the IFS variable and friends. Reading the BashGuide [1] cover to cover is an excellent base, I find. [1]: https://mywiki.wooledge.org/BashGuide https://mywiki.wooledge.org/BashGuide
- doctoboggan 6y ago
  Do you know if the `--` trick mentioned in the article to stop option scanning works with all bash programs? I thought parsing was up to the program itself. Is it just a widely used convention?
  - scbrg 6y ago
    You are correct, option parsing is up to each program. The double dash is just a convention.
  - joombaga 6y ago
    Yes, it's just a widely used convention, and a POSIX guideline. 12.2 Utility Syntax Guideline #10 https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1...
  - JdeBP 6y ago
    Talking of mental models: These are not "bash programs". Most of the ones where you need this mechanism have nothing to do with the Bourne Again shell, and are just generally-usable utility programs, usable even by things which aren't shells at all. They are, generally, programs that happen to use library mechanisms such as (but not limited to) getopt() or the popt library to parse their arguments.
ufo 6y ago
I think it is different than CSS here. With CSS I could never really understand how it worked and each new browser released introduced new concepts that added more and more corner cases to the mix. On the other hand, with shell scripting after a while I got a decent mental model of how things worked. It is just that there are some bad design choices that make it very easy to shoot yourself on the foot. For example, word splitting "feature" of POSIX shell is the source of many of the pitfalls in the list and the solution in those cases is always the same -- always quote your variables. By the way, check out Shellcheck if you haven't already. It can detect many of these common pitfalls and the error messages have a link to the explanation so it also works as a learning tool.
chubot 6y ago
The most important thing for understanding shell (without loads of special cases) is understanding the Unix kernel, as well as the kernel interface that the shell uses (i.e. which syscalls). If you strace the shell, that goes a long way. If you haven't used strace or don't know what it is, I recommend learning. These comics give more details, and so do Julia Evan's other comics (I think the first one was about strace): Three Comics For Understanding Unix Shell http://www.oilshell.org/blog/2020/04/comics.html http://www.oilshell.org/blog/2020/04/comics.html For example this will help you understand that filenames and argv arrays are byte strings in Unix, which may be different than the notion of strings in your programming language. ----- That will go a long way. However there is still a ton of accidental and needless complexity in bash, particularly related to string processing, and which is unrelated to the kernel. Hence the Oil project: http://www.oilshell.org/blog/2020/01/simplest-explanation.html http://www.oilshell.org/blog/2020/01/simplest-explanation.ht...

Spivak 6y ago

Some off the top of my head. * On Linux file and directory names can be anything except NUL. Because of how much software already assumes they’re UTF-8 you can probably safely assume it too and not run into much trouble. But since bash basically just does some text substitutions and then calls eval you have to be defensive. * Know how bash parses front and back. Almost all the pitfalls are unexpected consequences of the parser not behaving how you expect. https://web.archive.org/web/20200124085320/http://mywiki.wooledge.org/BashParser https://web.archive.org/web/20200124085320/http://mywiki.woo...

mfontani 6y ago
IIRC directory (and file) names can contain anything but slash and null, not just null.
bewuethr 6y ago
Pretty much this, and as a subset of it: quoting, word splitting, the IFS variable and friends. Reading the BashGuide [1] cover to cover is an excellent base, I find. [1]: https://mywiki.wooledge.org/BashGuide https://mywiki.wooledge.org/BashGuide
doctoboggan 6y ago
Do you know if the `--` trick mentioned in the article to stop option scanning works with all bash programs? I thought parsing was up to the program itself. Is it just a widely used convention?
- scbrg 6y ago
  You are correct, option parsing is up to each program. The double dash is just a convention.
- joombaga 6y ago
  Yes, it's just a widely used convention, and a POSIX guideline. 12.2 Utility Syntax Guideline #10 https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1...
- JdeBP 6y ago
  Talking of mental models: These are not "bash programs". Most of the ones where you need this mechanism have nothing to do with the Bourne Again shell, and are just generally-usable utility programs, usable even by things which aren't shells at all. They are, generally, programs that happen to use library mechanisms such as (but not limited to) getopt() or the popt library to parse their arguments.

ufo 6y ago

I think it is different than CSS here. With CSS I could never really understand how it worked and each new browser released introduced new concepts that added more and more corner cases to the mix. On the other hand, with shell scripting after a while I got a decent mental model of how things worked. It is just that there are some bad design choices that make it very easy to shoot yourself on the foot. For example, word splitting "feature" of POSIX shell is the source of many of the pitfalls in the list and the solution in those cases is always the same -- always quote your variables. By the way, check out Shellcheck if you haven't already. It can detect many of these common pitfalls and the error messages have a link to the explanation so it also works as a learning tool.

chubot 6y ago

The most important thing for understanding shell (without loads of special cases) is understanding the Unix kernel, as well as the kernel interface that the shell uses (i.e. which syscalls). If you strace the shell, that goes a long way. If you haven't used strace or don't know what it is, I recommend learning. These comics give more details, and so do Julia Evan's other comics (I think the first one was about strace): Three Comics For Understanding Unix Shell http://www.oilshell.org/blog/2020/04/comics.html http://www.oilshell.org/blog/2020/04/comics.html For example this will help you understand that filenames and argv arrays are byte strings in Unix, which may be different than the notion of strings in your programming language. ----- That will go a long way. However there is still a ton of accidental and needless complexity in bash, particularly related to string processing, and which is unrelated to the kernel. Hence the Oil project: http://www.oilshell.org/blog/2020/01/simplest-explanation.html http://www.oilshell.org/blog/2020/01/simplest-explanation.ht...