3 ms·
There are two big problems with the use of `echo $s` in bash/POSIX sh: 1. Never use echo to output untrusted content as the first argument Let's say `s='-e 1\
by dundarious 4y ago
There are two big problems with the use of `echo $s` in bash/POSIX sh:
1. Never use echo to output untrusted content as the first argument
Let's say `s='-e 1\n2'`, then `echo $s` will output:
> 1
> 2
Instead of:
> -e 1\n2
Always use printf if you want to start output with untrusted content, e.g., `printf %s\\n "$s"`.
2. Never use unquoted variable expansion when trying to exactly reproduce contents of the variable
Similarly, unquoted variable expansion re-tokenizes the contents and will not preserve spaces appropriately. Say `s='"a<space><space>b"'` (where each <space> is a literal ' ', HN seems to be collapsing 2 spaces down to 1), then `echo $s` will output:
> "a<space>b"
Instead of:
> "a<space><space>b"
You can get the latter with `echo "$s"` but use `printf %s\\n "$s"` to fix both issues.
PS: If you fail to use quoted expansion with printf, for example like so, `printf %s\\n $s`, then you'll notice the problem right away, as it will effectively turn that into `for i in $s ; do printf %s\\n "$i" ; done`. That's actually a very useful feature of printf if you know to use it.
Edit: These problems exist for bash/POSIX sh at least. Perhaps you're using a shell that works differently, like zsh, because otherwise issue 2 would probably have led to some checksum fails for you already.
- gavinuhma 4y agoThis is awesome. Thank you! I've been through so many iterations but it's been fun to improve
- gavinuhma 4y agoLike this? https://github.com/gavinuhma/checksum.sh/pull/2 https://github.com/gavinuhma/checksum.sh/pull/2
- dundarious 4y agoMissed the other `echo $s` piped into shasum. But I echo the sentiment of the another commenter that I'd rather rely on `shasum --check` to give the OK or not.
- gavinuhma 4y agoGot it. Thanks. Re --check, I suppose the way to do that would be to download the file to disk, which --check requires as fair as I can tell. So I could download the file to disk, --check, and then remove it. I think most of these installs scripts are trying not to leave any artifacts around from install, other than the resulting binary.
- dundarious 4y agoYou only need to create a temp file for the checksum file, not the downloaded contents. In the below example, no file exists on disk with the contents of `$s`. > $ s='1<space><space>2' > $ printf %s\\n "$s" | shasum -a 256 > tmp.sum > $ printf %s\\n "$s" | shasum --check tmp.sum > -: OK So you can just `printf '%s<space><space>-\n' "$c" > tmp.sum` and check with `printf %s\\n "$s" | shasum --check --status tmp.sum || { echo "checksum failed" > &2 ; exit 1 ; }` Having to create temp files is a wrinkle (could probably avoid it by using process substitution if you want to give up on POSIX sh), but so is writing bash scripts in general.
- gavinuhma 4y agoSolid! I couldn’t figure this out which I why I stopped using “—-check”. I’ll take a look
- gavinuhma 4y agoPr for that https://github.com/gavinuhma/checksum.sh/pull/4 https://github.com/gavinuhma/checksum.sh/pull/4 I ended up trying with process substitution so no tmp file. It works. Trying to decide if it’s more difficult to read
- googlryas 4y agoGreat post, you are wise in the ways of the shell. Minutiae like this is exactly why I stop writing shell scripts the moment I start, and reach for python or some other sane language. But, I can't help but respect when I see masters of sh work their magic.
- dundarious 4y agoHonestly, 90% of problems with scripts are people forgetting to put double quotes around stuff. The other stuff doesn't come up that much, and once you write a few decent scripts, the other stuff is as easy as noticing someone wrote `open = True` in Python, not realizing they've redefined a builtin function, and the fix is just do `is_open = True`. So just put double quotes around all your variable expansions unless you know you shouldn't -- 90% of scripts would be "fixed" with just that. And don't bother putting curly braces into the variable expansion unless you know you need to. People tend to think `echo ${s}` is somehow better than `echo $s` when it's exactly the same -- the curly braces are just a way to allow you to, e.g., write `"${s}_"` as distinct from `"${s_}"`. AFAIK in fish `${s}` is identical to `"$s"`, but that's a different kettle of sh.
- xelxebar 4y agoTry enabling shellcheck linting in your editor! It would immediately warn about unquoted variable expansions and the like. For the vast majority of "gotchas" shellcheck will prod you in the right direction. That said, to write good shell scripts requires actually learning the shell paradigm, instead of just trying to write "Python with Shell syntax".
- wolletd 4y agoI think there's place for both. When I build software installers for our embedded systems, the "frontend" tools production uses are written in a "sane" language, but the actual installation of software and configuration of the systems is done in shell. Because for running a bunch of apt/tar/cp/ln/sed commands to configure a Linux machine, it is the sane language.
- rnhmjoj 4y agoFor more caveats like this one I recommend reading: https://www.etalabs.net/sh_tricks.html https://www.etalabs.net/sh_tricks.html
- yjftsjthsd-h 4y agoIf I may pile on with a general suggestion for people writing shell scripts: Use shellcheck. Always. It will catch these things automatically for you:)