10 ms·
Filenames and pathnames in shell: How to do it correctly (2020)
- znxster 3y agoThis is interesting, he also provided commentary on Fixing Unix/Linux/POSIX Filenames[1] [1] https://dwheeler.com/essays/fixing-unix-linux-filenames.html https://dwheeler.com/essays/fixing-unix-linux-filenames.html
- tredre3 3y agoYour link was an interesting read, he does make good points along with clear examples. I'm one of those who believes that linux paths should have more limitations. I also believe they should be case insensitive. Both opinions are highly controversial, unfortunately I guess it comes down to whether you believe paths are there to serve humans or machines. If you believe it should serve humans, then you understand why allowing a - prefix or case sensitivity causes unnecessary problems. On the other hand if you believe that paths are there to serve the machine, then the fewer "arbitrary" limitations the better. The devs know best, after all.
- fbdab103 3y ago>If you believe it should serve humans, then you understand why allowing a - prefix or case sensitivity causes unnecessary problems. Serving humans feels like it should take whatever garbage I shove down its throat and be happy to receive. Similarly, I think many humans would like to differentiate between, "resumefinal.docx" and "resumeFINAL.docx"
- lmm 3y ago> Similarly, I think many humans would like to differentiate between, "resumefinal.docx" and "resumeFINAL.docx" I strongly disagree. Those sound exactly the same. If you had a physical filing cabinet and asked someone to get a document from the final folder (which is supposed to be the human-friendly metaphor we're using here), they would happily open a folder labeled FINAL, and be quite confused if you had distinct final and FINAL folders.
- smackeyacky 3y agoIf they shout the name, you know to look for the upper case version :-)
- eviks 3y agoHumans don't speak binary garbage, so it's a disservice to them to happily swallow/spit it out Cap sensitivity is a much lesser mistake
- coldtea 3y ago>Similarly, I think many humans would like to differentiate between, "resumefinal.docx" and "resumeFINAL.docx" Would they? Both are the "final resume" Word document... Humans don't care about case in distinguishing names. Considering case differences as important distinctions between names (of files in this case) is something instilled from using computers, not a human trait. That's a concern for coders, proof-readers, and other OCD types. Not to mention that the most popular Desktop OSes (Windows, MacOS) don't care for file case either. They preserve it for visual display is you set it, but two filenames with differences in case only are the same file - and can't coexist.
- jackthetab 3y ago> The devs know best, after all. Which may be why they gave us these options to put in our ~/.inputrc ``` set completion-ignore-case on set completion-prefix-display-length 2 set completion-map-case on ``` Technically filenames and paths are still case-sensitive, but with these settings they practically aren't.
- jackthetab 3y agoARGH! How does one do code on this site? I meant: set completion-ignore-case on set completion-prefix-display-length 2 set completion-map-case on
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- js2 3y agoIndent by 4 spaces. like this
- tsimionescu 3y agoYou actually indent by 2 spaces, not 4: Like this 4 spaces is a Markdown thing.
- js2 3y agoMy bad, I knew that at some point. The rules are actually linked to from the well-hidden "help" at the bottom right of the comment box: https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc
- chasil 3y agohttps://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc
- scbrg 3y agoEvery time I see someone argue for case insensitivity I remember this excellent Hacker News comment on the issue: https://news.ycombinator.com/item?id=29755865 https://news.ycombinator.com/item?id=29755865 It's definitely more complex than you'd naively think it is.
- arp242 3y agoPeople are making things more complex than they are. No one said you need to convert between scripts or muck about with fullwidth variants or stuff like that; "semantically identical" is not the same as "case insensitive". That post is going of on a "if you want case-insensitivity then you must also treat color and colour as identical!" tangent, which is just silly and not what anyone has ever argued for. The scripts with case translations that are more complex than a simple 1-to-1 mapping are the exception, not the rule (German, Greek, Turkish, Lithuanian). These can be dealt with. It's certainly not the case that it "will only work for English and a few European languages"; it will work for much of the world. The fact of the matter is, two out of the three most used systems today are case-insensitive, and Linux/POSIX being the exception is rather painful.
- lelanthran 3y ago> It's certainly not the case that it "will only work for English and a few European languages"; it will work for much of the world. I respectfully disagree. Consistently broken in the same way for everyone is better than unreliably working for some people, reliably working for other people, unreliably broken for a third group and reliably broken for a fourth group. Filenames should be created as the user typed it - don't change the input before storing it. Filenames should be displayed as the user typed it - don't render output different to what was input. Tools should match filenames as the user expects it - in this case (hehe) it should perform a case-insensitive match. Ambiguity resolution has to be performed in the case where there is more than one match. cat > MyFile.txt # Create 'MyFile.txt' ls # Display 'MyFile.txt' echo myfile* # Display 'MyFile.txt' vim myfile.txt # Opens 'MyFile.txt' The first problem with this is in automated shell scripts: there is no interactivity so the script can't prompt the user "Which of the following files did you mean to open: [MyFile.txt, myfile.txt]?". This means that a script which relies on 'foo.txt' will fail if someone creates a 'Foo.txt' in the same directory. Another issue with this is that userspace calls to `open()`, `stat()`, etc aren't able to fail and return a list of case-insensitive matches in the case of ambiguity. This makes handling the ambiguity the applications (very complex) problem - before any `open(fname)` call, the application will first have to `scandir()` to get all the filenames, then perform a case-insensitive match against the list to get all the CI matches, then prompt the user to select the correct one. > The fact of the matter is, two out of the three most used systems today are case-insensitive I think it's more that they are case-aware than case-insensitive; after all the filesystem certainly stores the case, and they both retrieve the case correctly. It's the tooling that faces the user which "fix" cases to prevent two files with the same name in different cases being created. You can certainly create file 'FOO' and file 'foo' on Windows in the same directory, and then the tools tend to randomly open only one of them no matter which on the user clicked on.[1] Which is why I say that the actual filesystem is case-sensitive. The user programs normalise the case for the user. [1] EDIT: I accidentally did this once, and had to write another small tool to remove files with the filename as typed because the normal tools (del, explorer.exe) just randomly choose one file to remove.
- dang 3y agoRelated: Filenames and Pathnames in Shell: How to Do It Correctly (2010) - https://news.ycombinator.com/item?id=9021786 https://news.ycombinator.com/item?id=9021786 - Feb 2015 (20 comments) Filenames and Pathnames in Shell: How to Do It Correctly - https://news.ycombinator.com/item?id=8337296 https://news.ycombinator.com/item?id=8337296 - Sept 2014 (2 comments)
- js2 3y agoThis is an excellent resource. I'd like to re-iterate the author's recommendation to use shellcheck: https://www.shellcheck.net/ https://www.shellcheck.net/ I also like using shfmt on top of that. (I run them via pre-commit[1] hooks[2,3].) FWIW, I find writing POSIX-compliant shell rarely necessary. In my 25 years of writing shell scripts, they've nearly always been for automation on a specific OS where portability is not a concern. Arrays are really nice to have. One place I like to use them is for making long commands easier to read. e.g. cmd=( some-command --some-long-option --another-long-option --an-option-with-a-value="$value" ) "${cmd[@]}" This avoids having to use backslashes so you can comment in-between lines or at the end of any line. I also almost always prefer the self-documenting long-option to single-letter switches when writing shell scripts. Those single-letter options are to save you typing on the command-line. There's no reason to use them in a script. When I do write a script for more than one OS (typically macOS and Linux), I have bash on both systems, so I use "#!/usr/bin/env bash" and take care to use only the features that the older version of bash that macOS ships with supports. Where I have to be more careful is in utilities like find, sed, etc which can differ quite a bit between BSD and Linux. 1. https://pre-commit.com/ https://pre-commit.com/ 2. https://github.com/shellcheck-py/shellcheck-py https://github.com/shellcheck-py/shellcheck-py 3. https://github.com/maxwinterstein/shfmt-py https://github.com/maxwinterstein/shfmt-py
- dreamcompiler 3y agoThe author suggests using shellcheck in addition to his recommendations.
- js2 3y agoYou're right! I totally missed that. I've updated my comment.
- dreamcompiler 3y agoI write lots of scripts that need to work on both Linux and MacOS. Although both are POSIX, filenames in MacOS are case-insensitive and often contain spaces. This stuff plays havoc with bash scripts written by naive Linux programmers. Following TFA's recommendations ensures that your scripts will work on both.* * well, except for the fact that MacOS defaults to zsh now.
- themk 3y agoAm I the only one that thinks setting -e is a huge footgun given that it only works in certain contexts? I'd much rather see explicit error handling.
- chungy 3y agoGiven that it makes a script terminate for any unhandled errors, I don't think your goal contradicts with setting -e. Rather, it makes the script author explicitly handle errors or the script dies.
- fbdab103 3y agoHandle errors in bash? More power to you, but if my script cannot just fail in place, then it will be written in Python. Even with shellcheck, I do not trust myself to write more than a handful of lines in bash.
- chungy 3y ago> Handle errors in bash? The || operator does that pretty well.
- js2 3y agoThe biggest issue with "set -e" is that it's inconsistent. It doesn't always cause the shell to exit where you'd expect it to, and in some cases, it causes the shell to exit where you would NOT expect it to. This is the most surprising: set -e i=0 (( i++ )) # shell will exit here Still, I tend to use it because even though it doesn't handle every case, I'd still usually rather my scripts exit when a command fails.
- deleted 3y ago[deleted]
- metadat 3y agoThis would be a significantly better resource if it explained the underlying "why's" inline. In it's current form, it comes off as cargo culting / magic spell recital. For example, why "./*" rather than "*". Not helpful to understand the core principal and reality. Personally I don't tend to fuck about with "*" and instead use `find . -print0 | xargs -0 -n1'. Hasn't bitten me yet, across a few thousand shell scripts (!). Why should I prefer this magic incantation compared to my own battle-tested proven one? And this is why, IMHO, the explanations should be inline.
- dwheeler 3y agoSadly you can't please everyone :-). The explanation is further down on the page. If I wrote in that recommended order, many other developers would complain "I don't want all the details, just tell me what to do". I wrote it this way because some developers just want the tl;dr version, that is, they "just want the answer"
- eviks 3y agoWhat do you think the chances are of removing the bag of bytes filename design bug in another decade?
- dwheeler 3y agoLow. It wouldn't be hard to create an LSM for Linux to reject file creation of certain filenames, though. Then at least sone security-minded people could enable it. The POSIX folks are finally adding formal support for null terminated filenames in filename lists (which is what a lot of people are doing). So that's something.
- semiquaver 3y agoI found it to be quite thorough and lucid in its explanation of the rationales behind each suggestion, including the one you mentioned. Perhaps you are referring to the first section which is introduced as being a brief tl;dr: > Here’s a quick summary about how to do it correctly, for the impatient who “just want the answer”. The full document below that tldr includes for example this explanation of the reason for using ./* over *. Namely, flag injection: > This is important because almost all commands will interpret a string beginning with dash as an option, not as a filename, until they see something that does not begin with dash. Globs are expanded by the shell into a list of filenames, and dash is earlier in the sort order compared to before alphanumerics, so it is easy for attackers to make this happen.
- mmphosis 3y agoOoh, I can put slashes in filenames! touch 2023∕06∕19 Or, maybe not. ls 2023/06/19 /usr/bin/ls: cannot access '2023/06/19': No such file or directory Or, maybe I can. ls 2023∕06∕19 2023∕06∕19
- rascul 3y ago> touch 2023∕06∕19 Won't work if the directory 2023/06 doesn't exist. The filename is 19, the relative path is 2023/06/19.
- mmphosis 3y ago2023∕06∕19 is a filename. 2023/06/19 is a directory. UTF-8 ∕ division slash
- meinheld111 3y agoLol Unescaped non-asci was a mistake
- layer8 3y ago2023∕06∕19 may be a filename, but it’s not a date, it’s an arithmetic expression. ;)
- rascul 3y agoOh geez I missed the utf8.
- coldtea 3y agoOr you quote them? ls "2023/06/19" etc?
- joveian 3y agoA nice collection of issues, although I'd disagree a bit with some of the solutions, particularly around prefixing. You first need to check if you have an absolute path and it doesn't work with input that isn't a path. So for me it seems best to focus on -- as the primary measure and prefixing as an additional measure if possible when needing to be extra careful. Most software uses some getopt variant so it should be quite rare that -- doesn't work. One nice thing about bash is the ${var@Q} syntax that is great for printing stuff and deals with the potential for stuff that affects the terminal in a much nicer way than piping through tr. IMO, it should always be used for pathnames so that you can always just copy and paste the output, which I do quite a bit. I wish it was in POSIX.
- Calzifer 3y ago> One nice thing about bash is the ${var@Q} syntax that is great for printing stuff and deals with the potential for stuff that affects the terminal in a much nicer way than piping through tr. An alternative which works in both Bash and Zsh is printf %q "$var" using /usr/bin/printf %q "$var" has even a good chance to work in other shells. And as fun fact: the Zsh equivalent for Bash's ${var@@} is ${(q)var} and it was not enough to have one quoting style but instead you can also provide qq, qqq, qqqq, q+ or q- all with a different result. https://zsh.sourceforge.io/Doc/Release/Expansion.html#Parameter-Expansion-Flags https://zsh.sourceforge.io/Doc/Release/Expansion.html#Parame...
- emmelaich 3y agoDoes this work as desired? "$(dirname "$file")" Is $file not actually quoted here? Since the quoted bits are "$(dirname " ")"
- two_handfuls 3y agoThis will correctly nest the quotes, yes. It’s somewhat counter-intuitive, but it works.
- gpvos 3y ago$() is a quote, like ``, and the "$file" is entirely within it, so it should work.
- emmelaich 3y agoThanks. I had thought the " quote was parsed / used much earlier in the tokenisation for sh.
- lenkite 3y agoThis is the reason its better to use a "Real Programming Language" leveraging libraries to do this job instead of a shell script. A ludicrous number of things to keep in mind for simple filenames and pathnames.
- atoav 3y agoIMO someone made the wrong trade-offs there. Shell scrips mainly deal with text output of different programs, so escaping and quoting should be as simple as possible.
- meinheld111 3y agoMessing with IFS can also solve a lot of problems
- diegocg 3y agoOr just use a sane shell (eg. the oil shell https://www.oilshell.org/blog/2020/10/osh-features.html#eliminate-quoting-hell-the-qefs-problem https://www.oilshell.org/blog/2020/10/osh-features.html#elim..., fish) where handling pathnames with spaces, etc. is not a problem and doesn't require arcane quoting knowledge or linters. The only reason why this is an issue is that people insists in using bash in the name of backwards compatibility. It's about time people start considering moving away from bash - its design is broken and the efforts and time it takes to write safe bash shells might not be worth it.
- meinheld111 3y agoBrokenness is really a tradeof. Imho using oil/fish/etc trades unfavorable because of network effects but it is perfectly acceptable choice in many circumstances
- planede 3y agoI don't disagree, but only some of the rules are really about the shell itself. A significant part of the problem is the stringly-typed interface of exec. Avoiding passing filenames starting with "-" has little to do with shells.
- water9 3y agoThe colon is always an interesting character, since it works on windows, but not linux
- otikik 3y agoSo many sharp corners to cut yourself with.
- saaspirant 3y agoUnrelated but if you want this page to look modern and responsive, you can use WaterCSS's Bookmarklet: https://watercss.kognise.dev/#bookmarklet https://watercss.kognise.dev/#bookmarklet Example (NOT NSFW , I don't know why imgur shows this warning): https://imgur.com/a/QxJGARN https://imgur.com/a/QxJGARN
- Anthony-G 3y agoThanks for the tip. I appreciate the minimalism of the original article (which is already responsive) but Water.css makes it visually prettier. It even has dark and light themes!
- casey2 3y agoWhy is the fact that filenames can contain newlines and executable shell commands always framed as a problem? You can have files that write to themselves and change their names. This is a massive boon for expressive power of a language. You can have a lot of fun on the command line with a couple files and eval *. "We have persistent "meta"objects, they're called files."
- billpg 3y ago"You can use any characters in filenames, except NUL and slash." "Including LF?" "Yes." "So what does find do when it encounters a file with an LF in its name?" "It outputs the name including the LF." "Without quoting it?" "Right." "But how can something reading the output from find know what's an LF in a filename or an LF separating lines?" "You can use -print0 so it separates files with a NUL character, which filenames can't use." "Wouldn't it be simpler to prohibit control characters from filenames?" "Never!"
- Anthony-G 3y agoYou would think that this would be common-sense. In 2010, David Wheeler (author of the featured article) proposed to the Austin Group that control characters – in particular newline (and TAB and ESC) – be forbidden in filenames: https://www.austingroupbugs.net/view.php?id=251 https://www.austingroupbugs.net/view.php?id=251 Unfortunately, it has yet to be implemented. :( I also note that in 2016 the authors of GNU coreutils took a lot of flak when they changed the default behaviour of ls to quote filenames with special characters¹². Personally, I’d be having stern words with anyone that produced path names with anything funkier than a space character. ¹ https://www.gnu.org/software/coreutils/quotes.html https://www.gnu.org/software/coreutils/quotes.html ² https://unix.stackexchange.com/a/258691 https://unix.stackexchange.com/a/258691
- its-summertime 3y ago> Set IFS to just newline and tab, if you can, to reduce the risk of mishandling filenames with spaces. should be: reduce the risk when mishandling filenames. If your handling of filenames depends on IFS, you are already in big trouble!
- lisper 3y agoI'm sorry, but anything that requires a programmer to memorize this many rules in order to avoid shooting themselves in the foot is deeply and fundamentally flawed and needs to be completely redesigned from the ground up. There is a reason that modern programming languages include actual data strutures and don't just treat everything as a string of bytes. That might have been a good idea in 1973. In 2023, not so much. So... use a shell for interactive work if you must, but never ever write a script for one. Use Python or Ruby or just about anything besides a shell.
- ape4 3y agoI would agree. It would be nice to see a new shell (that isn't PowerShell) that worked well with spaces in file names.
- nerdponx 3y agoZsh works fine with spaces and all other manner of weird whitespace, non-printable binary data, etc. The only unfortunate edge case where are you still need to quote things defensively is if a parameter is completely empty, then the entire word gets removed from the command if the expansion is not quoted. Other shells are fun, but Z Shell already solved all the worst problems of parameter expansion years ago. If anything, you should be migrating your Bash scripts to Zsh even if you never switch your interactive shell. That said, most of the problems here don't have much to do with parameter expansion and have more to do with the fact that text as a universal interface kind of sucks. Nushell and Powershell have better models here, but the latter lacks a convenient way to parse fixed-width tables so it's unfortunately frustrating to use with a lot of traditional Unix tools (there's no "ConvertFrom-Table" cmdlet, only "ConvertFrom-CSV").
- ElectricalUnion 3y ago> Zsh works fine with spaces and all other manner of weird whitespace, non-printable binary data, etc. > most of the problems here don't have much to do with parameter expansion Parameter expansion is 99% of the problem here as you said youself. Parameter expansion ends up being most of the time what other script languages would call eval, and everyone knows to avoid eval unless you really need to use it. Excluding some really annoying bugs, almost all shells works fine for any sort of binary data. It's the user shell script that is composed of unsafe word expansions that doesn't. It just takes someone messing with parameter expansion in zsh (SH_WORD_SPLIT) and then zsh becomes just another shell in a world of hurt again.
- tyingq 3y agoI would add the "-print0" option in find that causes it to output null separated file names. And the companion "-0" flag in things like xargs, perl, tar, etc...to parse them.
- watersb 3y agoSee also the HN comments on "Defensive Bash Programming (2015)" Archived: https://web.archive.org/web/20170728030103/http://www.kfirlavi.com/blog/2012/11/14/defensive-bash-programming https://web.archive.org/web/20170728030103/http://www.kfirla... HN: https://news.ycombinator.com/item?id=10736584 https://news.ycombinator.com/item?id=10736584 "The Hell that is Filename Encoding (2016), HN 2018" https://news.ycombinator.com/item?id=16991263 https://news.ycombinator.com/item?id=16991263
- watersb 3y agoThe "bag of bytes" POSIX(?) standard(?) can be overridden at the filesystem level: ZFS zpool create -O utf8only=on -O normalization=formD -O casesensitivity=insensitive mypool <pool geometry> <devices> My primary use case for ZFS is a home lab file server to my Mac clients; you might want to keep Unicode normalization and case sensitivity at the default settings. My suggestion here is just one [futile] step at reducing the madness; Postel's Law dictates that we should expect pathological inputs to shell scripts anyway. ("emit clean outputs, expect crazy inputs") https://en.m.wikipedia.org/wiki/Robustness_principle https://en.m.wikipedia.org/wiki/Robustness_principle
- Self-Perfection 3y agoThis is a nice comprehensive resource. I admire author's perseverance in expounding these edge cases and pursuing to amend them somehow. For many years! Yet filenames is not the only place that can unexpectedly include control chars. For instance here is creation of an user with partially colored username. Still work on GNU/Linux: $ UNAME=$(printf '%b' "iam\033[1;31mred\033[0m\n") $ sudo useradd --create-home "$UNAME" Lot's of tools break in various ways for such user.