11 ms·
You shouldn't parse the output of ls(1)
- enriquto 5y agoI disagree. Parsing the textual output of ls is such a natural idiom that I'm happy to renounce any other thing that causes trouble. Give me a "-o sanenames" option for mount, instead.
- spicybright 5y agoI think the point is ls has so many options, it's not safe to parse as is. I always use `find .` if I need a list of files from a directory for this reason
- rightbyte 5y agoYe that is what the maintainer said for the -z option for ls too.
- beermonster 5y agoI use `echo *`
- vidarh 5y ago`echo *` has many of the downsides of ls (doesn't escape e.g. space in filenames) and additionally breaks on directories where the expansion fills the command line buffer. EDIT: Also note that "find ..." is also not safe from all the quoting issues without "-print0" or equivalent options to make it separate the names with ASCII NUL rather than linefeed or otherwise taking steps to handle filenames with actual linefeeds in them.
- xorcist 5y agoIndeed. Those backticks only works in a shell, but in a shell why not just write * which is how filename expansion is supposed to work.
- beermonster 5y agoThe back ticks were supposed to be quotes, I meant to say echo * However as per replies, this suffers from the same issues as ls.
- vidarh 5y agoThe problem is not the backticks. They were just used as quote characters. The problem is that shell expansion doesn't escape the characters. E.g. this is a cut down output from my system now after I did a echo >'/tmp/ space ': $ echo /tmp/* /tmp/bspwm_0_0-socket /tmp/config-err-Q667kI /tmp/foolog /tmp/ space /tmp/... Parse that output and you get a broken list of filenames.
- traceroute66 5y agofind or indeed, the Rust-based fd[1] which is infinitesimally faster. [1]https://github.com/sharkdp/fd https://github.com/sharkdp/fd
- vidarh 5y agoIf you're listing just the filenames, the things that makes fd fast (the parallelised directory traversal when you do something that requires stat calls or similar) are irrelevant, as the getdents() calls are going to be more affected by your buffer size. So for the limited subset of tasks where you're ok with using a tool that might not be installed and need options that requires stat calls and the directories may be large enough, it might make a difference.
- mellavora 5y agoinfinitesimally https://en.wikipedia.org/wiki/Infinitesimal https://en.wikipedia.org/wiki/Infinitesimal "In mathematics, an infinitesimal or infinitesimal number is a quantity that is closer to zero than any standard real number, but that is not zero." I'll blame this one on auto-correct.
- CRConrad 5y agos/simal// There, FTFY.
- johnisgood 5y agoYeah. I often do "find . | grep 'foo'". Perhaps "find" can do it without the "| grep" bit, but I have not RTFM. :P
- aidenn0 5y agoYou probably want -name or -iname which match glob-like expressions against the filename (the "i" prefix means case insensitive). If you really need regex you can use -regex or -iregex, but be aware that they match the entire path (so if you do "find ." you will be matching a string that starts with "./"
- rightbyte 5y agoYe I mean when doing sysadmin stuff you know to avoid asking for it with, like, filenames with spaces. Why even bother handling newlines or what not.
- pkrumins 5y agoIt's not an idiom. It's a fireable offense.
- mattowen_uk 5y agoDead link (HN hug of death?) Cached copy: https://webcache.googleusercontent.com/search?q=cache:eZI_amfj-00J:https://mywiki.wooledge.org/ParsingLs+&cd=1&hl=en&ct=clnk&gl=uk https://webcache.googleusercontent.com/search?q=cache:eZI_am...
- artemonster 5y agoI wonder how world would look like if all standard unix tools gave two outputs: human readable and structured, json-like.
- wayoutthere 5y agoThen it would be PowerShell.
- disgruntledphd2 5y agoBut with shorter command names, and gnomic short options.
- laumars 5y agoNo. Powershell is a whole new CLI user land as well as a shell. If you want something that’s compatible with POSIX but still has smart pipelines and native support for JSON then you’re better off with Elvish or Murex as shells.
- xorcist 5y agoPerhaps take git as an example instead, with its plumbing and porcelain commands. They have historically been easier to keep backwards compatibility with than the dict-like structures of json.
- rnestler 5y agoI'd recommend to take a look at https://www.nushell.sh/ https://www.nushell.sh/ which has structured output, but displays it neatly when printing to the terminal.
- Kim_Bruning 5y agoYou don't need to wonder, because jc is a filter that does just that! https://kellyjonbrazil.github.io/jc/ https://kellyjonbrazil.github.io/jc/
- robert_tweed 5y ago
- nailer 5y agoJust add JSON output to ls like other tools have.
- asicsp 5y agoSee also: Why not parse `ls` (and what to do instead)? https://unix.stackexchange.com/questions/128985/why-not-parse-ls-and-what-to-do-instead https://unix.stackexchange.com/questions/128985/why-not-pars...
- unilynx 5y agoMore importantly, we need to get rid of the ability to put line feeds, tabs in file names and also disallow odd starting characters such as tab, dash and $ I wish someone would add a mount option for that and have eg fedora be a trailblazer to fix the few apps that break
- Tepix 5y agoIt would be almost trivial to create a fuse filesystem which completely hides these files if they exist (and doesn't allow the creation of new ones).
- patrec 5y agoAnd how would that be even remotely useful? Unless something changed recently, FUSE has so much overhead it's only useful for niche applications and prototyping.
- laumars 5y agoIt would be trivial to code but any such wrapper would add overhead to file system operations. FUSE is a fantastic set of APIs (I’ve used it personally) and performs remarkably well considering it is constantly swapping memory between kernel and user space but for wide spread adoption any feature like this would need to be part of the native file system options.
- patrec 5y agoA thousand times this. There is absolutely no reason to allow newlines in filenames, and it is pathetic that there isn't even yet a mount option to disallow totally idiotic filenames (at the minimum I don't want programs to create filenames with newlines or invalid utf-8).
- throwawaylinux 5y agoIt's great that file names in the user/kernel ABI are treated as unstructured NUL terminated byte streams so I can do what I want with my file names even if you don't like it. And you can do what you want with yours, including not creating ones you think are idiotic, or using filesystems with code or options that restrict what names can be used.
- hericium 5y agoFirst example suggests that `ls` should not be used but `ls -l` - the same program author advises against in the title, but with a parameter - works as expected and in this case would not result in "you can't tell". > The problem is that from the output of ls, neither you or the computer can tell what parts of it constitute a filename. Computer does not use console output of ls(1) to determine the list of files. It's for the user. The computer can tell what is a file here. The title could also be stricter with s/ls/"GNU coreutils ls"/g, too. I could not reproduce all the issues with FreeBSD's ls(1) under zsh.
- Tepix 5y ago"ls -l" has other issues, now it will show you user and group names which can contain unexpected characters, too.
- deleted 5y ago[deleted]
- vidarh 5y ago> First example suggests that `ls` should not be used but `ls -l` - the same program author advises against in the title, but with a parameter - works as expected and in this case would not result in "you can't tell". The first example is used to demonstrate the issue and to demonstrate that "-l" introduces other issues (inconsistent escaping). > Computer does not use console output of ls(1) to determine the list of files. It's for the user. The computer can tell what is a file here. But if you try to use the output of ls in a script to find filenames, the computer will be using ls to determine the list of files. Hence the advice not to do so. > The title could also be stricter with s/ls/"GNU coreutils ls"/g, too. I could not reproduce all the issues with FreeBSD's ls(1) under zsh. I think that just emphasises why you shouldn't, as it demonstrates you can't trust the output of ls to be consistent between systems either. If you are sure you'll never need to run your scripts on another system, you might not care, but when it's so easy to prevent this by e.g. using find with "-print0" or equivalent, it seems silly to not just unlearn the bad habit of using ls for this.
- deleted 5y ago[deleted]
- yagop 5y agoParsing UNIX command outputs is generally a pain and constantly a source of errors. PowerShell mostly solve that, I wish we can use that.
- 13of40 5y agoPowerShell has been available for Linux and Mac for a few years now. https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-linux https://docs.microsoft.com/en-us/powershell/scripting/instal...
- Tepix 5y agoVery valuable points that are too easily forgotten. Thanks
- gorgoiler 5y agoGreg aka graycat was a real IRC legend 20 years ago. I learned so much from him. Many a happy hour did I watch him flaming lazy newcomers looking for a quick fix in #debian, right about the time when Linux as a commercially viable server platform was taking off. Almost every admonishment was accompanied by sound technical advice which was useful to lurkers as well as the unfortunate noob who dared ask. Thanks :)
- ByThyGrace 5y agoGreg Wooledge's bash wiki is my goto resource for bash scripting. Everything I always need to find out is in there (Bash Guide + FAQ). I didn't know about his IRC persona which only improves my appreciation of him, so thanks for sharing.
- Fnoord 5y agoAh, greycat. Yeah I remember him from #debian on Freenode some 20 years ago. Smart, helpful fellow.
- dsr_ 5y agoOn occasion I have posted something in debian-user, adding "but Greg will have a better approach". Then he shows up and offers a better approach. Thanks to Greg, my bashrc contains: > stat=( ) > statcolor=("$Green" "$Red") > ... > PS1=... > ${statcolor[!!$?]}\]${stat[!!$?]}$ Which, if it's not entirely clear, puts a green checkmark or a red x in my prompt depending on the error value of the last run commandline.
- marcosdumay 5y agoOh, a long time ago (but not so long as that) I got this line from a HN thread on bash tricks: export PS1="\h:\w \$(if [ \$? = 0 ]; then echo :\\\); else echo :\\\(; fi) \$ " It's a non-colored version of it, with a happy or sad smiley.
- jmnicolas 5y agoI have a bash alias that creates a random playlist of videos or music with ls. I noticed that sometimes there were duplicates in the list. If I can't use ls, it's not going to be a one liner anymore, so I have to create a file, store it somewhere, assign execute privileges and link my alias to it. Much more complicated.
- theamk 5y agoUm, no? As examples show, "find -maxdepth" can do the same things, but safely. And in most cases, it'd still be one-liner, even if a longer one.
- rascul 5y ago> so I have to create a file, store it somewhere, assign execute privileges and link my alias to it Or put it in a function in the same file your alias is in.
- aidenn0 5y agoyou can't call shell functions via "find -exec" or xargs
- amptorn 5y agoWhy in the world does Unix allow newlines in a filename in the first place? That's just such an obviously brain-damaged idea. There's not a single rational use case for it, yet it breaks nearly every text-based tool you could possibly imagine...
- mistrial9 5y agomy imagined reason is -- because when that terrible day happens, and an important file with some new name, does in fact get a newline in it, the rest of the system now has predictable code paths. Q. Is this related to perl, who knows
- Latty 5y agoUnix filenames are just sequences of bytes, not defined as strings. Most programs parse them as utf-8, but there is nothing mandating that. Obviously that leads to problems.
- ninkendo 5y agoOne pedantic qualification: any byte except 0x2f (`/`) or 0x00. This actually rules out nearly any non-UTF8 character set (besides ASCII.) Quote from Linus, which reminds me of Henry Ford’s “you can have any color you want, so long as it’s black”: > And that one true format is UTF-8. End of story. If you try to talk to the kernel in UCS-2 or anything else, you _will_ fail. https://lore.kernel.org/all/Pine.LNX.4.58.0402141827200.14025@home.osdl.org/ https://lore.kernel.org/all/Pine.LNX.4.58.0402141827200.1402...
- yholio 5y agoThese all seem to be ls bugs. It's a common pattern when outputting data to format it such that the receiver can unambiguously separate the data from the formatting. If you use CR/LF in your output formatting, then those characters need to be escaped in the data. If your attacker can deceive you into printing fake output by crafting their filename as : "\n -rw-r--r-- 1 user group 12 Dec 15:55 mostly_harmless_planet" ...then you have already lost. Violating this pattern always leads to problems like format string vulnerabilities, SQL or executable injections etc. As the long history of fighting against these problems shows, "banning weird characters" without fixing the bugs will always lead to problems, some apparently harmless characters find devious uses etc. You can't unscramble eggs. The only real solutions are properly escaping the payload so that it can be unambiguously interpreted. And you can't claim that the authors of 'ls' don't expect their output to be consumed by other programs.
- josephcsible 5y agoInterestingly, ls does escape characters like \n in its output when it's printing to a terminal, but not when it's being piped into other programs. Try this by making a file with a newline in its name, and then comparing "ls" with "ls | cat".
- thibran 5y agoIf common CLI programs would have a --json option this would be no problem.
- ape4 5y agoIts pretty trivial to make a C program that lists a directory in the format you want.
- salmo 5y agoSome of these are why I bail for a “real language” in many seemingly simple scenarios. As soon as I care about datetimes, it’s just easier to use stat() and a proper datetime API. I can treat filenames as byte arrays and translate to Unicode or let the language do it for me. In dire circumstances, find … -print0 | xargs -0 second_script is usually my fallback, but that has pitfalls as well. Go has been a blessing there for me, not having to rely on a runtime across diverse hosts. But that’s a preference and doesn’t help on old kernels w/o epoll(). So many battle scars from inconsistency in Bash and GNU utilities over the years, especially on Unixes’ bundled versions (Solaris, etc) or supporting GNU, BSD, SysV, and HP-UX in the same script. Used to deploy a ksh88(ish) on all for SOME consistency. Luckily now I’m not supporting anything but Linux anymore. When I can’t Go, then I just hijack some tool’s bundled Ruby (eg Puppet), Python, etc when I have to handle that and stick to the standard library. I am too lazy to C these days like I used to. I’m usually dealing with an emergency (looking at you log4j) and don’t have the cycles to cover the gotchas there.
- chasil 5y agoAny POSIX system has an easy way to remove a file with a malformed/hostile name. Determine the inode number with "ls -li", then remove it with "find . -inum # -delete" to remove it. The GNU stat utility makes finding the inode slightly easier. This method is preferable when there is any doubt of wildcard expansion.
- scbrg 5y agoUnless I'm mistaken, POSIX find does not have -delete
- chasil 5y agoAlas, you are right, my mistake. It doesn't have inum either. Those GNUisms do creep in. https://pubs.opengroup.org/onlinepubs/9699919799/utilities/find.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/f...
- 5y ago
- tyingq 5y agoThis is one reason Perl was very popular even before CGI was a thing. You could get to things like stat() with an interpreted language that was very portable. It also has the "-0" flag to accept the null terminated output of "find -print0".
- rvieira 5y agoIsn't parsing `ls` the whole backbone of Emacs' `dired`?
- tyingq 5y agoIf you <ctrl-f> and search for "newline", you can see some of the hackery they do to get around newlines in file names: https://github.com/emacs-mirror/emacs/blob/master/lisp/dired.el https://github.com/emacs-mirror/emacs/blob/master/lisp/dired...
- NumberWangMan 5y ago"Quoted string notation" (https://www.oilshell.org/release/latest/doc/qsn.html https://www.oilshell.org/release/latest/doc/qsn.html) seems like a good way to solve this problem.
- chubot 5y ago(author here) Yes thanks, that is exactly the point! As I point out at the end of the doc, coreutils ls actually started quoting the names in 2016. However the format is confusing for people who can't read 2 or 3 types of shell strings, and not that readable. In contrast, QSN is simply Rust string literal syntax, which are a cleaned up version of C string literal syntax. $ touch $'foo\nbar' 'dq"dq' "sq'sq" # create 3 files with newline, double quote, single quote # coreutils is correct, though I'm not sure people will understand $'\n' $ ls 'dq"dq' eggs 'foo'$'\n''bar' "sq'sq" Pipe through cat mangles the name: $ ls|cat dq"dq foo bar sq'sq In Oil, write --qsn will ALWAYS give you 5 lines if you have 5 names, no matter what they are $ oil -c 'write --qsn -- *' 'dq"dq' 'foo\nbar' # more familiar encoding 'sq\'sq' Without --qsn it's like ls|cat: $ oil -c 'write -- *' dq"dq foo bar sq'sq I think it's important for something like QSN to be built into the shell, because quoting issues arise in many places, not just filenames and ls. Although this makes me think that we should have the inverse of `printf %q` to parse the output of coreutils ls. Oil does implement printf %q, but most people don't know about it. $ printf '%q\n' -- * dq\"dq $'foo\nbar' sq\'sq Again it is actually correct, but sort of a grab bag of formats derived from shell strings. QSN strings will be familiar to anyone using Python, Rust, etc. consistent with Oil's slogan: It's for Python and JavaScript users who avoid shell! ----- edit: Also reminds me that I wrote this page before designing QSN: https://github.com/oilshell/oil/wiki/Shell-Almost-Has-a-JSON-Analogue https://github.com/oilshell/oil/wiki/Shell-Almost-Has-a-JSON... So printf %q and %b are inverses in bash, but this doesn't work in other shells. QSN can represent NUL bytes, which are illegal in filenames, but are useful elsewhere.
- renewiltord 5y agoMy file system is my file system. I solve this problem by just not having weird file names on it.
- dvh 5y agoIn a similar way, ftp's "dir" command is only for humans. Every ftp library that is for accessing ftp API for programs is only guessing what in the "dir" output is filename.
- bravetraveler 5y agoThis is one of those things I always look out for in submissions at work (as bad as that may sound) It's one of the easy ways to guarantee a process will run into an edge case eventually... General rule of thumb: when possible, lean on shell features. Globbing, expansion, redirection. That replaces dozens of tools (eg: `seq`, `ls`, `cat`, and so on) Another example (though less severe) that comes to mind: subshells to simply read (cat) a file. Unless you're doing things at a ridiculous scale/pace, it doesn't usually matter - but redirection is 'cheaper'. (Talking about cases where you care about nproc/nofile ulimits) I wish I could contrive better examples. I feel like my ability to 'sniff' this kind of stuff out is usually what makes my best contributions at work, but without being in the moment... it's difficult. xargs is a good one. That's usually an indication you need an array, even though I think they're 'fake' in BASH
- wutbrodo 5y agoHonestly, I've yet to hear a good reason not to just use Python for scripting that's any more complex than a series of commands. I resisted this for years, but one day realized I was only doing it because it felt intuitively right that scripts were simple extensions of the way one engages with the terminal interactively. I had to come to terms with the fact that the intuition gap is due to the fact that bash is so dang awful.
- sethammons 5y agoI like that ruby and perl can integrate with shell commands more easily than python or go.
- wutbrodo 5y agoRight, I didn't mean to single out Python, but rather to single out Bash. My comment applies to Ruby and Perl as well.
- bravetraveler 5y agoSadly I'm still there. Largely because... I killed my development mojo as a kid. C/C++ and the early web crazes. Nowadays I'll do most ad-hoc complex things with BASH... but at a certain point I tend to use Ansible. I guess one could say I indirectly write Python through Ansible/YAML :)