8 ms·
Why you shouldn't parse the output of ls(1)
- deleted 2y ago[deleted]
- hprotagonist 2y agounless, of course, you use (gnu) ls -D and the application for which it was specifically earmarked. I am consistently amazed at how hard it is to mess dired up.
- SassyBird 2y agoWhen would you use it? Emacs dired works fine on other platforms, which don’t have GNU ls on them, so I’m guessing by default it doesn’t run ls on its own.
- User23 2y agoDired will warn in that case. ls does not support --dired; see `dired-use-ls-dired' for more details.
- hprotagonist 2y agoIf your ls program supports the ‘--dired’ option, Dired automatically passes it that option; this causes ls to emit special escape sequences for certain unusual file names, without which Dired will not be able to parse those names. The first time you run Dired in an Emacs session, it checks whether ls supports the ‘--dired’ option by calling it once with that option. If the exit code is 0, Dired will subsequently use the ‘--dired’ option; otherwise it will not. You can inhibit this check by customizing the variable dired-use-ls-dired. The value unspecified (the default) means to perform the check; any other non-nil value means to use the ‘--dired’ option; and nil means not to use the ‘--dired’ option. On MS-Windows and MS-DOS systems, and also on some remote systems, Emacs emulates ls. See Emulation of ls on MS-Windows, for options and peculiarities of this emulation. https://www.gnu.org/software/emacs/manual/html_node/emacs/Dired-Enter.html https://www.gnu.org/software/emacs/manual/html_node/emacs/Di... https://www.gnu.org/software/emacs/manual/html_node/emacs/ls-in-Lisp.html https://www.gnu.org/software/emacs/manual/html_node/emacs/ls...
- oefrha 2y agoGeez I didn't realize dired relies on shelling out to ls. Why the fuck would you go through all the trouble of - shelling out to an unreliable program and doing all the parsing work; - adding a special mode for yourself to another piece of software, which may or may not be present; - adding a user customization point; - adding an emulation of said software when it's not present; when you are in a real programming language, and all you need to get the clean data directly is readdir(3) and stat(3) (or their Windows equivalents)? This is unbelievable.
- TeMPOraL 2y ago`ls` is faster and more portable in Unix-land, fallback to emulation on non-Unix platforms is easier than asking users to install a port of half of Unix just to have `ls`, and user customization points are what Emacs is made of and for in the first place.
- oefrha 2y agoShelling out to `ls` (exec + opendir + readdir + stat + pipe IO) is not faster than opendir + readdir + stat. `ls --dired` is definitely not portable. readdir and stat (the libc functions) are portable, work across POSIX and other Unix-likes. And the customization point here doesn’t provide any benefit for the user, it serves to let the user shoulder the responsibility for its own design defect (has to guess whether the `ls` supports --dired or not).
- hprotagonist 2y agodired predates emacs, and was originally itself a standalone tool.
- oefrha 2y agoSure, but the dired mode of Emacs has been first party forever, and nobody thought to improve it?
- mickeyp 2y ago
- userbinator 2y agoI think that if an attacker can control filenames fully, this is the least of your worries. Otherwise this article could've just as well been titled "why you shouldn't create filenames with newlines", but that makes too much sense. Has anyone actually found a good use for filenames containing newlines?
- inopinatus 2y ago> Has anyone actually found a good use for filenames containing newlines? Yes - in tests for code that handles filenames, shell or otherwise.
- fshbbdssbbgdd 2y agoSomebody does it once and now we all have to handle it forever!
- kelnos 2y agoI think that falls under "no".
- II2II 2y agoSeeming as newlines may appear in file names, it is a good use. Otherwise something as mundane as a bug in program A (which produces a file name with a newline) has a potential to expose a bug in program B (which fails to interpret a file name with a newline).
- tyre 2y agoAt some point it feels fine that code fails. If you’re writing shell scripts against file names that you don’t control and someone puts a new line, that’s on them. They’re asking for a bad time.
- inopinatus 2y agoMany shell scripts we write run as root, or as a deployment user (which is often privileged in terms of its access to business-critical code, packages, data stores, user uploads). Or they may be supplied for use as utilities by folks with other domains of expertise. We should never plan to enable the possibility of uncontrolled outcomes due to bad/hostile third-party input; moralising blame after an incident isn't a productive activity.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- SoftTalker 2y agoUse find(1) with -print0
- sambazi 2y agoit's even posix complaint!111
- 0xml 2y agoThat's why I kind of like PowerShell, object-oriented instead of string-oriented
- g15jv2dp 2y agoAfter using powershell for a while, I just cannot get back into "unix" tools. Crazy shell syntax/parsing, inadequate formats...
- pastage 2y agoIt seems like a good idea but breaks down, a shell is text for good reasons. Psh is ok in Windows land on Microsoft tools, but IMO you might as well use Python if you want those features. I run the same bash script on five platforms while the power of bash disappears when you do not have all the 3rd party tools and Unix kernel, it is still easy to write and execute fast. That said the idea of power shell is great I use it from bash on Windows for very limited things. As a shell to integrate stuff not so much.
- SAI_Peregrinus 2y agoPOSIX Shell is text for good reasons, but file names aren't (necessarily) text. So POSIX shells aren't very well-suited to dealing with file names, requiring arcane hackery to be portable & safe. It'd be easier with any sort of type system to distinguish file names from text, object orientation isn't necessary at all.
- csydas 2y agoI get what you’re saying but the other weird limitations of powershell as well as microsoft’s strangely abusive security relationship with powershell makes it a non-starter for me, or at least makes me hesitate a ton before putting effort into a ps script since even in the same AD orgs which in theory have the same gpos applied across all machines, it’s a crapshoot if the ps script will even execute without having to trick windows into allowing it to run. object oriented nature of ps is also a curse as much as blessing as simple text parsing or iterating over files on a system are very slow compared to bash, never minding that from my experience it’s a crapshoot on how well a given ps module handles strings and how that will be reflected in the script. (e.g., one company’s ps module doesn’t handle square brackets ][ well and for an inexplicable reason the normal ps escaping does not work as advertised, requiring more backticks than usual and a different number depending on if you “” or ‘’ the input string. As best i can tell its joint fault of ps bug and poor module design but had no luck convincing the vendor to adjust their design because “it will break existing scripts”, which is a valid concern but it also means writing automation for their product is a headache since you’re going to see ][ in production environments as is a visually obvious delimiter when humans are reading it)
- djha-skin 2y agoSeriously though, has anyone encountered a file with a new line in its name? I have never encountered this "problem" in the wild.
- rurban 2y agoI often create them by accident in emacs with save-as, somehow a newline is often to easily appended
- makeitdouble 2y agoI'd assume you can also hit that case if your string library mishandles the encoding of the file name and it becomes a new line where there was none. Then auto generating files from another source will also do that, in particular from html fields, where SQL and sheer security related sanitization will obviously be done, but comestic stuff like new lines can be left untouched.
- fortran77 2y agoThis is why PowerShell is so great. Commands return objects with a structure so you can do what you want with the output from Get-Childitem safely.
- m3kw9 2y agoCan’t wait for the iOS 18 feature where it summarizes the page for you with a tap
- donatj 2y agoI have been writing shell scripts for over twenty years and don't think I have ever encountered a file name with a newline. I have hit file names starting with a dash - though being interpreted as flags at least a few times. Every command should support the -- path separation feature a lot of newer stuff like git does.
- arwineap 2y agoI don't place newlines in my filenames, but I also prefer my shell scripts to be portable. Learning and understanding the bash oddities is key for me because I use it so often. It's the same reason I quote every expansion; sure my text likely doesn't have spaces, but maybe it does and id rather use the extra quote characters to ensure I don't have to troubleshoot language grammar bugs in my code
- frou_dh 2y agoI stopped having to proactively learn most gotchas by just having the spectacularly useful shellcheck linter automatically run on save.
- userbinator 2y agoEvery command should support the -- path separation feature a lot of newer stuff like git does. That's a POSIX standard feature: https://pubs.opengroup.org/onlinepubs/009696799/basedefs/xbd_chap12.html#tag_12_02 https://pubs.opengroup.org/onlinepubs/009696799/basedefs/xbd... The argument -- should be accepted as a delimiter indicating the end of options. Any following arguments should be treated as operands, even if they begin with the '-' character. The -- argument should not be used as an option or as an operand.
- deleted 2y ago[deleted]
- 1vuio0pswjnm7 2y agoYou shouldn't parse the output of ls(1) (wooledge.org) 149 points by tosh on Dec 31, 2021 | past | 148 comments https://news.ycombinator.com/item?id=29747034 https://news.ycombinator.com/item?id=29747034 Parsing the output of ls is an antipattern (wooledge.org) 2 points by goranmoomin on Sept 6, 2021 | past https://news.ycombinator.com/item?id=28435532 https://news.ycombinator.com/item?id=28435532 Why you shouldn't parse the output of ls(1) (wooledge.org) 2 points by O1111OOO on Dec 24, 2018 | past | 2 comments https://news.ycombinator.com/item?id=18753980 https://news.ycombinator.com/item?id=18753980 Parsing ls (wooledge.org) 3 points by tambourine_man on Jan 31, 2016 | past https://news.ycombinator.com/item?id=11007601 https://news.ycombinator.com/item?id=11007601 Why you shouldn't parse the output of ls(1) (wooledge.org) 2 points by rosser on Jan 20, 2015 | past https://news.ycombinator.com/item?id=8914841 https://news.ycombinator.com/item?id=8914841 Why you shouldn't parse the output of ls (wooledge.org) 31 points by dgellow on July 6, 2014 | past | 18 comments https://news.ycombinator.com/item?id=7994720 https://news.ycombinator.com/item?id=7994720
- INTPenis 2y agoI hung around #bash@freenode (before libera) for a few years and picked up so many bashisms that I now can't look at any script without finding some nitpick. Especially scripts made by enterprise organisations to install proprietary software.
- cornel_io 2y agoSimple solution: if you ever run into this, blog about it and mark it as a freak occurrence, because it is. Don't ever in real work worry about stupid edge cases like this or you're wasting your employer's money. I'd reprimand an engineer for spending more than an hour on this, it serves nobody unless you're working on extremely widely used library code.
- TeMPOraL 2y agoThis is why Unix practice of being so thoroughly stringly-typed is bad, especially when combined with advice like yours. At every stage, someone is going to cut corners and don't parse just that one "freak occurrence". Then someone wants to wire a bunch of things together and they start to fail, as with so many corners cut on the way, you're bound to hit one. And it all starts with stringly typed data and asking everyone to make half-baked parsers and serializers on the fly.
- fire_lake 2y agoThis is why I won’t use bash for anything remotely complicated. It’s just too full of edge cases. Any general application language is better JavaScript, F#, heck even Python.
- jessekv 2y agoPython scripting based only on the standard library is underrated. There is no pip hell, it's readable and maintainable by the most junior team members, it's largely cross platform. This is the "batteries included" dream.
- jmclnx 2y agoI disagree, the whole purpose of UNIX is to feed output of one command into another command. All these file name issues all come from one source, Microsoft. POSIX should be updated to forbid file names with any of these characters: * space * NULL * New Line (as others said, I never ran across a file with a \n in it)
- mtoner23 2y agoNo one makes files with new line characters so I think I'll be fine
- kjkjadksj 2y agoThe more realistic reason is that ls is very slow in directories with a lot of files compared to find.
- Xophmeister 2y agoA number of people have commented that they’ve never experienced a filename with `\n`, or other weird characters, in the wild. On the contrary, I’ve seen this many many times, to the point that I’m defensive about how I write scripts, to compensate. In my case, these were files created by some other program that contained a bug, where the filename was accidentally set to something like the file contents, say. These other programs were often written by researchers, rather than engineers, who (reasonably so) cared more about their research, than functional correctness. They were also not incentivised to clean-up the dodgy files, if they fix the bug — a big if! — or may not have even be equipped to do so. You could argue that this is an edge case, but shit happens. Often.
- SkyPuncher 2y agoAnytime files, file systems, or binary files become involved I immediately assume everything is going to be harder and longer than estimated. There are simply so many dumb things that can happen. A library might handle 99% of cases correctly, but that 1% case basically requires an entire rewrite.
- Log_out_ 2y agoOf the filenames. Got FileRenamer CRON job, go and kill filenames. Dont solve edge cases, blunt edges at the bleeding edge. Im not gonna rewrite my software to acomodate down sign errors of the ancients.
- ykonstant 2y agoNot an edge case at all; to add another source of such file names, academics often copy-paste or manually type pdf titles into file names resulting in files with newlines. Sometimes their machines use ancient encodings which, together with non-ASCII author names and "helpful" encoding converters result in garbage file names. I've mentioned these situations before in this forum; the last time I did, I got the absurd reply "I don't care what your academic colleagues do" from someone who supposedly had sysadmin experience. So yeah, all you can do is write correct code and accept the fact that many others will refuse to.
- Genbox 2y agoPeople don't consider adversarial inputs when building scripts, which is one of the main reasons why I started a company to build an incident response platform in 2018. I was handling security incidents on a large scale with tools that attackers would purposely circumvent with special filenames/inputs. Instead of fighting against the grain, I decided to build a platform with a heavy focus on correctness. Any deviation from the data specification would stick out as a sore thumb and immediately detect anti-forensics and other tricks. Today, I have a vast compendium of anti-forensics tricks, such as commands not getting written to history, files that cannot be deleted or copied, and much more. Suffice to say that if you are parsing the output of any tool, you are vulnerable to a whole slew of adversarial techniques.