22 ms·
A common mistake involving wildcards and the find command
- dkarras 7y agoI also have the habit of always using -iname instead of -name to make case insensitive searches.
- manaskarekar 7y agoI used to do this until just replacing it with `fd`. https://github.com/sharkdp/fd https://github.com/sharkdp/fd
- dehrmann 7y agoShell (because this is technically a shell, not a find issue) is the worst language that everyone should learn. It's a language you'll actually encounter, and it's one that's hard to avoid (unlike PHP).
- thunderbong 7y agoPhew! I'm glad I've been hitting the "Happy Case" scenario all these years! Very useful article. And very informative. Summary: Instead of - find . -name *.jpg Use quotes around pattern i.e. find . -name '*.jpg' Edit: Oops, the double-quotes should have been single quotes! Thanks, @lucd. Happy case, like I said!
- lucd 7y agoWildcards are still expanded inside double quotes.. Yiu have to use single quotes.
- solarexplorer 7y ago...or a backslash.
- beamatronic 7y agoI use the backslash. I also specify -type f explicitly when I’m looking for files. find . -type f -name \*.txt Now the next article will talk about problems when there’s a space in the file names and you just piped the output into xargs :)
- chousuke 7y agoThat's not true. Variable expansion happens in double quotes, but globs stay untouched.
- waynecochran 7y agoYes but double quotes will work fine in this case -- globs won't be expanded...
- deleted 7y ago[deleted]
- mangix 7y agoI wonder if this is also a problem with fish. The globing there is different.
- Apofis 7y agoFish basically forces you to single quote commands that contain wildcard characters.
- faho 7y agoFish won't pass along the unmatched glob, so in the given example you'd notice that with find . -name *.py it will error initially, because there is no "*.py" in the current directory. Globbing is still a thing fish as the shell does, and you still need to protect asterisks that need to reach commands, but you're more likely to notice. (for bash there is also the "failglob" option that I personally recommend) Disclaimer: I am a fish developer.
- mynegation 7y agoCommon? Yes. Simple enough to stop doing this mistake after two times? Also yes. One you internalize in which case shell is responsible for globbing and in which case command itself, it is pretty clear cut.
- seiferteric 7y agoThis was obvious to me, but one version of this that surprised me is when using scp. If you glob a remote destination like "scp myserver:*.jpg ./" It will probably work! But how? Because the remote path will likely not match any local files and the path with the asterisk will be passed to scp and scp will do the globbing on the remote side.
- tyingq 7y agoI mostly use ssh and tar instead of scp, because I'm usually after more than one file. Like, "ssh remote 'cd /whatever; tar -cf - someglobpattern' | tar -xvf"
- emmelaich 7y agoTip: get into the habit of using && rather than ; So... > ssh remote 'cd /whatever && tar -cf - someglobpattern' | tar -xvf
- gdavisson 7y agoSeconded. Here https://apple.stackexchange.com/questions/378942/cant-initialize-osx-interrogation-folder https://apple.stackexchange.com/questions/378942/cant-initia... is an example of what can happen from not checking for errors after a `cd` command. Summary: "sudo find ... -delete" ran in the root directory rather than the intended location, and now the OS is gone (probably along with lots more).
- tyingq 7y agoFair observation. I will change my ways :)
- nine_k 7y agoCould also be faster because of the common compression dictionary.
- heinrichhartman 7y ago
- twic 7y agoI often set the nullglob option in scripts, because it makes the handling of globs which don't match anything a bit more predictable: http://bash.cumulonim.biz/NullGlob.html http://bash.cumulonim.biz/NullGlob.html There's a note at the end about how with nullglob set, ls on a glob with no matches does something surprising. This is a great illustration of how an empty list and the absence of a list are different. Sadly it's rather hard to make that distinction in shells! I do wish that either shells had a more explicit syntax for globbing, or other commands didn't use the same syntax for patterns. Then confusion like this couldn't occur. An example of the former would be if you had to write: ls $(glob *.txt) Here, the shell would not treat * specially, but rather you would have to explicitly expand it. This would be a pain, but at least you wouldn't do it by mistake!
- johncs 7y agoI set failglob: `shopt -s failglob`. Makes the whole command fail if there's no matches. That combined with `set -e` which aborts the script in the event of any command failing makes me feel somewhat safe. Indeed I add the following two lines to every bash script I write: set -exu shopt -s failglob
- nothrabannosir 7y agoIf you like set -e, I recommend looking at set -o pipefail.
- simias 7y agoI don't know if it's the default or if it's something I've set in my config a decade or two ago but my zsh behaves like this by default (i.e. I get an error for blobs that match nothing instead of silently passing the * along). That seems much saner to me: $ find . -name *.txt zsh: no matches found: *.txt That'll teach you to quote your `find` patterns real fast...
- daveFNbuck 7y agoIt's either default or part of oh my zsh, because I've had the same experience and I don't have any custom configs.
- StreamBright 7y agoIsn't this in the Unix hater manual? I never use -name without ''. I guess this is just muscle memory from early on when I run into this issue that in Unix *.py can mean very different things depending on where it gets resolved.
- svnpenn 7y agoAs another user mentioned, many POSIX and/or GNU utilities havent aged well. I respect trying to stay portable, but peoples needs change over time and these tools simply havent kept up. Like the other user, I use Fd now instead: https://github.com/sharkdp/fd https://github.com/sharkdp/fd As well as Silver Searcher: https://github.com/ggreer/the_silver_searcher https://github.com/ggreer/the_silver_searcher while Grep performance is pretty good, its also gotten pretty stale with regard to its defaults and options.
- kragen 7y agoWouldn't you have the same problem with fd if you invoke it on unquoted globs? The problem is the semantics of the shell, not find.
- wffurr 7y agoYes, but fd has implicit pattern matching without requiring the * character: Features: Convenient syntax: fd PATTERN instead of find -iname '*PATTERN*'
- pwdisswordfish2 7y agoSeems a misfeature. Much of the time, I use find to look for files with a specific extension, which makes the trailing asterisk harmful.
- johnp_ 7y agofd -e py Also neatly bypasses the glob issue.
- anonsivalley652 7y agoFor grepping, rg is faster and saner. https://github.com/BurntSushi/ripgrep https://github.com/BurntSushi/ripgrep On end-user systems with fast I/O, it is usually a better use of resources to have a battery-/suspend-/reboot-aware indexing system that has path/extension white/blacklists and prioritizes file monitored changes. This way, searching can happen against an optimized text search DBMS much faster than waiting for zillions of IOPS that are basically wasted by instead spending them ahead-of-time in the background/idle indexing text files/metadata/structured data once when it changed/s and already know exactly where all occurrences of kWhateverCondition_PP3V42_G3H or \A[AB]{1,3}c+d\z live under ~/Projects without reading any actual files.
- thundergolfer 7y agoPutting ‘shellcheck’ in your CI pipeline is a must for me now, after one too many mistakes. I just finished cleaning away all existing ‘error’ and ‘warning’ level issues in our codebase so that the ‘shellcheck’ CI step can be really strict on code quality.
- sys_64738 7y agofind . -name \*.jpg This is pretty elementary which any seasoned Linux person should know.
- tyingq 7y agoGlobbing is also more complex than it seems on first blush. Alternation is supported in bash. Stuff like "echo ∗.{png,jp{e,}g}" (utf8 asterisk to get around HN, cut/paste won't work). That bsd derived glob is useful for some sometimes useful tricks. Like in Perl: use File::Glob qw/bsd_glob/; my @list = bsd_glob('This {list|stuff} is nested {{very,quite} deeply,deep}');
- acqq 7y ago> "echo ∗.{png,jp{e,}g}" (utf8 asterisk to get around HN, cut/paste won't work) This works: echo *.{png,jp{e,}g} Note ∗ ⟵ U+2217 * ⟵ needed ASCII asterisk Just put the code examples in the separate lines, prefixed with spaces.
- mehrdada 7y agoStraight from The UNIX-HATERS Handbook. https://web.mit.edu/~simsong/www/ugh.pdf https://web.mit.edu/~simsong/www/ugh.pdf
- fctorial 7y agoA classic
- tzs 7y agoFor a long time (probably since the first time I forgot to quote something and got burned, so around 40 years), I've thought that there should be some mechanism for the shell to pass in information about how each argument came about. For each argument, it would tell the program if it was supplied directly, or came from wildcard expansion. For those from wildcard expansion, it would tell the program what the wildcard was. Most programs would not care, but some programs could use this to catch common quoting errors.
- kalium_xyz 7y agoThe unix haters guide lists the shell handling the wildcard like it does as a major flaw with unix shells.
- jolmg 7y ago> For those from wildcard expansion, it would tell the program what the wildcard was. Different shells have different globbing mechanisms. Why should all programs tie themselves to the mechanisms of any one particular shell? The simpler the calling mechanism for executables, the simpler it is to write them in any existing or future language. This also gives more flexibility to future shells. UNIX is pretty much designed thinking of users as programmers. Making it easy to write programs building on other programs is as important as being able to call them. With that in mind, I don't think it's a good compromise to complicate the writing of executables in order to protect users from their own mistakes.
- tzs 7y ago> Different shells have different globbing mechanisms. Why should all programs tie themselves to the mechanisms of any one particular shell? I don't think my proposal would do that. First, a clarification. When I said "For those from wildcard expansion, it would tell the program what the wildcard was" I meant the shell tells the program the full argument containing the wildcard, not just the wildcard part. E.g., if the argument was a?c which expanded to abc, the shell would tell the program it was a?c, not merely ?. Let's say myprog takes one argument, which it expects to either be a file name or a wildcard in some pattern language that is not necessarily the same pattern language used by the shell. Suppose myprog is invoked and it sees it has one argument, X. If the shell tells it that X did not come from shell wildcard expansion, myprog can safely use it. If it does come from wildcard expansion and the source was Y, then there are a few cases. 1. Neither X nor Y contains any myprog wildcards. In this case, I'd have myprog go ahead and use X. 2. X does not contain any myprog wildcards, but Y does. Myprog can infer from this that it and the shell probably have overlapping wildcard languages. Report a "needs quote?" error. 3. X contains a myprog wildcard, but Y does not. Probably myprog and the shell use different wildcard languages. I'd guess its pretty unlikely that the user expects that a filename will contain a myprog wildcard, so I'd report an error in this case. 4. Both X and Y contain myprog wildcards. Go ahead an accept X, applying myprog wildcard expanstion to it. (Maybe first check that X and Y contain the same myprog wildcards, in the same order). I'd probably add support in myprog for some optional flags to change the defaults in those cases so if you really want to do obscure things like have file names that contain myprog wildcards and use shell wildcards to provide those file names you can do it.
- monkeycantype 7y agoat my workplace I use git bash on windows, and because i'm always getting man: command not found I flick to a browser and type man <whatever I was looking for> About once a year I forget what happened all the previous times I typed man find into google. TLDR: looking for Dennis Ritchie, I found Chuck Tingle.
- joana035 7y agoThe title seems a bit off since shell expansions and arguments has nothing to do with the find command. Both features are also often covered in entry level material for introduction to shell.
- umanwizard 7y agoMost commands don’t accept the shell-like wildcard `*` as part of their command syntax; find does. That’s the connection.
- shmageggy 7y agoWell it seems the root of the problem is > * Most importantly, the 'find' command uses a different algorithm than shell globbing does when matching wildcard characters. More specifically, the find command will apply the search pattern against the base of the file name with all leading directories removed. This is contrasted from shell globbing which will expand the wildcard between each path component separately. When no path components are specified, the wildcard will match only files in the current directory.* So there does seem to be a `find` specific issue here
- deleted 7y ago[deleted]
- Izkata 7y agoThat's just ignorance about find. The globbing works fine, they're just not doing the search they think they are - they seem to think "-name" acts like "-path": $ ls Foobar/ one three two $ find . -path '*bar/t*' ./Foobar/two ./Foobar/three
- umanwizard 7y agoMy instant reaction to the example was “that won’t work; you’re shell will say something like ‘no matches’”. Using an unescaped star in a find command never works for me, which is a lot better than it sometimes working and sometimes breaking! Reading the article and the comments, it seems like bash doesn’t do this? I suppose it’s one nice thing about oh-my-zsh, whose default confit I use almost unchanged.
- CGamesPlay 7y agoDitto, fish shell also provides this behavior in the default config.
- erikbye 7y agoIt's a default in zsh, nevermind oh-my-zsh.
- umanwizard 7y agoThanks; I didn’t know whether it was or not while I was writing my comment, and it was difficult to check on mobile, so I went with what I knew for sure. Glad to know it’s a stock zsh default, too!
- tzs 7y agoAnother one I've seen a few times with find (although it is actually more an xargs thing than a find thing), usually not with any bad consequences at least, is something like this: find . -type f | xargs grep foo You expect to see all the "foo" lines from your files, each prefixed with the file name and a colon. And that's what you get most of the time. But sometimes you might get a foo line without the filename prefix. Why? Because grep only adds the filename prefix when there is more than one filename argument. There are two ways that might come about in the above command. The first is if the find only finds one file. The second is if the find finds so many files that xargs has to invoke grep more than once. It can happen that there is only one file left to do when it gets to the final grep invocation. Simple fix: find . -type f | xargs grep foo /dev/null
- Xophmeister 7y agoA “better” fix would be: find . -type f -exec grep -H foo {} +
- dmurray 7y agoYeah, find ¦ xargs is definitely an antipattern, but in practice I use it all the time. Either because I forget the syntax for find -exec, or because I'm feeling principled that find -exec breaks the Unix philosophy of doing one thing well and being composable, or because my script started using ls and got changed to use find.
- derekp7 7y agoxargs is also more efficient than -exec, as it stuffs multiple files to the target command. Esp. for rm, instead of executing a copy for each file.
- detaro 7y agodoesn't -exec <command> {} \+ do that too? (and we're back at the "remembering specific syntax" point: knowing xargs helps you everywhere)
- arcade79 7y agoSubject of the post is wrong. This is a common mistake between the user and whichever shell the user is using, not the user and the command itself. The find command works exactly as expected.
- jasonhansel 7y agoThis is the one thing I like about DOS/cmd.exe. In those shells, wildcard expansion is done by applications instead of the shell itself, so there's no need to resort to hacks like this.
- mkl 7y agoThat means the expansion is very limited and can be inconsistent. The bash etc. way is much more powerful and convenient.
- nothrabannosir 7y agoTangential: another safeguard you can adopt is avoiding "hard delete" commands like rm and find -delete. Untrain yourself from these commands by never using them. On Mac systems, the "trash" program (brew install trash) sends files to your system trash. You can use `trash [file]` and `find .. -print0 | xargs -0 trash --`. rm is a dangerous command you should only very rarely be using. I fish something out of the trash a few times a year and lemme tell ya; it's worth the investment. Another tip if you fancy debugging shells is using python -c "print(__import__('sys').argv[1:])" sample "arg here" * foo this provides the same functionality as the c program in TFA, without needing gcc.
- icebraining 7y agoTrash is the poor man's backup system.
- JulianWasTaken 7y agoYou use trash and backups. Rich men can still save a few dollars for pocket change.
- Cpoll 7y agoI learned from a sysadmin to use `mv` instead of `rm` when you're removing potentially critical files. Other tips include colouring your prompt highlighted red on production boxes so you never accidentally think you're somewhere safe. Nowadays I try to avoid SSHing into mission critical machines though.
- rezonant 7y ago> this provides the same functionality as the c program in TFA, without needing gcc. or just using bash: function showargs() { for x in "$@"; do echo "arg: $x"; done }
- mkl 7y agoI like it. I'm using a numbered version: function showargs() { i=1; for x in "$@"; do echo "arg $((i++)): $x"; done }
- rootusrootus 7y agoI almost appreciate the idea of writing a C program whose sole purpose is to show you the arguments send to it. That's some serious overkill. But I think echo *.py would not just be easier, but more effective at demonstrating what your find command line will actually look like after shell expansion.
- mkl 7y agoThat will be quite misleading when you have filenames with spaces and arguments with quotes.
- NotSammyHagar 7y agoIt is unexpected to new users, principle of lead surprise should show it's not a great idea. At the same time, globbing is a basic concept for beginning shell users. I don't know how I learned about it, if someone told me or something.
- jancsika 7y ago> You can type: man glob Translation: you cannot type "man" followed by an asterisk because that would have required forethought in how one learns a programming language. Argle: Hey Bargle, is that new bridge built to spec? Bargle: It's like I always say, man: good enough for shell script manual operator discoverability. Argle: Yeah, you're always saying that...
- vmchale 7y agoI use fd-find and I don't really miss find.
- emmelaich 7y agoQuiz; in any directory, full or empty I can run this command $ * and get * What allows me to do this? (the $ is the prompt, not what I typed)
- foota 7y agoDoes bash not do expansion for the program name?
- iudqnolq 7y agoYou can't, at least on bash version 5.0.3(1)-release with the default config Ubuntu packages it with. After a lag I get the error `$folder_name$: command not found`
- emmelaich 7y agoI promise you you can, with a previous particular incantation. Answer to come. PS. the response is not important.
- iudqnolq 7y agoOh, I thought you meant in an arbitrary directory. If the first file in sorted order is executable and prints "*" to stdout you could get that.
- emmelaich 7y agoI did mean an arbitrary directory. Your answer almost works; but only if PATH includes the current directory which is unusual. It does hint at another almost answer ... mkdir t cd t ln -s $(type -p ls) *
- ridiculous_fish 7y agoGLOBIGNORE=*; *(){ echo *; } welp
- 7y ago
- vikinghckr 7y ago`find` has one of the worst user experiences out of UNIX tools. I prefer to use `find . | grep foo` to find files.
- 7oi 7y agoAlso, a good thing to remember is to mind the order of your options. I know from experience that it matters a lot after thinking once: "I can just place this `-delete` option wherever, right?" and using it as my first option. Needless to say, I had a very bad time.
- alleycat5000 7y agoOne of my favorite features of fish shell is the wildcard expansion: https://fishshell.com/docs/current/tutorial.html#tut_wildcards https://fishshell.com/docs/current/tutorial.html#tut_wildcar... It's rare I reach for find nowadays since switching.
- shmerl 7y agoTL;DR: globbing kicks in before other things unless you turn it off like with single quotes: https://www.tldp.org/LDP/abs/html/globbingref.html https://www.tldp.org/LDP/abs/html/globbingref.html Don't forget about globbing when making Bash scripts.
- sengstrom 7y agopoints for spelling globbing correctly :)
- siddharthgoel88 7y agoIt is just awesome that I stumbled upon this post. I remember previously I had faced similar issue while running a command like find . -name *.gradle | blah blah Instead of finding the root cause, I by-passed it by find . | grep "\.gradle" | blah blah It just feels great to now connect the dot and know the real reason for the issue.
- ulrikrasmussen 7y agoI believe that programming languages should never make the meaning of a program depend on the context in which it is executed. So many obscure bugs are directly caused by such behaviors. There should be exactly one possible interpretation for a given statement, and if that cannot be executed, then the program should abort. In this case, the glob should never have been passed on to find. It should either have expanded to the empty array, or failed.
- madsbuch 7y agoOne could argue that it is merely a side effect that a shell constitutes a programming language. And one could also note that find should employ the same tradition as the shell (ie. Use /*.jpg to recourse into folders)
- fouc 7y agoI always write it like this: find . -name \*.jpg escape the glob
- deleted 7y ago[deleted]
- madsbuch 7y agoThat was a lot of text to explain that one should be cautious of the wildcard expansion some shells provide. Thanks! I would have jumped right in!
- tigrezno 7y agoApparently, the OP read the man page of "glob" but couldn't spend a single minute reading the man page of FIND. -name pattern ... Don't forget to enclose the pattern in quotes in order to protect it from expansion by the shell. So, a lengthy article on an already documented feature of find.
- Lio 7y agofind . -name *.jpg TL;DR if you don’t wrap your wildcard expressions in quotes the shell will expand them. find . -name ‘*.jpg’ so wrap them in single quotes so that they make their way to the find command unexpanded.
- JosefAssad 7y agoDo people really use find like that to clean up source code? git clean -fxdn # review what would be deleted, then git clean -fxd
- Hamuko 7y agoI do for .pyc files. find . -name "*.pyc" -delete
- JosefAssad 7y agoIs there a reason you prefer that rather than having something like *.py[cod] in your .gitignore?
- Hamuko 7y agoYeah, the reason is that the Python interpreter doesn't give a shit about .gitignore.
- enriquto 7y agoFind is an ugly beast. I only use it to print all the files and then grep my way on.