13 ms·
Show HN: A simple, fast and user-friendly alternative to find, written in Rust
- pmoriarty 9y agoI wonder how this compares to the speed of zsh's globbing, which is what I've mostly switched to using instead of find.
- sharkdp 9y agozsh globs are about a factor of 5 slower (for this example): > time fd -sIe jpg > /dev/null 1,24s user 0,77s system 758% cpu 0,265 total > time ls ~/**/*.jpg > /dev/null 0,53s user 0,97s system 98% cpu 1,518 total
- xfactor973 9y agoFor the benchmarks did you clear the page cache inbetween runs? It’s crazy fast and seems a little suspect
- infogulch 9y agoThe readme says the benchmarks are designed to run warm, so I would assume each was run once or twice beforehand.
- sharkdp 9y agoThank you for the feedback! The benchmarks that are mentioned in the README are performed for a "warm cache", i.e. I'm running one of the tools first to fill the caches. Then, I'm performing multiple runs of each tool (using bench for some nice statistics) such that both tools profit from the warmed-up cache. I also perform other benchmarks where I clear the caches. In this case, 'fd' is usually even faster. The scripts for both warm and cold cache are here: https://gist.github.com/sharkdp/4bc3e5f5ea9df2f29c02ede50634b16a https://gist.github.com/sharkdp/4bc3e5f5ea9df2f29c02ede50634...
- xfactor973 9y agoAwesome. Sorry I should’ve dug a little deeper before commenting. Nice job on the multithreading also!
- dbdr 9y ago> using bench for some nice statistics 'bench' turns out to be this tool: https://github.com/Gabriel439/bench https://github.com/Gabriel439/bench It seems to be quite useful, and I was not aware of it, thanks! Would probably be nice to have it packaged in distributions...
- the8472 9y agofor a narrow set of usecases with cold page caches I have written ffcnt[0]. As the name says I mostly use it to count how many million files have accumulated over time in a large directory tree. It could be extended to do regex matching on filenames a la find though, I just never had the need for it. [0] https://github.com/the8472/ffcnt https://github.com/the8472/ffcnt
- josteink 9y agoMy main usecase for find is not only finding files, but to do actions on them through -exec. Unless I’m missing something, that’s not supported with this tool.
- sharkdp 9y agoThank you for the feedback! '-exec' is not supported at the moment. It's definitely something I would consider adding. We do have '-0'/'--print0' to separate results by NULL, so 'fd' can be used in combination with 'xargs'.
- lloeki 9y agoOr parallel, which is slightly trickier to implement than xargs/-exec.
- jitl 9y agoYou can easily compose this with `xargs` using the`-0 / --print0` option: fd -0 '\.log$' | xargs -0 tail
- tyingq 9y agoxargs does have limits on input length. On some platforms they aren't terribly high. Sound like -exec is potentially coming though. Edit: Swore I had gotten ARG_MAX complaints out of xargs in the past. Replies are correct though. I'm wrong here...xargs adjusts on the fly. Perhaps long ago, or some obscure use of xargs?
- benmarten 9y agoIt is incredibly fast! Thanks a lot for this great tool!
- sharkdp 9y agoThank you for the feedback! Most of the credit for the speed goes to the amazing Rust modules 'ignore' and 'regex', which are also used by ripgrep (https://github.com/BurntSushi/ripgrep https://github.com/BurntSushi/ripgrep).
- nickcw 9y agoDid you do a benchmark with `--threads 1`? I suspect most of the speedup is coming from that as the the listing of the files is likely to be IO bound (or at least bound by calling into the kernel to read the directory listings). That has certainly been my experience in the past when experimenting with this sort of thing, that more threads makes a lot of difference.
- sharkdp 9y ago> Did you do a benchmark with `--threads 1`? I did. You are right, multi-threading does not give a linear speed up, but it makes fd about a factor of three faster on my machine (8 virtual cores). With `--threads 1`, fd is on-par with 'find -iname', but still faster than 'find -iregex'.
- smegel 9y ago> Ignores patterns from your .gitignore, by default. Do you need to be in the current git directory for this to happen, or all git dirs that happen to be traversed? And are you using an NFA based regex engine?
- sharkdp 9y agoIt will work for all git directories that are encountered. This behavior can be disabled with the '-I' flag, if needed. fd uses Rusts regex engine (https://github.com/rust-lang/regex https://github.com/rust-lang/regex) which is based on finite automata.
- JepZ 9y agoI am not sure if I find that a smart default as I frequently exclude generated files from git repositories and when want to find something I do not know why I would not like to find such a file if my pattern matches it. I would prefer to let the switch enable the .gitignore logic, but as I don't know the authors use-case, theirs might be valid too.
- Sean1708 9y agoSurely the common case is that you don't want to search generated files? Or at least I personally am almost never interested in generated files because they're not the ones I'm going to be changing/doing stuff with.
- adwhit 9y agoripgrep, exa and now fd... any other CLI tools I should cargo-install?
- seagreen 9y agouna (https://github.com/jwiegley/una https://github.com/jwiegley/una), a universal unarchiver which handles choosing between tar, unzip, etc. for you.
- rahiel 9y agoYou can't install this with cargo. Una is a Haskell program; you can get it with `stack install una`.
- seagreen 9y agoWhoops, good point. In the long run I think Una's a good candidate for being installed at the systems level, because you really want to install unzip, etc. along with it. EDIT: The idea being that if you install it with your system package manager it can have unzip etc. as dependencies so that's taken care of automatically.
- raimue 9y agoYou could just use bsdtar from libarchive, which handles all the compression and archive formats automatically without additional tools. https://github.com/libarchive/libarchive/wiki/LibarchiveFormats https://github.com/libarchive/libarchive/wiki/LibarchiveForm... GNU tar is so limited, but unfortunately still the default on most systems.
- jnwatson 9y agoHow does it compare in speed to 'ag' (i.e the silver searcher) -g option?
- rjzzleep 9y agoI didn't even know that existed, thank you. ripgrep seems to have something similar: rg -g '*.foo' --files
- coldtea 9y agorg (ripgrep -- the rust-made grep/ag style tool) is already quite faster than ag.
- deleted 9y ago[deleted]
- burntsushi 9y agoI haven't benchmarked them, but fd has a recursive parallel directory iterator. ag doesn't. The mysql-server repository[1] should be a fun one to try out, because of this: $ wc -l .gitignore 3122 .gitignore The combination of fast glob matching[2] and parallel traversal should be a boon. [1] - https://github.com/mysql/mysql-server https://github.com/mysql/mysql-server [2] - https://github.com/BurntSushi/ripgrep/blob/e7c06b92fb996adcbc40d1f025e29c44de5383e8/globset/src/lib.rs#L494 https://github.com/BurntSushi/ripgrep/blob/e7c06b92fb996adcb...
- sharkdp 9y agofd is about a factor of 2-3 faster (for this example): > time fd -HIe jpg > /dev/null 5,12s user 2,03s system 785% cpu 0,911 total > time ag --depth 100 -uuu -g '\.jpg$' > /dev/null 0,95s user 1,66s system 99% cpu 2,628 total
- tayo42 9y agoWhy is this faster then find?
- tyingq 9y ago"Concerning fd's speed, the main credit goes to the regex and ignore crates that are also used in ripgrep"
- sharkdp 9y agoYes. For simple searches, the main reason is that 'fd' walks the directory tree in a multithreaded fashion (thanks to the 'ignore' crate).
- tayo42 9y agoShould have been more clear heh. There was a benchmark that didn't use regex and seemed to turn off the ignore files feature. Im not familiar with rust or those crates Im assuming that's what those are there for?
- deleted 9y ago[deleted]
- coldtea 9y agoIt would be nice to have a collection of modern/faster/saner alternatives to common unix tools and such utilities (made in rust or not) that becomes standard -- or at least easily installable in linux distros and OS X. Not for replacing those tools, but for supplementing them. Thinking of stuff like fd and: ripgrep: https://github.com/BurntSushi/ripgrep https://github.com/BurntSushi/ripgrep (grep/ag alt) xsv: https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv (csv tool) exa: https://the.exa.website/ https://the.exa.website/ (ls) una: https://github.com/jwiegley/una https://github.com/jwiegley/una (multi-compression utils wrapper) tokei: https://github.com/Aaronepower/tokei https://github.com/Aaronepower/tokei (loc stats) And of course this: https://github.com/uutils/coreutils https://github.com/uutils/coreutils
- deleted 9y ago[deleted]
- deleted 9y ago[deleted]
- empath75 9y agoRedox already has a core untils library.
- kuwze 9y agovgrep: https://github.com/fmthoma/vgrep https://github.com/fmthoma/vgrep (Haskell vertical grep)
- nerdponx 9y agoDoes the Fish shell count? Consider also the "fuzzy finder" FZF.
- StavrosK 9y agoThe fish shell is the only one that counts. Fish is life.
- bm1362 9y agoI appreciate all these new incarnations of old school tools but end up sticking with the basics regardless. When I'm sshing into a box to figure out whats going on, my toolset is mostly limited to top, find, xargs, awk, grep, df, du etc. Even local development now, I'm mostly debugging on a docker container running alpine or ubuntu. Knowing the right incantations is useful in that context and keeps me from installing better tools until they become part of the distro we deploy with.
- dau 9y agoWith the obligatory sneaky Rust advertisement in the title, my bullshit detector rings all its bells. And once more it was right.
- acdha 9y agohttps://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html “Be civil. Don't say things you wouldn't say face-to-face. Don't be snarky. Comments should get more civil and substantive, not less, as a topic gets more divisive.” Calling something bullshit is both unwarranted in this case and unfair to the person whose work you’re dismissing. They built something and gave it to you for free – if you’re not interested just move along.
- tree_of_item 9y agoWhat exactly do you think was bullshit here?
- sidlls 9y agoThe fact that Rust is used is uninteresting, considering there's nothing special about it as it pertains to the other advertised features. "Bullshit" might be strong, but "trivia relevant only to a person interested in fomenting language flamewars" isn't a stretch.
- rootlocus 9y ago> The fact that Rust is used is uninteresting, considering there's nothing special about it as it pertains to the other advertised features. Considering the fact that a programming language's appeal is directly influenced by the size and accomplishments of its community, I'd say it's relevant. A high level, systems programming, relative new language is used to write software that outperforms the default tools. That's proof that the language has potential and encourages people to try it out. It's how languages and communities grow. If you think it's uninteresting, that's your subjective opinion, and you're free to dismiss it. It certainly doesn't make the language, the project or the title "bullshit" or "trivia relevant only to [...] language flamewars".
- wruza 9y agoAny plan to provide find compatibility for drop-in replacement? My find/locate usage patterns are not worth learning new sophisticated tool for 5sec speedup once a week, and I suspect that I’m not alone. This is the main problem of new tools — not repeating the old-familiar style makes it unattractive for non-aggressive users. Side question: why is speed important? Which tools use fd to gain required performance?
- MoBattah 9y agoExactly. Especially once you've come to a good understanding with a standard *nix tool/command.
- romgrk 9y agoThe other big advantage is `more user-friendly`. Thus no backward compatibility. No doubt some people are used to the old patterns, but lots of people also prefer saner defaults.
- wruza 9y agoI couldn’t use neither thing without man page for the first time. And remembering -name -iname -type* and -I -s comes only with practice. * does fd find files by type, mtime, depth, logical operators? I missed that in readme. If not, maybe “just pattern” argument is not too fair at all.
- walkingolof 9y agoI agree, but replacing find would be very hard, not only does people except it to work in a certain way, but tons of shell scripts also.
- deleted 9y ago[deleted]
- foota 9y agoIt's probably the "I'm searching the entire disk for something" use case.
- 9y ago
- z1mm32m4n 9y agoI've found that fzf has replaced all uses of find for me except in scripts. fzf has the benefits of being bound to a single key binding and showing me results as I type, rather than after the command runs.
- pmoriarty 9y agofzf is fine for searching just for filenames, but it does nothing else that find does. It can't even search specifically for directories, as far as I can see, nevermind searching for files/dirs with certain permissions, ages, etc. It's also uselesss for non-interactive uses such as scripting. I like fzf, but it's really not anywhere close to a complete find replacement.
- sa46 9y agoYou're right that fzf doesn't replace find. Fzf is just an interactive, filtering tool. You can pipe anything into it. find is a common way of populating it with data. I suspect the GP meant that fzf replaced the pattern 'find |xargs $binary' for small interactive use cases. It's much nicer to do '$binary <invoke fzf>'. I use fzf the same way with key bindings to select directories and files powered by find.
- igemnace 9y agoIt's not a find replacement at all, even! fzf just actually calls find by default, so a naked call to fzf will actually give you the same results as find | fzf. Search specifically for directories, that's find -type d | fzf.
- jpz 9y agothanks v much for recommendation
- majewsky 9y agoIt's strange to advertise this as a find(1) replacement when it covers maybe 1% of find(1) use cases. find -type d -empty -delete find -type l -exec readlink -f {} + etc. are not covered at all.
- raverbashing 9y ago1% of possible cases but 90% of actual usage I'd say
- majewsky 9y agoThen you have a different usage pattern than I do. For simple name globbing, I usually just use shell globs, e.g. ls src/**/*_test.go It might be marginally slower than find or FD, but it usually doesn't matter because I usually deal with a small number of files.
- lkbm 9y agoOh, [star][star]! What I usually end up doing is: 1. ls [star]/foo 2. ls [star]/[star]/foo 3. ls [star]/[star]/[star]/foo 4. "I guess it doesn't exist." [star][star] would have been useful to know. But now I have fd, so that's nice. :-) Edit: Silly markdown.
- raverbashing 9y agoI just tried this and: ls (some dir)/**/*.py zsh: argument list too long Yeah, I'll keep using find
- lkbm 9y ago100% of the time I consider using find, what I want is "dir/s foo". Step 1 is decide whether it's worth re-looking-up how to do it. I rarely remember--sometimes I think the man page will be helpful. It's not. ("man find" mentions "-name", which gives me an "illegal option" error. Maybe it's "--name"? Nope. Maybe it's "-n"? Nope. Maybe it's "-f"? Oooh, you have to do path and then add in options, unless those options are -H, -L, -P, -E, -X, -d, -s, or -x, in which case they go before the path? Of course, how silly of me.) "fd filename" does exactly what I want, and what I'd expect. Gave me what I wanted on the first try. For my purposes, find is fully replaced, with something actually usable. Small, simple utils are nifty.
- Karrot_Kream 9y agoWill we ever be able to promote a tool without talking about the programming language it is implemented in?
- dysoco 9y agoYeah when it's written in Python or C and not Go/Rust.
- mmirate 9y agoAye, that's because being written in Python, C or Go is - in my eyes - a strike against the project in an absolute sense, because it will be more buggy and difficult to maintain due to lacking the functional-programming abstractions and powerful static-type system of a good ML descendant such as Rust or OCaml. (Or of Haskell, but it tends to lack the practicality of the ML family.) So of course one does not trumpet that one's project is written in Python, C or Go. n.b. I said "absolute sense" because all of this this is, of course, inapplicable when searching for libraries specifically for a language such as Python, C, or Go.
- burntsushi 9y agoI don't buy it. People say, "X (written in Foo)" because there is fundamentally an interest in the fact that something is written in Foo (and this may indeed depend on the nature of X). Back when Go was new and shiny, it was the same situation.[1] :-) [1] - https://news.ycombinator.com/item?id=4685594 https://news.ycombinator.com/item?id=4685594
- mmirate 9y ago> there is fundamentally an interest in the fact that something is written in Foo (and this may indeed depend on the nature of X). That speaks to a deficit in human cognition, that slapping Foo's name on something makes it interesting only based on Foo's "shininess" rather than Foo's technical merits. Neophilia, perhaps? Anyway. Go has been in roughly the same state of "another purposefully-boring Java-like, but instead of generics it has a different shade of OO and a full gamut of integer machine-types" for its entire life thusfar, which is why I've never understood most of its hype. Rust, on the other hand, actually has some serious motivation to its learning curve and its developing ecosystem.
- fuzzygroup 9y agoYes I'll agree with the people who say that this isn't a drop in replacement (fewer features and different syntax) but I also just don't care. fd's defaults are brilliantly simple and just plain make sense. Yes I can use find but I always, always end up looking up an example. fd was simple enough that I very quickly figured it out by trial and error. And it really is fast. Thank you sharkdp -- really nicely done. Appreciated.
- sharkdp 9y agoThank you for the feedback, I'm glad you like it!
- falcolas 9y agoI'm sorry to be something of a negative Nancy - and I'm sure I'm a corner case - but this is not really an alternative to find. It's an alternative for your editor's fuzzy find and a better version of shell globs. The absense of -delete, -execdir, -mtime, and the ability to chain multiple patterns together for the non-trivial use-cases, means this is practically useless in most places where `find` is used in day-to-day work. Not to mention the "opinionated" choice to ignore a dynamic set of files and directories (what files were skipped? In any directory, or just git repos? Does it back track to find a git directory and .gitignore? Does it query out to git? Does it respect user/global .gitignore settings? Does it skip Windows hidden files from a unix prompt, or a dotted file when on Windows?), the options hidden behind two separate flags. Perhaps it's just because I'm used to using 'find', but when I reach for it, it's because I need -delete or -execdir, or I'm writing automation that really needs -mtime and chained patterns. So, I would suggest that you don't call this an alternative to find; it's not. A replacement for shell globs, sure. A replacement for poor file finding in text editors, OK. Just... not find. EDIT: Oh, 'fd' also already exists. https://github.com/knu/FDclone https://github.com/knu/FDclone
- zeotroph 9y ago> Not to mention the "opinionated" choice to ignore a dynamic set of files and directories That has bitten me a few times when using `rg`. Sometimes (but not often enough to remember the switch) I want to grep a lot of binary .so files for a string and am left wondering why nothing was found before just replacing rg with grep. Just calling `TOOL PATTERN` without switches is the most intuitive thing, and there I agree with `fd` to make `fd PATTERN` the default. Unlike BSD find which always needs at least a `.` to function. > chained patterns. The -or/-and and are indeed nice, but the -prune logic should get an update. Maybe when fd grows from 80 to 90% of use cases.
- burntsushi 9y agoIf you're grepping binary files, then even with grep, you need to pass the `-a/--text` flag. ripgrep has the exact same flag. Protips: `rg -u` stop looking at `.gitignore` files. `rg -uu` searches hidden files/dirs. `rg -uuu` searches binary files.
- hzhou321 9y agoFor me, the speed of find and grep is sufficiently never a concern, so I do not find these speedy alternatives sufficient to compensate for its non-universality. And when the speed of grep and find do become a concern, I would think the issues are somewhere else -- like have 10s thousands of jpg files scatter everywhere that you need to find in the first place. When issues are somewhere else, replacing it with better tool only hides and makes the problem worse.
- hehno 9y agoI'm honestly wondering when will we have the whole coreutils rewritten in Rust.
- dbaupp 9y agohttps://github.com/uutils/coreutils https://github.com/uutils/coreutils is a long way along.
- oblio 9y agoThe docs don't say: is this meant to replace Busybox or the actual GNU coreutils? As in, all GNU features and flags (which are generally much more comprehensive and IMO nicer than plain old POSIX features).
- dbaupp 9y agoI do think it (optionally) provides a busybox-esque single-binary interface, but I do not know whether this also translates into a busybox-esque command-line interface.
- wheresmyusern 9y agohey david, you can omit the 'static lifetime in the root directory string declarations if you want! https://github.com/rust-lang/rfcs/pull/1623 https://github.com/rust-lang/rfcs/pull/1623
- sharkdp 9y agoThank you for the hint! Unfortunately, this is not in rust 1.16, which I currently still want to support.
- alvil 9y agoPet: https://github.com/knqyf263/pet https://github.com/knqyf263/pet and Peco: https://github.com/peco/peco https://github.com/peco/peco
- Dowwie 9y agoIt's exciting to see so many problem solvers use Rust to improve core linux utilities. I've been using Exa as an 'ls' replacement for some time now. 'fd' looks promising but has room to grow, still. Finding a file and then acting upon it is an important feature that ought to be addressed.
- reacweb 9y agoIMHO, there is no demand for this kind of "improved tools collection". In the programming world, the programmer creates a sweet working environment to fit his needs by creating many alias and functions, by customizing his desktop. He complains that the tools collection is complex but has no opportunity to progress because he rarely uses them directly. In the "sysop" world, the user is always logged remotely on varying machines. There is no incentive to install and customize a machine because he will quickly have to work on another machine without this customization. I think we need a saner alternative to the man pages of the classic unix tools collection. Something less focus on the seldom used options, but more focus on the traps to avoid and the useful tricks. The bad quality of documentation is more annoying than the incongruous syntax of tools.