5 ms·
Speaking of faster coreutils replacements, I highly recommend ripgrep (rg) and fd-find (fd), which are Rust-based, incompatible replacements for grep and find.
by perihelions 5y ago
Speaking of faster coreutils replacements, I highly recommend ripgrep (rg) and fd-find (fd), which are Rust-based, incompatible replacements for grep and find.
I know I'm way behind the popularity curve* (they're already in debian-stable, for crying out loud); but for the benefit of anyone even more out of the loop than myself, do check these out. The speed increase is amazing: a significant quality-of-life improvement, essentially for free**. Particularly if you're on a modern NVME SSD and not utilizing it properly. (Incidentally, did you know the "t" in coreutil's "tar" stands for magnetic [t]ape?")
* (The now-top reply to this comment says 'ag' is even superior to 'rg', and they're probably right, but I had no clue about it! I did say I'm ignorant!)
**(Might have some cost if you're heavily invested in power-user syntax of the GNU or BSD versions, in which case incompatibility has a price).
https://github.com/BurntSushi/ripgrep https://github.com/BurntSushi/ripgrep
https://github.com/sharkdp/fd https://github.com/sharkdp/fd
- chx 5y agoWhen I looked ag https://github.com/ggreer/the_silver_searcher https://github.com/ggreer/the_silver_searcher was more featureful than ripgrep yet it's always ripgrep that is mentioned. :/
- rscnt 5y agoweird right? I don't know what ripgrep has, maybe it's just the name? the fact that you call `ag` but it's called `the_silver_searcher`?
- pdimitar 5y agoFrom my side, I knew about both `ripgrep` and `the_silver_searcher` but I will openly admit I've lost faith in C and C++'s abilities to protect against memory safety errors. Thus, Rust tools get a priority, at least for myself. There are probably other memory-safe languages but I haven't had the chance to give them a spin like I did with Rust. If I found out about them then I'll also prefer tools written in them if there's no Rust port and if the alternatives are C/C++ tools.
- chx 5y agoThis is what I suspected: Rust zealotry. We are searching for a string. Language matters to the guy writing it but the user? Nah. There's no security hole here.
- pdimitar 5y agoWho's the zealot here? The guy who doesn't want to risk and openly states so, or the guy proclaiming there's no risk, even with ample historical evidence for the opposite? I don't accept your labeling, especially when it's so egregiously misguided. Also, you're clearly seeing what you want to see.
- jhoechtl 5y agoAg is shorthand for argentum which in turn means silver. https://en.wikipedia.org/wiki/Silver https://en.wikipedia.org/wiki/Silver
- saagarjha 5y agoCode search on your own codebase is generally not a place where memory safety shines.
- pdimitar 5y agoWhen it's about security, I try not to pick and choose. Plus a memory error might lead to scary consequences (like a random script getting elevated privileges). Finally, and last I checked, `rg` is quite featureful and I've never felt constrained by it. So for me it's a win all around.
- saagarjha 5y agoThis is kind of my point: security requires a threat model, and you don't really have one. Rust has a lot going for it, and it does hold promise in improving the security of a lot of critical software. But in this case, it's not really doing that, so it's kind of misleading to say it is meaningfully doing anything for security.
- timerol 5y agoripgrep is noticeably faster than ag. I'm not sure what features you mention that ripgrep is missing, but I've been plenty happy with it for basic grepping around. I'm sure it's also partly because BurntSushi, the author of ripgrep, is reasonably active here. Some fun benchmarks, and a fair description of how text search works, can be found in this 2016 post https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/
- chx 5y ago> ripgrep is noticeably faster than ag. Since both are instant on every codebase I care about, not so sure about that. These tools were always chiefly I/O bound , SSDs have torn that down especially now. But for features: there's no lookaround in the regex.
- aidenn0 5y agoThese tools are I/o bound, but not disk bound. I often perform multiple searches in a row, and all but the first will hit the disk cache.
- burntsushi 5y agoThey are only I/O bound when searching data that isn't in cache. It's often the case that you're searching, for example, a repository of code repeatedly. > Since both are instant on every codebase I care about, not so sure about that. It is very possible that there is no performance difference between ripgrep and ag for your use cases, but that does not mean there isn't a performance difference between ripgrep and ag. For example, in my checkout of the chromium repository: $ time rg -c Openbox testing/xvfb.py:2 tools/metrics/histograms/enums.xml:1 ui/base/x/x11_util.cc:1 real 0.448 user 2.593 sys 2.490 maxmem 77 MB faults 0 $ time ag -c Openbox ui/base/x/x11_util.cc:1 tools/metrics/histograms/enums.xml:1 testing/xvfb.py:2 real 2.302 user 2.996 sys 10.462 maxmem 15 MB faults 0 > But for features: there's no lookaround in the regex. That's not true and hasn't been true for a long time. ripgrep supports PCRE2 with the -P/--pcre2 flag. You can even put `--engine auto` in an alias or ripgreprc file and have ripgrep automatically select the regex engine based on whether you're using "fancy" features or not. In general, I also claim that ripgrep has far fewer bugs than ag. ag doesn't really get gitignore support correct, although if you only have simple gitignores, its support might be good enough.
- burntsushi 5y agoIt's not weird because ripgrep has had more features than ag for a long time. Originally ripgrep didn't have multi-line support or support for fancy regex features like look-around, but it has both of those now. It also has support for automatic UTF-16 transcoding, preprocessors for searching non-text files and overall less buggy support for gitignore. (Look at ag's issue tracker.) And then there's also the fact that ag isn't that great as a general purpose grep tool. It really falls over in terms of speed when searching, say, bigger files: $ time rg 'Sherlock Holmes' OpenSubtitles2018.raw.en | wc -l 7673 real 1.475 user 1.115 sys 0.356 maxmem 12511 MB faults 0 $ time ag 'Sherlock Holmes' OpenSubtitles2018.raw.en | wc -l 7673 real 20.276 user 19.850 sys 0.413 maxmem 12508 MB faults 0 A lot of people like to comment and say, "well ag is fast enough for me." Well, OK, that's fine. But if you're wondering about why other people might mention ripgrep more, well, maybe it isn't just about the way you use the tools. For example, if you only ever search tiny repositories of code, then you aren't going to care that ripgrep is faster than ag. Which is perfectly reasonable, but it should also be reasonable to be aware that others might actually search bigger corpora than you.
- chx 5y agoripgrep regexp has lookaround? The linked docs still doesn't say that. https://docs.rs/regex/1.5.4/regex/#syntax https://docs.rs/regex/1.5.4/regex/#syntax > This crate provides a library for parsing, compiling, and executing regular expressions. Its syntax is similar to Perl-style regular expressions, but lacks a few features like look around and backreferences. Edit: I see. It can use pcre2 but it's a build time option which of course Ubuntu has off.
- burntsushi 5y ago> Edit: I see. It can use pcre2 but it's a build time option which of course Ubuntu has off. That's unfortunate. Perhaps a bug report to the packagers is in order? Archlinux enabled it: https://github.com/archlinux/svntogit-community/blob/0dc033f5d0a476671dc0971a02eb9baa51cabc73/trunk/PKGBUILD#L26 https://github.com/archlinux/svntogit-community/blob/0dc033f...
- ziml77 5y agoI still haven't seen anyone elaborate on what features ag has over ripgrep. Can you explain what you like about ag that ripgrep currently doesn't do?
- burntsushi 5y agoWhich features does it have that ripgrep doesn't?
- hakre 5y agoOkay, call me lazy. Not for ripgrep, installed it early from sources. However your fdfind (fd) mention made me curious. Thought I give it an apt install on Ubtunu 20.04 LTS but dead end. So perhaps in debian-stable but not in Ubuntu. Just saying, so I can feel less out of the loop ;)
- perihelions 5y agoUpstream says Ubuntu's package is 'fd-find' (and the executable is 'fdfind' -- both renamed from upstream's 'fd' because of a name collision with an unrelated Debian package. If you don't care about that one, you can safely alias fd="fdfind"). https://github.com/sharkdp/fd https://github.com/sharkdp/fd (I've edited my first comment in response to this reply: I originally wrote "fdfind". (For a comment about regexp tools, this is a uniquely, hilariously stupid oversight. Sorry!)).
- hakre 5y agoWell, even its late here, I could still have enough creative energy to insert the minus in there ... yeah, thanks for that. So now I feel really behind because sure, it's packaged in Ubuntu 20.04. Thanks again.
- sramam 5y agoIn an attempt to remove some of my ignorance, I tried this: (MBA, M1) time rg Query src > /tmp/junk.rg rg Query src > /tmp/junk.rg 0.01s user 0.04s system 156% cpu 0.037 total time pt Query src > /tmp/junk.pt pt Query src > /tmp/junk.pt 1.44s user 10.80s system 220% cpu 5.545 total time ag Query src > /tmp/junk.ag ag Query src > /tmp/junk.ag 4.32s user 9.82s system 204% cpu 6.918 total ls -l /tmp/junk\* -rw-r--r-- 1 sramam wheel 16492080 Nov 10 16:05 /tmp/junk.ag -rw-r--r-- 1 sramam wheel 16492080 Nov 10 16:04 /tmp/junk.pt -rw-r--r-- 1 sramam wheel 27477 Nov 10 16:04 /tmp/junk.rg This is a TypeScript module, turns out ripgrep correctly ignores the sourcemaps for the improved performance. I like tools that are smart by default. (EDIT: formatting)
- dan-robertson 5y agoI think rg isn’t ignoring sourcemaps to be fast. It tries to ignore based on .gitignore and similar files, but maybe the sorcemaps are ignored for being binary.
- sramam 5y agoI thought so too. The documentation for all state they honor .gitignore. In my case, the compiled code is committed. My query was for a generic term "src" - so that likely trapped the file-path embedded within the long lines of a source-map. Hence the massive file-size difference. (I did inspect the files to validate)
- burntsushi 5y agoThe main goal of respecting gitignore is generally to decrease noise in search results. But, this does of course also have the effect of improving performance in a lot of cases.
- nextaccountic 5y agoThere is also ouch, a fast multi-format compressor/decompressor written in Rust https://github.com/ouch-org/ouch https://github.com/ouch-org/ouch