11 ms·

Ugrep – a more powerful, fast, user-friendly, compatible grep

344 points by smartmic 3y ago

infamia 3y ago
Ugrep is also available in Debian based repos, which is super nice.
dmlerner 3y ago
Why not ripgrep?
- devraza 3y ago
  From the ugrep README: For an up-to-date performance comparison of the latest ugrep, please see the ugrep performance benchmarks [at https://github.com/Genivia/ugrep-benchmarks https://github.com/Genivia/ugrep-benchmarks]. Ugrep is faster than GNU grep, Silver Searcher, ack, sift. Ugrep's speed beats ripgrep in most benchmarks.
  - codetrotter 3y ago
    Does these performance comparison take into account the things BurntSushi (ripgrep author) pointed out in the ripgrep issue link elsewhere ITT? https://github.com/BurntSushi/ripgrep/discussions/2597 https://github.com/BurntSushi/ripgrep/discussions/2597 Either way, ripgrep is awesome and I’m staying with it.
    - devraza 3y ago
      Agreed - ripgrep is great, and I'm not planning to switch either. The performance improvement is tiny, anyways.
- 0cf8612b2e1e 3y ago
  I assume the grep compatible bit is attractive to some people. Not me, but they exist.
  - derriz 3y ago
    I find myself returning to grep from my default of rg because I'm just too lazy to learn a new regex language. Stuff like word boundaries "\<word\>" or multiple patterns "$one\|two$".
    - masklinn 3y ago
      That seems like the weirdest take ever: ripgrep uses pretty standard PCRE patterns, which are a lot more common than posix’s bre monstrosity. To me the regex langage is very much a reason to not use grep.
      - pbhjpbhj 3y ago
        `pgrep`, or `grep -P`, uses PCRE though, AFAIUI.
        deleted 3y ago
        [deleted]
      - derriz 3y ago
        A bit hyperbolic, no? If you consider it "the weirdest ever", I'm guessing that I'm probably older than you. I've certainly been using regex long before PCRE became common. As a vim user I compose 10s if not 100s of regexes a day. It does not use PCRE. Nor does sed, a tool I've been using for decades. Do you also recommend not using these?
        comex 3y ago
        I use all of those tools but the inconsistency drives me crazy as it's hard to remember which syntax to use where. Here's how to match the end of a word: ripgrep, Python, JavaScript, and practically every other non-C language: \b vim: \> BSD sed: [[:>:]] GNU sed, GNU grep: \> or \b BSD grep: \>, \b, or [[:>:]] less: depends on the OS it's running on
- meindnoch 3y ago
  Because this is faster?
- bsdpufferfish 3y ago
  ripgrep stole the name but doesn’t follow the posix standard.
- Conscat 3y ago
  The best practical reason to choose this is its interactive features, like regexp building.
- tredre3 3y ago
  One thing I never liked about ripgrep is that it doesn't have a pager. Yes, it can be configured to use the system-wide ones, but it's an extra step (and every time I have to google how to preserve colors) and on Windows you're SOL unless you install gnu utils or something. The author always refused to fix that. Ugrep not only has a pager built in, but it also allows searching the results which is super nice! And that feature works on all supported platforms!
  - burntsushi 3y ago
    This is what I do personally: $ cat ~/bin/rgp #!/bin/sh exec rg -p "$@" | less -RFX Should work just fine. For Windows, you can install `bat` to use a pager if you don't otherwise have one. You don't need GNU utils to have a pager.
    - anjanb 3y ago
      hi @burntsushi, fan of your tool. like it's speed and defaults. I use windows : didn't understand what you mean by "install `bat`" to use a pager. I use cygwin and WSL for my unix needs. I have more and less in cygwin for use in windows.
      - burntsushi 3y ago
        I referenced bat because I've found that suggesting cygwin sometimes provokes a negative reaction. The GP also mentioned needing to install GNU tooling as if it were a negative. bat is fancy pager written in Rust. It's on GitHub: https://github.com/sharkdp/bat https://github.com/sharkdp/bat
        anjanb 3y ago
        I'm sure you know but windows command prompt always came with its inbuilt pager -- more. So, you could always do "dir | more" or "rg -p "%*" | more ". (more is good with colors without flags)
- GuB-42 3y ago
  Why not ugrep? They are more or less equivalent. One has obscure feature X other has obscure feature Y, one is a bit faster on A, other is a bit faster on B, the defaults are a bit different, and one is written in Rust, the other in C++. Pick the one you like, or both. I have both on my machine, and tend to use the one that does what I want with the least options. I also use GNU grep when I don't need the speed or features of either ug and rg.
- MrDrMcCoy 3y ago
  For me, it's a lot easier to compile a static binary of a C++ app than a Rust one. Never got that to work. Also nice to have compatibility with all of grep's arguments.
  - datadeft 3y ago
    > to compile a static binary Cargo is one of the main reasons to use Rust of C++. I am pretty sure there is more involved with C++ than this: rustup target add x86_64-unknown-linux-musl cargo build --target=x86_64-unknown-linux-musl
- jedisct1 3y ago
  Fuzzy matching is the main reason I switched to ugrep. This is insanely useful.
- philkrylov 3y ago
  Although being faster in some cases, ripgrep lacks archive search support (no, transparent decompression ignoring the archive structure is not enough) which works great in ugrep.
graphe 3y ago
Here's a thread on performance vs rg (ripgrep). https://github.com/BurntSushi/ripgrep/discussions/2597 https://github.com/BurntSushi/ripgrep/discussions/2597 didn't know about hypergrep either.
- infocollector 3y ago
  I think you should try it before you read these conflicting benchmarks from the authors: https://github.com/Genivia/ugrep-benchmarks https://github.com/Genivia/ugrep-benchmarks
- ashvardanian 3y ago
  Haven't benchmarked *grep implementations, but assuming those are just CLI wrappers around RegEx libraries, I'd expect the RegEx benchmarks to be broader and more representative. There, hyperscan is generally the king, which means hypergrep numbers are likely accurate: https://github.com/p-ranav/hypergrep?tab=readme-ov-file#directory-search-usr https://github.com/p-ranav/hypergrep?tab=readme-ov-file#dire... Disclaimer: I rarely use any *grep utilities, but often implement string libraries.
  - burntsushi 3y ago
    I'm the author of ripgrep and its regex engine. Your claim is true to a first approximation. But greps are line oriented, and that means there are optimizations that can be done that are hard to do in a general regex library. You can read more about that here: https://blog.burntsushi.net/ripgrep/#anatomy-of-a-grep https://blog.burntsushi.net/ripgrep/#anatomy-of-a-grep (greps are more than simple CLI wrappers around a regex engine). If you read my commentary in the ripgrep discussion above, you'll note that it isn't just about the benchmarks themselves being accurate, but the model they represent. Nevertheless, I linked the hypergrep benchmarks not because of Hyperscan, but because they were done by someone who isn't the author of either ripgrep or ugrep. As for regex benchmarks, you'll want to check out rebar: https://github.com/BurntSushi/rebar https://github.com/BurntSushi/rebar You can see my full thoughts around benchmark design and philosophy if you read the rebar documentation. Be warned though, you'll need some time. There is a fork of ripgrep with Hyperscan support: https://sr.ht/~pierrenn/ripgrep/ https://sr.ht/~pierrenn/ripgrep/ Hyperscan also has some preculiarities on how it reports matches. You won't notice it in basic usage, but it will appear when using something like the -o/--only-matching flag. For example, Hyperscan will report matches of a, b and c for the regex \w+, where as a normal grep will just report a match of abc. (And this makes sense given the design and motivation for Hyperscan.) Hypergrep goes to some pain to paper over this, but IIRC the logic is not fully correct. I'm on mobile, otherwise I would link to the reddit thread where I had a convo about this with the hypergrep author.
mathverse 3y ago
Also look at https://github.com/stealth/grab https://github.com/stealth/grab from Sebastian Krahmer.
- meowface 3y ago
  ripgrep, grab, ugrep, hypergrep... Any of the four are probably fast enough for any of my use cases but I suddenly feel tempted to micro-optimize and spend ages comparing them all.
jraph 3y ago
Okay, this solves a feature I was occasionally missing for a long time: searching for several terms in files (the "Googling files" feature). I wrote a 8 line script a few weeks ago to do this, that I will gladly throw away. I'll look into the TUI too. (I've been using ripgrep for quite some time now, how does this otherwise compare to it? would I be able to just replace rg with ug?)
- Levitating 3y ago
  Is it that different from using fzf? I currently use ripgrep-all (which can search into anything, video captions or pdfs) and fzf.
- bishfish 3y ago
  ugrep+ has this feature similar to ripgrep-all. For regular use, I use ugrep’s %u option with its format feature to only get one match per line same as other grep tools. Overall, I’m a happy user of ugrep. ugrep works as well as ripgrep for me. It’s VERY fast and has built-in option to search archives within archives recursively.
o11c 3y ago
Important note: not actually compatible. It took me seconds to find an option that does something completely different than the GNU version.
- fwip 3y ago
  Which option is that? I'm scanning the ugrep page, but nothing is popping out to me.
- e12e 3y ago
  I would assume compatible meant posix/bsd - unless explicitly advertised AS "GNU grep compatible"?
  - burntsushi 3y ago
    From the OP: "Ugrep is compatible with GNU grep and supports GNU grep command-line options."
- burntsushi 3y ago
  Indeed. And here are some concrete examples around locale: $ grep -V | head -n1 grep (GNU grep) 3.11 $ alias ugrep-grep="ugrep-4.4.1 -G -U -Y -. --sort -Dread -dread" $ echo 'pokémon' | LC_ALL=en_US.UTF-8 grep 'pok[[=e=]]mon' pokémon $ echo 'pokémon' | LC_ALL=en_US.UTF-8 ugrep-grep 'pok[[=e=]]mon' $ echo 'γ' | LC_ALL=en_US.UTF-8 grep -i 'Γ' γ $ echo 'γ' | LC_ALL=en_US.UTF-8 ugrep-grep -i 'Γ' BSD grep works like GNU grep too: $ grep -V grep (BSD grep, GNU compatible) 2.6.0-FreeBSD $ echo 'pokémon' | LC_ALL=en_US.UTF-8 grep 'pok[[=e=]]mon' pokémon $ echo 'γ' | LC_ALL=en_US.UTF-8 grep -i 'Γ' γ
MightyBuzzard 3y ago
[dead]
ww520 3y ago
Just tried it out. It's blazingly fast. The interactive TUI search is pretty sweet.
comex 3y ago
Interesting, it supports an n-gram indexer. ripgrep has had this planned for a few years now [1] but hasn't implemented it yet. For large codebases I've been using csearch, but it has a lot of limitations. Unfortunately... I just tried the indexer and it's extremely slow on my machine. It took 86 seconds to index a Linux kernel tree, while csearch's cindex tool took 8 seconds. [1] https://github.com/BurntSushi/ripgrep/issues/1497 https://github.com/BurntSushi/ripgrep/issues/1497
- dtgriscom 3y ago
  That's close to a gig of disk reads; I trust you didn't try ugrep first and then cindex second, without taking into account caching.
  - comex 3y ago
    I ran both multiple times, alternating (and making sure to clean out the indexes in between). Results were reasonably consistent across runs.
- jgalt212 3y ago
  If you're gonna go the csearch route, you should also consider hound. I use it many times per day. https://github.com/hound-search/hound https://github.com/hound-search/hound
- bishfish 3y ago
  It creates per-directory index files on its first run. ugrep-indexer is also labeled as beta. A couple of relevant quotes from its GitHub site: “Indexing adds a hidden index file ._UG#_Store to each directory indexed.” “Re-indexing is incremental, so it will not take as much time as the initial indexing process.”
joshka 3y ago
There's a few ripgrep based tuis: - https://github.com/acheronfail/repgrep https://github.com/acheronfail/repgrep - https://github.com/konradsz/igrep https://github.com/konradsz/igrep - https://github.com/seg-mx/grep_tui https://github.com/seg-mx/grep_tui - https://github.com/Robertleoj/grepedit https://github.com/Robertleoj/grepedit
- nsagent 3y ago
  You can also use fzf with ripgrep to great effect: [1]: https://github.com/junegunn/fzf/blob/master/ADVANCED.md#using-fzf-as-interactive-ripgrep-launcher https://github.com/junegunn/fzf/blob/master/ADVANCED.md#usin...
stevebmark 3y ago
There are many grep variations. The Unix philosophy: do one thing well. The Unix reality: do many things poorly* *grep, awk, sed
jedberg 3y ago
I feel like if you're going to make a new grep and put a web page for it, your webpage should start with why your grep is better than the default (or all the other ones). Why did you build a new grep?
- deleted 3y ago
  [deleted]
- infamia 3y ago
  > I feel like if you're going to make a new grep and put a web page for it, your webpage should start with why your grep is better than the default (or all the other ones). No snark here, but is the subtitle not enough to start? "a more powerful, ultra fast, user-friendly, compatible grep"
  - fsckboy 3y ago
    no snark here, but the subtitle was the start of my confusion: what does "user-friendly" mean in the context of grep, and why should I believe the claim? regular expressions are not friendly, but the user friendly way for a cli filter to behave is to return retvals appropriately, output to stdout, error messages to stderr... does user friendly mean copious output to stderr? what else could it possibly mean? do I want copious output to stderr?
    - infamia 3y ago
      > no snark here, but the subtitle was the start of my confusion: what does "user-friendly" mean in the context of grep, and why should I believe the claim? Granted, it is far from a thing of beauty, but there is a large, captioned screenshot of the included text user interface just beneath. Then again, it is a website for a command line tool. "Many Bothans died to bring us this information."
  - jedberg 3y ago
    Not really. * a more powerful -- This is meaningless without some sort of examples. Powerful how? What does it do that's better than grep? * ultra fast -- This at least means something, but it should be quantified in some way. "50%+ faster for most uses cases" or something like that. * user-friendly -- not even sure what this means. Seems kind of subjective anyway. I find grep plenty user friendly, for a command line tool. * compatible grep -- I mean, they all are pretty much, but I guess it's good to know this?
Sparkyte 3y ago
Cool, but in a real life scenario where the system is not able to pull from external packages because it is in a secured environment makes myself think this is moot as you'll be out of practice of actually running grep. I would avoid not staying out of practice with grep. On the other hand for a non-work environment where security isn't in question this is cool.
- seanp2k2 3y ago
  This. I had to beg and wait about a year to get jq added to our base image once it passed sec review and all that.
- kyawzazaw 3y ago
  I find bat pretty useful on my local machine
- deleted 3y ago
  [deleted]
jftuga 3y ago
I really like the fuzzy match feature. Useful for typos or off by 1-2 characters. https://github.com/Genivia/ugrep#fuzzy https://github.com/Genivia/ugrep#fuzzy
Joel_Mckay 3y ago
Someone please just standardize the grep flags across all platforms. Specifically -P / --perl-regexp support on MacOS and FreeBSD It really would reduce the WTF moments for the students. Insert jokes about standards below... =)
- burntsushi 3y ago
  That's what POSIX was supposed to be. It's easier IMO to just use the same tool on all platforms. Which you can of course do.
  - Joel_Mckay 3y ago
    Not sure if brew's grep is as NERF'ed, but POSIX standard often is just a subset of minimal features for the GNU version. Cheers, =)
    - burntsushi 3y ago
      Yes, that's the problem. You need to maintain a close attention level to know which things are POSIX. And in the case of GNU grep, you actually need to set POSIXLY_CORRECT=1. Otherwise its behavior is not a subset. POSIX also forbids greps from searching UTF-16 because it mandates that certain characters always use a single byte. ripgrep, for example, doesn't have this constraint and thus can transparently search UTF-16 correctly via BOM sniffing.
ijustlovemath 3y ago
Any particular reason why newer tools don't follow the well-established XDG standard for config files? Those folder structures probably already exist on end user machines, and keep your home directory from getting cluttered with tens of config files
- burntsushi 3y ago
  For ripgrep at least, you set an environment variable telling it where to look for a config file. You can put it anywhere, so you don't need to put it in $HOME. I didn't do XDG because this route seemed simpler, and XDG isn't something that is used everywhere.
  - Hendrikto 3y ago
    > I didn't do XDG because this route seemed simpler Simpler how? This requires custom config, instead of following what I set system-wide. > and XDG isn't something that is used everywhere. Yeah, that‘s why it defines defaults to fall back on.
    - burntsushi 3y ago
      It's far simpler to implement. No, you don't understand. I'm not saying The XDG variables might not be defined. Give me a little credit here lol. I have more than a passing familiarity with XDG. I've implemented it before. I'm saying the XDG convention itself may not apply. For example, Windows. And its controversial whether to use them on macOS when I last looked into it. I don't see any significant problem with defining an environment variable. You likely already have dozens defined. I know I do. I'm not trying to convince you of anything. Someone asked why. This is why for ripgrep at least.
      - ijustlovemath 3y ago
        Could ripgrep not simply add a check for the XDG environment variables and use those, if no rg environment variable is given? Of course if both are not available you would use the default.
zaidhaan 3y ago
A little off-topic, but I'd love to see a tool similar to this that provides real-time previews for an entire shell pipeline which, most importantly, integrates into the shell. This allows for leveraging the completion system to complete command-line flags and using the line editor to navigate the pipeline. In zsh, the closest thing I've gotten to this was to bind Ctrl-\ to the `accept-and-hold` zle widget, which executes what is in the current buffer while still retaining it and the cursor position. That gets me close (no more ^P^B^B^B^B for editing), but I'd much rather see the result of the pipeline in real-time rather than having to manually hit a key whenever I want to see the result.
- wazzaps 3y ago
  Sounds similar to this: https://github.com/akavel/up https://github.com/akavel/up
- tacone 3y ago
  I guess Alt+a is the default zsh shortcut for that.
nineteen96 3y ago
this is slick! easily the best of these new grep tools. thanks for sharing. i’ll use this when grep(1) doesn’t quite cut it
tarun_anand 3y ago
Very insightful discussion. Is there a regex library that is tuned for in-memory data/strings? Similar to in-memory databases? I recall using hyperscan but isn't it discontinued.
karakanb 3y ago
Slightly off topic, but how does one publish so many installable versions of a binary across all the package managers? I figured out how to do it for Brew, but the rest seems like a billion different steps that need to be done and I feel like I am missing something.
- wint3rmute 3y ago
  You only have to set up CI/CD once for each package type, afterwards all the packaging work is done for you automatically. Ripgrep is also quite a large project (judging on both star count and contribution cout), so people probably volunteer to support their platform/package manager of choice.
louwrentius 3y ago
I will never learn this tool I will not even contemplate using this tool. The reason is very simple: I can trust 'grep' to be on any system I ever touch. Learning ugrep doesn't make any sense as I can't trust it to be available. I could still use it on my own systems, but I work on customer systems which won't have this tool installed. And I'm proficient enough with grep that it's 'good enough', I'm not focussing on a better grep. I'm focussing on fixing a problem, or trying something new. I'd rather invest my time into something that will benefit me across all environments I work with. Because a tool may be 'better' (whatever that means) doesn't mean it will see adoption. This is not about being closeminded, but it's about focus on what's really important.

344 points by smartmic 3y ago

infamia 3y ago

Ugrep is also available in Debian based repos, which is super nice.

dmlerner 3y ago

Why not ripgrep?

devraza 3y ago
From the ugrep README: For an up-to-date performance comparison of the latest ugrep, please see the ugrep performance benchmarks [at https://github.com/Genivia/ugrep-benchmarks https://github.com/Genivia/ugrep-benchmarks]. Ugrep is faster than GNU grep, Silver Searcher, ack, sift. Ugrep's speed beats ripgrep in most benchmarks.
- codetrotter 3y ago
  Does these performance comparison take into account the things BurntSushi (ripgrep author) pointed out in the ripgrep issue link elsewhere ITT? https://github.com/BurntSushi/ripgrep/discussions/2597 https://github.com/BurntSushi/ripgrep/discussions/2597 Either way, ripgrep is awesome and I’m staying with it.
  - devraza 3y ago
    Agreed - ripgrep is great, and I'm not planning to switch either. The performance improvement is tiny, anyways.
0cf8612b2e1e 3y ago
I assume the grep compatible bit is attractive to some people. Not me, but they exist.
- derriz 3y ago
  I find myself returning to grep from my default of rg because I'm just too lazy to learn a new regex language. Stuff like word boundaries "\<word\>" or multiple patterns "$one\|two$".
  - masklinn 3y ago
    That seems like the weirdest take ever: ripgrep uses pretty standard PCRE patterns, which are a lot more common than posix’s bre monstrosity. To me the regex langage is very much a reason to not use grep.
    - pbhjpbhj 3y ago
      `pgrep`, or `grep -P`, uses PCRE though, AFAIUI.
      - deleted 3y ago
        [deleted]
    - derriz 3y ago
      A bit hyperbolic, no? If you consider it "the weirdest ever", I'm guessing that I'm probably older than you. I've certainly been using regex long before PCRE became common. As a vim user I compose 10s if not 100s of regexes a day. It does not use PCRE. Nor does sed, a tool I've been using for decades. Do you also recommend not using these?
      - comex 3y ago
        I use all of those tools but the inconsistency drives me crazy as it's hard to remember which syntax to use where. Here's how to match the end of a word: ripgrep, Python, JavaScript, and practically every other non-C language: \b vim: \> BSD sed: [[:>:]] GNU sed, GNU grep: \> or \b BSD grep: \>, \b, or [[:>:]] less: depends on the OS it's running on
meindnoch 3y ago
Because this is faster?
bsdpufferfish 3y ago
ripgrep stole the name but doesn’t follow the posix standard.
Conscat 3y ago
The best practical reason to choose this is its interactive features, like regexp building.
tredre3 3y ago
One thing I never liked about ripgrep is that it doesn't have a pager. Yes, it can be configured to use the system-wide ones, but it's an extra step (and every time I have to google how to preserve colors) and on Windows you're SOL unless you install gnu utils or something. The author always refused to fix that. Ugrep not only has a pager built in, but it also allows searching the results which is super nice! And that feature works on all supported platforms!
- burntsushi 3y ago
  This is what I do personally: $ cat ~/bin/rgp #!/bin/sh exec rg -p "$@" | less -RFX Should work just fine. For Windows, you can install `bat` to use a pager if you don't otherwise have one. You don't need GNU utils to have a pager.
  - anjanb 3y ago
    hi @burntsushi, fan of your tool. like it's speed and defaults. I use windows : didn't understand what you mean by "install `bat`" to use a pager. I use cygwin and WSL for my unix needs. I have more and less in cygwin for use in windows.
    - burntsushi 3y ago
      I referenced bat because I've found that suggesting cygwin sometimes provokes a negative reaction. The GP also mentioned needing to install GNU tooling as if it were a negative. bat is fancy pager written in Rust. It's on GitHub: https://github.com/sharkdp/bat https://github.com/sharkdp/bat
      - anjanb 3y ago
        I'm sure you know but windows command prompt always came with its inbuilt pager -- more. So, you could always do "dir | more" or "rg -p "%*" | more ". (more is good with colors without flags)
GuB-42 3y ago
Why not ugrep? They are more or less equivalent. One has obscure feature X other has obscure feature Y, one is a bit faster on A, other is a bit faster on B, the defaults are a bit different, and one is written in Rust, the other in C++. Pick the one you like, or both. I have both on my machine, and tend to use the one that does what I want with the least options. I also use GNU grep when I don't need the speed or features of either ug and rg.
MrDrMcCoy 3y ago
For me, it's a lot easier to compile a static binary of a C++ app than a Rust one. Never got that to work. Also nice to have compatibility with all of grep's arguments.
- datadeft 3y ago
  > to compile a static binary Cargo is one of the main reasons to use Rust of C++. I am pretty sure there is more involved with C++ than this: rustup target add x86_64-unknown-linux-musl cargo build --target=x86_64-unknown-linux-musl
jedisct1 3y ago
Fuzzy matching is the main reason I switched to ugrep. This is insanely useful.
philkrylov 3y ago
Although being faster in some cases, ripgrep lacks archive search support (no, transparent decompression ignoring the archive structure is not enough) which works great in ugrep.

graphe 3y ago

Here's a thread on performance vs rg (ripgrep). https://github.com/BurntSushi/ripgrep/discussions/2597 https://github.com/BurntSushi/ripgrep/discussions/2597 didn't know about hypergrep either.

infocollector 3y ago
I think you should try it before you read these conflicting benchmarks from the authors: https://github.com/Genivia/ugrep-benchmarks https://github.com/Genivia/ugrep-benchmarks
ashvardanian 3y ago
Haven't benchmarked *grep implementations, but assuming those are just CLI wrappers around RegEx libraries, I'd expect the RegEx benchmarks to be broader and more representative. There, hyperscan is generally the king, which means hypergrep numbers are likely accurate: https://github.com/p-ranav/hypergrep?tab=readme-ov-file#directory-search-usr https://github.com/p-ranav/hypergrep?tab=readme-ov-file#dire... Disclaimer: I rarely use any *grep utilities, but often implement string libraries.
- burntsushi 3y ago
  I'm the author of ripgrep and its regex engine. Your claim is true to a first approximation. But greps are line oriented, and that means there are optimizations that can be done that are hard to do in a general regex library. You can read more about that here: https://blog.burntsushi.net/ripgrep/#anatomy-of-a-grep https://blog.burntsushi.net/ripgrep/#anatomy-of-a-grep (greps are more than simple CLI wrappers around a regex engine). If you read my commentary in the ripgrep discussion above, you'll note that it isn't just about the benchmarks themselves being accurate, but the model they represent. Nevertheless, I linked the hypergrep benchmarks not because of Hyperscan, but because they were done by someone who isn't the author of either ripgrep or ugrep. As for regex benchmarks, you'll want to check out rebar: https://github.com/BurntSushi/rebar https://github.com/BurntSushi/rebar You can see my full thoughts around benchmark design and philosophy if you read the rebar documentation. Be warned though, you'll need some time. There is a fork of ripgrep with Hyperscan support: https://sr.ht/~pierrenn/ripgrep/ https://sr.ht/~pierrenn/ripgrep/ Hyperscan also has some preculiarities on how it reports matches. You won't notice it in basic usage, but it will appear when using something like the -o/--only-matching flag. For example, Hyperscan will report matches of a, b and c for the regex \w+, where as a normal grep will just report a match of abc. (And this makes sense given the design and motivation for Hyperscan.) Hypergrep goes to some pain to paper over this, but IIRC the logic is not fully correct. I'm on mobile, otherwise I would link to the reddit thread where I had a convo about this with the hypergrep author.

mathverse 3y ago

Also look at https://github.com/stealth/grab https://github.com/stealth/grab from Sebastian Krahmer.

meowface 3y ago
ripgrep, grab, ugrep, hypergrep... Any of the four are probably fast enough for any of my use cases but I suddenly feel tempted to micro-optimize and spend ages comparing them all.

jraph 3y ago

Okay, this solves a feature I was occasionally missing for a long time: searching for several terms in files (the "Googling files" feature). I wrote a 8 line script a few weeks ago to do this, that I will gladly throw away. I'll look into the TUI too. (I've been using ripgrep for quite some time now, how does this otherwise compare to it? would I be able to just replace rg with ug?)

Levitating 3y ago
Is it that different from using fzf? I currently use ripgrep-all (which can search into anything, video captions or pdfs) and fzf.
bishfish 3y ago
ugrep+ has this feature similar to ripgrep-all. For regular use, I use ugrep’s %u option with its format feature to only get one match per line same as other grep tools. Overall, I’m a happy user of ugrep. ugrep works as well as ripgrep for me. It’s VERY fast and has built-in option to search archives within archives recursively.

o11c 3y ago

Important note: not actually compatible. It took me seconds to find an option that does something completely different than the GNU version.

fwip 3y ago
Which option is that? I'm scanning the ugrep page, but nothing is popping out to me.
e12e 3y ago
I would assume compatible meant posix/bsd - unless explicitly advertised AS "GNU grep compatible"?
- burntsushi 3y ago
  From the OP: "Ugrep is compatible with GNU grep and supports GNU grep command-line options."
burntsushi 3y ago
Indeed. And here are some concrete examples around locale: $ grep -V | head -n1 grep (GNU grep) 3.11 $ alias ugrep-grep="ugrep-4.4.1 -G -U -Y -. --sort -Dread -dread" $ echo 'pokémon' | LC_ALL=en_US.UTF-8 grep 'pok[[=e=]]mon' pokémon $ echo 'pokémon' | LC_ALL=en_US.UTF-8 ugrep-grep 'pok[[=e=]]mon' $ echo 'γ' | LC_ALL=en_US.UTF-8 grep -i 'Γ' γ $ echo 'γ' | LC_ALL=en_US.UTF-8 ugrep-grep -i 'Γ' BSD grep works like GNU grep too: $ grep -V grep (BSD grep, GNU compatible) 2.6.0-FreeBSD $ echo 'pokémon' | LC_ALL=en_US.UTF-8 grep 'pok[[=e=]]mon' pokémon $ echo 'γ' | LC_ALL=en_US.UTF-8 grep -i 'Γ' γ

MightyBuzzard 3y ago

[dead]

ww520 3y ago

Just tried it out. It's blazingly fast. The interactive TUI search is pretty sweet.

comex 3y ago

Interesting, it supports an n-gram indexer. ripgrep has had this planned for a few years now [1] but hasn't implemented it yet. For large codebases I've been using csearch, but it has a lot of limitations. Unfortunately... I just tried the indexer and it's extremely slow on my machine. It took 86 seconds to index a Linux kernel tree, while csearch's cindex tool took 8 seconds. [1] https://github.com/BurntSushi/ripgrep/issues/1497 https://github.com/BurntSushi/ripgrep/issues/1497

dtgriscom 3y ago
That's close to a gig of disk reads; I trust you didn't try ugrep first and then cindex second, without taking into account caching.
- comex 3y ago
  I ran both multiple times, alternating (and making sure to clean out the indexes in between). Results were reasonably consistent across runs.
jgalt212 3y ago
If you're gonna go the csearch route, you should also consider hound. I use it many times per day. https://github.com/hound-search/hound https://github.com/hound-search/hound
bishfish 3y ago
It creates per-directory index files on its first run. ugrep-indexer is also labeled as beta. A couple of relevant quotes from its GitHub site: “Indexing adds a hidden index file ._UG#_Store to each directory indexed.” “Re-indexing is incremental, so it will not take as much time as the initial indexing process.”

joshka 3y ago

There's a few ripgrep based tuis: - https://github.com/acheronfail/repgrep https://github.com/acheronfail/repgrep - https://github.com/konradsz/igrep https://github.com/konradsz/igrep - https://github.com/seg-mx/grep_tui https://github.com/seg-mx/grep_tui - https://github.com/Robertleoj/grepedit https://github.com/Robertleoj/grepedit

nsagent 3y ago
You can also use fzf with ripgrep to great effect: [1]: https://github.com/junegunn/fzf/blob/master/ADVANCED.md#using-fzf-as-interactive-ripgrep-launcher https://github.com/junegunn/fzf/blob/master/ADVANCED.md#usin...

stevebmark 3y ago

There are many grep variations. The Unix philosophy: do one thing well. The Unix reality: do many things poorly* *grep, awk, sed

jedberg 3y ago

I feel like if you're going to make a new grep and put a web page for it, your webpage should start with why your grep is better than the default (or all the other ones). Why did you build a new grep?

deleted 3y ago
[deleted]
infamia 3y ago
> I feel like if you're going to make a new grep and put a web page for it, your webpage should start with why your grep is better than the default (or all the other ones). No snark here, but is the subtitle not enough to start? "a more powerful, ultra fast, user-friendly, compatible grep"
- fsckboy 3y ago
  no snark here, but the subtitle was the start of my confusion: what does "user-friendly" mean in the context of grep, and why should I believe the claim? regular expressions are not friendly, but the user friendly way for a cli filter to behave is to return retvals appropriately, output to stdout, error messages to stderr... does user friendly mean copious output to stderr? what else could it possibly mean? do I want copious output to stderr?
  - infamia 3y ago
    > no snark here, but the subtitle was the start of my confusion: what does "user-friendly" mean in the context of grep, and why should I believe the claim? Granted, it is far from a thing of beauty, but there is a large, captioned screenshot of the included text user interface just beneath. Then again, it is a website for a command line tool. "Many Bothans died to bring us this information."
- jedberg 3y ago
  Not really. * a more powerful -- This is meaningless without some sort of examples. Powerful how? What does it do that's better than grep? * ultra fast -- This at least means something, but it should be quantified in some way. "50%+ faster for most uses cases" or something like that. * user-friendly -- not even sure what this means. Seems kind of subjective anyway. I find grep plenty user friendly, for a command line tool. * compatible grep -- I mean, they all are pretty much, but I guess it's good to know this?

Sparkyte 3y ago

Cool, but in a real life scenario where the system is not able to pull from external packages because it is in a secured environment makes myself think this is moot as you'll be out of practice of actually running grep. I would avoid not staying out of practice with grep. On the other hand for a non-work environment where security isn't in question this is cool.

seanp2k2 3y ago
This. I had to beg and wait about a year to get jq added to our base image once it passed sec review and all that.
kyawzazaw 3y ago
I find bat pretty useful on my local machine
deleted 3y ago
[deleted]

jftuga 3y ago

I really like the fuzzy match feature. Useful for typos or off by 1-2 characters. https://github.com/Genivia/ugrep#fuzzy https://github.com/Genivia/ugrep#fuzzy

Joel_Mckay 3y ago

Someone please just standardize the grep flags across all platforms. Specifically -P / --perl-regexp support on MacOS and FreeBSD It really would reduce the WTF moments for the students. Insert jokes about standards below... =)

burntsushi 3y ago
That's what POSIX was supposed to be. It's easier IMO to just use the same tool on all platforms. Which you can of course do.
- Joel_Mckay 3y ago
  Not sure if brew's grep is as NERF'ed, but POSIX standard often is just a subset of minimal features for the GNU version. Cheers, =)
  - burntsushi 3y ago
    Yes, that's the problem. You need to maintain a close attention level to know which things are POSIX. And in the case of GNU grep, you actually need to set POSIXLY_CORRECT=1. Otherwise its behavior is not a subset. POSIX also forbids greps from searching UTF-16 because it mandates that certain characters always use a single byte. ripgrep, for example, doesn't have this constraint and thus can transparently search UTF-16 correctly via BOM sniffing.

ijustlovemath 3y ago

Any particular reason why newer tools don't follow the well-established XDG standard for config files? Those folder structures probably already exist on end user machines, and keep your home directory from getting cluttered with tens of config files

burntsushi 3y ago
For ripgrep at least, you set an environment variable telling it where to look for a config file. You can put it anywhere, so you don't need to put it in $HOME. I didn't do XDG because this route seemed simpler, and XDG isn't something that is used everywhere.
- Hendrikto 3y ago
  > I didn't do XDG because this route seemed simpler Simpler how? This requires custom config, instead of following what I set system-wide. > and XDG isn't something that is used everywhere. Yeah, that‘s why it defines defaults to fall back on.
  - burntsushi 3y ago
    It's far simpler to implement. No, you don't understand. I'm not saying The XDG variables might not be defined. Give me a little credit here lol. I have more than a passing familiarity with XDG. I've implemented it before. I'm saying the XDG convention itself may not apply. For example, Windows. And its controversial whether to use them on macOS when I last looked into it. I don't see any significant problem with defining an environment variable. You likely already have dozens defined. I know I do. I'm not trying to convince you of anything. Someone asked why. This is why for ripgrep at least.
    - ijustlovemath 3y ago
      Could ripgrep not simply add a check for the XDG environment variables and use those, if no rg environment variable is given? Of course if both are not available you would use the default.

zaidhaan 3y ago

A little off-topic, but I'd love to see a tool similar to this that provides real-time previews for an entire shell pipeline which, most importantly, integrates into the shell. This allows for leveraging the completion system to complete command-line flags and using the line editor to navigate the pipeline. In zsh, the closest thing I've gotten to this was to bind Ctrl-\ to the `accept-and-hold` zle widget, which executes what is in the current buffer while still retaining it and the cursor position. That gets me close (no more ^P^B^B^B^B for editing), but I'd much rather see the result of the pipeline in real-time rather than having to manually hit a key whenever I want to see the result.

wazzaps 3y ago
Sounds similar to this: https://github.com/akavel/up https://github.com/akavel/up
tacone 3y ago
I guess Alt+a is the default zsh shortcut for that.

nineteen96 3y ago

this is slick! easily the best of these new grep tools. thanks for sharing. i’ll use this when grep(1) doesn’t quite cut it

tarun_anand 3y ago

Very insightful discussion. Is there a regex library that is tuned for in-memory data/strings? Similar to in-memory databases? I recall using hyperscan but isn't it discontinued.

karakanb 3y ago

Slightly off topic, but how does one publish so many installable versions of a binary across all the package managers? I figured out how to do it for Brew, but the rest seems like a billion different steps that need to be done and I feel like I am missing something.

wint3rmute 3y ago
You only have to set up CI/CD once for each package type, afterwards all the packaging work is done for you automatically. Ripgrep is also quite a large project (judging on both star count and contribution cout), so people probably volunteer to support their platform/package manager of choice.

louwrentius 3y ago

I will never learn this tool I will not even contemplate using this tool. The reason is very simple: I can trust 'grep' to be on any system I ever touch. Learning ugrep doesn't make any sense as I can't trust it to be available. I could still use it on my own systems, but I work on customer systems which won't have this tool installed. And I'm proficient enough with grep that it's 'good enough', I'm not focussing on a better grep. I'm focussing on fixing a problem, or trying something new. I'd rather invest my time into something that will benefit me across all environments I work with. Because a tool may be 'better' (whatever that means) doesn't mean it will see adoption. This is not about being closeminded, but it's about focus on what's really important.