5 ms·
And ripgrep is quite a lot faster than GNU grep[1], especially if you need unicode support. [1] https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.ne
by opencl 7y ago
And ripgrep is quite a lot faster than GNU grep[1], especially if you need unicode support.
[1] https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/
- dewey 7y agoIt depends, I benchmarked all of the usual suspects (fgrep, GNU grep, BSD grep, ripgrep, sift grep) on the same 30GB line separated JSON file last week and GNU grep was the fastest one. I didn't have a scientific testing setup though. I just ran them one by one like this: time cat title_complete_json.txt | grep tt9184994 >> result
- Scarbutt 7y agoThe infamous "useless use of cat"
- adrusi 7y agoThat's relevant in some scripts, where unnecessary forks might actually add up to some meaningful resource usage. Not relevant when typed at a tty.
- nsajko 7y agoTo clarify, it would have been prettier (conceptually less clunky and shorter) to run this: time grep tt9184994 < title_complete_json.txt >> result
- catucia 7y agogrep takes a file directly, so this could simply be: time grep tt9184994 title_complete_json.txt >> result
- hyper_reality 7y agoIt annoys me whenever somebody brings this up. People do this because it's easier to modify commands at the end of your prompt than near the beginning. Especially in this case, where the grandparent wants to quickly switch between benchmarking grep, ripgrep etc. Not useless at all.
- naniwaduni 7y agoThough a neat thing to know is that redirections can appear anywhere on the line in Bourneish shells: time < title_complete_json.txt grep tt9184994 As it happens, when benchmarking grep-like tools on large files, this is actually somewhat likely to make a significant difference since the "useless" use of cat actually forces the file to be read sequentially as a stream (because it's piped through), while a process receiving a file on standard input can treat it as a file and e.g. mmap it.
- hyper_reality 7y agoI found a benchmark that suggests the performance gap between a pipe and input redirection is insignificant for most workloads, but there would indeed start to be a difference for the exact case discussed here (grepping a large file): http://oletange.blogspot.com/2013/10/useless-use-of-cat.html http://oletange.blogspot.com/2013/10/useless-use-of-cat.html Thanks for your tip, it's clearly the best of both worlds for this purpose.
- burntsushi 7y agoIncidentally, ripgrep will actually be faster in this case generally (on Linux at least) when given an explicit file path, since that lets it use memory maps. For example: $ time rg QZQZQZQZ < subtitles.en.txt real 1.842 user 0.552 sys 1.287 maxmem 9 MB faults 0 $ time rg QZQZQZQZ subtitles.en.txt real 1.210 user 0.776 sys 0.433 maxmem 9510 MB faults 0 (See my other comments in this thread for where to get subtitles.en.txt.)
- mnbvkhgvmj 7y agoSurely using cat like this is more in line with the unix philosophy?
- mihaitodor 7y agoDid you try The Silver Searcher https://github.com/ggreer/the_silver_searcher https://github.com/ggreer/the_silver_searcher? :)
- dewey 7y agoI didn't, but will try it out of curiosity when I'm back on that machine.
- kodz4 7y agoYou have to throw parallels in there - https://stackoverflow.com/questions/9066609/fastest-possible-grep https://stackoverflow.com/questions/9066609/fastest-possible...
- opencl 7y agoYeah, rg is definitely not always the fastest. Test for your own use case and all. In pretty much any case involving more complicated queries and/or unicode I have found rg to be the fastest by a very wide margin. Also if you're searching multiple files rg is multithreaded by default without having to pass through something like xargs or parallel. If you're just searching for an ASCII string literal in a single file rg probably gets slowed down by its UTF-8 validation.
- burntsushi 7y agoThanks for noticing ripgrep's performance on Unicode. :-) > If you're just searching for an ASCII string literal in a single file rg probably gets slowed down by its UTF-8 validation. This is most definitely wrong. Firstly, ripgrep doesn't do any UTF-8 validation. That would make it much much slower. Secondly, searching a simple literal is one of the cases where ripgrep should actually do better than GNU grep in many cases. This is because of the slightly smarter way in which ripgrep keeps its vectorized loop active more often than GNU grep does by trying to search rarer bytes. I talk about this quite a bit here, which even includes a benchmark where I measured ripgrep at 2x the speed of GNU grep for a simple literal search: https://blog.burntsushi.net/ripgrep/#subtitles-literal https://blog.burntsushi.net/ripgrep/#subtitles-literal Anyone can try this for themselves: $ cd /tmp $ curl 'https://object.pouta.csc.fi/OPUS-OpenSubtitles/v2016/mono/en.txt.gz' | gzip -cd > subtitles.en.txt $ pv < subtitles.en.txt > /dev/null 9.28GiB 0:00:01 [4.89GiB/s] [======================================================================================================================================================>] 100% $ time grep ZQZQZQZQ < subtitles.en.txt real 8.713 user 7.495 sys 1.212 maxmem 9 MB faults 0 $ time rg ZQZQZQZQ < subtitles.en.txt real 1.857 user 0.573 sys 1.282 maxmem 9 MB faults 0 On my system, /tmp is a ramdisk, so this doesn't factor in I/O. If you look up this thread, you'll note that someone was trying to search a 30GB file. Unless that person could fit that file into memory, it's very likely that they were just measuring the I/O bandwidth of their machine.
- opencl 7y agoThanks for the detailed explanation and your work on rg, I use it all the time and strongly appreciate the performance with unicode because I do a lot of searching of large directories of files that are largely in Japanese.
- Someone 7y agoWhat’s the rationale for using >> ? I would try to get as small an output as possible, and not hit the file system at all, for example by using time cat title_complete_json.txt | grep tt9184994 | grep -v tt9184994
- davemp 7y agoAFAIK ripgrep always uses regular expressions. So for straight substring matching grep will usually be faster.
- burntsushi 7y agoIt does not. Could you please say where you got this impression so I could fix it? See my other comments in this very thread. ripgrep actually has more sophisticated literal extraction & searching than GNU grep does. Moreover, many of these optimizations are inside the regex engine itself, so even if it looks like you're using a regex, literal optimizations may still be applied.
- davemp 7y agoMy mistake. I read the first line of the ripgrep readme and assumed because utils like `find` run slower with regex matching for the same searches that ripgrep would be similar. > ripgrep is a line-oriented search tool that recursively searches your current directory for a regex pattern I'm not sure if many other people would jump to similar conclusions. So the readme might not really need a revision.