5 ms·
I'm the author of ag[1]. File searching stuff interests me, so I took a look at grab. Sadly, grab only compiles on Linux right now. Some of the mmap flags and c
by ggreer 12y ago
I'm the author of ag[1]. File searching stuff interests me, so I took a look at grab. Sadly, grab only compiles on Linux right now. Some of the mmap flags and cpu_set_t aren't defined on OS X. I was curious about the performance claims, so I benchmarked grab and ag on my code directory. Times are medians of five runs. I used a Lenovo X140e running Ubuntu 14.10. It has a 160GB Intel SSD, 8GB of RAM, and an AMD A4-5000 (4 x 1.5Ghz Jaguar).
ggreer@boron:~/code% du -sh .
8.3G .
ggreer@boron:~/code% time ag cpu_set_t
ag cpu_set_t 4.45s user 5.25s system 295% cpu 3.285 total
ggreer@boron:~/code% time grab -R cpu_set_t .
grab -R cpu_set_t . 13.31s user 21.67s system 35% cpu 1:38.28 total
30x faster, but these benchmarks aren't a fair fight. Ag ignores binary and hidden files by default. If I tell ag to do an unrestricted search, it's still 2x faster (43 seconds vs 98 seconds). Even with cold caches (echo 3 | sudo tee /proc/sys/vm/drop_caches between each run), ag beats grab handily:
ag -u cpu_set_t 19.62s user 32.56s system 90% cpu 57.433 total
grab -R cpu_set_t . 15.48s user 37.89s system 37% cpu 2:22.67 total
I haven't profiled grab yet, but there's definitely some low-hanging fruit. For example, it looks like grab could get a big speedup by detecting literal patterns and using strstr() instead of a whole PCRE engine. Also, FileGrep::find is calling pthread_mutex_lock/unlock even if there are no matches to print. Adding a condition around that makes grab 1.5x faster.
I'm glad I took a look at grab. Despite its shortcomings, I learned from it. I'll definitely try out a few tricks grab uses that ag doesn't, such as thread affinity.
One more thing: The author of grab is right about counting newlines. It does hurt performance. Still, I enable it by default in ag. I think the tradeoff is worthwhile.
1. https://github.com/ggreer/the_silver_searcher https://github.com/ggreer/the_silver_searcher
Edit: The mutex locking change was so straightforward that I submitted a PR: https://github.com/stealth/grab/pull/2 https://github.com/stealth/grab/pull/2
- stefantalpalaru 12y ago> Sadly, grab only compiles on Linux right now. the_silver_searcher is a great piece of software and I use it every day, but why would your main development platform be anything but Linux in this day and age? OS X has inferior performance, no official package manager and it's being produced by people hostile to open source developers. Why in the name of Jah would you subject yourself to that?
- anon4 12y agoBecause the UI is better. I personally make the tradeoff for more openness and power and use Linux, but I can see why people use MacOS.
- stevenjohns 12y agoJust a slight recommendation for those interested in a good OSX-y looking GNU/Linux distribution: check out Elementary OS [1]. It is heavily inspired by OSX and overall it is very responsive and attractive. [1] http://elementaryos.org/ http://elementaryos.org/
- caryhartline 12y agoPeople like what they like. This is not the place for a flame war.
- stefantalpalaru 12y agoNo, this is the place where you get with the program or get downvoted into oblivion.
- infamouscow 12y agoHaving a faster grep-like tool installed on a production BSD or illumos machine sounds reasonable.
- tsomctl 12y agoBecause the GUI on OS X is actually useable, and upgrades don't automatically cause audio to stop working.
- stefantalpalaru 12y agoThere are tens of popular Linux distributions out there. Choose one that doesn't rely on PulseAudio instead of caving in for the shiny GUI and the used car salesman bullshit.
- 12y ago
- Touche 12y agoThank you so much for Silver Searcher. I depend on it every day. One feature I would like is the ability to ignore very large files that get in the way when I'm searching.
- bhuga 12y agoThanks for ag. It was a game changer for me.
- _delirium 12y ago> using strstr() instead of a whole PCRE engine This is probably faster on a lot of platforms with the common grep use-cases, yeah (i.e. very short literal strings). But as a common practice imo using the C library string functions in performance-sensitive code is tricky, because it can lead to some wildly variable performance between platforms, compared to directly using a portable-C implementation of a good algorithm. I was bitten by strstr() in particular in the past. I had a large slowdown when taking something pretty simple (filter/extraction code) that ran fine on Linux, and trying it on FreeBSD. It turns out that strstr() on FreeBSD implements the naive O(nm) algorithm: a loop around strncmp() that tries a match at each possible starting location [1]. That's fine for short search strings but increasingly bad for long ones. I had assumed that a typical strstr() would use Boyer-Moore or something of that sort past a threshold, but on many platforms it doesn't (GNU libc, at least in recent versions, does switch to Two-Way [2]). [1] http://svnweb.freebsd.org/base/head/lib/libc/string/strstr.c?revision=251069&view=markup#l44 http://svnweb.freebsd.org/base/head/lib/libc/string/strstr.c... [2] http://www-igm.univ-mlv.fr/~lecroq/string/node26.html http://www-igm.univ-mlv.fr/~lecroq/string/node26.html
- ComputerGuru 12y agoI was coming here to post about ag, thank you for the silver searcher.
- ww520 12y agoAg is amazing. Thank you for such a useful tool. I don't have to use find + grep after using ag. One thing I wish for is to cache the search result of a search term so that next search of the same term would be instant. This is good for large static file set.
- nightwolf 12y agoI just wanted to thank you for ag. It's literally the most used command in my shell history and I consider it crucial to my work. So, thanks!