2 ms·

> Ag has a multi-threaded architecture that can search multiple files Interesting. I would have guessed you were i/o bound there, and file systems usually give

by xorcist 10y ago

> Ag has a multi-threaded architecture that can search multiple files Interesting. I would have guessed you were i/o bound there, and file systems usually give higher throughput on sequential reads. Tried it on my bog standard Linux box but couldn't get it close to grep performance wise. Can you share how you did it? > Ag ignores binary and hidden files by default Like grep -I? That could easily cause unexpected results. Many files are treated as binary just because they contain parts of a non native character set (such as latin-1 files on a utf-8 system).

ggreer 10y ago
I've documented most of ag's tricks in various blog posts.[1] You're right that sequential reads are faster than random reads, but searching a bunch of source code is going to be a bunch of short reads no matter what. Each file is on the order of 50kB, or about 0.5ms if read at 100MB/sec. That's far less than the seek time of a hard drive. I've experimented a lot, and found it best to read ASAP, then let the OS & device figure out the best order to service the requests. Also, modern SSDs (and even HDDs) have tricks like NCQ. With caches cleared (cold reads), multi-threaded ag is 2x faster on SSDs. Disabling NCQ makes multi-threaded perform the same as single-threaded.[2] (Again, with caches cleared.) > … That could easily cause unexpected results. … I've received very few complaints about ag's binary detection. It does a nice job of balancing false positives, false negatives, and performance. See is_binary() in src/utils.c for the exact code[3]. 1. http://geoff.greer.fm/ag#related-posts http://geoff.greer.fm/ag#related-posts 2. http://geoff.greer.fm/2012/09/07/the-silver-searcher-adding-pthreads/#comment-644691480 http://geoff.greer.fm/2012/09/07/the-silver-searcher-adding-... 3. https://github.com/ggreer/the_silver_searcher/blob/7d52078cb6460db0c4725a34da45cec4d6d349e8/src/util.c#L288 https://github.com/ggreer/the_silver_searcher/blob/7d52078cb...

by xorcist 10y ago

ggreer 10y ago

I've documented most of ag's tricks in various blog posts.[1] You're right that sequential reads are faster than random reads, but searching a bunch of source code is going to be a bunch of short reads no matter what. Each file is on the order of 50kB, or about 0.5ms if read at 100MB/sec. That's far less than the seek time of a hard drive. I've experimented a lot, and found it best to read ASAP, then let the OS & device figure out the best order to service the requests. Also, modern SSDs (and even HDDs) have tricks like NCQ. With caches cleared (cold reads), multi-threaded ag is 2x faster on SSDs. Disabling NCQ makes multi-threaded perform the same as single-threaded.[2] (Again, with caches cleared.) > … That could easily cause unexpected results. … I've received very few complaints about ag's binary detection. It does a nice job of balancing false positives, false negatives, and performance. See is_binary() in src/utils.c for the exact code[3]. 1. http://geoff.greer.fm/ag#related-posts http://geoff.greer.fm/ag#related-posts 2. http://geoff.greer.fm/2012/09/07/the-silver-searcher-adding-pthreads/#comment-644691480 http://geoff.greer.fm/2012/09/07/the-silver-searcher-adding-... 3. https://github.com/ggreer/the_silver_searcher/blob/7d52078cb6460db0c4725a34da45cec4d6d349e8/src/util.c#L288 https://github.com/ggreer/the_silver_searcher/blob/7d52078cb...