4 ms·

Would using ag help? No longer maintained AFAIK, but fast for me.

• chrismonsanto 12y ago
  where did you get that idea? last commit was 7 days ago, no mention of abandonment
  • porker 12y ago
    From the status of the Ubuntu ppa, and (I thought) comments on his site. Thanks for pointing out that's not the case.
• e12e 12y ago
  As far as I know, it still holds that ag isn't fast, ack is slow ;-) That is to say: grep is pretty fast (too). In other words, ack improved the user-interface and api for search across files, with an eye towards programming and editors, but was relatively slow -- ag aims to keep the improved ui/api/output but bring speed back up to gnu grep-like levels.
  • porker 12y ago
    I think if your grep has the -s flag (the GNU 'treat as string' option) you're correct. If not, it may be faster :)
    • e12e 12y ago
      You might be thinking of -F/--fixed-strings. -s is slient (long option --no-messages). For GNU grep 2.12, anyway. Or you might be thinking of BSD grep: http://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html http://lists.freebsd.org/pipermail/freebsd-current/2010-Augu... edit: Eg, with warm cache: :~/tmp/riak/riak-2.0.0pre5/deps $ time (find . -type f -exec cat '{}' \; |wc -l) 2765699 real 0m5.021s user 0m0.144s sys 0m0.792s :~/tmp/riak/riak-2.0.0pre5/deps $ time (find . -type f -exec cat '{}' \; |grep -E 'Some pattern' -v -c) 2765700 real 0m5.133s user 0m0.264s sys 0m0.852s :~/tmp/riak/riak-2.0.0pre5/deps $ time (find . -type f -exec cat '{}' \; |grep -E 'Some..pattern' -v -c) 2765700 real 0m5.144s user 0m0.400s sys 0m0.768s # "%% " used for leading comment lines in some of this code: :~/tmp/riak/riak-2.0.0pre5/deps $ time (find . -type f -exec cat '{}' \; |grep -E '^%% ' -c) 27535 real 0m5.597s user 0m0.520s sys 0m0.788s :~/tmp/riak/riak-2.0.0pre5/deps $ du -hcs . 405M . 405M total :~/tmp/riak/riak-2.0.0pre5/deps $ time (find . -type f -exec cat '{}' \; |ag '^%% ' >/dev/null) real 0m5.735s user 0m1.480s sys 0m0.876s #actually find/cat is pretty slow -- I guess both GNU grep and ag #use nmap to good effect: $ time rgrep '^%% ' . > /dev/null real 0m0.539s user 0m0.404s sys 0m0.128s :~/tmp/riak/riak-2.0.0pre5/deps $ time ag '^%% ' . |wc -l 27500 real 0m0.252s user 0m0.284s sys 0m0.068s :~/tmp/riak/riak-2.0.0pre5/deps $ time rgrep -E '^%% ' . |wc -l 27553 real 0m0.535s user 0m0.396s sys 0m0.140s Note that grep clearly goes looking in more files here (more mathcing lines). Still, I guess ag is indeed faster than grep in some cases (even if it might not be apples to apples depending how you count -- of course the whole point of ag is to help search just the right files). :~/tmp/riak/riak-2.0.0pre5/deps $ time rgrep -E 'Some pattern' . |wc -l 0 real 0m0.266s user 0m0.128s sys 0m0.132s :~/tmp/riak/riak-2.0.0pre5/deps $ time rgrep -E 'Some..pattern' . |wc -l 0 real 0m0.338s user 0m0.212s sys 0m0.120s :~/tmp/riak/riak-2.0.0pre5/deps $ time ag 'Some..pattern' . |wc -l 0 real 0m0.111s user 0m0.100s sys 0m0.076s I guess ag is indeed faster, even if it might not be due to fixed string search... [edit2: For those wondering that's an (old) ssd, on an old machine -- but with ~4G ram the working set should fit, as soon as some of my open tabs in ff are paged to disk...]
      • e12e 12y ago
        Quickly reviewing some of the posts on the ag blog/page[1], I'm guessing the speedup is mainly from a custom dir scanning algorithm and possibly from running two threads. In the course of checking out ag (again) I also learned about gnu id-utils[2]. [1] http://geoff.greer.fm/ag/ http://geoff.greer.fm/ag/ [2} http://www.delorie.com/gnu/docs/id-utils/id-utils_1.html http://www.delorie.com/gnu/docs/id-utils/id-utils_1.html
      • ggreer 12y ago
        Thanks for benchmarking ag against grep. You're right that it's not exactly apples to apples. Ag doesn't search as many files, but it does parse and match against rules in .ag/.git/.hgignore. Also, ag prints line numbers by default, which can be an expensive operation on larger files. I think most of the slowdown you're seeing with "find -exec | cat" is forking at least two processes (ag and cat) for each file. Also, each process has to be run sequentially (to prevent garbled output), which makes use of only one CPU core most of the time. I've tried to keep ag's startup time fast so that classic find-style commands still run quickly. (This is why ag doesn't support a ~/.agrc or similar.) Just FYI, you can use ag --stat to see how many files/bytes were searched, how long it took, etc. I think I'll add some stats about obeying ignore rules, since some of those can be outright pathological in terms of runtime cost. In many cases, ag spends more time figuring out what to search than actually searching.
        • e12e 12y ago
          I tried to gauge the cpu usage (just looking at the percentage as listed in xmobar) -- but both grep and ag are too fast on the ~400mb set of files for that to work... As I have two cores on this machine, the difference between ag and rgrep could indeed be ag's use of threads. Many thanks for not just writing and sharing ag as free software, but for the nice articles describing the design and optimizations! At least this brief benchmarking run convinced me that I should probably try to integrate ag in my work flow :-)
      • pimlottc 12y ago
        This was very confusing until I realized HN had invisible code boxes with a fixed width.