4 ms·

A faster string search would be nice. I used to (and still sometimes do) use less to analyze large trace files. With a few hundred MB the searching becomes a re

by nn3 12y ago

A faster string search would be nice. I used to (and still sometimes do) use less to analyze large trace files. With a few hundred MB the searching becomes a real bottle neck. Couldn't be that hard to do a boyer moore for non RE substrings.

porker 12y ago
Would using ag help? No longer maintained AFAIK, but fast for me.
- chrismonsanto 12y ago
  where did you get that idea? last commit was 7 days ago, no mention of abandonment
  - porker 12y ago
    From the status of the Ubuntu ppa, and (I thought) comments on his site. Thanks for pointing out that's not the case.
- e12e 12y ago
  As far as I know, it still holds that ag isn't fast, ack is slow ;-) That is to say: grep is pretty fast (too). In other words, ack improved the user-interface and api for search across files, with an eye towards programming and editors, but was relatively slow -- ag aims to keep the improved ui/api/output but bring speed back up to gnu grep-like levels.
  - porker 12y ago
    I think if your grep has the -s flag (the GNU 'treat as string' option) you're correct. If not, it may be faster :)
    - e12e 12y ago
      You might be thinking of -F/--fixed-strings. -s is slient (long option --no-messages). For GNU grep 2.12, anyway. Or you might be thinking of BSD grep: http://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html http://lists.freebsd.org/pipermail/freebsd-current/2010-Augu... edit: Eg, with warm cache: :~/tmp/riak/riak-2.0.0pre5/deps $ time (find . -type f -exec cat '{}' \; |wc -l) 2765699 real 0m5.021s user 0m0.144s sys 0m0.792s :~/tmp/riak/riak-2.0.0pre5/deps $ time (find . -type f -exec cat '{}' \; |grep -E 'Some pattern' -v -c) 2765700 real 0m5.133s user 0m0.264s sys 0m0.852s :~/tmp/riak/riak-2.0.0pre5/deps $ time (find . -type f -exec cat '{}' \; |grep -E 'Some..pattern' -v -c) 2765700 real 0m5.144s user 0m0.400s sys 0m0.768s # "%% " used for leading comment lines in some of this code: :~/tmp/riak/riak-2.0.0pre5/deps $ time (find . -type f -exec cat '{}' \; |grep -E '^%% ' -c) 27535 real 0m5.597s user 0m0.520s sys 0m0.788s :~/tmp/riak/riak-2.0.0pre5/deps $ du -hcs . 405M . 405M total :~/tmp/riak/riak-2.0.0pre5/deps $ time (find . -type f -exec cat '{}' \; |ag '^%% ' >/dev/null) real 0m5.735s user 0m1.480s sys 0m0.876s #actually find/cat is pretty slow -- I guess both GNU grep and ag #use nmap to good effect: $ time rgrep '^%% ' . > /dev/null real 0m0.539s user 0m0.404s sys 0m0.128s :~/tmp/riak/riak-2.0.0pre5/deps $ time ag '^%% ' . |wc -l 27500 real 0m0.252s user 0m0.284s sys 0m0.068s :~/tmp/riak/riak-2.0.0pre5/deps $ time rgrep -E '^%% ' . |wc -l 27553 real 0m0.535s user 0m0.396s sys 0m0.140s Note that grep clearly goes looking in more files here (more mathcing lines). Still, I guess ag is indeed faster than grep in some cases (even if it might not be apples to apples depending how you count -- of course the whole point of ag is to help search just the right files). :~/tmp/riak/riak-2.0.0pre5/deps $ time rgrep -E 'Some pattern' . |wc -l 0 real 0m0.266s user 0m0.128s sys 0m0.132s :~/tmp/riak/riak-2.0.0pre5/deps $ time rgrep -E 'Some..pattern' . |wc -l 0 real 0m0.338s user 0m0.212s sys 0m0.120s :~/tmp/riak/riak-2.0.0pre5/deps $ time ag 'Some..pattern' . |wc -l 0 real 0m0.111s user 0m0.100s sys 0m0.076s I guess ag is indeed faster, even if it might not be due to fixed string search... [edit2: For those wondering that's an (old) ssd, on an old machine -- but with ~4G ram the working set should fit, as soon as some of my open tabs in ff are paged to disk...]
nhaehnle 12y ago
Not for string search, but I got fed up with the extremely long time it takes less to precalculate line counts on large files when I was working with log files of a size in the order of GBs. The result is here: https://github.com/nhaehnle/less https://github.com/nhaehnle/less Since it was only a small hack to scratch the itch I was having at the time, I never really completed that project. For example, backwards line counting is not sped up, which can sometimes be noticeable. If you feel like working on less-speedup issues, feel free to drop me a line.
- jingo 12y ago
  "... I got fed up with the extremely long time it takes less to precalculate line counts on large files..." When you were experiencing long wait times, did you turn off line numbering? less -n logfile
  - sounds 12y ago
    You can also turn off line numbering _during_ the long "calculating line numbers" phase. Hit Ctrl-C and it says "line numbering off" or somesuch.
  - nhaehnle 12y ago
    Sure, but that's not an option if you actually want the line numbers. Given that it was possible to speed up the line number calculation by more than an order of magnitude, I do believe that fixing the code was the right way to go :)
morsch 12y ago
One of the lesser known less features is filtering only matching lines using &pattern. This is also very cool in combination with F, ie. tail -f mode. Unfortunately it tends to be extremely slow in large files, even though grep seemingly has no problem with them. I suspect it's related to search performance. Overall I think less is one of those tools where it's really valuable to spend 10 minutes a day in the man page for a week, which should be enough to learn essentially all of its functionality. Markers are also very useful, particularly paired with the functionality to pipe data to another file or shell command. E.g. to extract the instance of a server error plus some lines for context from an otherwise unwieldy log file. :) I use markers rarely enough that I invariable need to reread the man-/help-page, but being aware of the functionality is half the battle. :) Another tip: within less, press -S to toggle line wrap. (Works for most other command line options, too.)
- mercurial 12y ago
  > Markers are also very useful, particularly paired with the functionality to pipe data to another file or shell command. E.g. to extract the instance of a server error plus some lines for context from an otherwise unwieldy log file. :) I use markers rarely enough that I invariable need to reread the man-/help-page, but being aware of the functionality is half the battle. :) Isn't that just grep -C?
  - morsch 12y ago
    Sort of an interactive version of grep -C.
NickM 12y ago
I wholeheartedly agree, and I'm honestly surprised nobody has done this already. I often find that grep is ridiculously fast compared to less. It seems like a huge shame that a tool uses by so many people on such a regular basis is so slow at such a simple, commonplace task. Honestly I'd take a stab at myself if I had the time. Maybe I should start a kickstarter or something like that.

by nn3 12y ago

porker 12y ago

Would using ag help? No longer maintained AFAIK, but fast for me.

chrismonsanto 12y ago
where did you get that idea? last commit was 7 days ago, no mention of abandonment
- porker 12y ago
  From the status of the Ubuntu ppa, and (I thought) comments on his site. Thanks for pointing out that's not the case.
e12e 12y ago
As far as I know, it still holds that ag isn't fast, ack is slow ;-) That is to say: grep is pretty fast (too). In other words, ack improved the user-interface and api for search across files, with an eye towards programming and editors, but was relatively slow -- ag aims to keep the improved ui/api/output but bring speed back up to gnu grep-like levels.
- porker 12y ago
  I think if your grep has the -s flag (the GNU 'treat as string' option) you're correct. If not, it may be faster :)
  - e12e 12y ago
    You might be thinking of -F/--fixed-strings. -s is slient (long option --no-messages). For GNU grep 2.12, anyway. Or you might be thinking of BSD grep: http://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html http://lists.freebsd.org/pipermail/freebsd-current/2010-Augu... edit: Eg, with warm cache: :~/tmp/riak/riak-2.0.0pre5/deps $ time (find . -type f -exec cat '{}' \; |wc -l) 2765699 real 0m5.021s user 0m0.144s sys 0m0.792s :~/tmp/riak/riak-2.0.0pre5/deps $ time (find . -type f -exec cat '{}' \; |grep -E 'Some pattern' -v -c) 2765700 real 0m5.133s user 0m0.264s sys 0m0.852s :~/tmp/riak/riak-2.0.0pre5/deps $ time (find . -type f -exec cat '{}' \; |grep -E 'Some..pattern' -v -c) 2765700 real 0m5.144s user 0m0.400s sys 0m0.768s # "%% " used for leading comment lines in some of this code: :~/tmp/riak/riak-2.0.0pre5/deps $ time (find . -type f -exec cat '{}' \; |grep -E '^%% ' -c) 27535 real 0m5.597s user 0m0.520s sys 0m0.788s :~/tmp/riak/riak-2.0.0pre5/deps $ du -hcs . 405M . 405M total :~/tmp/riak/riak-2.0.0pre5/deps $ time (find . -type f -exec cat '{}' \; |ag '^%% ' >/dev/null) real 0m5.735s user 0m1.480s sys 0m0.876s #actually find/cat is pretty slow -- I guess both GNU grep and ag #use nmap to good effect: $ time rgrep '^%% ' . > /dev/null real 0m0.539s user 0m0.404s sys 0m0.128s :~/tmp/riak/riak-2.0.0pre5/deps $ time ag '^%% ' . |wc -l 27500 real 0m0.252s user 0m0.284s sys 0m0.068s :~/tmp/riak/riak-2.0.0pre5/deps $ time rgrep -E '^%% ' . |wc -l 27553 real 0m0.535s user 0m0.396s sys 0m0.140s Note that grep clearly goes looking in more files here (more mathcing lines). Still, I guess ag is indeed faster than grep in some cases (even if it might not be apples to apples depending how you count -- of course the whole point of ag is to help search just the right files). :~/tmp/riak/riak-2.0.0pre5/deps $ time rgrep -E 'Some pattern' . |wc -l 0 real 0m0.266s user 0m0.128s sys 0m0.132s :~/tmp/riak/riak-2.0.0pre5/deps $ time rgrep -E 'Some..pattern' . |wc -l 0 real 0m0.338s user 0m0.212s sys 0m0.120s :~/tmp/riak/riak-2.0.0pre5/deps $ time ag 'Some..pattern' . |wc -l 0 real 0m0.111s user 0m0.100s sys 0m0.076s I guess ag is indeed faster, even if it might not be due to fixed string search... [edit2: For those wondering that's an (old) ssd, on an old machine -- but with ~4G ram the working set should fit, as soon as some of my open tabs in ff are paged to disk...]

nhaehnle 12y ago

Not for string search, but I got fed up with the extremely long time it takes less to precalculate line counts on large files when I was working with log files of a size in the order of GBs. The result is here: https://github.com/nhaehnle/less https://github.com/nhaehnle/less Since it was only a small hack to scratch the itch I was having at the time, I never really completed that project. For example, backwards line counting is not sped up, which can sometimes be noticeable. If you feel like working on less-speedup issues, feel free to drop me a line.

jingo 12y ago
"... I got fed up with the extremely long time it takes less to precalculate line counts on large files..." When you were experiencing long wait times, did you turn off line numbering? less -n logfile
- sounds 12y ago
  You can also turn off line numbering _during_ the long "calculating line numbers" phase. Hit Ctrl-C and it says "line numbering off" or somesuch.
- nhaehnle 12y ago
  Sure, but that's not an option if you actually want the line numbers. Given that it was possible to speed up the line number calculation by more than an order of magnitude, I do believe that fixing the code was the right way to go :)

morsch 12y ago

One of the lesser known less features is filtering only matching lines using &pattern. This is also very cool in combination with F, ie. tail -f mode. Unfortunately it tends to be extremely slow in large files, even though grep seemingly has no problem with them. I suspect it's related to search performance. Overall I think less is one of those tools where it's really valuable to spend 10 minutes a day in the man page for a week, which should be enough to learn essentially all of its functionality. Markers are also very useful, particularly paired with the functionality to pipe data to another file or shell command. E.g. to extract the instance of a server error plus some lines for context from an otherwise unwieldy log file. :) I use markers rarely enough that I invariable need to reread the man-/help-page, but being aware of the functionality is half the battle. :) Another tip: within less, press -S to toggle line wrap. (Works for most other command line options, too.)

mercurial 12y ago
> Markers are also very useful, particularly paired with the functionality to pipe data to another file or shell command. E.g. to extract the instance of a server error plus some lines for context from an otherwise unwieldy log file. :) I use markers rarely enough that I invariable need to reread the man-/help-page, but being aware of the functionality is half the battle. :) Isn't that just grep -C?
- morsch 12y ago
  Sort of an interactive version of grep -C.

NickM 12y ago

I wholeheartedly agree, and I'm honestly surprised nobody has done this already. I often find that grep is ridiculously fast compared to less. It seems like a huge shame that a tool uses by so many people on such a regular basis is so slow at such a simple, commonplace task. Honestly I'd take a stab at myself if I had the time. Maybe I should start a kickstarter or something like that.