4 ms·

> I use grep on a flat file, and testing now it takes 200ms for a query across my 180k history entries 200ms is order of magnitude more than I'd prefer for int

by zokier 3y ago

> I use grep on a flat file, and testing now it takes 200ms for a query across my 180k history entries 200ms is order of magnitude more than I'd prefer for interactive use

fnordpiglet 3y ago
Yeah that was my thought too, and I thought about fishing out the unbranded $1 1Gb microsd in my drawer I use for 3d printing to help him out with his 0.5Gb storage challenge
PhilipRoman 3y ago
200ms for 180k entries sounds way too slow. Just tested on my 150k line log file and it takes about 10ms on average for various kinds of patterns with various hit frequencies. On an unrelated note, from a quick check with "strace", gnu grep seems to detect when output is redirected to /dev/null as no write syscalls are being made in that case.
- watermelon0 3y ago
  Unrelated note intrigued me, so I went looking in the source code; detection is here: https://git.savannah.gnu.org/cgit/grep.git/tree/src/grep.c#n2886 https://git.savannah.gnu.org/cgit/grep.git/tree/src/grep.c#n... Interesting that they would optimize for this use case.
  - remram 3y ago
    Especially since grep -q exists. It would never come to my mind to direct grep's output to /dev/null, I wonder how this optimization came to be?
jefftk 3y ago
I was going to say that it was "instant", because that's how it feels when using it, but got 200ms from running "time" on grepping it to include a number in the post.
- doodlesdev 3y ago
  Have you tried out using ripgrep [0] or fzf [1]? 200ms is quite a lot for history search, I'd expect search to be 20ms or less. In fact, it should be instantaneous as it should all fit in memory really. I've been using fzf and the searches are all faster than my monitor can display frames (so less than 16ms), although I only have 10k entries on my history as I clear it up once in a while. [0]: https://github.com/BurntSushi/ripgrep https://github.com/BurntSushi/ripgrep [1]: https://github.com/junegunn/fzf https://github.com/junegunn/fzf
  - jefftk 3y ago
    Trying now, "cat > /dev/null" takes ~150ms, so I suspect it's related to keeping my history in Google Drive (with local mirroring). If it gets annoyingly slows I'll stop doing that and use a different syncing system.
    - jefftk 3y ago
      Actually, I was an idiot and misreading these numbers. It's 0.020s and 0.015s, which are 20ms and 15ms, not 200ms and 150ms.

by zokier 3y ago

> I use grep on a flat file, and testing now it takes 200ms for a query across my 180k history entries 200ms is order of magnitude more than I'd prefer for interactive use

fnordpiglet 3y ago

Yeah that was my thought too, and I thought about fishing out the unbranded $1 1Gb microsd in my drawer I use for 3d printing to help him out with his 0.5Gb storage challenge

PhilipRoman 3y ago

200ms for 180k entries sounds way too slow. Just tested on my 150k line log file and it takes about 10ms on average for various kinds of patterns with various hit frequencies. On an unrelated note, from a quick check with "strace", gnu grep seems to detect when output is redirected to /dev/null as no write syscalls are being made in that case.

watermelon0 3y ago
Unrelated note intrigued me, so I went looking in the source code; detection is here: https://git.savannah.gnu.org/cgit/grep.git/tree/src/grep.c#n2886 https://git.savannah.gnu.org/cgit/grep.git/tree/src/grep.c#n... Interesting that they would optimize for this use case.
- remram 3y ago
  Especially since grep -q exists. It would never come to my mind to direct grep's output to /dev/null, I wonder how this optimization came to be?

jefftk 3y ago

I was going to say that it was "instant", because that's how it feels when using it, but got 200ms from running "time" on grepping it to include a number in the post.

doodlesdev 3y ago
Have you tried out using ripgrep [0] or fzf [1]? 200ms is quite a lot for history search, I'd expect search to be 20ms or less. In fact, it should be instantaneous as it should all fit in memory really. I've been using fzf and the searches are all faster than my monitor can display frames (so less than 16ms), although I only have 10k entries on my history as I clear it up once in a while. [0]: https://github.com/BurntSushi/ripgrep https://github.com/BurntSushi/ripgrep [1]: https://github.com/junegunn/fzf https://github.com/junegunn/fzf
- jefftk 3y ago
  Trying now, "cat > /dev/null" takes ~150ms, so I suspect it's related to keeping my history in Google Drive (with local mirroring). If it gets annoyingly slows I'll stop doing that and use a different syncing system.
  - jefftk 3y ago
    Actually, I was an idiot and misreading these numbers. It's 0.020s and 0.015s, which are 20ms and 15ms, not 200ms and 150ms.