14 ms·

Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, tar.gz

684 points by angrygoat 6y ago

ssivark 6y ago
I love that we’re seeing fast & flexible solutions for personal search. I’ve recently been playing with Recoll for full-text-search on content. Since it indexes content up front, the search is pretty fast. It can also easily accommodate tag metadata on files. It would be interesting to consider how ripgrep based tools can fit into generically broad “search your database of content” workflows (as opposed to remember or go through your file system paths).
- rjzzleep 6y ago
  rga also indexes them when you search. To be honest I like that approach a lot more since it saves space and I generally know where I'm looking for things ls -sh ~/.cache/rga/ total 336M 336M data.mdb 4.0K lock.mdb
  - curious_tenet 6y ago
    Wow that is so cool!
  - ssivark 6y ago
    That kind of caching is an interesting solution to incrementally building a database instead of spending hours up-front indexing. So the tool is ready for immediate use. Quite nifty :-)
- simias 6y ago
  FZF + ripgrep is really killer for me. I don't even bother organizing my notes anymore, I just throw everything markdown files in a flat directory and then I have a script that uses FZF + ripgrep to search through it when I need it. I search by "last modified first" so unless I'm digging for something very old the results are instant. Code snippets, finances, TODO lists, cake recipes... It's all in there. I use the same system in Vim to browse source code. It's very powerful, very fast, works with any language and requires zero configuration.
  - polyrand 6y ago
    I couldn't agree more with that. I wrapped a bash function to search through my notes folder with fzf + rg and it works perfectly. Also, I have a specific pattern to write some tags inside files that I can parse with ripgrep.
  - rshm 6y ago
    Can you share your script.
    - maxioatic 6y ago
      I'd love some more info as well!
    - simias 6y ago
      This is the main one (that actually only uses FZF, not ripgrep): https://gist.github.com/simias/b1d8356469d2a9386deeb7c45984b961 https://gist.github.com/simias/b1d8356469d2a9386deeb7c45984b... You'll need to set NOTES_DIR in your environment to wherever you want your notes to be stored. Then you can write `note something` to create or open $NOTES_DIR/something.md with your $EDITOR. If you type "note" without parameter you'll start a search on all the note names, ordered by last use. If you type "note -f" it starts a full text search. For best results you should have the fzf.vim's preview.sh somewhere in your fs, otherwise it'll use "cat" but it won't be as good looking (see FZF_PREVIEW in the script). Hopefully despite being shell it should be readable enough to tweak to your liking. Note that it was written and used exclusively on Linux, but I did try to avoid GNU-isms so hopefully it should work on BSDs and maybe even on MacOS with a bit of luck.
  - tasuki 6y ago
    About a year ago, I discovered it was very helpful for me to have git branches ordered by "recently modified first": From my `~/.gitconfig`: [alias] brt = "!git for-each-ref refs/heads --color=always --sort -committerdate --format='%(HEAD)%(color:reset);%(color:yellow)%(refname:short)%(color:reset);%(contents:subject);%(color:green)(%(committerdate:relative))%(color:blue);<%(authorname)>' | column -t -s ';'" I always spent a lot of time being confused about branches, and never realised how easy the solution was.
diimdeep 6y ago
Is anyone preferring some other search tool other than Spotlight on macOS ?
- qppo 6y ago
  I use ripgrep on macos
- cpach 6y ago
  I like Spotlight, and its CLI companion mdfind.
- michaelcampbell 6y ago
  ripgrep in emacs for me.
vmchale 6y ago
Wonderful! pdfgrep is good but slow.
- phiresky 6y ago
  pdfgrep has a --cache option since a while ago :) Not sure why they don't enable it by default. Still, this is much faster.
hobofan 6y ago
Big fan of rga! I use it almost every day for the academic part of my life, when I want to know the location of some specific keywords in my lecture slides, books or papers I've been reading. Even for single ebooks, it is often more useful than the search in Acrobat Reader.
- durnygbur 6y ago
  > search in Acrobat Reader The search in PDF viewers is an anti-feature in terms of UI and performance. Their advantage is that they allow to scroll to and highlight the found phrase back in the document.
  - mssdvd 6y ago
    The search in most application is an anti-feature.
  - solstice 6y ago
    The search in Tracker Software's PDF X-Change Viewer/Editor is really great. Effective and easy to use
SamuelAdams 6y ago
Any advantages to this over something like Agent Ransack? https://www.mythicsoft.com/agentransack/ https://www.mythicsoft.com/agentransack/
- tfigment 6y ago
  Command line, linux, and open source immediately come to mind
- fnord123 6y ago
  Works on non-Windows. ripgrep is notoriously fast. Command line interface. Not comically priced at 59.95 USD.
  - cpach 6y ago
    Why is there an expectation that every application should be free or cheap? IMHO $60 is very reasonable for a program that can save a lot of time for the user. And developers also have to eat, and might want to some day retire.
    - michaelcampbell 6y ago
      $60 may not be much for you; it may be worth it for you. For others it may not be.
    - fnord123 6y ago
      I'm not the boss of you. If you want to spend 60 USD on a program that is mostly built into Finder and Nautilus, fill your boots.
akavel 6y ago
The "Integration with fzf" example looks really cool: https://github.com/phiresky/ripgrep-all#integration-with-fzf https://github.com/phiresky/ripgrep-all#integration-with-fzf
globular-toast 6y ago
I have mixed feelings about these kinds of tools. I can understand it might be nice to have a personal library of PDF books and searching in them. I can't think of a time I've ever wished I could search my bookshelf in that way, but you never know. Obviously I use tools like ripgrep for searching codebases and the like. But the extreme flexibility of this one in particular (and others like MacOs Spotlight) makes it seem more like a data recovery tool for me. If my directory structures and databases ever completely failed for some reason I might need to search through everything to find the data again. It's good to know such tools exist, I suppose. But my fear is that tools like this teach people to not worry about organisation of data and to just fill up their disks with no structure at all. I think that unless something goes terribly wrong nobody should ever need a tool like this. Once you rely on it, you're out of luck it if it ever fails you. What if you just can't remember a single searchable phrase from some document, but you just know it must exist somewhere? It's similar to what Google has done to the web. When I was growing up it used to be a skill to use the web. People used tools like bookmarks and followed links from one place to another. Now it's just type it into Google and if Google doesn't know, it doesn't exist.
- armoredkitten 6y ago
  I mean, I understand what you mean when it comes to Google -- the web essentially becomes locked into a particular proprietary solution to finding information. I definitely still have hundreds (maybe into the thousands?) of bookmarks of sites that store information I care about. But I don't think this tool deserves the same sort of mixed feelings. I don't think this replaces structure -- there's still value to having a conceptual mapping of where documents are stored, and for grouping sets of documents together. It's just that having a structure doesn't help if you don't know where in the structure something is stored. This sort of tool is a bottom-up approach for the times when the top-down approach doesn't work very well. Do you have similarly mixed feelings if sometimes, even with my carefully-crafted set of bookmarks with all their nested folders, I use the search tool to find the bookmark I'm looking for? It's the same idea. Sometimes a top-down structure is beneficial. But sometimes things get misclassified, or you forget about some piece of the structure, or you aren't familiar with some new structure, and in those cases, having bottom-up tools are immensely useful. There's no risk of vendor lock-in here. It's just a difference of approach in information retrieval.
antegamisou 6y ago
I always found useful something along the lines of pdftotext -layout file.pdf | grep -E ... for PDFs, good to see a Swiss Army knife utility for all sorts of file though!
- phiresky 6y ago
  rga uses pdftotext (from poppler) internally for pdfs, except wraps it in parallelization and a very fast cache layer, since you usually want to do multiple queries per file :)
faitswulff 6y ago
I noticed that you can use Tesseract as an OCR adapter for rga. Tesseract is written in python, IIRC, and in the OP it comes with a warning that it’s slow and not enabled by default. Are there any other fast, reliable OCR libs out there? Or any rust OCR backends?
- mouldysammich 6y ago
  https://github.com/tesseract-ocr/tesseract https://github.com/tesseract-ocr/tesseract seems to be written in c++ not python
  - faitswulff 6y ago
    Ah, my mistake then.
- hobofan 6y ago
  I don't think the problem necessarily is that Tesseract is slow, but that the whole process of rendering a PDF to a series of PNGs on which you can then run OCR is slow (which is what it does in the background).
  - undebuggable 6y ago
    The process of converting all pages to raster images and then OCR-ing each one takes hours for PDFs hundreds of pages long. This workflow is not suitable for instant search. For non OCR-ed PDFs it's worth to pregenerate the text.
    - hobofan 6y ago
      That's why rga comes with a cache. I've occasionally used the Tesseract adapter with good success (results-wise), and after the inital rendering and indexing, it's fast enough to use.
maxioatic 6y ago
This is great. I have 100+ ebooks/pdfs of programming and textbooks of which I've been extracting the index pages of. My intention was to always make some sort of search index out of them. I will definitely be trialing this (initial few searches seem promising!)
- skanga 6y ago
  Great idea. Please update on whether this use case works or not! And other tips, examples, etc.
ghoomketu 6y ago
One a related note there is one program that I absolutely miss on Linux called everything (on windows). The closest I can find is mlocate but it does not have a GUI but more importantly it does not index my Windows or NTFS drives. Would appreciate any suggestions if someone knows something like 'everything' for Ubuntu.
- demosito666 6y ago
  There is about a dozen dedicated file indexers [1] on linux (some are gui, some are not) and also DE-integrated ones like Baloo for KDE and Tracker for Gnome. [1] https://lmgtfy.app/?q=Is+there+a+file+search+engine+like+%E2%80%9CEverything%E2%80%9D+in+Linux%3F https://lmgtfy.app/?q=Is+there+a+file+search+engine+like+%E2...
- opan 6y ago
  Haven't used this, but heard of it years ago, and it aims to be similar. https://github.com/dotheevo/angrysearch/ https://github.com/dotheevo/angrysearch/
  - ghoomketu 6y ago
    Thanks! That's super nice and very close to what I was looking for all this time. I just learned how to mount all my Windows drive under /mnt using (using the `disks` software), so hopefully this should index those files too.
- ronjouch 6y ago
  Everything is great on Windows to pick files/folders. From the linux command-line, I like fzf ( https://github.com/junegunn/fzf https://github.com/junegunn/fzf ), that you can instruct to use the faster fd ( https://github.com/junegunn/fzf#environment-variables https://github.com/junegunn/fzf#environment-variables ). Fzf even offers keybindings for your shell. For example, it binds Alt+C to fuzzy-finding a directory, and Enter cds to it ( https://github.com/junegunn/fzf#key-bindings-for-command-line https://github.com/junegunn/fzf#key-bindings-for-command-lin... ). Fzf is great for other things too; here is a fish function to bing Alt+G to fuzzy-pick a Git branch and jump to it: function fish_user_key_bindings bind \eg 'test -d .git; or git rev-parse --git-dir > /dev/null 2>&1; and git checkout (string trim -- (git branch | fzf)); and commandline -f repaint' bind \eG 'test -d .git; or git rev-parse --git-dir > /dev/null 2>&1; and git checkout (string trim -- (git branch --all | fzf)); and commandline -f repaint' end
durnygbur 6y ago
No ripgrep-all through the package manager: $ sudo dnf install -y ripgrep-all [...] No match for argument: ripgrep-all Error: Unable to find a match: ripgrep-all Rust's package manager fails: $ cargo install ripgrep_all [...] failed to select a version for the requirement `cachedir = "^0.1.1"` candidate versions found which didn't match: 0.2.0 location searched: crates.io index required by package `ripgrep_all v0.9.6` Quick search on the web shows that more people have problems with cachedir version.
- ChrisSD 6y ago
  It looks like cachedir yanked version 0.1.1. This is usually only done when a very serious issue is discovered, though I don't know what the reason is in this case. https://crates.io/crates/cachedir https://crates.io/crates/cachedir
- est31 6y ago
  You can do cargo install --locked ripgrep_all as a workaround. It uses the lockfile that's part of the ripgrep_all package, so you miss out on some package updates, but can also get the cachedir version required. There is a github issue to make this the default behaviour of cargo, but you miss out on updates which might fix security bugs so the cargo team is unwilling to change the default. https://github.com/rust-lang/cargo/issues/7169 https://github.com/rust-lang/cargo/issues/7169
hiq 6y ago
It would be nice to have a direct comparison with ugrep. In the case of rg the benchmarks are already enough to switch. Why should I use rga instead of ugrep?
- burntsushi 6y ago
  I've called the ugrep benchmarks into question, and I elaborated on it here (and this includes a frustrating exchange between myself and the ugrep author): https://old.reddit.com/r/rust/comments/i6pfb2/ugrep_new_ultrafast_c_grep_claims_to_be_faster/g0xybge/ https://old.reddit.com/r/rust/comments/i6pfb2/ugrep_new_ultr... I've also re-run my original set of benchmarks[1] with ugrep included: https://github.com/BurntSushi/ripgrep/blob/master/benchsuite/runs/2020-10-14-archlinux-frink/summary https://github.com/BurntSushi/ripgrep/blob/master/benchsuite... [1] - https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/
  - hiq 6y ago
    Just to be clear, I meant that I had switched to ripgrep because its speed was convincing enough on its own (so I did not even extra features to switch). I'm currently not using any of ugrep or rga, although I have used pdfgrep in the past. It'd be nice for casual users like me to know more about why I should use rga over ugrep (or vice-versa).
fock 6y ago
can it produce links to open the file yet (don't know rust, so can't add a PR easily). At least gnome-terminal supports that (and normally it should also support opening a specific pdf page)!
- mcintyre1994 6y ago
  Not sure if the implementation is in rg or zsh, but that combination produces cmd-clickable file names for me.
kovek 6y ago
How could I use Rga to search my browsing history?
- simonw 6y ago
  Your browser history (if you use Chrome or Firefox at least) is stored in a SQLite database. It looks like rga can handle SQLite out of the box, so just making sure your history .db file is visible to rga may be all you need. You can also use my Datasette tool to get a web UI against your history, see https://docs.datasette.io/en/stable/getting_started.html#using-datasette-on-your-own-computer https://docs.datasette.io/en/stable/getting_started.html#usi...
  - kovek 6y ago
    Are there tools to store the all history webpage contents to be viewed later?
chris_st 6y ago
Curious why this isn't a pull request to ripgrep? Maybe it was, and rejected? It'd be nice to just have one tool, and this doesn't feel like it's a stretch to add to ripgrep.
- burntsushi 6y ago
  It's a stretch. A big one. I answered this a while back: https://old.reddit.com/r/rust/comments/c1bjw4/rga_ripgrep_but_also_search_in_pdfs_ebooks_office/erhy4wd/ https://old.reddit.com/r/rust/comments/c1bjw4/rga_ripgrep_bu...
root_axis 6y ago
This is really great.
awinter-py 6y ago
thanks but it's way faster to have my stuff in G drive that way I can open a browser tab, wait 5 seconds for it to load, locate the new screen location of the search bar, click it, wait for javascript to finish loading so I can click the search bar, click it for real this time, mistype because there's some kind of contenteditable event jank, wait 5 seconds for my results to come up, fix the typo, and just have my results waiting for me I'm not going to learn a new tool when web is fine
- ffpip 6y ago
  If you're using Duckduckgo, just search ''!drive search-term'' or ''search term !drive'' or ''search !drive term'' More !operators here - https://duckduckgo.com/bang https://duckduckgo.com/bang
  - deleted 6y ago
    [deleted]
  - spappal 6y ago
    Firefox supports custom search engines, the most bang for the buck custom search engine must be https://duckduckgo.com/?q=%s https://duckduckgo.com/?q=%s with keyword being the letter d. Then you get all these 13000+ bangs without having to configure the custom search engines. E.g. write "d !drive term" in url bar. And "d !w hacker news" sends you directly to https://en.wikipedia.org/wiki/Hacker_News https://en.wikipedia.org/wiki/Hacker_News
    - codethief 6y ago
      Or you just set DDG as your default search engine and then you don't even have to type the "d" anymore. :)
    - t0astbread 6y ago
      Slightly off-topic but this is an important PSA. Firefox keyword search has one little known killer feature: You can combine it with data URIs and JavaScript to run small "command line snippets" stored in your bookmarks from your browser bar. To get started, create a keyword search from any form (like the search bar on duckduckgo.com) and edit the URL of the entry in the bookmark manager to point to data:text/html,<script>alert("%s")</script> instead. What you can do with this is (fortunately) limited by cross-origin restrictions but there are some useful applications. For example, I use this snippet data:text/html,<script>i="%s";firstSep=i.indexOf(" ");if(firstSep==-1)firstSep=i.length;subreddit=i.substr(0,firstSep);searchTerm=i.substr(firstSep+1);location=`https://old.reddit.com/r/${encodeURIComponent(subreddit)}${(searchTerm!=null&&searchTerm.replace(/\s/,"").length>0)?`/search?q=${encodeURIComponent(searchTerm)}&restrict_sr=1`:""}`</script> as a nice shortcut to Reddit ("<keyword> <subreddit>" to jump to a Subreddit, "<keyword> <subreddit> <search string>" to search within a Subreddit). You can also insert content directly into the document which opens the possibility for instant marquee data:text/html,<marquee>%s</marquee>
cb321 6y ago
NOTE: ripgrep already has --pre. (No pre-built indexing, of course.)
- burntsushi 6y ago
  That's exactly what ripgrep-all uses to implement this. There's a lot of integration work required to make this nice. The --pre flag is just a small hook. More info on it here: https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md#preprocessor https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md#p...
  - cb321 6y ago
    Yup. Something perhaps more helpful but so far unmentioned (and somewhat OS-specific) is that statically linked executables usually fork & exec (especially exec) much faster than dynamically linked ones. This difference is usually only like 50..150 us vs 500..3000 us but can multiply up over thousands of files. This only matters on the first run of `rga`, of course. While the dispatched-to decoder is likely mostly out of one's linking control, this overhead can be saved for the dispatcher, at least. So, I would suggest `rga-preproc` should have a static linking option/suggestion, at least on Linux. Of course, this overhead may also fall below the noise of PDF/ebook/etc. parsing, but maybe not the decompression of small files in some dark horse format. :-)
soferio 6y ago
Can it (or any tool) perform proximity searches on scanned PDFs? E.g word1 within 20 words of word2, on scanned PDFs? (I think this is non trivial but very useful.)
- phiresky 6y ago
  Scanned PDFs only work well if they already have an OCR layer. There's some optional integration of rga with tesseract, but it's pretty slow and less good than external OCR tools. ripgrep-all can do the same regexes as rg on any filetypes it supports. So you can could do something like --multiline and foo(\w+[\s\n]+){,20}bar It won't work exactly like this, but something similar should do it: --multiline enables multiline matching * foo searches for foo * \w+ searches for at least one word character * [\W]+ searches for at least one space/nonword character like sentence marks * {,20} searches for at most 20 iterations of the word-space combination bar searches for bar
- ballmerspeak 6y ago
  If its a scanned PDF (essentially a collection of 1 image per page), there would need to be an OCR step to get some text out first. Tesseract would work for this. Once that's done, you have all the options available to perform that search. But I don't know of a search tool that does the OCR for you. I did read a blog post of someone uploading PDFs to google drive (they OCR them on upload) as an easy way to do this.
gopty 6y ago
Sounds like a poor man's version of recoll https://www.lesbonscomptes.com/recoll/ https://www.lesbonscomptes.com/recoll/ A PDF in a Zip file, in an email attachment. recoll can index it and do OCR if you like
phiresky 6y ago
Developer of the tool here :) Glad to see it posted here, I still actively use it myself. Also check out the fzf integration in the README: https://github.com/phiresky/ripgrep-all/blob/master/doc/rga-fzf.gif https://github.com/phiresky/ripgrep-all/blob/master/doc/rga-... Currently the main branch is undergoing a refactor to add support for having custom extractors (calling out to other tools), and more flexible chains of extractors. Ripgrep itself has functionality integrated to call custom extractors with the `--pre` flag, but by adding it here we can retain the benefits of the rga wrapper (more accurate file type matchers, caching, recursion into archives, adapter chaining, no slow shell scripts in between, etc). Sadly, during rewriting it to allow this, I kind of got hung up and couldn't manage to figure out how to cleanly design that in Rust. I'd be really glad if a Rust expert could help me out here: In the currently stable version, the main interface of each "adapter" is `fn(Read, Write) -> ()`. To allow custom adapter chaining I have to change it to be `fn(Read) -> Read` where each chained adapter wraps the read stream and converts it while reading. But then I get issues with how to handle threading etc, as well as a random deadlock that I haven't figured out how to solve so far :/
- cb321 6y ago
  One possibility is the almost dirt-simple solution wherein you just have a "make"/"Makefile" (or your favorite other build system) maintain a shadow tree of parallel pre-translated files. You get parallelism via `make -j$(nproc)` or its equivalent. Every name in the shadow is built from the name in the origin but maybe with ".txt" added (or .txt.gz if you want to keep the compressed with whatever is the fastest decompressor builtin to ripgrep as a library not called as a program). Untranslated names can be just symbolic/hard links back to the origin. Build rules become as flexible as your build system. This also scales to deployments that have more disk space than memory. Admittedly, in that case, the whole procedure probably becomes disk-IO bound, but maybe not. Maybe some translations cannot even keep up with disk IO - NVMe storage is pretty fast, for example. Or available memory may vary dynamically a lot, sometimes allowing the shadow to be fully in the buffer cache, other times not. It strikes me as less presumptuous to assume you can find disk space vs. having that much memory available. (EDIT2: though I may be confused about how `rga` operates - your doc says "memory cache", though.) On the pro-side, but for updating the shadows based on origins, the user could even just `rg` from within the shadow and translate filenames "in their head", although stripping an always present string is obviously trivial. Indeed, you won't need `rg --pre` at all and the grep itself could become pluggable. I doubt any of your other `fzf`/etc. integrations would be made more complicated by this design, either. This all strikes me as simple/nice enough that someone has probably already done it...EDIT1: Oh, I see from thumbs ups and other comments over at [1] and [2] that @phiresky is probably already aware of this design idea, but maybe some HN person knows of an existing solution along these lines. [1] https://github.com/BurntSushi/ripgrep/issues/978 https://github.com/BurntSushi/ripgrep/issues/978 [2] https://github.com/BurntSushi/ripgrep/pull/981 https://github.com/BurntSushi/ripgrep/pull/981
nikisweeting 6y ago
Aww hell yeah we should definitely use this in place of ripgrep for the new ArchiveBox.io full-text search backend. https://github.com/ArchiveBox/ArchiveBox/pull/543 https://github.com/ArchiveBox/ArchiveBox/pull/543
alexruf 6y ago
Idea behind Rga is cool. Anyway, I tried it on Mac and installed via Homebrew. The formula already says it depends on ripgrep (that's fine since I have ripgrep already installed and use it regularly). I still was surprised when I executed Rga for the first time and got an error message that 'pdftotext' was not found. Since pdftotext has been officially discontinued, I am not sure if I want to install an old version just to make Rga work on my machine. Don't think it's an good idea to rely on a project which is not maintained actively.
- burntsushi 6y ago
  > Since pdftotext has been officially discontinued Do you have a link for that? That's news to me.
  - alexruf 6y ago
    brew info pdftotext https://formulae.brew.sh/cask/pdftotext#default https://formulae.brew.sh/cask/pdftotext#default
    - burntsushi 6y ago
      That looks like a problem with that specific package, and not pdftotext that is in poppler. I don't even know what that package is. It links to bluem.net?
- there_the_and 6y ago
  I don't see any indication that pdftotext has been discontinued [1]. It looks like a Mac-specific installer available via Homebrew Cask has been discontinued [2], but pdftotext is still available through the normal poppler formula [3]. 1. https://poppler.freedesktop.org/releases.html https://poppler.freedesktop.org/releases.html 2. https://formulae.brew.sh/cask/pdftotext https://formulae.brew.sh/cask/pdftotext 3. https://formulae.brew.sh/formula/poppler https://formulae.brew.sh/formula/poppler
- phiresky 6y ago
  Yeah, In my opinion poppler should be a dependency of rga in homebrew (since it's kinda useless without having the default adapters), but I don't maintain that package.
edm0nd 6y ago
Big fan of ripgrep. Use it on Windows to search through 100s of GBs of data really quickly.
- CamperBob2 6y ago
  Do you have a version that works with DOS pipes? I have to resort to an older grep version if I need to do something like: dir *.c* | rg somesubstringinfilename
dang 6y ago
If curious see also 2019 https://news.ycombinator.com/item?id=20196982 https://news.ycombinator.com/item?id=20196982
supernova87a 6y ago
For PDFs, how does it (does it?) deal with for example, when phrases get ripped apart by the layout? Like if you search for a multiple word phrase, it's often foiled by word wrap or being in a table.
0df8dkdf 6y ago
Great tool!!!
lafrenierejm 6y ago
If anyone is interested gron [0], I have an open PR [1] to add it as an adapter to ripgrep-all. The patch was based on the most recent release, since master is currently not functional. 0: https://github.com/TomNomNom/gron https://github.com/TomNomNom/gron 1: https://github.com/phiresky/ripgrep-all/pull/77 https://github.com/phiresky/ripgrep-all/pull/77
patricktlo 6y ago
Thanks! This is a godsend for someone like me who needs to search through many PDFs/docx documents to find information for work!
lenkite 6y ago
Is there a way to alias adapters ? So that .jar and .esa can be used for .zip ?

684 points by angrygoat 6y ago

I love that we’re seeing fast & flexible solutions for personal search. I’ve recently been playing with Recoll for full-text-search on content. Since it indexes content up front, the search is pretty fast. It can also easily accommodate tag metadata on files. It would be interesting to consider how ripgrep based tools can fit into generically broad “search your database of content” workflows (as opposed to remember or go through your file system paths).

rjzzleep 6y ago
rga also indexes them when you search. To be honest I like that approach a lot more since it saves space and I generally know where I'm looking for things ls -sh ~/.cache/rga/ total 336M 336M data.mdb 4.0K lock.mdb
- curious_tenet 6y ago
  Wow that is so cool!
- ssivark 6y ago
  That kind of caching is an interesting solution to incrementally building a database instead of spending hours up-front indexing. So the tool is ready for immediate use. Quite nifty :-)
simias 6y ago
FZF + ripgrep is really killer for me. I don't even bother organizing my notes anymore, I just throw everything markdown files in a flat directory and then I have a script that uses FZF + ripgrep to search through it when I need it. I search by "last modified first" so unless I'm digging for something very old the results are instant. Code snippets, finances, TODO lists, cake recipes... It's all in there. I use the same system in Vim to browse source code. It's very powerful, very fast, works with any language and requires zero configuration.
- polyrand 6y ago
  I couldn't agree more with that. I wrapped a bash function to search through my notes folder with fzf + rg and it works perfectly. Also, I have a specific pattern to write some tags inside files that I can parse with ripgrep.
- rshm 6y ago
  Can you share your script.
  - maxioatic 6y ago
    I'd love some more info as well!
  - simias 6y ago
    This is the main one (that actually only uses FZF, not ripgrep): https://gist.github.com/simias/b1d8356469d2a9386deeb7c45984b961 https://gist.github.com/simias/b1d8356469d2a9386deeb7c45984b... You'll need to set NOTES_DIR in your environment to wherever you want your notes to be stored. Then you can write `note something` to create or open $NOTES_DIR/something.md with your $EDITOR. If you type "note" without parameter you'll start a search on all the note names, ordered by last use. If you type "note -f" it starts a full text search. For best results you should have the fzf.vim's preview.sh somewhere in your fs, otherwise it'll use "cat" but it won't be as good looking (see FZF_PREVIEW in the script). Hopefully despite being shell it should be readable enough to tweak to your liking. Note that it was written and used exclusively on Linux, but I did try to avoid GNU-isms so hopefully it should work on BSDs and maybe even on MacOS with a bit of luck.
- tasuki 6y ago
  About a year ago, I discovered it was very helpful for me to have git branches ordered by "recently modified first": From my `~/.gitconfig`: [alias] brt = "!git for-each-ref refs/heads --color=always --sort -committerdate --format='%(HEAD)%(color:reset);%(color:yellow)%(refname:short)%(color:reset);%(contents:subject);%(color:green)(%(committerdate:relative))%(color:blue);<%(authorname)>' | column -t -s ';'" I always spent a lot of time being confused about branches, and never realised how easy the solution was.

diimdeep 6y ago

Is anyone preferring some other search tool other than Spotlight on macOS ?

qppo 6y ago
I use ripgrep on macos
cpach 6y ago
I like Spotlight, and its CLI companion mdfind.
michaelcampbell 6y ago
ripgrep in emacs for me.

vmchale 6y ago

Wonderful! pdfgrep is good but slow.

phiresky 6y ago
pdfgrep has a --cache option since a while ago :) Not sure why they don't enable it by default. Still, this is much faster.

hobofan 6y ago

Big fan of rga! I use it almost every day for the academic part of my life, when I want to know the location of some specific keywords in my lecture slides, books or papers I've been reading. Even for single ebooks, it is often more useful than the search in Acrobat Reader.

durnygbur 6y ago
> search in Acrobat Reader The search in PDF viewers is an anti-feature in terms of UI and performance. Their advantage is that they allow to scroll to and highlight the found phrase back in the document.
- mssdvd 6y ago
  The search in most application is an anti-feature.
- solstice 6y ago
  The search in Tracker Software's PDF X-Change Viewer/Editor is really great. Effective and easy to use

SamuelAdams 6y ago

Any advantages to this over something like Agent Ransack? https://www.mythicsoft.com/agentransack/ https://www.mythicsoft.com/agentransack/

tfigment 6y ago
Command line, linux, and open source immediately come to mind
fnord123 6y ago
Works on non-Windows. ripgrep is notoriously fast. Command line interface. Not comically priced at 59.95 USD.
- cpach 6y ago
  Why is there an expectation that every application should be free or cheap? IMHO $60 is very reasonable for a program that can save a lot of time for the user. And developers also have to eat, and might want to some day retire.
  - michaelcampbell 6y ago
    $60 may not be much for you; it may be worth it for you. For others it may not be.
  - fnord123 6y ago
    I'm not the boss of you. If you want to spend 60 USD on a program that is mostly built into Finder and Nautilus, fill your boots.

akavel 6y ago

The "Integration with fzf" example looks really cool: https://github.com/phiresky/ripgrep-all#integration-with-fzf https://github.com/phiresky/ripgrep-all#integration-with-fzf

globular-toast 6y ago

I have mixed feelings about these kinds of tools. I can understand it might be nice to have a personal library of PDF books and searching in them. I can't think of a time I've ever wished I could search my bookshelf in that way, but you never know. Obviously I use tools like ripgrep for searching codebases and the like. But the extreme flexibility of this one in particular (and others like MacOs Spotlight) makes it seem more like a data recovery tool for me. If my directory structures and databases ever completely failed for some reason I might need to search through everything to find the data again. It's good to know such tools exist, I suppose. But my fear is that tools like this teach people to not worry about organisation of data and to just fill up their disks with no structure at all. I think that unless something goes terribly wrong nobody should ever need a tool like this. Once you rely on it, you're out of luck it if it ever fails you. What if you just can't remember a single searchable phrase from some document, but you just know it must exist somewhere? It's similar to what Google has done to the web. When I was growing up it used to be a skill to use the web. People used tools like bookmarks and followed links from one place to another. Now it's just type it into Google and if Google doesn't know, it doesn't exist.

armoredkitten 6y ago
I mean, I understand what you mean when it comes to Google -- the web essentially becomes locked into a particular proprietary solution to finding information. I definitely still have hundreds (maybe into the thousands?) of bookmarks of sites that store information I care about. But I don't think this tool deserves the same sort of mixed feelings. I don't think this replaces structure -- there's still value to having a conceptual mapping of where documents are stored, and for grouping sets of documents together. It's just that having a structure doesn't help if you don't know where in the structure something is stored. This sort of tool is a bottom-up approach for the times when the top-down approach doesn't work very well. Do you have similarly mixed feelings if sometimes, even with my carefully-crafted set of bookmarks with all their nested folders, I use the search tool to find the bookmark I'm looking for? It's the same idea. Sometimes a top-down structure is beneficial. But sometimes things get misclassified, or you forget about some piece of the structure, or you aren't familiar with some new structure, and in those cases, having bottom-up tools are immensely useful. There's no risk of vendor lock-in here. It's just a difference of approach in information retrieval.

antegamisou 6y ago

I always found useful something along the lines of pdftotext -layout file.pdf | grep -E ... for PDFs, good to see a Swiss Army knife utility for all sorts of file though!

phiresky 6y ago
rga uses pdftotext (from poppler) internally for pdfs, except wraps it in parallelization and a very fast cache layer, since you usually want to do multiple queries per file :)

faitswulff 6y ago

I noticed that you can use Tesseract as an OCR adapter for rga. Tesseract is written in python, IIRC, and in the OP it comes with a warning that it’s slow and not enabled by default. Are there any other fast, reliable OCR libs out there? Or any rust OCR backends?

mouldysammich 6y ago
https://github.com/tesseract-ocr/tesseract https://github.com/tesseract-ocr/tesseract seems to be written in c++ not python
- faitswulff 6y ago
  Ah, my mistake then.
hobofan 6y ago
I don't think the problem necessarily is that Tesseract is slow, but that the whole process of rendering a PDF to a series of PNGs on which you can then run OCR is slow (which is what it does in the background).
- undebuggable 6y ago
  The process of converting all pages to raster images and then OCR-ing each one takes hours for PDFs hundreds of pages long. This workflow is not suitable for instant search. For non OCR-ed PDFs it's worth to pregenerate the text.
  - hobofan 6y ago
    That's why rga comes with a cache. I've occasionally used the Tesseract adapter with good success (results-wise), and after the inital rendering and indexing, it's fast enough to use.

maxioatic 6y ago

This is great. I have 100+ ebooks/pdfs of programming and textbooks of which I've been extracting the index pages of. My intention was to always make some sort of search index out of them. I will definitely be trialing this (initial few searches seem promising!)

skanga 6y ago
Great idea. Please update on whether this use case works or not! And other tips, examples, etc.

ghoomketu 6y ago

One a related note there is one program that I absolutely miss on Linux called everything (on windows). The closest I can find is mlocate but it does not have a GUI but more importantly it does not index my Windows or NTFS drives. Would appreciate any suggestions if someone knows something like 'everything' for Ubuntu.

demosito666 6y ago
There is about a dozen dedicated file indexers [1] on linux (some are gui, some are not) and also DE-integrated ones like Baloo for KDE and Tracker for Gnome. [1] https://lmgtfy.app/?q=Is+there+a+file+search+engine+like+%E2%80%9CEverything%E2%80%9D+in+Linux%3F https://lmgtfy.app/?q=Is+there+a+file+search+engine+like+%E2...
opan 6y ago
Haven't used this, but heard of it years ago, and it aims to be similar. https://github.com/dotheevo/angrysearch/ https://github.com/dotheevo/angrysearch/
- ghoomketu 6y ago
  Thanks! That's super nice and very close to what I was looking for all this time. I just learned how to mount all my Windows drive under /mnt using (using the `disks` software), so hopefully this should index those files too.
ronjouch 6y ago
Everything is great on Windows to pick files/folders. From the linux command-line, I like fzf ( https://github.com/junegunn/fzf https://github.com/junegunn/fzf ), that you can instruct to use the faster fd ( https://github.com/junegunn/fzf#environment-variables https://github.com/junegunn/fzf#environment-variables ). Fzf even offers keybindings for your shell. For example, it binds Alt+C to fuzzy-finding a directory, and Enter cds to it ( https://github.com/junegunn/fzf#key-bindings-for-command-line https://github.com/junegunn/fzf#key-bindings-for-command-lin... ). Fzf is great for other things too; here is a fish function to bing Alt+G to fuzzy-pick a Git branch and jump to it: function fish_user_key_bindings bind \eg 'test -d .git; or git rev-parse --git-dir > /dev/null 2>&1; and git checkout (string trim -- (git branch | fzf)); and commandline -f repaint' bind \eG 'test -d .git; or git rev-parse --git-dir > /dev/null 2>&1; and git checkout (string trim -- (git branch --all | fzf)); and commandline -f repaint' end

durnygbur 6y ago

No ripgrep-all through the package manager: $ sudo dnf install -y ripgrep-all [...] No match for argument: ripgrep-all Error: Unable to find a match: ripgrep-all Rust's package manager fails: $ cargo install ripgrep_all [...] failed to select a version for the requirement `cachedir = "^0.1.1"` candidate versions found which didn't match: 0.2.0 location searched: crates.io index required by package `ripgrep_all v0.9.6` Quick search on the web shows that more people have problems with cachedir version.

ChrisSD 6y ago
It looks like cachedir yanked version 0.1.1. This is usually only done when a very serious issue is discovered, though I don't know what the reason is in this case. https://crates.io/crates/cachedir https://crates.io/crates/cachedir
est31 6y ago
You can do cargo install --locked ripgrep_all as a workaround. It uses the lockfile that's part of the ripgrep_all package, so you miss out on some package updates, but can also get the cachedir version required. There is a github issue to make this the default behaviour of cargo, but you miss out on updates which might fix security bugs so the cargo team is unwilling to change the default. https://github.com/rust-lang/cargo/issues/7169 https://github.com/rust-lang/cargo/issues/7169

hiq 6y ago

It would be nice to have a direct comparison with ugrep. In the case of rg the benchmarks are already enough to switch. Why should I use rga instead of ugrep?

burntsushi 6y ago
I've called the ugrep benchmarks into question, and I elaborated on it here (and this includes a frustrating exchange between myself and the ugrep author): https://old.reddit.com/r/rust/comments/i6pfb2/ugrep_new_ultrafast_c_grep_claims_to_be_faster/g0xybge/ https://old.reddit.com/r/rust/comments/i6pfb2/ugrep_new_ultr... I've also re-run my original set of benchmarks[1] with ugrep included: https://github.com/BurntSushi/ripgrep/blob/master/benchsuite/runs/2020-10-14-archlinux-frink/summary https://github.com/BurntSushi/ripgrep/blob/master/benchsuite... [1] - https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/
- hiq 6y ago
  Just to be clear, I meant that I had switched to ripgrep because its speed was convincing enough on its own (so I did not even extra features to switch). I'm currently not using any of ugrep or rga, although I have used pdfgrep in the past. It'd be nice for casual users like me to know more about why I should use rga over ugrep (or vice-versa).

fock 6y ago

can it produce links to open the file yet (don't know rust, so can't add a PR easily). At least gnome-terminal supports that (and normally it should also support opening a specific pdf page)!

mcintyre1994 6y ago
Not sure if the implementation is in rg or zsh, but that combination produces cmd-clickable file names for me.

kovek 6y ago

How could I use Rga to search my browsing history?

simonw 6y ago
Your browser history (if you use Chrome or Firefox at least) is stored in a SQLite database. It looks like rga can handle SQLite out of the box, so just making sure your history .db file is visible to rga may be all you need. You can also use my Datasette tool to get a web UI against your history, see https://docs.datasette.io/en/stable/getting_started.html#using-datasette-on-your-own-computer https://docs.datasette.io/en/stable/getting_started.html#usi...
- kovek 6y ago
  Are there tools to store the all history webpage contents to be viewed later?

chris_st 6y ago

Curious why this isn't a pull request to ripgrep? Maybe it was, and rejected? It'd be nice to just have one tool, and this doesn't feel like it's a stretch to add to ripgrep.

burntsushi 6y ago
It's a stretch. A big one. I answered this a while back: https://old.reddit.com/r/rust/comments/c1bjw4/rga_ripgrep_but_also_search_in_pdfs_ebooks_office/erhy4wd/ https://old.reddit.com/r/rust/comments/c1bjw4/rga_ripgrep_bu...

root_axis 6y ago

This is really great.

awinter-py 6y ago

thanks but it's way faster to have my stuff in G drive that way I can open a browser tab, wait 5 seconds for it to load, locate the new screen location of the search bar, click it, wait for javascript to finish loading so I can click the search bar, click it for real this time, mistype because there's some kind of contenteditable event jank, wait 5 seconds for my results to come up, fix the typo, and just have my results waiting for me I'm not going to learn a new tool when web is fine

ffpip 6y ago
If you're using Duckduckgo, just search ''!drive search-term'' or ''search term !drive'' or ''search !drive term'' More !operators here - https://duckduckgo.com/bang https://duckduckgo.com/bang
- deleted 6y ago
  [deleted]
- spappal 6y ago
  Firefox supports custom search engines, the most bang for the buck custom search engine must be https://duckduckgo.com/?q=%s https://duckduckgo.com/?q=%s with keyword being the letter d. Then you get all these 13000+ bangs without having to configure the custom search engines. E.g. write "d !drive term" in url bar. And "d !w hacker news" sends you directly to https://en.wikipedia.org/wiki/Hacker_News https://en.wikipedia.org/wiki/Hacker_News
  - codethief 6y ago
    Or you just set DDG as your default search engine and then you don't even have to type the "d" anymore. :)
  - t0astbread 6y ago
    Slightly off-topic but this is an important PSA. Firefox keyword search has one little known killer feature: You can combine it with data URIs and JavaScript to run small "command line snippets" stored in your bookmarks from your browser bar. To get started, create a keyword search from any form (like the search bar on duckduckgo.com) and edit the URL of the entry in the bookmark manager to point to data:text/html,<script>alert("%s")</script> instead. What you can do with this is (fortunately) limited by cross-origin restrictions but there are some useful applications. For example, I use this snippet data:text/html,<script>i="%s";firstSep=i.indexOf(" ");if(firstSep==-1)firstSep=i.length;subreddit=i.substr(0,firstSep);searchTerm=i.substr(firstSep+1);location=`https://old.reddit.com/r/${encodeURIComponent(subreddit)}${(searchTerm!=null&&searchTerm.replace(/\s/,"").length>0)?`/search?q=${encodeURIComponent(searchTerm)}&restrict_sr=1`:""}`</script> as a nice shortcut to Reddit ("<keyword> <subreddit>" to jump to a Subreddit, "<keyword> <subreddit> <search string>" to search within a Subreddit). You can also insert content directly into the document which opens the possibility for instant marquee data:text/html,<marquee>%s</marquee>

cb321 6y ago

NOTE: ripgrep already has --pre. (No pre-built indexing, of course.)

burntsushi 6y ago
That's exactly what ripgrep-all uses to implement this. There's a lot of integration work required to make this nice. The --pre flag is just a small hook. More info on it here: https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md#preprocessor https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md#p...
- cb321 6y ago
  Yup. Something perhaps more helpful but so far unmentioned (and somewhat OS-specific) is that statically linked executables usually fork & exec (especially exec) much faster than dynamically linked ones. This difference is usually only like 50..150 us vs 500..3000 us but can multiply up over thousands of files. This only matters on the first run of `rga`, of course. While the dispatched-to decoder is likely mostly out of one's linking control, this overhead can be saved for the dispatcher, at least. So, I would suggest `rga-preproc` should have a static linking option/suggestion, at least on Linux. Of course, this overhead may also fall below the noise of PDF/ebook/etc. parsing, but maybe not the decompression of small files in some dark horse format. :-)

soferio 6y ago

Can it (or any tool) perform proximity searches on scanned PDFs? E.g word1 within 20 words of word2, on scanned PDFs? (I think this is non trivial but very useful.)

phiresky 6y ago
Scanned PDFs only work well if they already have an OCR layer. There's some optional integration of rga with tesseract, but it's pretty slow and less good than external OCR tools. ripgrep-all can do the same regexes as rg on any filetypes it supports. So you can could do something like --multiline and foo(\w+[\s\n]+){,20}bar It won't work exactly like this, but something similar should do it: --multiline enables multiline matching * foo searches for foo * \w+ searches for at least one word character * [\W]+ searches for at least one space/nonword character like sentence marks * {,20} searches for at most 20 iterations of the word-space combination bar searches for bar
ballmerspeak 6y ago
If its a scanned PDF (essentially a collection of 1 image per page), there would need to be an OCR step to get some text out first. Tesseract would work for this. Once that's done, you have all the options available to perform that search. But I don't know of a search tool that does the OCR for you. I did read a blog post of someone uploading PDFs to google drive (they OCR them on upload) as an easy way to do this.

gopty 6y ago

Sounds like a poor man's version of recoll https://www.lesbonscomptes.com/recoll/ https://www.lesbonscomptes.com/recoll/ A PDF in a Zip file, in an email attachment. recoll can index it and do OCR if you like

phiresky 6y ago

Developer of the tool here :) Glad to see it posted here, I still actively use it myself. Also check out the fzf integration in the README: https://github.com/phiresky/ripgrep-all/blob/master/doc/rga-fzf.gif https://github.com/phiresky/ripgrep-all/blob/master/doc/rga-... Currently the main branch is undergoing a refactor to add support for having custom extractors (calling out to other tools), and more flexible chains of extractors. Ripgrep itself has functionality integrated to call custom extractors with the `--pre` flag, but by adding it here we can retain the benefits of the rga wrapper (more accurate file type matchers, caching, recursion into archives, adapter chaining, no slow shell scripts in between, etc). Sadly, during rewriting it to allow this, I kind of got hung up and couldn't manage to figure out how to cleanly design that in Rust. I'd be really glad if a Rust expert could help me out here: In the currently stable version, the main interface of each "adapter" is `fn(Read, Write) -> ()`. To allow custom adapter chaining I have to change it to be `fn(Read) -> Read` where each chained adapter wraps the read stream and converts it while reading. But then I get issues with how to handle threading etc, as well as a random deadlock that I haven't figured out how to solve so far :/

cb321 6y ago
One possibility is the almost dirt-simple solution wherein you just have a "make"/"Makefile" (or your favorite other build system) maintain a shadow tree of parallel pre-translated files. You get parallelism via `make -j$(nproc)` or its equivalent. Every name in the shadow is built from the name in the origin but maybe with ".txt" added (or .txt.gz if you want to keep the compressed with whatever is the fastest decompressor builtin to ripgrep as a library not called as a program). Untranslated names can be just symbolic/hard links back to the origin. Build rules become as flexible as your build system. This also scales to deployments that have more disk space than memory. Admittedly, in that case, the whole procedure probably becomes disk-IO bound, but maybe not. Maybe some translations cannot even keep up with disk IO - NVMe storage is pretty fast, for example. Or available memory may vary dynamically a lot, sometimes allowing the shadow to be fully in the buffer cache, other times not. It strikes me as less presumptuous to assume you can find disk space vs. having that much memory available. (EDIT2: though I may be confused about how `rga` operates - your doc says "memory cache", though.) On the pro-side, but for updating the shadows based on origins, the user could even just `rg` from within the shadow and translate filenames "in their head", although stripping an always present string is obviously trivial. Indeed, you won't need `rg --pre` at all and the grep itself could become pluggable. I doubt any of your other `fzf`/etc. integrations would be made more complicated by this design, either. This all strikes me as simple/nice enough that someone has probably already done it...EDIT1: Oh, I see from thumbs ups and other comments over at [1] and [2] that @phiresky is probably already aware of this design idea, but maybe some HN person knows of an existing solution along these lines. [1] https://github.com/BurntSushi/ripgrep/issues/978 https://github.com/BurntSushi/ripgrep/issues/978 [2] https://github.com/BurntSushi/ripgrep/pull/981 https://github.com/BurntSushi/ripgrep/pull/981

nikisweeting 6y ago

Aww hell yeah we should definitely use this in place of ripgrep for the new ArchiveBox.io full-text search backend. https://github.com/ArchiveBox/ArchiveBox/pull/543 https://github.com/ArchiveBox/ArchiveBox/pull/543

alexruf 6y ago

Idea behind Rga is cool. Anyway, I tried it on Mac and installed via Homebrew. The formula already says it depends on ripgrep (that's fine since I have ripgrep already installed and use it regularly). I still was surprised when I executed Rga for the first time and got an error message that 'pdftotext' was not found. Since pdftotext has been officially discontinued, I am not sure if I want to install an old version just to make Rga work on my machine. Don't think it's an good idea to rely on a project which is not maintained actively.

burntsushi 6y ago
> Since pdftotext has been officially discontinued Do you have a link for that? That's news to me.
- alexruf 6y ago
  brew info pdftotext https://formulae.brew.sh/cask/pdftotext#default https://formulae.brew.sh/cask/pdftotext#default
  - burntsushi 6y ago
    That looks like a problem with that specific package, and not pdftotext that is in poppler. I don't even know what that package is. It links to bluem.net?
there_the_and 6y ago
I don't see any indication that pdftotext has been discontinued [1]. It looks like a Mac-specific installer available via Homebrew Cask has been discontinued [2], but pdftotext is still available through the normal poppler formula [3]. 1. https://poppler.freedesktop.org/releases.html https://poppler.freedesktop.org/releases.html 2. https://formulae.brew.sh/cask/pdftotext https://formulae.brew.sh/cask/pdftotext 3. https://formulae.brew.sh/formula/poppler https://formulae.brew.sh/formula/poppler
phiresky 6y ago
Yeah, In my opinion poppler should be a dependency of rga in homebrew (since it's kinda useless without having the default adapters), but I don't maintain that package.

edm0nd 6y ago

Big fan of ripgrep. Use it on Windows to search through 100s of GBs of data really quickly.

CamperBob2 6y ago
Do you have a version that works with DOS pipes? I have to resort to an older grep version if I need to do something like: dir *.c* | rg somesubstringinfilename

dang 6y ago

If curious see also 2019 https://news.ycombinator.com/item?id=20196982 https://news.ycombinator.com/item?id=20196982

supernova87a 6y ago

For PDFs, how does it (does it?) deal with for example, when phrases get ripped apart by the layout? Like if you search for a multiple word phrase, it's often foiled by word wrap or being in a table.

0df8dkdf 6y ago

Great tool!!!

lafrenierejm 6y ago

If anyone is interested gron [0], I have an open PR [1] to add it as an adapter to ripgrep-all. The patch was based on the most recent release, since master is currently not functional. 0: https://github.com/TomNomNom/gron https://github.com/TomNomNom/gron 1: https://github.com/phiresky/ripgrep-all/pull/77 https://github.com/phiresky/ripgrep-all/pull/77

patricktlo 6y ago

Thanks! This is a godsend for someone like me who needs to search through many PDFs/docx documents to find information for work!

lenkite 6y ago

Is there a way to alias adapters ? So that .jar and .esa can be used for .zip ?