5 ms·

Recoll: A desktop full-text search tool

111 points by karlicoss 5y ago

readyplayeremma 5y ago
Is there any desktop search engine that can do topic modelling and/or semantic clustering to allow you to discover related documents or do other kinds of browsing and discovery of large sets of documents where you may not immediately know the specific search terms to directly go to what you need?
- Veen 5y ago
  DEVONthink's See Also feature does this, but it's Apple-only (and rather expensive). https://www.devontechnologies.com/apps/devonthink https://www.devontechnologies.com/apps/devonthink https://download.devontechnologies.com/download/devonthink/3.8/DEVONthink%20Manual.pdf https://download.devontechnologies.com/download/devonthink/3...
kohlerm 5y ago
I ran recoll on a Linux Laptop for a while. It worked pretty well and was very fast because of the index it created.
sooheon 5y ago
Other "personal search engines" https://thesephist.com/posts/monocle/ https://thesephist.com/posts/monocle/ written in personal language, takes multiple doc sources https://apse.io https://apse.io interesting because it takes periodic screenshots and performs OCR https://keminglabs.com/finda/ https://keminglabs.com/finda/ rust, fast, seemingly abandonded
- amai 5y ago
  I would like to have a personal search engine for the web. It should only index my content on the web (social media, blog post, comments on hacker news, etc...)
- encryptluks2 5y ago
  Recoll isn't just a personal search engine and it does all the things you mentioned and more. It is built on Xapian.
  - sooheon 5y ago
    Does it OCR periodic screencaps? I.e. the subtitle text of something you're watching on netflix? There's no saved file to index.
    - encryptluks2 5y ago
      If you have periodic screen caps saved to a folder then it can, however it makes more sense to make your own plugin to Xapian which you can do and share with Recoll.
    - walterbell 5y ago
      iOS now OCRs all photos, wish there was a way to bulk export that text alongside the photos, https://9to5mac.com/2021/09/21/how-iphone-live-text-ocr-works-in-ios-15/ https://9to5mac.com/2021/09/21/how-iphone-live-text-ocr-work...
- hulitu 5y ago
  Orher personal search engines: locate and grep.
encryptluks2 5y ago
Recoll and the underlying search Xapian are incredibly powerful. I am surprised that almost all server side search use such bloated frameworks. Xapian seemed really easy to theme but their site and docs need some love. I also don't recall, but if they had a way to parse JSON search hints I'm sure that it would be perfect as a lightweight server side search for static sites. Not that JS search isn't cool, but until browsers implement a native site searcb feature, it is nice to have a way to search from non-JS browsers and clients.
- hagbard_c 5y ago
  Recoll can be used server-side, I've been doing this for years now - no gui needed nor even installed: https://news.ycombinator.com/item?id=28954294 https://news.ycombinator.com/item?id=28954294
  - encryptluks2 5y ago
    Wow, you also solved the JSON issue. I've been wanting to implement Xapian as a server-side search server for Hugo. I believe by just creating the template using Hugo, it should be pretty easy to have it search the JSON. I've got quite a bit of work to do though to figure it out.
1vuio0pswjnm7 5y ago
This uses Xapian. Some of the Linux mailing lists, like linux-netdev, use Xapian. I use Xapian on the desktop but I only store text documents so I have no need for Recoll. It is refreshing to use search that is not tuned for "popularity" and selling online ad services. If search is too easy, then IMO it is not really search. Search should require some skill. It should not be a game where a commercial entity Hoovering up user data and selling ad services runs the search and tries to guess what the searcher is searching for. I never tried it, but this looks somewhat similar to Recoll: https://github.com/kendling/pinot-search https://github.com/kendling/pinot-search
ackatz 5y ago
As a consultant, I am routinely given hundreds of new files from new clients which are usually a mix of Word documents and PDFs of policies, procedures, and standards. Then, throughout the course of the engagement, I often have to pick out certain phrases or topics from these large datasets in order to reference them in a report. Windows File Explorer can do this to some degree. I am not sure if File Explorer looks at the actual content of the documents or just titles. Either way, Recoll is magnitudes better and is absolutely worth it for my case.
dash2 5y ago
Doesn’t Spotlight on Mac do this already? What’s the added value?
- 72deluxe 5y ago
  Spotlight does indeed do this so Mac users don't need this. Might help Windows and Linux users though. I use voidtools.com's Everything tool to search for filenames quickly on Windows as it uses the NTFS index to do near-instant searches, since the Windows Indexing Service has been poor since XP... This might help if you are looking for content instead of just filenames.
hagbard_c 5y ago
Recoll is not just a desktop search engine, it can also be used as an intranet search engine either 'stand-alone' [1] or as an 'engine' in Searx [2]. I made the Searx engine a few years ago to extend Searx to intranet search, the QT qui is not used (an, on Debian, not even installed - only the 'recollcmd' package is needed). [1] https://github.com/koniu/recoll-webui https://github.com/koniu/recoll-webui [2] https://github.com/searx/searx/pull/1257 https://github.com/searx/searx/pull/1257
siboehm 5y ago
I've been running a setup with Recoll and https://github.com/ArchiveBox/ArchiveBox https://github.com/ArchiveBox/ArchiveBox for a few months now [1]. Each morning archivebox scrapes all new links that I've put into my (text-based) notes and saves them as HTML singlefiles. Then Recoll indexes them. It's very fast and ~4 lines of code. It's surprising how often I rediscover old blog posts & papers that are much better than what Google yields me. From my experience Recoll isn't very good at searching for aliases sadly. [1] https://siboehm.com/articles/21/a-local-search-engine https://siboehm.com/articles/21/a-local-search-engine
- vageli 5y ago
  Tangential but I really love the look of your site, especially the footnotes displayed adjacent to the text. Did you create the layout yourself? I have been looking for a theme like this for a paper discussion website.
  - czottmann 5y ago
    Not my site, but it looks like a theme based on Tufte CSS: https://edwardtufte.github.io/tufte-css/ https://edwardtufte.github.io/tufte-css/
- leodriesch 5y ago
  Sounds cool, thanks for the inspiration!
Tomte 5y ago
I wouldn't mind paying a few bucks for the Windows version, but it is unclear to me if the Windows version can index PDFs. It seems to depend on pdftotext on Linux. Can I get that for Windows? If so, is an executable somewhere in my path enough?
- mattowen_uk 5y ago
  PanDoc is available for Windows, which can also extract text from PDF files. Not sure if Recoll supports it though. However if you want to OCR a PDF which contains image based 'pages' then you'll need something like Tesseract OCR.
- flyingpant 5y ago
  Yes, it can, I'm using it personally and can confirm that the PDF indexing works without you having to install anything else yourself, it seems like every binary needed for it to run is bundled together with the main app.
streamofdigits 5y ago
recoll is a killer app. it should somehow integrate deeper with linux desktops and take personal information management to the next level. it is incredibly useful in itself, has a CLI/scripting interface etc, but its homemade GUI is quirky and using its output for next steps in workflows can be cumbersome (may need coding). better integration with file managers and other desktop apps would help it really takeoff
albertzeyer 5y ago
Many projects really miss some section on their homepage which briefly compares itself to similar or related projects. I assume there are many related projects. Most big desktop environments (commercial or open source) provide some similar service. I wonder about e.g. Baloo (Baloo is the file indexing and file search framework for KDE Plasma, https://community.kde.org/Baloo https://community.kde.org/Baloo). Is this similar? Or different? How is it different? Just yesterday my dad complained about a couple of Baloo error messages, Baloo crash reports, disk full, Xsession log full of Baloo messages. And he was not really aware about what Baloo actually is. The problem is similar to the one here: https://askubuntu.com/questions/1214572/how-do-i-stop-and-remove-kde-baloo-that-is-broken-and-filling-up-my-syslog-file https://askubuntu.com/questions/1214572/how-do-i-stop-and-re... So I disabled the content indexing, as it was suggested there. The stack trace of the Baloo crashes also suggested that it was related to indexing the content of some documents. I think this is a problem for many such file indexing services. Even Spotlight on MacOSX frequently crashes when it tries to index some documents. Does Recoll handle this differently? If Recoll is in general better than Baloo, maybe it should replace Baloo? Or Baloo can be extended to use Recoll under the hood?
- vhanda 5y ago
  Baloo Author here, though I haven't contributed code for over 4 years now. Baloo used to use Xapian under the hood, like Recoll, when initially launched, but in 2014 (I think?) I moved away from it to a custom DB built on top of LMDB. It resulted in a massive performance increase, one of the main criteria was for results to always take less than 'x milliseconds' (I can't remember exactly how much) so that KRunner (similar to OSX Spotlight) would always feel snappy. Another criteria was for Baloo to operate well with minimal memory and CPU, which wasn't the case with Recoll (Things might have changed since then) I do remember ensuring that Baloo always did content indexing in another process, though I can't remember at what point it blacklists a file. Sorry, this isn't the best answer to your question, but most of the details are now quite fuzzy to me, and I haven't kept up with how the project has changed after I stopped contributing to KDE. In retrospect though, I definitely should have spent far far more time ensuring that Baloo doesn't end up in situations like the one you described, and putting far more limits on the IO and CPU usage it's allowed to use.
avapark 5y ago
https://prestigecityavalonpark.in/contact.html https://prestigecityavalonpark.in/contact.html
mellosouls 5y ago
For Windows users, a shout out for Copernic Desktop Search (content is indexed so search results are instant and presented via an appropriate embedded view for some types; you can filter but it doesn't support grep), which I've used for many years. It's not free, but one of the utils I wouldn't be without. https://copernic.com/en/desktop/ https://copernic.com/en/desktop/

111 points by karlicoss 5y ago

Is there any desktop search engine that can do topic modelling and/or semantic clustering to allow you to discover related documents or do other kinds of browsing and discovery of large sets of documents where you may not immediately know the specific search terms to directly go to what you need?

Veen 5y ago
DEVONthink's See Also feature does this, but it's Apple-only (and rather expensive). https://www.devontechnologies.com/apps/devonthink https://www.devontechnologies.com/apps/devonthink https://download.devontechnologies.com/download/devonthink/3.8/DEVONthink%20Manual.pdf https://download.devontechnologies.com/download/devonthink/3...

kohlerm 5y ago

I ran recoll on a Linux Laptop for a while. It worked pretty well and was very fast because of the index it created.

sooheon 5y ago

Other "personal search engines" https://thesephist.com/posts/monocle/ https://thesephist.com/posts/monocle/ written in personal language, takes multiple doc sources https://apse.io https://apse.io interesting because it takes periodic screenshots and performs OCR https://keminglabs.com/finda/ https://keminglabs.com/finda/ rust, fast, seemingly abandonded

amai 5y ago
I would like to have a personal search engine for the web. It should only index my content on the web (social media, blog post, comments on hacker news, etc...)
encryptluks2 5y ago
Recoll isn't just a personal search engine and it does all the things you mentioned and more. It is built on Xapian.
- sooheon 5y ago
  Does it OCR periodic screencaps? I.e. the subtitle text of something you're watching on netflix? There's no saved file to index.
  - encryptluks2 5y ago
    If you have periodic screen caps saved to a folder then it can, however it makes more sense to make your own plugin to Xapian which you can do and share with Recoll.
  - walterbell 5y ago
    iOS now OCRs all photos, wish there was a way to bulk export that text alongside the photos, https://9to5mac.com/2021/09/21/how-iphone-live-text-ocr-works-in-ios-15/ https://9to5mac.com/2021/09/21/how-iphone-live-text-ocr-work...
hulitu 5y ago
Orher personal search engines: locate and grep.

encryptluks2 5y ago

Recoll and the underlying search Xapian are incredibly powerful. I am surprised that almost all server side search use such bloated frameworks. Xapian seemed really easy to theme but their site and docs need some love. I also don't recall, but if they had a way to parse JSON search hints I'm sure that it would be perfect as a lightweight server side search for static sites. Not that JS search isn't cool, but until browsers implement a native site searcb feature, it is nice to have a way to search from non-JS browsers and clients.

hagbard_c 5y ago
Recoll can be used server-side, I've been doing this for years now - no gui needed nor even installed: https://news.ycombinator.com/item?id=28954294 https://news.ycombinator.com/item?id=28954294
- encryptluks2 5y ago
  Wow, you also solved the JSON issue. I've been wanting to implement Xapian as a server-side search server for Hugo. I believe by just creating the template using Hugo, it should be pretty easy to have it search the JSON. I've got quite a bit of work to do though to figure it out.

1vuio0pswjnm7 5y ago

This uses Xapian. Some of the Linux mailing lists, like linux-netdev, use Xapian. I use Xapian on the desktop but I only store text documents so I have no need for Recoll. It is refreshing to use search that is not tuned for "popularity" and selling online ad services. If search is too easy, then IMO it is not really search. Search should require some skill. It should not be a game where a commercial entity Hoovering up user data and selling ad services runs the search and tries to guess what the searcher is searching for. I never tried it, but this looks somewhat similar to Recoll: https://github.com/kendling/pinot-search https://github.com/kendling/pinot-search

ackatz 5y ago

As a consultant, I am routinely given hundreds of new files from new clients which are usually a mix of Word documents and PDFs of policies, procedures, and standards. Then, throughout the course of the engagement, I often have to pick out certain phrases or topics from these large datasets in order to reference them in a report. Windows File Explorer can do this to some degree. I am not sure if File Explorer looks at the actual content of the documents or just titles. Either way, Recoll is magnitudes better and is absolutely worth it for my case.

dash2 5y ago

Doesn’t Spotlight on Mac do this already? What’s the added value?

72deluxe 5y ago
Spotlight does indeed do this so Mac users don't need this. Might help Windows and Linux users though. I use voidtools.com's Everything tool to search for filenames quickly on Windows as it uses the NTFS index to do near-instant searches, since the Windows Indexing Service has been poor since XP... This might help if you are looking for content instead of just filenames.

hagbard_c 5y ago

Recoll is not just a desktop search engine, it can also be used as an intranet search engine either 'stand-alone' [1] or as an 'engine' in Searx [2]. I made the Searx engine a few years ago to extend Searx to intranet search, the QT qui is not used (an, on Debian, not even installed - only the 'recollcmd' package is needed). [1] https://github.com/koniu/recoll-webui https://github.com/koniu/recoll-webui [2] https://github.com/searx/searx/pull/1257 https://github.com/searx/searx/pull/1257

siboehm 5y ago

I've been running a setup with Recoll and https://github.com/ArchiveBox/ArchiveBox https://github.com/ArchiveBox/ArchiveBox for a few months now [1]. Each morning archivebox scrapes all new links that I've put into my (text-based) notes and saves them as HTML singlefiles. Then Recoll indexes them. It's very fast and ~4 lines of code. It's surprising how often I rediscover old blog posts & papers that are much better than what Google yields me. From my experience Recoll isn't very good at searching for aliases sadly. [1] https://siboehm.com/articles/21/a-local-search-engine https://siboehm.com/articles/21/a-local-search-engine

vageli 5y ago
Tangential but I really love the look of your site, especially the footnotes displayed adjacent to the text. Did you create the layout yourself? I have been looking for a theme like this for a paper discussion website.
- czottmann 5y ago
  Not my site, but it looks like a theme based on Tufte CSS: https://edwardtufte.github.io/tufte-css/ https://edwardtufte.github.io/tufte-css/
leodriesch 5y ago
Sounds cool, thanks for the inspiration!

Tomte 5y ago

I wouldn't mind paying a few bucks for the Windows version, but it is unclear to me if the Windows version can index PDFs. It seems to depend on pdftotext on Linux. Can I get that for Windows? If so, is an executable somewhere in my path enough?

mattowen_uk 5y ago
PanDoc is available for Windows, which can also extract text from PDF files. Not sure if Recoll supports it though. However if you want to OCR a PDF which contains image based 'pages' then you'll need something like Tesseract OCR.
flyingpant 5y ago
Yes, it can, I'm using it personally and can confirm that the PDF indexing works without you having to install anything else yourself, it seems like every binary needed for it to run is bundled together with the main app.

streamofdigits 5y ago

recoll is a killer app. it should somehow integrate deeper with linux desktops and take personal information management to the next level. it is incredibly useful in itself, has a CLI/scripting interface etc, but its homemade GUI is quirky and using its output for next steps in workflows can be cumbersome (may need coding). better integration with file managers and other desktop apps would help it really takeoff

albertzeyer 5y ago

Many projects really miss some section on their homepage which briefly compares itself to similar or related projects. I assume there are many related projects. Most big desktop environments (commercial or open source) provide some similar service. I wonder about e.g. Baloo (Baloo is the file indexing and file search framework for KDE Plasma, https://community.kde.org/Baloo https://community.kde.org/Baloo). Is this similar? Or different? How is it different? Just yesterday my dad complained about a couple of Baloo error messages, Baloo crash reports, disk full, Xsession log full of Baloo messages. And he was not really aware about what Baloo actually is. The problem is similar to the one here: https://askubuntu.com/questions/1214572/how-do-i-stop-and-remove-kde-baloo-that-is-broken-and-filling-up-my-syslog-file https://askubuntu.com/questions/1214572/how-do-i-stop-and-re... So I disabled the content indexing, as it was suggested there. The stack trace of the Baloo crashes also suggested that it was related to indexing the content of some documents. I think this is a problem for many such file indexing services. Even Spotlight on MacOSX frequently crashes when it tries to index some documents. Does Recoll handle this differently? If Recoll is in general better than Baloo, maybe it should replace Baloo? Or Baloo can be extended to use Recoll under the hood?

vhanda 5y ago
Baloo Author here, though I haven't contributed code for over 4 years now. Baloo used to use Xapian under the hood, like Recoll, when initially launched, but in 2014 (I think?) I moved away from it to a custom DB built on top of LMDB. It resulted in a massive performance increase, one of the main criteria was for results to always take less than 'x milliseconds' (I can't remember exactly how much) so that KRunner (similar to OSX Spotlight) would always feel snappy. Another criteria was for Baloo to operate well with minimal memory and CPU, which wasn't the case with Recoll (Things might have changed since then) I do remember ensuring that Baloo always did content indexing in another process, though I can't remember at what point it blacklists a file. Sorry, this isn't the best answer to your question, but most of the details are now quite fuzzy to me, and I haven't kept up with how the project has changed after I stopped contributing to KDE. In retrospect though, I definitely should have spent far far more time ensuring that Baloo doesn't end up in situations like the one you described, and putting far more limits on the IO and CPU usage it's allowed to use.

avapark 5y ago

https://prestigecityavalonpark.in/contact.html https://prestigecityavalonpark.in/contact.html

mellosouls 5y ago

For Windows users, a shout out for Copernic Desktop Search (content is indexed so search results are instant and presented via an appropriate embedded view for some types; you can filter but it doesn't support grep), which I've used for many years. It's not free, but one of the utils I wouldn't be without. https://copernic.com/en/desktop/ https://copernic.com/en/desktop/