3 ms·

Xapian: Open source search engine library

172 points by Bluestein 2y ago

dvdkon 2y ago
Xapian is nice. I've used it before to add interactive autocomplete to a Python web app, since my previous favourite, Whoosh, is unmaintained and somehow slower than grep on a folder (I remember it being pretty fast years back, I'd love to know what happened). I'd say my favourite thing about Xapian is that it's just a simple library you can embed in any app, no need for a separate database and JVM tuning. For simple usecases and small-to-medium datasets, it just works.
infocollector 2y ago
This project has been around and maintained for more than a decade! Small footprint, good speed. One downside might be GPL v2 for commercial use.
- Bluestein 2y ago
  In fact, AIUI it's roots go back about 3 decades. The "about" page has a nice historical overview.-
- the_mitsuhiko 2y ago
  The project at one point started tracking files for potential rewrite to rid itself of GPL history. I used it many years ago and I quite enjoyed it (pre elastic search times) but unfortunately the license situation didn’t help the project to become popular.
- frenchman99 2y ago
  You can always build a small search webservice that you open source and that your proprietary software calls out too, removing the need to open source everything. Linux is GPL too, didn't hinder companies making trillions on top of it.
  - synergy20 2y ago
    you mean, don't compile it and link it within my application, instead wrap it as a separate program then call it via rpc remotely or locally?
    - frenchman99 2y ago
      Yes, exactly that.
  - the_mitsuhiko 2y ago
    Linux is not a good example because of the syscall exemption. The licensing situation is not at all comparable as xapian’s original point of existence was embedding.
- synergy20 2y ago
  that's true, wonder if there is alternative that is not gpl
  - bearjaws 2y ago
    Sonic search https://github.com/valeriansaliou/sonic https://github.com/valeriansaliou/sonic Maybe not exactly the same, its a server that you can store documents and then retrieve their ID using a search string.
  - Bluestein 2y ago
    Elastic Search and its Amazon fork Opensearch perhaps?
    - JackSlateur 2y ago
      Xapian is a library, while elastic has a client-server model Xapian is more like sqlite while elastic would be mariadb
      - Bluestein 2y ago
        Thanks for the spot-on, very illuminated comparison.-
      - inertiatic 2y ago
        Lucene which is what ES builds upon is a library with bindings in languages other than Java, and it's Apache licensed.
- rbanffy 2y ago
  For what kind of use do you think GPLv2 would be a blocker?
- donio 2y ago
  A lot more than a decade. I've been using it for 15 years at it was a very mature project even then. Repo history goes back to 1999 and according to the history page the project's roots go back to the 80s. A bit like Postgres in this respect. https://xapian.org/history https://xapian.org/history https://sigir.org/files/forum/S2000/MUSCAT_note.pdf https://sigir.org/files/forum/S2000/MUSCAT_note.pdf
- rurban 2y ago
  I used it commercially, at a very big international company. All users had access to its unmodified source code and templates. There was no trouble at all.
jfmc 2y ago
Xapian is used in https://www.djcbsoftware.nl/code/mu/ https://www.djcbsoftware.nl/code/mu/ for indexing emails.
- calvinmorrison 2y ago
  And by Fastmail
  - nathell 2y ago
    And by Notmuch https://notmuchmail.org/ https://notmuchmail.org/
    - Bluestein 2y ago
      In itsel, Notmuch, a very interesting tool.-
      - Bluestein 2y ago
        s/itsel/itself/
openrisk 2y ago
used also by recoll, the desktop search app: https://www.recoll.org/ https://www.recoll.org/
- nanna 2y ago
  I use recoll to index and search thousands of pdfs. Because I always have the author name in the filename I can filter queries like this: Cybernetics OR steering filename:Heidegger ext:pdf It's an absolute power tool.
  - nickpsecurity 2y ago
    I do it like this: Title Year Author Name.pdf Same benefits as you mentioned. You can also filter by time that way.
    - nanna 2y ago
      My way is to do year__author1_author2_author-n~~title-of-book~subtitle##tag1#tag-2#tagn.pdf This means that files are automatically organised by year of publication,that I can search by tag name, and that I dont have to escape chars in the terminal. One day I hope to get round to building an Emacs mode to filter by the different elements.
  - donio 2y ago
    If your PDFs have Author properties then you might be able to do "author:Heidegger" too. The Recoll PDF filter extracts some of these fields and if I remember right it can be configured to extract additional custom properties too.
    - nanna 2y ago
      That would be great, however most of my pdfs don't have the author property set.
- Beijinger 2y ago
  Recoll is magic...
rbanffy 2y ago
I remember having used, a very long time ago, a self-hosted search engine on my library of PDFs, and it was unbelievably useful. I dream about a similar thing that can do OCR on scanned docs and extract text from my also sprawling library of epub and mobi files. If someone builds something like this, with maybe a LOCAL LLM to extract text descriptions from photos and movies as well as indexing metadata for everything, subtitles from movies and lyrics for songs, and add that to a NAS appliance, it’d be a killer.
- andyfilms1 2y ago
  Evernote will do this, you can feed it a bunch of PDFs and other documents, it will OCR them and make them all searchable. it's not perfect, but you can also add manual tags for things you know are important.
  - rbanffy 2y ago
    At some point I'd love to further train an LLM on all my PDFs and be able to ask it questions.
    - Jzush 2y ago
      For txt based pdfs, doc files etc, this is super easy to do and requires very little technical expertise or configuration. Download the gpt4all client. Pick an LLM the new meta llama3 model works really well then configure the local documents plugin. I wrote a scraper to download all of the California EdCode from the governments site, convert them all to txt docs and I can ask questions about California EdCode in plain English. I work in a shared governance capacity that requires us to refer to the Ed code for contractual negotiations and it’s been extremely helpful.
- kordlessagain 2y ago
  I have most of this code for doing this - just needs to get rewritten for local storage (I was running it on Google Cloud). Need to pick something that doesn't run Solr as a service for local use. With Ollama, we have function calls running, so should be doable. I was also thinking about using the Open WebUI for use.
importsaas 2y ago
Much better docs here at https://getting-started-with-xapian.readthedocs.io/en/latest/ https://getting-started-with-xapian.readthedocs.io/en/latest...
donio 2y ago
Love Xapian, been using it for many years via notmuch (mail) and recoll (document indexing, mainly PDFs in my case). It's been trouble free and very performant, a real workhorse. https://notmuchmail.org/ https://notmuchmail.org/ https://www.recoll.org/ https://www.recoll.org/
Vuizur 2y ago
I once wanted to compile a program that used Xapian on Windows. It was basically impossible for mortals. Imo people should use cross-platform alternatives.

172 points by Bluestein 2y ago

Xapian is nice. I've used it before to add interactive autocomplete to a Python web app, since my previous favourite, Whoosh, is unmaintained and somehow slower than grep on a folder (I remember it being pretty fast years back, I'd love to know what happened). I'd say my favourite thing about Xapian is that it's just a simple library you can embed in any app, no need for a separate database and JVM tuning. For simple usecases and small-to-medium datasets, it just works.

infocollector 2y ago

This project has been around and maintained for more than a decade! Small footprint, good speed. One downside might be GPL v2 for commercial use.

Bluestein 2y ago
In fact, AIUI it's roots go back about 3 decades. The "about" page has a nice historical overview.-
the_mitsuhiko 2y ago
The project at one point started tracking files for potential rewrite to rid itself of GPL history. I used it many years ago and I quite enjoyed it (pre elastic search times) but unfortunately the license situation didn’t help the project to become popular.
frenchman99 2y ago
You can always build a small search webservice that you open source and that your proprietary software calls out too, removing the need to open source everything. Linux is GPL too, didn't hinder companies making trillions on top of it.
- synergy20 2y ago
  you mean, don't compile it and link it within my application, instead wrap it as a separate program then call it via rpc remotely or locally?
  - frenchman99 2y ago
    Yes, exactly that.
- the_mitsuhiko 2y ago
  Linux is not a good example because of the syscall exemption. The licensing situation is not at all comparable as xapian’s original point of existence was embedding.
synergy20 2y ago
that's true, wonder if there is alternative that is not gpl
- bearjaws 2y ago
  Sonic search https://github.com/valeriansaliou/sonic https://github.com/valeriansaliou/sonic Maybe not exactly the same, its a server that you can store documents and then retrieve their ID using a search string.
- Bluestein 2y ago
  Elastic Search and its Amazon fork Opensearch perhaps?
  - JackSlateur 2y ago
    Xapian is a library, while elastic has a client-server model Xapian is more like sqlite while elastic would be mariadb
    - Bluestein 2y ago
      Thanks for the spot-on, very illuminated comparison.-
    - inertiatic 2y ago
      Lucene which is what ES builds upon is a library with bindings in languages other than Java, and it's Apache licensed.
rbanffy 2y ago
For what kind of use do you think GPLv2 would be a blocker?
donio 2y ago
A lot more than a decade. I've been using it for 15 years at it was a very mature project even then. Repo history goes back to 1999 and according to the history page the project's roots go back to the 80s. A bit like Postgres in this respect. https://xapian.org/history https://xapian.org/history https://sigir.org/files/forum/S2000/MUSCAT_note.pdf https://sigir.org/files/forum/S2000/MUSCAT_note.pdf
rurban 2y ago
I used it commercially, at a very big international company. All users had access to its unmodified source code and templates. There was no trouble at all.

jfmc 2y ago

Xapian is used in https://www.djcbsoftware.nl/code/mu/ https://www.djcbsoftware.nl/code/mu/ for indexing emails.

calvinmorrison 2y ago
And by Fastmail
- nathell 2y ago
  And by Notmuch https://notmuchmail.org/ https://notmuchmail.org/
  - Bluestein 2y ago
    In itsel, Notmuch, a very interesting tool.-
    - Bluestein 2y ago
      s/itsel/itself/

openrisk 2y ago

used also by recoll, the desktop search app: https://www.recoll.org/ https://www.recoll.org/

nanna 2y ago
I use recoll to index and search thousands of pdfs. Because I always have the author name in the filename I can filter queries like this: Cybernetics OR steering filename:Heidegger ext:pdf It's an absolute power tool.
- nickpsecurity 2y ago
  I do it like this: Title Year Author Name.pdf Same benefits as you mentioned. You can also filter by time that way.
  - nanna 2y ago
    My way is to do year__author1_author2_author-n~~title-of-book~subtitle##tag1#tag-2#tagn.pdf This means that files are automatically organised by year of publication,that I can search by tag name, and that I dont have to escape chars in the terminal. One day I hope to get round to building an Emacs mode to filter by the different elements.
- donio 2y ago
  If your PDFs have Author properties then you might be able to do "author:Heidegger" too. The Recoll PDF filter extracts some of these fields and if I remember right it can be configured to extract additional custom properties too.
  - nanna 2y ago
    That would be great, however most of my pdfs don't have the author property set.
Beijinger 2y ago
Recoll is magic...

rbanffy 2y ago

I remember having used, a very long time ago, a self-hosted search engine on my library of PDFs, and it was unbelievably useful. I dream about a similar thing that can do OCR on scanned docs and extract text from my also sprawling library of epub and mobi files. If someone builds something like this, with maybe a LOCAL LLM to extract text descriptions from photos and movies as well as indexing metadata for everything, subtitles from movies and lyrics for songs, and add that to a NAS appliance, it’d be a killer.

andyfilms1 2y ago
Evernote will do this, you can feed it a bunch of PDFs and other documents, it will OCR them and make them all searchable. it's not perfect, but you can also add manual tags for things you know are important.
- rbanffy 2y ago
  At some point I'd love to further train an LLM on all my PDFs and be able to ask it questions.
  - Jzush 2y ago
    For txt based pdfs, doc files etc, this is super easy to do and requires very little technical expertise or configuration. Download the gpt4all client. Pick an LLM the new meta llama3 model works really well then configure the local documents plugin. I wrote a scraper to download all of the California EdCode from the governments site, convert them all to txt docs and I can ask questions about California EdCode in plain English. I work in a shared governance capacity that requires us to refer to the Ed code for contractual negotiations and it’s been extremely helpful.
kordlessagain 2y ago
I have most of this code for doing this - just needs to get rewritten for local storage (I was running it on Google Cloud). Need to pick something that doesn't run Solr as a service for local use. With Ollama, we have function calls running, so should be doable. I was also thinking about using the Open WebUI for use.

importsaas 2y ago

Much better docs here at https://getting-started-with-xapian.readthedocs.io/en/latest/ https://getting-started-with-xapian.readthedocs.io/en/latest...

donio 2y ago

Love Xapian, been using it for many years via notmuch (mail) and recoll (document indexing, mainly PDFs in my case). It's been trouble free and very performant, a real workhorse. https://notmuchmail.org/ https://notmuchmail.org/ https://www.recoll.org/ https://www.recoll.org/

Vuizur 2y ago

I once wanted to compile a program that used Xapian on Windows. It was basically impossible for mortals. Imo people should use cross-platform alternatives.