8 ms·

Whoosh – Fast, full-text indexing and searching library in Python

146 points by albertzeyer 13y ago

cardamomo 13y ago
I have found Whoosh to be easy to use, though I haven't put it to use with a large number of documents, so I don't know how well it holds up.
piqufoh 13y ago
Whoosh is great in that it fits the django philosophy of building sites fast, but I wouldn't use if for anything harder than a small site search. Once my site grew larger than 50 mb of text queries started slowing things down. Indexing (python, single threaded) took a while and the larger the index the slower the queries were returned. It's probably the best first iteration search app that I've come across, and you can always slip in solr or something with more umph when you need it.
- WoodenChair 13y ago
  Can you clarify why you would not use it on anything harder than a small site search? Without context, the comment is not especially helpful.
  - mattdeboard 13y ago
    > Once my site grew larger than 50 mb of text queries started slowing things down. Indexing (python, single threaded) took a while and the larger the index the slower the queries were returned. I think this makes it pretty clear. Not sure how much more explanation you need.
    - WoodenChair 13y ago
      Wasn't in the original comment... yet another example of why HN should not have edit.
  - ra 13y ago
    For very small datasets, Whoosh is fine. For anything larger than 'site search' on a website that has no UGC, you're much better off with SOLR (or ES, Xapian or Sphinx - whatever your poison). When document counts get into the hundreds you see orders of magnitude faster queries with SOLR (etc), not to mention much more sophisticated querying options.
- hackerboos 13y ago
  ElasticSearch with Haystack is the next logical move after outgrowing Whoosh.
  - andrewingram 13y ago
    Based on my own experiences, by the time search is an important enough feature to outgrow Whoosh, you've also outgrown Haystack's API.
- nknighthb 13y ago
  > Indexing (python, single threaded) took a while The documentation indicates there is a multiprocessing option. (Also configurable memory limits, if you didn't see that.) http://whoosh.readthedocs.org/en/latest/batch.html http://whoosh.readthedocs.org/en/latest/batch.html
  - piqufoh 13y ago
    Ooh - handy, I hadn't seen that thanks. I'll try it out.
davb 13y ago
Whoosh is fantastic. I'm using it on an ecommerce web project just now for indexing CMS pages, products, datasheets and product hierarchy (categories, product families, etc). FWIW I'm using Flask, Flask-SQLAlchemy, Whoosh and Flask-WhooshAlchemy [0]. Quick to get going with but the mix-and-match approach lets me easily rip pieces out as the project grows. [0] http://pythonhosted.org/Flask-WhooshAlchemy/ http://pythonhosted.org/Flask-WhooshAlchemy/
deleted 13y ago
[deleted]
daemonk 13y ago
Sounds cool. Does it scale well? Has anyone used it for a large-ish amount of data (gigs)?
- tbatterii 13y ago
  not in my experience. i put id3 tags from ~32K songs in it and a query for an artist was around 700ms. same query in ES was like 10ms, same qry to sqlite db (select * from track where artist=...) was about the same as ES. the docs for whoosh make these claims that it's fast, but I haven't seen anything to backup the claims in unit tests or otherwise. I'm sure it's still useful for some purposes but ES is so easy to setup that I don't bother.
- davb 13y ago
  It works well enough for me. I can't say how large my index set is (because I honestly don't have the figures to hand) but on the current project, a search of around 50,000 products, 10,000 product families and a whole lot of associated data (product attributes, datasheets, etc), an uncached search takes around 65ms. To put it in context, the psycopg2 calls to PostgreSQL take about 100ms to retrieve the associated data once I've found my search results with Whoosh. (Most of my response time sits with SQLAlchemy ORM, building up matrices of data, which is why in production I'm caching the more complex queries with memcached). Overall, for a project of this size (I can't imagine having to index more than a few hundred thousand objects) I'm very happy with Whoosh. If I get to the point of indexing millions of objects, I'll optimize it then.
caioariede 13y ago
If you are using it for Django, take a look on Haystack [1]. It supports Whoosh as well as Solr, Elasticsearch, etc... For example, you could use Whoosh for development environment and Elasticsearch for production if you need something more robust. [1] http://haystacksearch.org/ http://haystacksearch.org/
- j_s 13y ago
  https://github.com/toastdriven/django-haystack https://github.com/toastdriven/django-haystack
- mixmastamyk 13y ago
  And Xapian, which I've had good results with.
mattdeboard 13y ago
Solr is just so easy to get up and running, so powerful with so many options for scaling, I don't see the benefit of "pure Python" here. (To be fair, despite living and breathing Python most days, I've never thought "pure Python" is a selling point.)
- ra 13y ago
  Solr, Sphinx and ElasticSearch are all reliable search-at-scale options for those of us without petabyte problems. Whoosh is fine for very small datasets (megabytes) and low loads, just watch out for file permission issues.
  - Torn 13y ago
    ElasticSearch is actually designed to scale to petabytes and beyond: http://elasticsearch.com/product/ http://elasticsearch.com/product/ ("petabytes of data with ease")
- nknighthb 13y ago
  Having had nothing but extremely negative experiences with production JVM deployments, anything being not-JVM-based is a huge selling point for me. Furthermore, "pure python" itself means that I can run it in any Python implementation I want, including pypy, without worrying about any C modules.
  - koolkao 13y ago
    what kinds of JVM issues did you run into in deployment? for personal projects I have always used C++, Python, RoR, MEAN, etc. but at work where my role is non-dev, our software dev team likes Java and the heavy weighted GWT... frankly shutting me out. I'd love to know how to convince them to switch out of Java.
    - nknighthb 13y ago
      Memory issues out the ass. Which are obviously all in my head because there are no memory leaks in Java. I must be doing it wrong. Am I sure I know how to multiply? Of course you shouldn't be using version 1.2.3.4.5, our code only works with 1.2.3.4.4, everybody knows that. Well, except for that code, which of course you have to run on 1.2.3.4.4c. And this third-party daemon over here, that only runs on an obscure 6-year-old JVM and you have to put it in /var/deadchicken/ and insert three squirrels before executing. You won't convince them, they've already jumped off the cliff. It's all perfectly normal to them. If you want to do development, find a company that hasn't been contaminated yet.
shavenwarthog2 13y ago
I recommend taking a look at Whoosh -- I worked with it extensively a few years back, adding a keyvalue backend for it. It seems to be designed for indexing manpages. That is, a medium-sized semi-static database with a few different dimensions per document.

146 points by albertzeyer 13y ago

I have found Whoosh to be easy to use, though I haven't put it to use with a large number of documents, so I don't know how well it holds up.

piqufoh 13y ago

Whoosh is great in that it fits the django philosophy of building sites fast, but I wouldn't use if for anything harder than a small site search. Once my site grew larger than 50 mb of text queries started slowing things down. Indexing (python, single threaded) took a while and the larger the index the slower the queries were returned. It's probably the best first iteration search app that I've come across, and you can always slip in solr or something with more umph when you need it.

WoodenChair 13y ago
Can you clarify why you would not use it on anything harder than a small site search? Without context, the comment is not especially helpful.
- mattdeboard 13y ago
  > Once my site grew larger than 50 mb of text queries started slowing things down. Indexing (python, single threaded) took a while and the larger the index the slower the queries were returned. I think this makes it pretty clear. Not sure how much more explanation you need.
  - WoodenChair 13y ago
    Wasn't in the original comment... yet another example of why HN should not have edit.
- ra 13y ago
  For very small datasets, Whoosh is fine. For anything larger than 'site search' on a website that has no UGC, you're much better off with SOLR (or ES, Xapian or Sphinx - whatever your poison). When document counts get into the hundreds you see orders of magnitude faster queries with SOLR (etc), not to mention much more sophisticated querying options.
hackerboos 13y ago
ElasticSearch with Haystack is the next logical move after outgrowing Whoosh.
- andrewingram 13y ago
  Based on my own experiences, by the time search is an important enough feature to outgrow Whoosh, you've also outgrown Haystack's API.
nknighthb 13y ago
> Indexing (python, single threaded) took a while The documentation indicates there is a multiprocessing option. (Also configurable memory limits, if you didn't see that.) http://whoosh.readthedocs.org/en/latest/batch.html http://whoosh.readthedocs.org/en/latest/batch.html
- piqufoh 13y ago
  Ooh - handy, I hadn't seen that thanks. I'll try it out.

davb 13y ago

Whoosh is fantastic. I'm using it on an ecommerce web project just now for indexing CMS pages, products, datasheets and product hierarchy (categories, product families, etc). FWIW I'm using Flask, Flask-SQLAlchemy, Whoosh and Flask-WhooshAlchemy [0]. Quick to get going with but the mix-and-match approach lets me easily rip pieces out as the project grows. [0] http://pythonhosted.org/Flask-WhooshAlchemy/ http://pythonhosted.org/Flask-WhooshAlchemy/

deleted 13y ago

[deleted]

daemonk 13y ago

Sounds cool. Does it scale well? Has anyone used it for a large-ish amount of data (gigs)?

tbatterii 13y ago
not in my experience. i put id3 tags from ~32K songs in it and a query for an artist was around 700ms. same query in ES was like 10ms, same qry to sqlite db (select * from track where artist=...) was about the same as ES. the docs for whoosh make these claims that it's fast, but I haven't seen anything to backup the claims in unit tests or otherwise. I'm sure it's still useful for some purposes but ES is so easy to setup that I don't bother.
davb 13y ago
It works well enough for me. I can't say how large my index set is (because I honestly don't have the figures to hand) but on the current project, a search of around 50,000 products, 10,000 product families and a whole lot of associated data (product attributes, datasheets, etc), an uncached search takes around 65ms. To put it in context, the psycopg2 calls to PostgreSQL take about 100ms to retrieve the associated data once I've found my search results with Whoosh. (Most of my response time sits with SQLAlchemy ORM, building up matrices of data, which is why in production I'm caching the more complex queries with memcached). Overall, for a project of this size (I can't imagine having to index more than a few hundred thousand objects) I'm very happy with Whoosh. If I get to the point of indexing millions of objects, I'll optimize it then.

caioariede 13y ago

If you are using it for Django, take a look on Haystack [1]. It supports Whoosh as well as Solr, Elasticsearch, etc... For example, you could use Whoosh for development environment and Elasticsearch for production if you need something more robust. [1] http://haystacksearch.org/ http://haystacksearch.org/

j_s 13y ago
https://github.com/toastdriven/django-haystack https://github.com/toastdriven/django-haystack
mixmastamyk 13y ago
And Xapian, which I've had good results with.

mattdeboard 13y ago

Solr is just so easy to get up and running, so powerful with so many options for scaling, I don't see the benefit of "pure Python" here. (To be fair, despite living and breathing Python most days, I've never thought "pure Python" is a selling point.)

ra 13y ago
Solr, Sphinx and ElasticSearch are all reliable search-at-scale options for those of us without petabyte problems. Whoosh is fine for very small datasets (megabytes) and low loads, just watch out for file permission issues.
- Torn 13y ago
  ElasticSearch is actually designed to scale to petabytes and beyond: http://elasticsearch.com/product/ http://elasticsearch.com/product/ ("petabytes of data with ease")
nknighthb 13y ago
Having had nothing but extremely negative experiences with production JVM deployments, anything being not-JVM-based is a huge selling point for me. Furthermore, "pure python" itself means that I can run it in any Python implementation I want, including pypy, without worrying about any C modules.
- koolkao 13y ago
  what kinds of JVM issues did you run into in deployment? for personal projects I have always used C++, Python, RoR, MEAN, etc. but at work where my role is non-dev, our software dev team likes Java and the heavy weighted GWT... frankly shutting me out. I'd love to know how to convince them to switch out of Java.
  - nknighthb 13y ago
    Memory issues out the ass. Which are obviously all in my head because there are no memory leaks in Java. I must be doing it wrong. Am I sure I know how to multiply? Of course you shouldn't be using version 1.2.3.4.5, our code only works with 1.2.3.4.4, everybody knows that. Well, except for that code, which of course you have to run on 1.2.3.4.4c. And this third-party daemon over here, that only runs on an obscure 6-year-old JVM and you have to put it in /var/deadchicken/ and insert three squirrels before executing. You won't convince them, they've already jumped off the cliff. It's all perfectly normal to them. If you want to do development, find a company that hasn't been contaminated yet.

shavenwarthog2 13y ago

I recommend taking a look at Whoosh -- I worked with it extensively a few years back, adding a keyvalue backend for it. It seems to be designed for indexing manpages. That is, a medium-sized semi-static database with a few different dimensions per document.