5 ms·

Whoosh: a fast pure-Python search engine

60 points by bdfh42 18y ago

thomaspaine 18y ago
Looks very cool...it says it's as fast or faster than PyLucene and Xappy, but it'd be nice to see some concrete benchmarks.
nihilocrat 18y ago
There's definitely some clever coding afoot when a pure-python library can outpace a C wrapper library.
- adamtj 18y ago
  Well written python basically is a C wrapper.
  - teej 18y ago
    So is Ruby - that doesn't make it fast.
- timf 18y ago
  It does not look like it is outpacing them at all. ( http://news.ycombinator.com/item?id=478654 http://news.ycombinator.com/item?id=478654 )
delano 18y ago
... a typical single-term search takes 0.003 seconds in Xappy, while Whoosh takes 0.01 seconds. The Whoosh index was less than 4 MB, while the Xappy/Xapian index was 30 MB on disk. I should hope it's fast with an index of 4MB. That aside, it's more like a lookup engine rather than search engine. Search refers to calculating multi-dimensional relevance in real time. Here's a quick, two-dimensional example: A search for "fender" could be configured to have related keywords: "music", "gisbon", and "automobile", "ford". The strength of the relation should also be configurable as it's used to create a score for the weight of each keyword. Then expand each keyword: fender, fenders, music, musical, etc... These expansions are given slightly lower scores than the keywords. Then we consider what fields in the data set to look for these values. Each field has a score. We determine which records to include by making a two-dimensional calculation between the keywords and fields. Now we sort the result set based on the same or other additional criteria and return. There are many possible dimensions include misspellings, synonyms, wildcards, user history, brands. Basically anything. And the data-set get much larger. That's why search is so difficult. I'm not criticizing Whoosh specifically because other open source "search engine" libraries have the same weaknesses. But it is important to understand the difference between search and lookup.
- timf 18y ago
  "I should hope it's fast with an index of 4MB." But even in that case it's saying it is slower at search (0.003 < 0.01) than Xappy (that also had a 7.5x bigger index). Only claiming "as fast or faster" at indexing (and has no data to support that claim). I agree with you about search being hard, but it does mention BM25F so it doesn't look like "just" a lookup engine...
  - delano 18y ago
    The scoring in Whoosh is used only for sorting the result set. It's not used in determining the records within that set which is the primary reason why I refer to it as a lookup engine.
    - timf 18y ago
      OK, fair enough. But it looks like there's some awareness of these issues in that project, I thought you were trying to pigeonhole this as a glorified hashtable or something. Quoting from the Whoosh docs: "Like its ancestor Lucene, Whoosh is not really a search engine, it's a toolkit for creating a search engine." Lucene has support for proximity queries, fuzzy "minimumSimilarity", etc., and it looks like Whoosh wants to be a tool like this. Would "search engine library" be a more appropriate term for you? Something that is used to construct a useful search engine.
      - delano 18y ago
        Search engine library is better, ya. However, creating the search engine is the hard problem. It helps to have tools to make some of the outlying problems a bit easier, but the challenge is being able to calculate a configurable set of multi-dimensional scores. There are 3 places the scoring takes place: indexing-time, run-time, and search-time. Every solution (commercial and open-source) has an indexing process. Most have some run-time processing but only a few have multi-dimensional scoring at search-time and that is the coup de grâce. When you make the relations configurable and increase the data set to GB or TB you have several incredibly difficult problems. There are commercial products that do it, big and small, but to my knowledge there are no open source projects that can. In other words, the state of commercial products is so far ahead of open source ones that it's misleading to refer to both as "search engines".
deleted 18y ago
[deleted]

60 points by bdfh42 18y ago

Looks very cool...it says it's as fast or faster than PyLucene and Xappy, but it'd be nice to see some concrete benchmarks.

nihilocrat 18y ago

There's definitely some clever coding afoot when a pure-python library can outpace a C wrapper library.

adamtj 18y ago
Well written python basically is a C wrapper.
- teej 18y ago
  So is Ruby - that doesn't make it fast.
timf 18y ago
It does not look like it is outpacing them at all. ( http://news.ycombinator.com/item?id=478654 http://news.ycombinator.com/item?id=478654 )

delano 18y ago

... a typical single-term search takes 0.003 seconds in Xappy, while Whoosh takes 0.01 seconds. The Whoosh index was less than 4 MB, while the Xappy/Xapian index was 30 MB on disk. I should hope it's fast with an index of 4MB. That aside, it's more like a lookup engine rather than search engine. Search refers to calculating multi-dimensional relevance in real time. Here's a quick, two-dimensional example: A search for "fender" could be configured to have related keywords: "music", "gisbon", and "automobile", "ford". The strength of the relation should also be configurable as it's used to create a score for the weight of each keyword. Then expand each keyword: fender, fenders, music, musical, etc... These expansions are given slightly lower scores than the keywords. Then we consider what fields in the data set to look for these values. Each field has a score. We determine which records to include by making a two-dimensional calculation between the keywords and fields. Now we sort the result set based on the same or other additional criteria and return. There are many possible dimensions include misspellings, synonyms, wildcards, user history, brands. Basically anything. And the data-set get much larger. That's why search is so difficult. I'm not criticizing Whoosh specifically because other open source "search engine" libraries have the same weaknesses. But it is important to understand the difference between search and lookup.

timf 18y ago
"I should hope it's fast with an index of 4MB." But even in that case it's saying it is slower at search (0.003 < 0.01) than Xappy (that also had a 7.5x bigger index). Only claiming "as fast or faster" at indexing (and has no data to support that claim). I agree with you about search being hard, but it does mention BM25F so it doesn't look like "just" a lookup engine...
- delano 18y ago
  The scoring in Whoosh is used only for sorting the result set. It's not used in determining the records within that set which is the primary reason why I refer to it as a lookup engine.
  - timf 18y ago
    OK, fair enough. But it looks like there's some awareness of these issues in that project, I thought you were trying to pigeonhole this as a glorified hashtable or something. Quoting from the Whoosh docs: "Like its ancestor Lucene, Whoosh is not really a search engine, it's a toolkit for creating a search engine." Lucene has support for proximity queries, fuzzy "minimumSimilarity", etc., and it looks like Whoosh wants to be a tool like this. Would "search engine library" be a more appropriate term for you? Something that is used to construct a useful search engine.
    - delano 18y ago
      Search engine library is better, ya. However, creating the search engine is the hard problem. It helps to have tools to make some of the outlying problems a bit easier, but the challenge is being able to calculate a configurable set of multi-dimensional scores. There are 3 places the scoring takes place: indexing-time, run-time, and search-time. Every solution (commercial and open-source) has an indexing process. Most have some run-time processing but only a few have multi-dimensional scoring at search-time and that is the coup de grâce. When you make the relations configurable and increase the data set to GB or TB you have several incredibly difficult problems. There are commercial products that do it, big and small, but to my knowledge there are no open source projects that can. In other words, the state of commercial products is so far ahead of open source ones that it's misleading to refer to both as "search engines".

deleted 18y ago

[deleted]