3 ms·

Yes, for typo correction + instant search, Lucene definitely is not fast enough on large datasets. There are also some limitations with fuzzy searching when you

Yes, for typo correction + instant search, Lucene definitely is not fast enough on large datasets. There are also some limitations with fuzzy searching when you also want to sort/rank documents at the same time. Lucene is also a very generic mature library for a wider set of usecases.

msla 7y ago
Lucene is nice because it has a pretty well-defined plugin architecture where you can write a plugin which gets a parse tree, mutate the tree (to add fuzziness to the matching, for example), and then pass the tree to the next stage. It's a nicely composable way to extend functionality.
jillesvangurp 7y ago
What do you mean not fast enough? Lucene/Elasticsearch/Solr are basically routinely used on datasets that are way beyond the scope of what this or Algolia is used on (i.e. petabyte scale). I've worked with teams indexing billions of documents. At that scale measuring ranking quality is a much bigger concern than raw performance, which is just a matter of throwing more hardware at it (i.e. it's more of a cost concern than a performance concern). When it comes to ranking quality, a one size fits all, non sharded solution like this is definitely not going to stand up to much scrutiny. Either it fits, or more likely it doesn't and you need the knobs to tune it (which Lucene provides you plenty of). Most of the tricks that Algolia and (presumably) this product do to be fast have more to do with how they manage memory than with their implementation language. Basically, Algolia is a non sharded search index that is loaded into memory in its entirety. They don't have to worry about disk seeks, file cache misses, etc. Not a problem when your entire index fits in memory. Of course that puts an upper limit on what they can index and limits the uses to smallish datasets that in no way pose any challenge whatsoever to Lucene (especially when your index easily fits in memory). I'm not saying it's a bad approach. I think it's a great approach for smallish data sets with very loose ranking requirements where a one size fits all solution does the job and is good enough. For many companies search is not really core to their experience and it just needs to be idiot proof and low hassle to set up for their non expert tech teams and product managers.
- karterk 7y ago
  I agree with everything you've said. I was a bit sloppy in not mentioning the trade-offs. My "fast enough" remark was certainly not for handling large datasets. The parent comment had asked why Lucene was not used for Typesense and my response was specifically to that point. Typesense targets a different set of use cases involving small-medium datasets where an interactive search experience is importnat.

msla 7y ago

Lucene is nice because it has a pretty well-defined plugin architecture where you can write a plugin which gets a parse tree, mutate the tree (to add fuzziness to the matching, for example), and then pass the tree to the next stage. It's a nicely composable way to extend functionality.

jillesvangurp 7y ago

What do you mean not fast enough? Lucene/Elasticsearch/Solr are basically routinely used on datasets that are way beyond the scope of what this or Algolia is used on (i.e. petabyte scale). I've worked with teams indexing billions of documents. At that scale measuring ranking quality is a much bigger concern than raw performance, which is just a matter of throwing more hardware at it (i.e. it's more of a cost concern than a performance concern). When it comes to ranking quality, a one size fits all, non sharded solution like this is definitely not going to stand up to much scrutiny. Either it fits, or more likely it doesn't and you need the knobs to tune it (which Lucene provides you plenty of). Most of the tricks that Algolia and (presumably) this product do to be fast have more to do with how they manage memory than with their implementation language. Basically, Algolia is a non sharded search index that is loaded into memory in its entirety. They don't have to worry about disk seeks, file cache misses, etc. Not a problem when your entire index fits in memory. Of course that puts an upper limit on what they can index and limits the uses to smallish datasets that in no way pose any challenge whatsoever to Lucene (especially when your index easily fits in memory). I'm not saying it's a bad approach. I think it's a great approach for smallish data sets with very loose ranking requirements where a one size fits all solution does the job and is good enough. For many companies search is not really core to their experience and it just needs to be idiot proof and low hassle to set up for their non expert tech teams and product managers.

karterk 7y ago
I agree with everything you've said. I was a bit sloppy in not mentioning the trade-offs. My "fast enough" remark was certainly not for handling large datasets. The parent comment had asked why Lucene was not used for Typesense and my response was specifically to that point. Typesense targets a different set of use cases involving small-medium datasets where an interactive search experience is importnat.