3 ms·

For the search index, I wonder why they did not go with ElasticSearch or Solr, and decided to roll their own replication/distribution layer. RAFT is much simple

by karterk 13y ago

For the search index, I wonder why they did not go with ElasticSearch or Solr, and decided to roll their own replication/distribution layer. RAFT is much simpler than Paxos, but it's still something pretty difficult to get right (as with any distributed system). Given that both ES and Solr have already got this figured out, what were the specific reasons for rolling your own C++/Nginx layer?

zmitri 13y ago
I use Algolia on my site, and I believe it has to do with speed. Algolia is significantly faster than ElasticSearch.
xroche 13y ago
I believe this has to do with performances - ie. performing in few milliseconds rather than in tens/hundreds of milliseconds (at least that's what their benchmarks claim: http://blog.algolia.com/full-text-search-in-your-database-algolia-versus-elasticsearch/ http://blog.algolia.com/full-text-search-in-your-database-al...)
jlemoine 13y ago
Search engine needs have evolved a lot since Lucene was designed (Lucene is the search library behind ElasticSearch and Solr). Users now want an instant search that search at the first keystroke and are able to tolerate typos. Lucene does not propose anything special for that, so you have mainly two options: - slow down indexing: index all ngrams of words (this also require a lot of disk) - slow down search: you need to find all words that match the prefix query in the dictionary and then retrieve all inverted lists. ES and Solr have added autocomplete module that is basically a compiled Trie that associate a string to a results set, but this is not a real instant-search engine, there is no intersection of inverted lists and this produce mismatch between the autocomplete and search results. On our side, we have redesigned all search algorithms and data-structures from scratch to run on a mobile devices. We are for example able to search all cities from Geonames (> 3M) on a old iPhone 3GS in less than 50ms (including prefix search & typo tolerance) with only 1KB or RAM and a flash disk. You can't do that with Lucene. So I let you imagine what we can do with a high-end server ! For the High-availability & Consensus, this is something complex and we have spent a lot of time to test it by killing machines of a cluster, and we still continue to test it all the days since every deployment kill index builder process !

by karterk 13y ago

zmitri 13y ago

I use Algolia on my site, and I believe it has to do with speed. Algolia is significantly faster than ElasticSearch.

xroche 13y ago

I believe this has to do with performances - ie. performing in few milliseconds rather than in tens/hundreds of milliseconds (at least that's what their benchmarks claim: http://blog.algolia.com/full-text-search-in-your-database-algolia-versus-elasticsearch/ http://blog.algolia.com/full-text-search-in-your-database-al...)

jlemoine 13y ago

Search engine needs have evolved a lot since Lucene was designed (Lucene is the search library behind ElasticSearch and Solr). Users now want an instant search that search at the first keystroke and are able to tolerate typos. Lucene does not propose anything special for that, so you have mainly two options: - slow down indexing: index all ngrams of words (this also require a lot of disk) - slow down search: you need to find all words that match the prefix query in the dictionary and then retrieve all inverted lists. ES and Solr have added autocomplete module that is basically a compiled Trie that associate a string to a results set, but this is not a real instant-search engine, there is no intersection of inverted lists and this produce mismatch between the autocomplete and search results. On our side, we have redesigned all search algorithms and data-structures from scratch to run on a mobile devices. We are for example able to search all cities from Geonames (> 3M) on a old iPhone 3GS in less than 50ms (including prefix search & typo tolerance) with only 1KB or RAM and a flash disk. You can't do that with Lucene. So I let you imagine what we can do with a high-end server ! For the High-availability & Consensus, this is something complex and we have spent a lot of time to test it by killing machines of a cluster, and we still continue to test it all the days since every deployment kill index builder process !