4 ms·

Realtime Search: Solr vs Elasticsearch

67 points by kimchy 15y ago

sbochins 15y ago
What does the author mean by while indexing? You have to send a commit message to make the newly indexed doc searchable. Its silly to do this after every document. I'm assuming that is what the author was doing to get search poor search results. You could also just use Lucene's NRT abilities in Solr w/ a simple patch. I believe it is going to be working out of the box in the next version of solr. I wrote one myself to deal with real time search. If we want a fair comparison, we should be comparing Solr using NRT and ElasticSearch.
- kimchy 15y ago
  NRT is just part of the story, since in order to provide a proper (near) real time features in a distributed architecture you need to build it around it. For example, not doing pull replication, but push replication. The whole distributed nature of elasticsearch compared to Solr was not discussed in this blog post, but its a big differentiator.
- diego 15y ago
  It is not silly if you have asynchronous indexing because your content is generated by your users, one document at a time. If you want real-time indexing, that's the way to do it. This is the default behavior for IndexTank. When an indexing call returns, whatever you indexed WILL show up in search results. We don't give you the option to commit or not commit, we simply designed IndexTank to be fast enough when indexing in real time. I'd like to see us included in that comparison as well.
  - true_religion 15y ago
    What's IndexTank using as a backend?
    - diego 15y ago
      IndexTank :) Meaning, we wrote a search engine from the ground up. We probably should write a blog post to explain why. We did use a couple of things from Lucene: - The tokenizers, to be compatible with the query syntax - The index format for persistence on disk. It's a great format, no point in reinventing that. Analogous to how Open Office can save in .doc format.
  - sbochins 15y ago
    Its not the only way to do it. Twitter doesn't do it that way, neither does LinkedIn. They have their own Search Engines as well build around Lucene. I haven't looked at them in a while, but I don't think they were commiting every document. They also get their content from their users.
    - Yeroc 15y ago
      FYI, LinkedIn open-sourced the work they did to build real-time indexing/search on top of Lucene. It's called Zoie (http://javasoze.github.com/zoie/ http://javasoze.github.com/zoie/), I've played with it and it's nice from a performance point of view...
    - diego 15y ago
      Correct, Twitter in fact did a lot of work on the Lucene core specifically for this. The OP was talking about standard Solr though.
  - khill 15y ago
    Lucene doesn't have to commit document updates to make them searchable. You just need to grab the IndexReader via the IndexWriter.getReader() method instead of opening an IndexReader on the index directory. We use this for NRT indexing currently on our site and it works fine - there's a small (500 millisecond or less) delay in changes appearing but it's close enough for us. We don't use Solr, though - it's a custom search engine based on Lucene.
vibragiel 15y ago
Granted, percolation sounds pretty cool, but if the article is referring only to single-server/single-core Solr architectures, then that speed comparison is unfair.
sidmitra 15y ago
The article doesn't mention the setup he used for the benchmark. Is anyone using ES on production, how is the support for things like lat/long(geo) search. If you're on django then your options are limited if you want to use django-haystack. Solr is pretty much the only way to go. xapian geo support hasn't landed yet, and waiting for solr 4 to land. I've been using JTeam SSP 1.0 plugin(http://blog.jteam.nl http://blog.jteam.nl) for solr, with django-haystack(https://github.com/sidmitra/django-haystack https://github.com/sidmitra/django-haystack) for the same and it's been really useful and responsive so far. I still want to give ES a try, but will have to look into support with haystack + for geo stuff.
- true_religion 15y ago
  ES doesn't have haystack support, and shoe-horning it in is pretty hard to do because ES semantics are quite different from the search engines Haystack supports. I use ES via Python, and ended up writing my own backend for it that approximates the Django ORM.
  - sidmitra 15y ago
    Are you using it in production anywhere? The stack i mentioned above powers the location search for teaspiller.com right now, but i'm not that invested into haystack not to look at alternatives if it suits us well.
    - true_religion 15y ago
      I am, but only for my side-project: http://9cloud.us/about/ http://9cloud.us/about/ For everything else, I'm using Sphinx. I chose Spinx because at the time (starting 2 years ago), it seemed like the best choice for real time indexing, set up was easy, and it integrated well with MySQL. But now the Lucrene backend DBs are in decent completion with it---Elastic Search and Solandra.
eck 15y ago
I'm a bit confused by the usage of "percolation". Generally speaking that refers to, in a tiered retrieval system, a document being initially indexed in a small temporary index then gradually "percolated" upwards to larger more permanent indices. Such systems allow updates without merging into the main index. See: http://nlp.stanford.edu/IR-book/html/htmledition/dynamic-indexing-1.html http://nlp.stanford.edu/IR-book/html/htmledition/dynamic-ind...
espeed 15y ago
Twitter added realtime search to Lucene, and it is should be in the 4.0 release.
- khill 15y ago
  Lucene has supported near real-time search since 2.9 or so. By calling IndexWriter.getReader(), your index reader has access to both committed and non-committed changes. In my experience, this near real-time search picks up changes within the last 500 milliseconds or so. I think it's basically a workaround for the latency related to disk I/O on commits. What is Twitter doing to make that faster? I'd appreciate any links you could post which explain their changes.
  - squarecog 15y ago
    Detailed slides here (pdf): http://www.lucenerevolution.org/sites/default/files/Lucene%20Rev%20Preso%20Busch%20Realtime_Search_LR1010.pdf http://www.lucenerevolution.org/sites/default/files/Lucene%2... Video: http://vimeo.com/16063395 http://vimeo.com/16063395

67 points by kimchy 15y ago

What does the author mean by while indexing? You have to send a commit message to make the newly indexed doc searchable. Its silly to do this after every document. I'm assuming that is what the author was doing to get search poor search results. You could also just use Lucene's NRT abilities in Solr w/ a simple patch. I believe it is going to be working out of the box in the next version of solr. I wrote one myself to deal with real time search. If we want a fair comparison, we should be comparing Solr using NRT and ElasticSearch.

kimchy 15y ago
NRT is just part of the story, since in order to provide a proper (near) real time features in a distributed architecture you need to build it around it. For example, not doing pull replication, but push replication. The whole distributed nature of elasticsearch compared to Solr was not discussed in this blog post, but its a big differentiator.
diego 15y ago
It is not silly if you have asynchronous indexing because your content is generated by your users, one document at a time. If you want real-time indexing, that's the way to do it. This is the default behavior for IndexTank. When an indexing call returns, whatever you indexed WILL show up in search results. We don't give you the option to commit or not commit, we simply designed IndexTank to be fast enough when indexing in real time. I'd like to see us included in that comparison as well.
- true_religion 15y ago
  What's IndexTank using as a backend?
  - diego 15y ago
    IndexTank :) Meaning, we wrote a search engine from the ground up. We probably should write a blog post to explain why. We did use a couple of things from Lucene: - The tokenizers, to be compatible with the query syntax - The index format for persistence on disk. It's a great format, no point in reinventing that. Analogous to how Open Office can save in .doc format.
- sbochins 15y ago
  Its not the only way to do it. Twitter doesn't do it that way, neither does LinkedIn. They have their own Search Engines as well build around Lucene. I haven't looked at them in a while, but I don't think they were commiting every document. They also get their content from their users.
  - Yeroc 15y ago
    FYI, LinkedIn open-sourced the work they did to build real-time indexing/search on top of Lucene. It's called Zoie (http://javasoze.github.com/zoie/ http://javasoze.github.com/zoie/), I've played with it and it's nice from a performance point of view...
  - diego 15y ago
    Correct, Twitter in fact did a lot of work on the Lucene core specifically for this. The OP was talking about standard Solr though.
- khill 15y ago
  Lucene doesn't have to commit document updates to make them searchable. You just need to grab the IndexReader via the IndexWriter.getReader() method instead of opening an IndexReader on the index directory. We use this for NRT indexing currently on our site and it works fine - there's a small (500 millisecond or less) delay in changes appearing but it's close enough for us. We don't use Solr, though - it's a custom search engine based on Lucene.

vibragiel 15y ago

Granted, percolation sounds pretty cool, but if the article is referring only to single-server/single-core Solr architectures, then that speed comparison is unfair.

sidmitra 15y ago

The article doesn't mention the setup he used for the benchmark. Is anyone using ES on production, how is the support for things like lat/long(geo) search. If you're on django then your options are limited if you want to use django-haystack. Solr is pretty much the only way to go. xapian geo support hasn't landed yet, and waiting for solr 4 to land. I've been using JTeam SSP 1.0 plugin(http://blog.jteam.nl http://blog.jteam.nl) for solr, with django-haystack(https://github.com/sidmitra/django-haystack https://github.com/sidmitra/django-haystack) for the same and it's been really useful and responsive so far. I still want to give ES a try, but will have to look into support with haystack + for geo stuff.

true_religion 15y ago
ES doesn't have haystack support, and shoe-horning it in is pretty hard to do because ES semantics are quite different from the search engines Haystack supports. I use ES via Python, and ended up writing my own backend for it that approximates the Django ORM.
- sidmitra 15y ago
  Are you using it in production anywhere? The stack i mentioned above powers the location search for teaspiller.com right now, but i'm not that invested into haystack not to look at alternatives if it suits us well.
  - true_religion 15y ago
    I am, but only for my side-project: http://9cloud.us/about/ http://9cloud.us/about/ For everything else, I'm using Sphinx. I chose Spinx because at the time (starting 2 years ago), it seemed like the best choice for real time indexing, set up was easy, and it integrated well with MySQL. But now the Lucrene backend DBs are in decent completion with it---Elastic Search and Solandra.

eck 15y ago

I'm a bit confused by the usage of "percolation". Generally speaking that refers to, in a tiered retrieval system, a document being initially indexed in a small temporary index then gradually "percolated" upwards to larger more permanent indices. Such systems allow updates without merging into the main index. See: http://nlp.stanford.edu/IR-book/html/htmledition/dynamic-indexing-1.html http://nlp.stanford.edu/IR-book/html/htmledition/dynamic-ind...

espeed 15y ago

Twitter added realtime search to Lucene, and it is should be in the 4.0 release.

khill 15y ago
Lucene has supported near real-time search since 2.9 or so. By calling IndexWriter.getReader(), your index reader has access to both committed and non-committed changes. In my experience, this near real-time search picks up changes within the last 500 milliseconds or so. I think it's basically a workaround for the latency related to disk I/O on commits. What is Twitter doing to make that faster? I'd appreciate any links you could post which explain their changes.
- squarecog 15y ago
  Detailed slides here (pdf): http://www.lucenerevolution.org/sites/default/files/Lucene%20Rev%20Preso%20Busch%20Realtime_Search_LR1010.pdf http://www.lucenerevolution.org/sites/default/files/Lucene%2... Video: http://vimeo.com/16063395 http://vimeo.com/16063395