5 ms·

Elasticsearch has its flaws, but are there really any alternatives that do what it does? In my case, I need to do fulltext search across millions of documents -

by Meekro 11y ago

Elasticsearch has its flaws, but are there really any alternatives that do what it does? In my case, I need to do fulltext search across millions of documents -- nearly a TB of text in all, and still growing. Many database engines can do basic fulltext, but what else can scale like Elasticsearch does while offering powerful fulltext features like fuzzy matching and "more like this?"

nl 11y ago
There is Solr of course. But I think the recommendations the post are entirely reasonable: My recommendations for Elasticsearch users are unchanged: store your data in a database with better safety guarantees, and continuously upsert every document from that database into Elasticsearch. If your search engine is missing a few documents for a day, it’s not a big deal; they’ll be reinserted on the next run and appear in subsequent searches. Not using Elasticsearch as a system of record also insulates you from having to worry about ES downtime during elections.
- NDizzle 11y ago
  This makes me feel good because that's the same conclusion I came to with my ES implementation almost 2 years ago.
mavdi 11y ago
Exactly... ElasticSearch doesn't market itself as the best fault tolerant DB. If you're using ES as your main data persistence point, you're missing the point of it.
tylertreat 11y ago
I think the main point is that Elasticsearch should not be used as a primary store. It seems obvious (or maybe not?), but some people do use it as such. Just like Redis should only be used as a cache.
rch 11y ago
Riak search works well. The indexing is just Solr, but the cluster and data itself is managed differently.
- sargun 11y ago
  Riak Search 2.0, or Yokozuna is what you're thinking about. DISCLAIMER: I work for Basho.
- sargun 11y ago
  Riak Search 2.0, or Yokozuna is what you're thinking about. DISCLAIMER: I work for Basho.
threeseed 11y ago
There are a few options e.g. Microsoft FAST or you can roll your own. However Lucene based systems can leverage the wide variety of addons and is battle tested so it's hard to recommend against it. Just remember the golden rule of architecture. Always assume every component will catastrophically fail at some point and design accordingly. In the case of ElasticSearch/Solr it means don't use it as a primary data store, backup regularly and have a mechanism to insert deltas.
- vosper 11y ago
  Rolling your own is surely the worst possible option - there's very little chance you'll build something better than ElasticSearch on your own, and a very high chance that you'll build something much worse.
- mewwts 11y ago
  In order to use any remains of FAST you would need a sharepoint installation. Which is not anywhere near free or open source, though. I think the DB + ES/Solr is the best alternative. Possibly doing snapshots often from ES could help eliminate the need for a db?
curun1r 11y ago
I haven't used it in ~5 years, but Sphinx (http://sphinxsearch.com http://sphinxsearch.com) might be a reasonable competitor. Back then it was pretty pleasant to use...especially it's indexing abilities, which were significantly faster than ES. I can't speak to it's distributed ability since our index was never that large, but the documentation purports to have that capability. Sphinx has some curious features, like being able to run as a MySQL storage engine (no doubt the result of the author being ex-MySQL), but it's also got most of the features that you come to expect in search engine, so I think it's fair to call it a competitor. That said, we still use ES.
- gpsarakis 11y ago
  Sphinx does indeed a very good job and is continuously improved. It is very fast and with small memory footprint. We also built https://techusearch.com https://techusearch.com which uses Sphinx indexes (think of a SaaS version of Elasticsearch backed by Sphinx).
platform 11y ago
It is commercially licensed, but MarkLogic scales horizontally and is capable of full text search and structured queries (via xquery) https://developer.marklogic.com/learn/2009-07-search-api-walkthrough https://developer.marklogic.com/learn/2009-07-search-api-wal... They also have Alert queries, which are similar to ElasticSearch percolation feature. MarkLogic is also an ACID compliant store with configurable HA, as well as asynch replication to a remote site. it could potentially serve if correctly configured, as the primary store. --- ArangoDB might have these capabilities as well. And again, it is an ACID compliant database so the data committed, the indices, and the resulting searches should respect transactional boundaries at any moment of time. I had not looked at ArangoDB deep enough to understand what limitations it might have though. It does not have similarity search capability, and hierarchical data like ElasticSearch has https://docs.arangodb.com/2.3/IndexHandling/Fulltext.html https://docs.arangodb.com/2.3/IndexHandling/Fulltext.html ---- I checked RethinkDB, and it appears that it suggests to integrate with ElasticSearch, meaning that elastic search will have a copy of the data from Rethink (and therefore subject to commit protocol of ElasticSearch) --- Somebody already mentioned Riak with search capability. I think this one is much more similar to Marklogic's capability (and it is free) The advantage of these types of systems is that your primary store and text indexes are sitting on top of the single copy of data (or replicated invisibly within the infrastructure) http://docs.basho.com/riak/latest/dev/using/search/ http://docs.basho.com/riak/latest/dev/using/search/
deleted 11y ago
[deleted]
digitalzombie 11y ago
ES is probably to easiest to cluster. There's Solr which is a much more matured technology but wasn't cluster in mind, I think it uses zookeeper. Another Lucene base tech is RavenDb but IIRC it's for microsoft. what else offer something like ES? Any lucene based nosql, ES is lucened base anyway. You can roll your own and use Lucene library.

by Meekro 11y ago

nl 11y ago

There is Solr of course. But I think the recommendations the post are entirely reasonable: My recommendations for Elasticsearch users are unchanged: store your data in a database with better safety guarantees, and continuously upsert every document from that database into Elasticsearch. If your search engine is missing a few documents for a day, it’s not a big deal; they’ll be reinserted on the next run and appear in subsequent searches. Not using Elasticsearch as a system of record also insulates you from having to worry about ES downtime during elections.

NDizzle 11y ago
This makes me feel good because that's the same conclusion I came to with my ES implementation almost 2 years ago.

mavdi 11y ago

Exactly... ElasticSearch doesn't market itself as the best fault tolerant DB. If you're using ES as your main data persistence point, you're missing the point of it.

tylertreat 11y ago

I think the main point is that Elasticsearch should not be used as a primary store. It seems obvious (or maybe not?), but some people do use it as such. Just like Redis should only be used as a cache.

rch 11y ago

Riak search works well. The indexing is just Solr, but the cluster and data itself is managed differently.

sargun 11y ago
Riak Search 2.0, or Yokozuna is what you're thinking about. DISCLAIMER: I work for Basho.
sargun 11y ago
Riak Search 2.0, or Yokozuna is what you're thinking about. DISCLAIMER: I work for Basho.

threeseed 11y ago

There are a few options e.g. Microsoft FAST or you can roll your own. However Lucene based systems can leverage the wide variety of addons and is battle tested so it's hard to recommend against it. Just remember the golden rule of architecture. Always assume every component will catastrophically fail at some point and design accordingly. In the case of ElasticSearch/Solr it means don't use it as a primary data store, backup regularly and have a mechanism to insert deltas.

vosper 11y ago
Rolling your own is surely the worst possible option - there's very little chance you'll build something better than ElasticSearch on your own, and a very high chance that you'll build something much worse.
mewwts 11y ago
In order to use any remains of FAST you would need a sharepoint installation. Which is not anywhere near free or open source, though. I think the DB + ES/Solr is the best alternative. Possibly doing snapshots often from ES could help eliminate the need for a db?

curun1r 11y ago

I haven't used it in ~5 years, but Sphinx (http://sphinxsearch.com http://sphinxsearch.com) might be a reasonable competitor. Back then it was pretty pleasant to use...especially it's indexing abilities, which were significantly faster than ES. I can't speak to it's distributed ability since our index was never that large, but the documentation purports to have that capability. Sphinx has some curious features, like being able to run as a MySQL storage engine (no doubt the result of the author being ex-MySQL), but it's also got most of the features that you come to expect in search engine, so I think it's fair to call it a competitor. That said, we still use ES.

gpsarakis 11y ago
Sphinx does indeed a very good job and is continuously improved. It is very fast and with small memory footprint. We also built https://techusearch.com https://techusearch.com which uses Sphinx indexes (think of a SaaS version of Elasticsearch backed by Sphinx).

platform 11y ago

It is commercially licensed, but MarkLogic scales horizontally and is capable of full text search and structured queries (via xquery) https://developer.marklogic.com/learn/2009-07-search-api-walkthrough https://developer.marklogic.com/learn/2009-07-search-api-wal... They also have Alert queries, which are similar to ElasticSearch percolation feature. MarkLogic is also an ACID compliant store with configurable HA, as well as asynch replication to a remote site. it could potentially serve if correctly configured, as the primary store. --- ArangoDB might have these capabilities as well. And again, it is an ACID compliant database so the data committed, the indices, and the resulting searches should respect transactional boundaries at any moment of time. I had not looked at ArangoDB deep enough to understand what limitations it might have though. It does not have similarity search capability, and hierarchical data like ElasticSearch has https://docs.arangodb.com/2.3/IndexHandling/Fulltext.html https://docs.arangodb.com/2.3/IndexHandling/Fulltext.html ---- I checked RethinkDB, and it appears that it suggests to integrate with ElasticSearch, meaning that elastic search will have a copy of the data from Rethink (and therefore subject to commit protocol of ElasticSearch) --- Somebody already mentioned Riak with search capability. I think this one is much more similar to Marklogic's capability (and it is free) The advantage of these types of systems is that your primary store and text indexes are sitting on top of the single copy of data (or replicated invisibly within the infrastructure) http://docs.basho.com/riak/latest/dev/using/search/ http://docs.basho.com/riak/latest/dev/using/search/

deleted 11y ago

[deleted]

digitalzombie 11y ago

ES is probably to easiest to cluster. There's Solr which is a much more matured technology but wasn't cluster in mind, I think it uses zookeeper. Another Lucene base tech is RavenDb but IIRC it's for microsoft. what else offer something like ES? Any lucene based nosql, ES is lucened base anyway. You can roll your own and use Lucene library.