5 ms·
Have worked with ES... If you want the out-of-the-box distributed experience, then you can't go wrong with Elasticsearch because of the built-in sharding mecha
by yeukhon 9y ago
Have worked with ES...
If you want the out-of-the-box distributed experience, then you can't go wrong with Elasticsearch because of the built-in sharding mechanism. However, figuring out how many shards you need for each index, where should each shard sits, and how to balance out the cluster has always been a trial and error exercise. Indices continue to grow like there is infinite space, and performance will degrade. What I find useful in the end is to have multiple ES clusters, which seems obvious right? One of the most frequently asked questions is "how many nodes do I need", and the answer is always "it depends" which is true, but gives me a chill. With Cassandra, from my experience, adding more nodes (scale horizontally) so data spread more thinly, and revisit some data structure change, would be an acceptable answer.
The other thing is a lot of folks I have worked with just dump stuff into ES because ES feels like a dumping ground for "json"-like documents (reminds of me MongoDB). Please compress, understand mappings and different fields to make the document smaller...
I am still very convinced SQL databases are robust enough to take any lazy dumping.
- threeseed 9y agoWhy not just run ElasticSearch on a single instance then ? That way you're not having to completely redo your search infrastructure at some point in time. You can seamlessly scale from 1 document to billions without any change to your code, backup approach, monitoring etc.
- yeukhon 9y ago> Why not just run ElasticSearch on a single instance then ? Sorry, I am not sure why you brought up a single instance. If you run on a single instance, you lose half of the benefits of ES which is trying to be highly available. If we have an index which does not need any backups, single instance is fine, but for most people single instance is not an option. Furthermore, if one is already struggling with 3-4 nodes, how helpful is single instance?
- threeseed 9y agoI brought up single instance since you suggested using a SQL database most of which don't support a distributed deployment in the same way ElasticSearch does. My point was merely that ElasticSearch scales from very small use cases to very large ones. You don't have to setup a cluster. And that using your SQL database isn't going to be significantly easier to manage.
- moe 9y ago> SQL database isn't going to be significantly easier to manage. A SQL database doesn't run out of memory and crash when you add documents to it "too fast". ElasticSearch does.
- rpedela 9y agoFYI you can now split [1] and shrink [2] the number of shards. 1. https://www.elastic.co/guide/en/elasticsearch/reference/6.1/indices-split-index.html https://www.elastic.co/guide/en/elasticsearch/reference/6.1/... 2. https://www.elastic.co/guide/en/elasticsearch/reference/6.1/indices-shrink-index.html https://www.elastic.co/guide/en/elasticsearch/reference/6.1/...
- ddorian43 9y agoIts not while live indexing and has many caveats
- deleted 9y ago[deleted]