5 ms·

I know the project doesn't claim it, but the title somewhat implies this: I honestly don't understand people claiming ElasticSearch is hard to operate, especial

by heipei 7y ago

I know the project doesn't claim it, but the title somewhat implies this: I honestly don't understand people claiming ElasticSearch is hard to operate, especially not at small scales. If anything, ElasticSearch for me has been one of the easiest pieces of infrastructure to operate, for me pretty much "zero-config". Let me elaborate: You can run ElasticSearch via Docker command-line, if you want a cluster you just supply IPs of the other nodes. Then you start indexing documents with simple HTTP calls. You can add or remove nodes at any time and don't have to do anything but to start another ElasticSearch instance. If you run out of space or performance just start another node. Everything needed for management, indexing, search is available through HTTP APIs, no tools needed. Clustered ElasticSearch has been rock-solid for me and I've used it in anger many times. The level of maintenance needed is close to zero, both initially and long-term. Compare that with the abysmal experience of setting up a sharded MongoDB cluster for example... Please enlighten me how ElasticSearch is "a lot of work to operate" (heard that one multiple times), and what you're comparing it to.

Kerollmops 7y ago
MeiliSearch is "zero-config" compared to ElasticSearch in terms of setup to make it work for end-user instant and relevant search engine. Our engine follows the Algolia engine in terms of typo-tolerance, relevancy, and speed. Here is a little comparison to enlighten your questions: https://docs.meilisearch.com/resources/comparison_to_alternatives.html#meilisearch-vs-elasticsearch https://docs.meilisearch.com/resources/comparison_to_alterna....
- heipei 7y ago
  Thanks, hadn't seen that, that makes a lot more sense. I agree that ElasticSearch is definitely not "zero-config" when it comes to building certain bespoke applications on it that go beyond simple filtering or query-relevance document search.
- ksec 7y ago
  May be adding Vespa [1] to comparison? [1] https://vespa.ai https://vespa.ai
winrid 7y ago
I've had issues scaling writes to it. You can get around it, but maybe this would be better in a high write environment.
rodgerd 7y ago
> I honestly don't understand people claiming ElasticSearch is hard to operate, especially not at small scales. The problem is that ES is deceptively simple to operate. As millions of people who have found things like their medical records shared with the world can attest.
dijit 7y ago
Elasticsearch is easier than mongo in some ways and harder in others. I run a few 10TiB ES clusters (which, is not much to be fair) but infrequently find that I have to reindex or reshard the cluster because I can’t just add another node. There’s something to be said for understanding the index rotation too, and access patterns. It’s easy to make an ES cluster, it’s difficult to maintain one, it’s nearly impossible to debug one. - if you consider that “it’s slow” is what you have to debug.
- jniedrauer 7y ago
  > if you consider that “it’s slow” is what you have to debug. This is exactly it. This is a problem you encounter with every database engine, but in most of them you can quickly find the bottleneck and fix it. With elasticsearch... it's a frustrating and expensive game of trial and error.
- kstrauser 7y ago
  That's approximately how large our clusters are. Fortunately, ours are read-only, so our admin story is: - Hey, a node died! - Run Terraform to stand up a whole new cluster and restore it from a snapshot. - Update the app to point at the new cluster. - Run Terraform to delete the old cluster. I'm pretty happy with this arrangement.
- heipei 7y ago
  Would be interested to hear why you can't add another node in some cases.
  - dijit 7y ago
    If you have 5 shards on an index, adding a 6th node isn’t going to improve performance. (Contrived example for the sake of illustrating the point)
- antpls 7y ago
  > I run a few 10TiB ES clusters For information, what does "10TiB" refer to in this context? Is it the size of what ES takes in RAM, or the size of ES' index, or is it the total size of the corpus that ES must index? Or corpus size + index ?
jniedrauer 7y ago
I've been bitten by elasticsearch twice in my career, and I've seen others bitten by it as well. Once you put it in production, you can't just run it from docker on your workstation. You have to set up a cluster with enough capacity for whatever load you're going to throw at it, gracefully handle failures, updates, scaling up as load increases, etc. There are so many switches and dials to tune, and unless you really learn it in depth, you won't know which ones you need. It's difficult to even determine what hardware requirements you have. And it's a hard sell to tell your business guys "I think elasticsearch will work better if we give it more... CPU? Memory? Disk speed? I'm not really sure." and can't provide any concrete metrics to back that up. Another place where footguns abound is upgrading from one version to another, especially if you've got plugins installed. There are tricks that you have to learn the hard way. At this point, I think long and hard before reaching for a solution like elasticsearch. If I've got a DBA whose entire job it is to master the tech and wield it expertly, that's one thing. But if I'm part of an early stage startup, I just can't justify the lost time and potential for catastrophe.
- cormacrelf 7y ago
  I once wasted a whole day trying to get two instances up on GKE. Permissions problems, about ten configs for the JVM alone, many more for ElasticSearch. You would fix one, restart, wait ten minutes, browse 50 pages of logs, google for half an hour, add a config, and goto 1. Never got it going in the end.
- natefox 7y ago
  > Once you put it in production, you can't just run it from docker on your workstation. But that's true for any data store. This isn't any different. Nor is an RDBMS. They all need HA/replication. And that is rarely trivial. Honestly, I think this is why managed/hosted solutions (AWS RDS for example) are so popular - they remove a large part of the complexity for you.
  - hashhar 7y ago
    Not all data stores. You can go quite far with an out of the box Redis instance or even PostgreSQL. No fiddling needed unless you are in triple digit QPS ranges.
atombender 7y ago
Elasticsearch is very memory-intensive, and it's difficult to know exactly how much memory it will actually use, so you just have to throw a lot of RAM at it to avoid OOMing, then monitor it carefully, and hope your query concurrency won't accidentally blow the limits. Understanding why Elasticsearch is caving unpredictably under load is difficult, and GC pausing can be a significant performance sink.
fareesh 7y ago
We built a "Yelp for Colleges" product several years ago. The product needed a unified search where students could search for either a course or a college or a question from the forums with typeahead / autocomplete to get them to where they wanted to go quickly, with support for misspellings. In all there were about 50k documents, and we mostly cared about the title field. Elasticsearch would randomly bloat up to occupy a huge amount of RAM. Restarting it would make it work for a few days. It would also occasionally crash. We got rid of it and went with some levenshtein distance based database query I'd love to use it again sometime but the experience was not good, and Googling for information brought up all kinds of very complex use-cases shared by others
- Kerollmops 7y ago
  Go on and try MeiliSearch, 50k documents are easily handled by the engine and with not much RAM usage. It will take you something like 10 minutes to start and populate MeiliSearch, you will be able to test it just by going to the server HTTP url in no time!
- specialist 7y ago
  Neat. I implemented the student facing course catalog web interface for a single org. One of the funnests (most fun) parts was the heuristics in the query parser. Like patterns for recognizing course numbers and boosting those exact match results. Really helps you appreciate all the fit & finish that goes into proper search engines. This was the olden days, when we just used Lucene directly.

by heipei 7y ago

Kerollmops 7y ago

MeiliSearch is "zero-config" compared to ElasticSearch in terms of setup to make it work for end-user instant and relevant search engine. Our engine follows the Algolia engine in terms of typo-tolerance, relevancy, and speed. Here is a little comparison to enlighten your questions: https://docs.meilisearch.com/resources/comparison_to_alternatives.html#meilisearch-vs-elasticsearch https://docs.meilisearch.com/resources/comparison_to_alterna....

heipei 7y ago
Thanks, hadn't seen that, that makes a lot more sense. I agree that ElasticSearch is definitely not "zero-config" when it comes to building certain bespoke applications on it that go beyond simple filtering or query-relevance document search.
ksec 7y ago
May be adding Vespa [1] to comparison? [1] https://vespa.ai https://vespa.ai

winrid 7y ago

I've had issues scaling writes to it. You can get around it, but maybe this would be better in a high write environment.

rodgerd 7y ago

> I honestly don't understand people claiming ElasticSearch is hard to operate, especially not at small scales. The problem is that ES is deceptively simple to operate. As millions of people who have found things like their medical records shared with the world can attest.

dijit 7y ago

Elasticsearch is easier than mongo in some ways and harder in others. I run a few 10TiB ES clusters (which, is not much to be fair) but infrequently find that I have to reindex or reshard the cluster because I can’t just add another node. There’s something to be said for understanding the index rotation too, and access patterns. It’s easy to make an ES cluster, it’s difficult to maintain one, it’s nearly impossible to debug one. - if you consider that “it’s slow” is what you have to debug.

jniedrauer 7y ago
> if you consider that “it’s slow” is what you have to debug. This is exactly it. This is a problem you encounter with every database engine, but in most of them you can quickly find the bottleneck and fix it. With elasticsearch... it's a frustrating and expensive game of trial and error.
kstrauser 7y ago
That's approximately how large our clusters are. Fortunately, ours are read-only, so our admin story is: - Hey, a node died! - Run Terraform to stand up a whole new cluster and restore it from a snapshot. - Update the app to point at the new cluster. - Run Terraform to delete the old cluster. I'm pretty happy with this arrangement.
heipei 7y ago
Would be interested to hear why you can't add another node in some cases.
- dijit 7y ago
  If you have 5 shards on an index, adding a 6th node isn’t going to improve performance. (Contrived example for the sake of illustrating the point)
antpls 7y ago
> I run a few 10TiB ES clusters For information, what does "10TiB" refer to in this context? Is it the size of what ES takes in RAM, or the size of ES' index, or is it the total size of the corpus that ES must index? Or corpus size + index ?

jniedrauer 7y ago

I've been bitten by elasticsearch twice in my career, and I've seen others bitten by it as well. Once you put it in production, you can't just run it from docker on your workstation. You have to set up a cluster with enough capacity for whatever load you're going to throw at it, gracefully handle failures, updates, scaling up as load increases, etc. There are so many switches and dials to tune, and unless you really learn it in depth, you won't know which ones you need. It's difficult to even determine what hardware requirements you have. And it's a hard sell to tell your business guys "I think elasticsearch will work better if we give it more... CPU? Memory? Disk speed? I'm not really sure." and can't provide any concrete metrics to back that up. Another place where footguns abound is upgrading from one version to another, especially if you've got plugins installed. There are tricks that you have to learn the hard way. At this point, I think long and hard before reaching for a solution like elasticsearch. If I've got a DBA whose entire job it is to master the tech and wield it expertly, that's one thing. But if I'm part of an early stage startup, I just can't justify the lost time and potential for catastrophe.

cormacrelf 7y ago
I once wasted a whole day trying to get two instances up on GKE. Permissions problems, about ten configs for the JVM alone, many more for ElasticSearch. You would fix one, restart, wait ten minutes, browse 50 pages of logs, google for half an hour, add a config, and goto 1. Never got it going in the end.
natefox 7y ago
> Once you put it in production, you can't just run it from docker on your workstation. But that's true for any data store. This isn't any different. Nor is an RDBMS. They all need HA/replication. And that is rarely trivial. Honestly, I think this is why managed/hosted solutions (AWS RDS for example) are so popular - they remove a large part of the complexity for you.
- hashhar 7y ago
  Not all data stores. You can go quite far with an out of the box Redis instance or even PostgreSQL. No fiddling needed unless you are in triple digit QPS ranges.

atombender 7y ago

Elasticsearch is very memory-intensive, and it's difficult to know exactly how much memory it will actually use, so you just have to throw a lot of RAM at it to avoid OOMing, then monitor it carefully, and hope your query concurrency won't accidentally blow the limits. Understanding why Elasticsearch is caving unpredictably under load is difficult, and GC pausing can be a significant performance sink.

fareesh 7y ago

We built a "Yelp for Colleges" product several years ago. The product needed a unified search where students could search for either a course or a college or a question from the forums with typeahead / autocomplete to get them to where they wanted to go quickly, with support for misspellings. In all there were about 50k documents, and we mostly cared about the title field. Elasticsearch would randomly bloat up to occupy a huge amount of RAM. Restarting it would make it work for a few days. It would also occasionally crash. We got rid of it and went with some levenshtein distance based database query I'd love to use it again sometime but the experience was not good, and Googling for information brought up all kinds of very complex use-cases shared by others

Kerollmops 7y ago
Go on and try MeiliSearch, 50k documents are easily handled by the engine and with not much RAM usage. It will take you something like 10 minutes to start and populate MeiliSearch, you will be able to test it just by going to the server HTTP url in no time!
specialist 7y ago
Neat. I implemented the student facing course catalog web interface for a single org. One of the funnests (most fun) parts was the heuristics in the query parser. Like patterns for recognizing course numbers and boosting those exact match results. Really helps you appreciate all the fit & finish that goes into proper search engines. This was the olden days, when we just used Lucene directly.