3 ms·

In my experience, the care and feeding that goes into an Elastic Search cluster feels like it's often substantially higher than that involved in the primary dat

In my experience, the care and feeding that goes into an Elastic Search cluster feels like it's often substantially higher than that involved in the primary data store, which has always struck me as a little odd (particularly in cases where the primary data store is an RDBMS). I'd be very happy to use simpler more bulletproof solutions with a subset of ES's features for different use cases.

dewey 1y ago
To add another data point: After working with ES for the past 10 years in production I have to say that ES is never giving us any headaches. We've had issues with ScyllaDB, Redis etc. but ES is just chugging along and just works. The one issue I remember is: On ES 5 we once had an issue early on where it regularly went down, turns out that some _very long_ input was being passed into the search by some scraper and killed the cluster.
- everfrustrated 1y ago
  How big is the team that looks after it?
  - dewey 1y ago
    Nobody is actively looking after it. Good alerting + monitoring and if there's an alert like a node going down because of some Kubernetes node shuffling or a version upgrade that has to be performed one of our few infra people will do that. It's really not something that needs much attention in my experience.
- itpragmatik 1y ago
  how many clusters, how many indexes and how many documents per index? do you use self hosted es or aws managed opensearch?
  - dewey 1y ago
    12 nodes, 200 million documents / node, very high number of searches and indexing operations. Self-hosted ES on GCP managed Kubernetes.
  - binarymax 1y ago
    Lots of other options here if you don't like managing. You can use Elastic cloud, Bonsai.io, and others
    - lisbbb 1y ago
      A lot of places can't put their data just anywhere.
      - chatmasta 1y ago
        And they can pay the vendors for "bring your own cloud" or similar. If data sovereignty is important to them, then they can probably afford it. And if cost is an issue, then they wouldn't be looking at hosted solutions in the first place.
      - dewey 1y ago
        They manage it in your GCP project, so you can also make use of your commitments etc.
- heipei 1y ago
  I agree, and I don't get where the claims that ES is hard to operate originate from. Yeah, if you allow arbitrary aggregations that exceed the heap space, or if you allow expensive queries that effectively iterate over everything you're gonna have a bad time. But apart from those, as long as you understand your data model, your searches and how data is indexed, ES is absolutely rock-solid, scales and performs like a beast. We run a 35-node cluster with ~ 240TB of disk, 4.5TB of RAM, and about 100TB of documents and are able to serve hundreds of queries. The whole thing does not require any maintenance apart from replacing nodes that failed from unrelated causes (hardware, hosting). Version upgrades are smooth as well. The only bigger issue we had was when we initially added 10 nodes to double the initial capacity of the cluster. Performance tanked as a result, and it took us about half a day until we finally figured out that the new nodes were using dmraid (Linux RAID0) and as a result the block devices had a really high default read-ahead value (8192) compared to the existing nodes, which resulted in heavy read amplification. The ES manual specifically documents this, but since we hadn't run into this issue ourselves it took us a while to realise what was at fault.
- lisbbb 1y ago
  The thing I like about ES: When the business comes around and adds new requirements out of nowhere, the answer is always: "Yup, we can do it!" Unlike other tools such as Cassandra that force a data design from the get go and make it expensive to change later on.
unsuitable 1y ago
In my experience Elastic Search lacks fundamental tooling, like a CLI that copies data between nodes.
- WASDx 1y ago
  The `/_cluster/reroute` endpoint lets you do that with a curl. We have aliases for common operations so I've never felt that I lack a CLI. I'm happy with Elasticsearch overall having a few years of experience.
nchmy 1y ago
Check out manticoresearch - it's older than Lucene (which elasticsearch is built on), faster and simpler.
- mdaniel 1y ago
  GPLv3 https://github.com/manticoresoftware/manticoresearch/blob/13.6.7/LICENSE https://github.com/manticoresoftware/manticoresearch/blob/13...
  - nchmy 1y ago
    Are you running customized, or even embedded, versions of your search engine? Highly doubtful. So, GPL is irrelevant. Just run it and interface with it via api.
lisbbb 1y ago
I'm interested in this detail because a few years back I was involved in a major big data project at a health insurance company and I cooked up a solution that involved ElasticSearch that was workable only to be shot down--it was political, but they had to do it with Kafka, full stop. The problem was, at that time, Kafka wasn't very mature and it wasn't a good solution for the problem, regardless. So our ES version got shelved.

dewey 1y ago

To add another data point: After working with ES for the past 10 years in production I have to say that ES is never giving us any headaches. We've had issues with ScyllaDB, Redis etc. but ES is just chugging along and just works. The one issue I remember is: On ES 5 we once had an issue early on where it regularly went down, turns out that some _very long_ input was being passed into the search by some scraper and killed the cluster.

everfrustrated 1y ago
How big is the team that looks after it?
- dewey 1y ago
  Nobody is actively looking after it. Good alerting + monitoring and if there's an alert like a node going down because of some Kubernetes node shuffling or a version upgrade that has to be performed one of our few infra people will do that. It's really not something that needs much attention in my experience.
itpragmatik 1y ago
how many clusters, how many indexes and how many documents per index? do you use self hosted es or aws managed opensearch?
- dewey 1y ago
  12 nodes, 200 million documents / node, very high number of searches and indexing operations. Self-hosted ES on GCP managed Kubernetes.
- binarymax 1y ago
  Lots of other options here if you don't like managing. You can use Elastic cloud, Bonsai.io, and others
  - lisbbb 1y ago
    A lot of places can't put their data just anywhere.
    - chatmasta 1y ago
      And they can pay the vendors for "bring your own cloud" or similar. If data sovereignty is important to them, then they can probably afford it. And if cost is an issue, then they wouldn't be looking at hosted solutions in the first place.
    - dewey 1y ago
      They manage it in your GCP project, so you can also make use of your commitments etc.
heipei 1y ago
I agree, and I don't get where the claims that ES is hard to operate originate from. Yeah, if you allow arbitrary aggregations that exceed the heap space, or if you allow expensive queries that effectively iterate over everything you're gonna have a bad time. But apart from those, as long as you understand your data model, your searches and how data is indexed, ES is absolutely rock-solid, scales and performs like a beast. We run a 35-node cluster with ~ 240TB of disk, 4.5TB of RAM, and about 100TB of documents and are able to serve hundreds of queries. The whole thing does not require any maintenance apart from replacing nodes that failed from unrelated causes (hardware, hosting). Version upgrades are smooth as well. The only bigger issue we had was when we initially added 10 nodes to double the initial capacity of the cluster. Performance tanked as a result, and it took us about half a day until we finally figured out that the new nodes were using dmraid (Linux RAID0) and as a result the block devices had a really high default read-ahead value (8192) compared to the existing nodes, which resulted in heavy read amplification. The ES manual specifically documents this, but since we hadn't run into this issue ourselves it took us a while to realise what was at fault.
lisbbb 1y ago
The thing I like about ES: When the business comes around and adds new requirements out of nowhere, the answer is always: "Yup, we can do it!" Unlike other tools such as Cassandra that force a data design from the get go and make it expensive to change later on.

unsuitable 1y ago

In my experience Elastic Search lacks fundamental tooling, like a CLI that copies data between nodes.

WASDx 1y ago
The `/_cluster/reroute` endpoint lets you do that with a curl. We have aliases for common operations so I've never felt that I lack a CLI. I'm happy with Elasticsearch overall having a few years of experience.

nchmy 1y ago

Check out manticoresearch - it's older than Lucene (which elasticsearch is built on), faster and simpler.

mdaniel 1y ago
GPLv3 https://github.com/manticoresoftware/manticoresearch/blob/13.6.7/LICENSE https://github.com/manticoresoftware/manticoresearch/blob/13...
- nchmy 1y ago
  Are you running customized, or even embedded, versions of your search engine? Highly doubtful. So, GPL is irrelevant. Just run it and interface with it via api.

lisbbb 1y ago

I'm interested in this detail because a few years back I was involved in a major big data project at a health insurance company and I cooked up a solution that involved ElasticSearch that was workable only to be shot down--it was political, but they had to do it with Kafka, full stop. The problem was, at that time, Kafka wasn't very mature and it wasn't a good solution for the problem, regardless. So our ES version got shelved.