3 ms·

I read all releases log since 2.3 and its always been enlightening, and I start using those features. Elasticsearch is my favorite tool and it has been very use

by CSDude 6y ago

I read all releases log since 2.3 and its always been enlightening, and I start using those features. Elasticsearch is my favorite tool and it has been very useful over the years, just wanted to share my thanks. I just hope they can survive in the cloud defying broken AWS Elasticsearch abomination.

vosper 6y ago
> cloud defying broken AWS Elasticsearch abomination Heh, I'm pretty sure our attempt to use AWS-managed ES went badly, and we quickly gave up on it and stuck with self-hosted. And this is at a company that's pretty keen to use AWS-managed services when we can.
- WrtCdEvrydy 6y ago
  Well... this ought to be a fun year (just got told to go to AWS-managed)
  - chucky_z 6y ago
    Before going any further, take a _really close look_ at https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-supported-es-operations.html https://docs.aws.amazon.com/elasticsearch-service/latest/dev... Make darn well sure you aren't using any unsupported operations. In their favor, if I were to revisit them I could use it now; however in the past they didn't support templates and a few other things which turned out to be major blockers.
  - 0x202020 6y ago
    In typically AWS managed fashion: If you want to get it set up and running most of the way without tuning a bunch of stuff it’s pretty good. It also has significant cost overhead compared to self-managed and limited instance types. My team now is working on migrating a 2.3 cluster from AWS ES to a 7.8 self-managed cluster (probably 7.9 now I guess). We do lots of indexing throughout the day and the i3 instances available are pretty weak, just switching to r5d/r6gd instances greatly reduced our indexing and search latency
  - oars 6y ago
    I highly recommend you show your lead/management these two articles, then ask if they want to proceed: https://spun.io/2019/10/10/aws-elasticsearch-a-fundamentally-flawed-offering/ https://spun.io/2019/10/10/aws-elasticsearch-a-fundamentally... https://code972.com/blog/2017/12/why-you-shouldnt-use-aws-elasticsearch-service-111 https://code972.com/blog/2017/12/why-you-shouldnt-use-aws-el...
    - ollyculverhouse 6y ago
      From a few years ago? We have used managed ES for a while now and see no issues however I realise its different for everyone.
  - ak47surve 6y ago
    We moved in the early days. Today, we are fairly advanced with regards to our understanding of the internals of Elasticsearch - data nodes, master nodes, indexing, shards, replication strategy and query performance. In hindsight, it would be fair to say that we have been extremely happy with our choice given our limited understanding of running a production scale Elasticsearch cluster 6 months back. With Amazon Elasticsearch we were able to start small, move fast, learn on the go, and fine-tune our cluster for production workloads without having to understand every single aspect of managing and scaling a production scale Elasticsearch cluster. More details here - https://www.akshaysurve.com/2018/01/what-we-like-about-amazons-elasticsearch-on-aws/ https://www.akshaysurve.com/2018/01/what-we-like-about-amazo...
- ClikeX 6y ago
  We've been looking at doing managed because we use ES for some projects. But at they're always running on the same server as the application. And we have nobody with the knowledge or ambition to maintain it. But reading these comments I get wary of doing it that way.
  - vosper 6y ago
    I’m sure it depends what you’re doing - our cluster is medium-largeish, I’d guess (wish there was more info on the clusters that are out there!). Several hundred big EC2 instances, all NVMe, billions of documents, indexes in the terabytes, high update rate in addition to inserts. It requires fairly active management and optimization. We might just be outside of the sweet spot for the AWS offering.
    - ClikeX 6y ago
      If that is medium-largish, then I'd probably be more than fine within the use-case of my current customers.
    - ignoramous 6y ago
      > Several hundred big EC2 instances, all NVMe, billions of documents, indexes in the terabytes, high update rate in addition to inserts. It requires fairly active management and optimization. AWS Elasticsearch may not be for power users but they did announce Petabyte and Ultrawarm for über scale clusters in run up to last year's re:Invent: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/ultrawarm.html https://docs.aws.amazon.com/elasticsearch-service/latest/dev... and https://aws.amazon.com/blogs/database/run-a-petabyte-scale-cluster-in-amazon-elasticsearch-service/ https://aws.amazon.com/blogs/database/run-a-petabyte-scale-c...
gchamonlive 6y ago
Have you tried elastic cloud? I believe they offer the full featured xpack product and manage all your instances for you.

by CSDude 6y ago

vosper 6y ago

> cloud defying broken AWS Elasticsearch abomination Heh, I'm pretty sure our attempt to use AWS-managed ES went badly, and we quickly gave up on it and stuck with self-hosted. And this is at a company that's pretty keen to use AWS-managed services when we can.

WrtCdEvrydy 6y ago
Well... this ought to be a fun year (just got told to go to AWS-managed)
- chucky_z 6y ago
  Before going any further, take a _really close look_ at https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-supported-es-operations.html https://docs.aws.amazon.com/elasticsearch-service/latest/dev... Make darn well sure you aren't using any unsupported operations. In their favor, if I were to revisit them I could use it now; however in the past they didn't support templates and a few other things which turned out to be major blockers.
- 0x202020 6y ago
  In typically AWS managed fashion: If you want to get it set up and running most of the way without tuning a bunch of stuff it’s pretty good. It also has significant cost overhead compared to self-managed and limited instance types. My team now is working on migrating a 2.3 cluster from AWS ES to a 7.8 self-managed cluster (probably 7.9 now I guess). We do lots of indexing throughout the day and the i3 instances available are pretty weak, just switching to r5d/r6gd instances greatly reduced our indexing and search latency
- oars 6y ago
  I highly recommend you show your lead/management these two articles, then ask if they want to proceed: https://spun.io/2019/10/10/aws-elasticsearch-a-fundamentally-flawed-offering/ https://spun.io/2019/10/10/aws-elasticsearch-a-fundamentally... https://code972.com/blog/2017/12/why-you-shouldnt-use-aws-elasticsearch-service-111 https://code972.com/blog/2017/12/why-you-shouldnt-use-aws-el...
  - ollyculverhouse 6y ago
    From a few years ago? We have used managed ES for a while now and see no issues however I realise its different for everyone.
- ak47surve 6y ago
  We moved in the early days. Today, we are fairly advanced with regards to our understanding of the internals of Elasticsearch - data nodes, master nodes, indexing, shards, replication strategy and query performance. In hindsight, it would be fair to say that we have been extremely happy with our choice given our limited understanding of running a production scale Elasticsearch cluster 6 months back. With Amazon Elasticsearch we were able to start small, move fast, learn on the go, and fine-tune our cluster for production workloads without having to understand every single aspect of managing and scaling a production scale Elasticsearch cluster. More details here - https://www.akshaysurve.com/2018/01/what-we-like-about-amazons-elasticsearch-on-aws/ https://www.akshaysurve.com/2018/01/what-we-like-about-amazo...
ClikeX 6y ago
We've been looking at doing managed because we use ES for some projects. But at they're always running on the same server as the application. And we have nobody with the knowledge or ambition to maintain it. But reading these comments I get wary of doing it that way.
- vosper 6y ago
  I’m sure it depends what you’re doing - our cluster is medium-largeish, I’d guess (wish there was more info on the clusters that are out there!). Several hundred big EC2 instances, all NVMe, billions of documents, indexes in the terabytes, high update rate in addition to inserts. It requires fairly active management and optimization. We might just be outside of the sweet spot for the AWS offering.
  - ClikeX 6y ago
    If that is medium-largish, then I'd probably be more than fine within the use-case of my current customers.
  - ignoramous 6y ago
    > Several hundred big EC2 instances, all NVMe, billions of documents, indexes in the terabytes, high update rate in addition to inserts. It requires fairly active management and optimization. AWS Elasticsearch may not be for power users but they did announce Petabyte and Ultrawarm for über scale clusters in run up to last year's re:Invent: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/ultrawarm.html https://docs.aws.amazon.com/elasticsearch-service/latest/dev... and https://aws.amazon.com/blogs/database/run-a-petabyte-scale-cluster-in-amazon-elasticsearch-service/ https://aws.amazon.com/blogs/database/run-a-petabyte-scale-c...

gchamonlive 6y ago

Have you tried elastic cloud? I believe they offer the full featured xpack product and manage all your instances for you.