3 ms·

It's more a matter of configuring it right. I'd recommend trying out Elastic Cloud. It's a bit easier to deal with than Amazon's offering and much better suppor

It's more a matter of configuring it right. I'd recommend trying out Elastic Cloud. It's a bit easier to deal with than Amazon's offering and much better supported. AWS has always been a bit hands-off on that front. Their opensearch project does not seem to break that pattern so far. Also, with Elastic Cloud you get some access to useful features for logging (like life cycle management and data streams) that will help you scale the setup. Kibana in recent iterations has actually improved quite a bit. The version you are getting from Amazon is probably a bit bare bones in comparison. One nice thing with Elastic is that going with the defaults gets you some useful dashboards out of the box if you use e.g. file or docker beats for collecting logs.

nuker 5y ago
Thanks mate. I did tried Elastic before settling on AWS ES, it was slower somehow. As for Kibana features, im happy with barebones :) > you use e.g. file or docker beats for collecting logs. My setup is custom app reading from CW Logs. But can Elastic cluster scale automatically on indexing latency spikes, so my apps writes do not time out? If yes then how, please?
- jillesvangurp 5y ago
  Autoscaling is of course something they can do: https://www.elastic.co/guide/en/cloud/current/ec-autoscaling.html https://www.elastic.co/guide/en/cloud/current/ec-autoscaling... If you use life cycle management and data streams (which AWS doesn't have), you'd be able to control the sizes of your hot indices (i.e. the ones you write to). Basically keeping your hot indices small helps keeping things fast. If you have issues with app writes spiking, I'd use some queuing solution in between. Plenty of solutions for that. The rest is just a matter of configuring things right in terms of number of shards and setting up properly. Basically, you get what you pay for in the end.
  - nuker 5y ago
    > If you use life cycle management and data streams (which AWS doesn't have), you'd be able to control the sizes of your hot indices (i.e. the ones you write to) AWS has lifecycle stuff, a year ago. And index management is not cluster scaling, its a good advice, but not what I asked about.
  - nuker 5y ago
    To add. Your link to autoscaling describes storage scaling up, not the cluster scaling based on performance metrics.

nuker 5y ago

Thanks mate. I did tried Elastic before settling on AWS ES, it was slower somehow. As for Kibana features, im happy with barebones :) > you use e.g. file or docker beats for collecting logs. My setup is custom app reading from CW Logs. But can Elastic cluster scale automatically on indexing latency spikes, so my apps writes do not time out? If yes then how, please?

jillesvangurp 5y ago
Autoscaling is of course something they can do: https://www.elastic.co/guide/en/cloud/current/ec-autoscaling.html https://www.elastic.co/guide/en/cloud/current/ec-autoscaling... If you use life cycle management and data streams (which AWS doesn't have), you'd be able to control the sizes of your hot indices (i.e. the ones you write to). Basically keeping your hot indices small helps keeping things fast. If you have issues with app writes spiking, I'd use some queuing solution in between. Plenty of solutions for that. The rest is just a matter of configuring things right in terms of number of shards and setting up properly. Basically, you get what you pay for in the end.
- nuker 5y ago
  > If you use life cycle management and data streams (which AWS doesn't have), you'd be able to control the sizes of your hot indices (i.e. the ones you write to) AWS has lifecycle stuff, a year ago. And index management is not cluster scaling, its a good advice, but not what I asked about.
- nuker 5y ago
  To add. Your link to autoscaling describes storage scaling up, not the cluster scaling based on performance metrics.