5 ms·

Is there alternative to ES that scales nicely? I'm running ELK stack for logging using AWS Elasticsearch. Logs have unpredictable traffic volume and even overpr

by nuker 5y ago

Is there alternative to ES that scales nicely? I'm running ELK stack for logging using AWS Elasticsearch. Logs have unpredictable traffic volume and even overprovisioned ES cluster gets clogged sometimes. I wonder is there something more scalable than ES, and have nice GUI like Kibana?

shard972 5y ago
loki
jillesvangurp 5y ago
It's more a matter of configuring it right. I'd recommend trying out Elastic Cloud. It's a bit easier to deal with than Amazon's offering and much better supported. AWS has always been a bit hands-off on that front. Their opensearch project does not seem to break that pattern so far. Also, with Elastic Cloud you get some access to useful features for logging (like life cycle management and data streams) that will help you scale the setup. Kibana in recent iterations has actually improved quite a bit. The version you are getting from Amazon is probably a bit bare bones in comparison. One nice thing with Elastic is that going with the defaults gets you some useful dashboards out of the box if you use e.g. file or docker beats for collecting logs.
- nuker 5y ago
  Thanks mate. I did tried Elastic before settling on AWS ES, it was slower somehow. As for Kibana features, im happy with barebones :) > you use e.g. file or docker beats for collecting logs. My setup is custom app reading from CW Logs. But can Elastic cluster scale automatically on indexing latency spikes, so my apps writes do not time out? If yes then how, please?
  - jillesvangurp 5y ago
    Autoscaling is of course something they can do: https://www.elastic.co/guide/en/cloud/current/ec-autoscaling.html https://www.elastic.co/guide/en/cloud/current/ec-autoscaling... If you use life cycle management and data streams (which AWS doesn't have), you'd be able to control the sizes of your hot indices (i.e. the ones you write to). Basically keeping your hot indices small helps keeping things fast. If you have issues with app writes spiking, I'd use some queuing solution in between. Plenty of solutions for that. The rest is just a matter of configuring things right in terms of number of shards and setting up properly. Basically, you get what you pay for in the end.
    - nuker 5y ago
      > If you use life cycle management and data streams (which AWS doesn't have), you'd be able to control the sizes of your hot indices (i.e. the ones you write to) AWS has lifecycle stuff, a year ago. And index management is not cluster scaling, its a good advice, but not what I asked about.
    - nuker 5y ago
      To add. Your link to autoscaling describes storage scaling up, not the cluster scaling based on performance metrics.
hnarn 5y ago
I don't know what features AWS provides but in general terms logs benefit a lot from compression, so if I were to set up this on my own I'd probably want to try something like a VDO or ZFS backed storage system as well as compressed transfers (perhaps in batch if that's required).
- nuker 5y ago
  > something like a VDO or ZFS backed storage system And what text search engine?
  - hnarn 5y ago
    I have no idea, but I’m assuming that regardless of which one you use there will be disk level duplication that could be taken care of to alleviate uncontrolled growth somewhat.
sofixa 5y ago
Loki scales very nicely, and is for logs only, so much more lightweight than ES. (No full text search though, you have to define indexed labels from which to start your search when ingesting)

by nuker 5y ago

shard972 5y ago

loki

jillesvangurp 5y ago

It's more a matter of configuring it right. I'd recommend trying out Elastic Cloud. It's a bit easier to deal with than Amazon's offering and much better supported. AWS has always been a bit hands-off on that front. Their opensearch project does not seem to break that pattern so far. Also, with Elastic Cloud you get some access to useful features for logging (like life cycle management and data streams) that will help you scale the setup. Kibana in recent iterations has actually improved quite a bit. The version you are getting from Amazon is probably a bit bare bones in comparison. One nice thing with Elastic is that going with the defaults gets you some useful dashboards out of the box if you use e.g. file or docker beats for collecting logs.

nuker 5y ago
Thanks mate. I did tried Elastic before settling on AWS ES, it was slower somehow. As for Kibana features, im happy with barebones :) > you use e.g. file or docker beats for collecting logs. My setup is custom app reading from CW Logs. But can Elastic cluster scale automatically on indexing latency spikes, so my apps writes do not time out? If yes then how, please?
- jillesvangurp 5y ago
  Autoscaling is of course something they can do: https://www.elastic.co/guide/en/cloud/current/ec-autoscaling.html https://www.elastic.co/guide/en/cloud/current/ec-autoscaling... If you use life cycle management and data streams (which AWS doesn't have), you'd be able to control the sizes of your hot indices (i.e. the ones you write to). Basically keeping your hot indices small helps keeping things fast. If you have issues with app writes spiking, I'd use some queuing solution in between. Plenty of solutions for that. The rest is just a matter of configuring things right in terms of number of shards and setting up properly. Basically, you get what you pay for in the end.
  - nuker 5y ago
    > If you use life cycle management and data streams (which AWS doesn't have), you'd be able to control the sizes of your hot indices (i.e. the ones you write to) AWS has lifecycle stuff, a year ago. And index management is not cluster scaling, its a good advice, but not what I asked about.
  - nuker 5y ago
    To add. Your link to autoscaling describes storage scaling up, not the cluster scaling based on performance metrics.

hnarn 5y ago

I don't know what features AWS provides but in general terms logs benefit a lot from compression, so if I were to set up this on my own I'd probably want to try something like a VDO or ZFS backed storage system as well as compressed transfers (perhaps in batch if that's required).

nuker 5y ago
> something like a VDO or ZFS backed storage system And what text search engine?
- hnarn 5y ago
  I have no idea, but I’m assuming that regardless of which one you use there will be disk level duplication that could be taken care of to alleviate uncontrolled growth somewhat.

sofixa 5y ago

Loki scales very nicely, and is for logs only, so much more lightweight than ES. (No full text search though, you have to define indexed labels from which to start your search when ingesting)