3 ms·
I can't say I agree with this article. Elasticsearch is a bit of a Swiss army knife and definitely needs some setup and configuring. But if you set it up proper
by jillesvangurp 5y ago
I can't say I agree with this article. Elasticsearch is a bit of a Swiss army knife and definitely needs some setup and configuring. But if you set it up properly, it does logs extremely well.
And it does actually come with useful defaults and well integrated tools. For example, if you use the various Beats agents to push data to Elasticsearch, they come with a default schema and out of the box dashboards that they can create for you. These are actually useful. I don't know that many tools in this space that have out of the box dashboards pre-configured, with matching schemas, etc.
Its also not that hard to create your own schemas and dashboards. Dynamic mapping is a rookie mistake that you might get away with if you don't have a lot of structured data. The classic mistake here would be mapping hundreds/thousands of fields. Not a great idea when you have billions of documents. The per document overhead adds up pretty quickly (disk, memory, indexing speed, querying overhead, etc). Much better is to only map those fields that matter to you and disable dynamic mapping for everything else.
And then there are features like rollups, data streams, index lifecycle management, etc. that make a lot of sense if you are storing logs at large scale. But you have to of course know to use them. As these were x-pack features, you won't be getting that with Opensearch.
Now there are of course many other solutions. But frankly it's a bit of a messy field of half integrated tools and a high DIY degree of how to set this up. Log analytics is only as good as the dashboards you get with them. Usually that means Grafana these days. How the data is stored is only part of the problem.
Grafana is basically an old fork of Kibana (v3) that has had lots of love and attention since then. It's a bit less feature rich perhaps but very capable. Incidentally, they went down the same re-licensing path as Elasticsearch: https://www.infoq.com/news/2021/04/grafana-licence-agpl/ https://www.infoq.com/news/2021/04/grafana-licence-agpl/. So, you might want to think twice about putting that on a server unless you are completely fine having AGPLv3 licensed code close to where our own code is deployed (hint, if you are uncertain, you probably should not be doing that). LOKI is licensed similarly.
You can definitely gobble together some dashboards pretty easily using e.g. Grafana and any of its backends. But the out of the box experience for that is a lot of work figuring out what to use, how top operate it properly, etc. Elastic Cloud is expensive at scale but kind of plug and play on this front.
Probably the next best thing would be Opensearch. It's a fairly recent fork of Elasticsearch and Kibana. And unlike Elasticsearch/Kibana or Grafana/Loki, it remains under the Apache 2.0 license. Most of what you would need for a decent log analytics system is in there.
What you get by default from Google Cloud, AWS, etc. is pretty bare-bones in comparison in their cloud logging.
- thinkharderdev 5y agoMy experience with ElasticSearch is that it is amazing for log aggregation until you reach a certain volume and then the operational cost (both $ and time/complexity) of running a large ES cluster starts to really bite hard. To take one recent(ish) example from my last job. We built a bunch of streaming pipelines to enrich log data during ingest so we could use the ES ML jobs to do some proactive alerting on our application logs. It was all really nice but then we had to set up a regular patching cycle for our ES cluster. Generally everything is ES is stored in an index so doing a blue/green deployment works pretty well. Except that the ML jobs have some sort of in-memory state that can't be migrated, so the ML jobs would get migrated but in a weird, half-working state. So we ended up having to manually delete and recreate the "half-working" ML jobs whenever we rolled out the new patched cluster. Having an "all in one" solution like ES is great until you hit a certain scale and the inherent complexity of such a system starts to really make itself apparent.
- OldOneEye 5y agoThanks for sharing your experience with ML. We're experienced with managing a big scale elastic cluster, but never managed the ML jobs, so knowing about this limitation is definitely useful!
- jcims 5y agoAnyone that's used Splunk for any amount of time will feel like they are in a straightjacket with ES. Anyone stuck paying for Splunk will think they probably should be.
- dvdkon 5y agoAGPL is just GPL where usage over a network requires giving users the source code of the AGPL codebase. I'll take it over a custom licence where I have to worry if I need to release the source code of everything around it (SSPL) or if I'm providing it as a managed service (EL).