My experience with ElasticSearch is that it is amazing for log aggregation until you reach a certain volume and then the operational cost (both $ and time/compl

My experience with ElasticSearch is that it is amazing for log aggregation until you reach a certain volume and then the operational cost (both $ and time/complexity) of running a large ES cluster starts to really bite hard. To take one recent(ish) example from my last job. We built a bunch of streaming pipelines to enrich log data during ingest so we could use the ES ML jobs to do some proactive alerting on our application logs. It was all really nice but then we had to set up a regular patching cycle for our ES cluster. Generally everything is ES is stored in an index so doing a blue/green deployment works pretty well. Except that the ML jobs have some sort of in-memory state that can't be migrated, so the ML jobs would get migrated but in a weird, half-working state. So we ended up having to manually delete and recreate the "half-working" ML jobs whenever we rolled out the new patched cluster. Having an "all in one" solution like ES is great until you hit a certain scale and the inherent complexity of such a system starts to really make itself apparent.