5 ms·
Simple Alerting for the ELK Stack
- be_erik 10y agoIs anyone running this in production and care to share their experiences? This seems to be a pretty good replacement for Splunk's alerting mechanisms in the ELK stack, which has always been one of the parts I miss the most.
- ApsOps 10y agoI have been running this. It works well. I also wrote some custom code to send the stack-traces, if we get any, to a slack channel.
- be_erik 10y agoThat sounds fantastic. Do you have an example of how you extracted the trace? Was it simple to extend ElastAlert or did you have to fork it to add that kind of option?
- ApsOps 10y agoThe trace is basically extracted using Logstash's multiline filter. It adds a tag "stacktrace" to the log entry. I just made a few changes in SlackAlerter class to customize for my use-case. I'd try to clean-up and send a PR soon.
- sciurus 10y agoFWIW the company behind Elasticsearch is happy to sell you a solution for that: https://www.elastic.co/products/watcher https://www.elastic.co/products/watcher
- tzakrajs 10y agoI am using it right now in production. It's worst trait is that it is not clustered and does not share state about what alerts have been sent already. This means you can have no more than one agent running without having duplication of alerts. This also complicates high-availability for obvious reasons.
- tzakrajs 10y ago(Correction/Public Shaming: Its)
- yahyaheee 10y agoMy biggest issue with this was that it is purely event driven. So what if your trying to alert on no event? They had no answer for that
- bogomipz 10y agoRight. This is something you can get quite easily using Riemann.
- jonaf 10y agoHow does this compare with the log stash alerting plugin that's available? Does it work without log stash (i.e., just elastic search)?
- be_erik 10y agoIt works against elasticsearch and is somewhat agnostic as to how those documents got there in the first place. I've been playing with it as part of an EFK (fluentd instead of logstash) install.
- snorremd 10y agoDo you have a link to the logstash alerting plugin? I'm not familiar with it and I'm interested in alert tools for the elk (and beats) stack.
- bogomipz 10y agoAre you referring to watcher? That's a paid product I believe no?
- chrissnell 10y agoWe're using this. We're not running the full ELK stack; rather, we've replaced Logstash with fluentd. Our devs have two choices for acting on their logs: they can use elastalert (running in Kubernetes) to alert on events that are recorded in Elasticsearch. We've provided a sample elastalert template that can be easily customized to a developers needs. They can also deploy a custom fluentd parser/transformer in Kubernetes. To make this work, they apply a special label in their Kubernetes replication controller that specifies the name of the custom fluentd parser service. The primary fluentd service pulls the logs from Docker and when it detects this label for a particular log entry, it routes that entry to the custom parser service. This allows us to have a standard log pipeline that works out-of-the-box for most projects but also self-serve custom parsing for the apps that need it.
- sciurus 10y agoStack Exchange's monitoring system, Bosun (https://bosun.org/ https://bosun.org/), also has support for alerting on the results on Elasticsearch queries.
- otterley 10y agoIt sounds like they're using a search engine as a substitute for a proper CEP/ESP like Apache Flink. Was the latter considered? I'm rather curious as to why they chose a search engine instead.
- twic 10y agoELK is a very standard stack for collecting and disseminating logs. It's a fairly natural, and i think sensible, step to build a pattern-based alerting tool as a bag on the side of ELK. If you were going to do it more properly, the first choice would probably be Riemann, which has already seen some adoption in this role. There are already plugins on both sides to forward events to the other: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-riemann.html https://www.elastic.co/guide/en/logstash/current/plugins-out... http://riemann.io/api/riemann.logstash.html http://riemann.io/api/riemann.logstash.html
- otterley 10y agoWith all respect to the very smart author of Riemann, as a practical matter, its use of Clojure is simply not a hurdle most SRE types are going to overcome to use it. ElasticSearch is a great search engine. It's not a CEP or SEP. I don't mean to discourage its use, but for this purpose it's not a good fit. It does make sense to tee incoming logs to both as they suit different purposes, but I'd delegate pattern analysis to the latter.
- bogomipz 10y agoSEP? I know CEP. Riemann is great but I think you are right, there's a lot of people who aren't willing to pick up some clojure for whatever reason.
- zenlikethat 10y agoUsing Clojure, especially to slap together Riemann configs, is really not an insurmountable obstacle, especially when something as useful as Riemann is on the line. The more the self-defeating attitude that "Clojure's too hard" propagates the less likely people will be to even try. Most SRE's are pretty smart, I believe in them.
- yahyaheee 10y agoIt's a nice product, just be aware that it is purely event driven. So if you are trying to alert on no event then things get messy
- Liuser 10y agoUnless I'm misunderstanding your comment, isn't that a flatline type of alert, which is supported by Elastalert?
- yahyaheee 10y agoNo that is just if a value drops off, but that rule will never fire unless an event occurs in elasticsearch matching that query. I found this out the hard way