4 ms·
Another good package to look at is Etsy's Skyline - https://github.com/etsy/skyline https://github.com/etsy/skyline Their introduction of the Kale stack (which
by iandanforth 12y ago
Another good package to look at is Etsy's Skyline - https://github.com/etsy/skyline https://github.com/etsy/skyline
Their introduction of the Kale stack (which includes Skyline) is a great read - https://codeascraft.com/2013/06/11/introducing-kale/ https://codeascraft.com/2013/06/11/introducing-kale/
I spent a month or so evaluating anomaly detection systems and I can tell you a few things the twitter post fails to mention
1. You can get a long way with an ensemble of simple techniques. And it's always better than any single technique.
I wouldn't recommend trying to install skyline but re-implementing the ensemble of anomaly classifiers they have might take you a day or two and it will get you 90% of the way there.
2. The False Positive rate is probably your most important metric.
Detecting anomalies is good, but your ops team already has plenty of alerts to deal with. If you throw false positives at them from a new system they will hate you and ignore the new system. Most papers report ROC curve data as the metric for classification, that can be ok too.
3. Don't build something complex when a threshold will do.
If a point anomaly is obvious to a human, you absolutely should not build a complex system to detect it, just use thresholds. It's only when you want to detect anomalies before they cross a threshold that you should start on this kind of task. That leads me to
4. Almost all anomalies have a temporal component.
If your detector isn't ultimately looking at multiple sources of data and finding patterns that initially look like normal behavior (or odd behavior over time like a change in frequency) then its not adding as much value as it could. Slow trends, increased predictability, absent spikes which are still within threshold, those are the kind of anomalies your simple systems will miss and which will add a lot of value.
Ultimately anything that makes ops life easier and alerts them sooner to real problems is good. But in anomaly detection it is easy to fool yourself into thinking you need something complex to start out with, and when you've built a complex thing that it is "working" because it finds 95% of the obvious outliers.
- Eridrus 12y agoWhy would you not recommend installing skyline? I would like to have something that watches metrics for me, and we're already using graphite for stats.
- iandanforth 12y agoGetting the full package working was (several months ago) a painful process. It was much easier to just grab the algos.