3 ms·

Interesting stuff! I've actually been working on the same idea recently, starting with reading about anomaly detection. In particular, this survey: http://www-u

Interesting stuff! I've actually been working on the same idea recently, starting with reading about anomaly detection. In particular, this survey: http://www-users.cs.umn.edu/~kumar/papers/anomaly-survey.php http://www-users.cs.umn.edu/~kumar/papers/anomaly-survey.php I would like to know more about the performance of Skyline in practice: - what are the accuracy and recall like? - what is CPU consumption like? Regarding the latter, I had a quick look at the implemented algorithms and them seemed very inefficient. Basically recomputing over the entire series at every change. I think with a bit of work most of the algorithms could be reimplemented in an incremental way. I also wouldn't use Python for something that is going to be CPU bound. (I await the "We rewrote in Go and it's 10x faster!" blog post ;-)

aba_sababa 13y ago
Author here. Accuracy is okay - we err on the side of noise, but it does routinely pick up anomalies. It doesn't currently account for seasonal trends, though. We aim for 100% CPU consumption. Analyzing is very CPU intensive process, and there are two parts in particular that are expensive: decoding the Redis string from MessagePack to Python, and running the algorithms. As for the algorithm inefficiencies, pull requests encouraged :) Rewriting it in Go is a plan for a rainy weekend :) The problem with Go is that it doesn't have as great statistics support as Python does.
- noelwelsh 13y ago
  Thanks for replying! A little while ago I watched part of your Bacon Conf talk (http://devslovebacon.com/conferences/bacon-2013/talks/bring-the-noise-continuously-deploying-under-a-hailstorm-of-metrics http://devslovebacon.com/conferences/bacon-2013/talks/bring-...) and read the slides. There were a few things I thought were a bit odd about the architecture, as I recall it. IIRC you poll Graphite for metrics. Why not push them from StatsD directly into Skyline? This would probably be more efficient. If you used incremental / online / streaming algorithms you'll have a compact summary at each time step, so you can throw away the raw data. 250K metrics would fit in memory quite easily (we're just talking approximately a number and a string each, right?) and you have 4000+ cycles per second to process them, which should be sufficient. Python lack of good threading would possibly be a problem. I would use the JVM (Scala in my case). Apache Commons Math is pretty good (http://commons.apache.org/proper/commons-math/ http://commons.apache.org/proper/commons-math/). Java's verbose interfaces are a bit annoying, but the JVM is damn efficient, and you can wrap the crap is something more aesthetic. It's a solid choice no matter what the hipsters say. ;-)
  - aba_sababa 13y ago
    Ah! We could, but StatsD provides support for more complex metrics like aggregated sums over time. Not something that lends itself easily to a discrete datapoint. It is. But we use multiprocessing, which is basically the same API. Still, you can't beat the awesome Python stats libraries: Numpy, SciPy, Statsmodels, Pandas..
- boothead 13y ago
  If you're analyzing streams of data Haskell would be a good choice.* *assumes that you've already put in the requisite couple of years in monastic contemplation to learn Haskell :-)

aba_sababa 13y ago

Author here. Accuracy is okay - we err on the side of noise, but it does routinely pick up anomalies. It doesn't currently account for seasonal trends, though. We aim for 100% CPU consumption. Analyzing is very CPU intensive process, and there are two parts in particular that are expensive: decoding the Redis string from MessagePack to Python, and running the algorithms. As for the algorithm inefficiencies, pull requests encouraged :) Rewriting it in Go is a plan for a rainy weekend :) The problem with Go is that it doesn't have as great statistics support as Python does.

noelwelsh 13y ago
Thanks for replying! A little while ago I watched part of your Bacon Conf talk (http://devslovebacon.com/conferences/bacon-2013/talks/bring-the-noise-continuously-deploying-under-a-hailstorm-of-metrics http://devslovebacon.com/conferences/bacon-2013/talks/bring-...) and read the slides. There were a few things I thought were a bit odd about the architecture, as I recall it. IIRC you poll Graphite for metrics. Why not push them from StatsD directly into Skyline? This would probably be more efficient. If you used incremental / online / streaming algorithms you'll have a compact summary at each time step, so you can throw away the raw data. 250K metrics would fit in memory quite easily (we're just talking approximately a number and a string each, right?) and you have 4000+ cycles per second to process them, which should be sufficient. Python lack of good threading would possibly be a problem. I would use the JVM (Scala in my case). Apache Commons Math is pretty good (http://commons.apache.org/proper/commons-math/ http://commons.apache.org/proper/commons-math/). Java's verbose interfaces are a bit annoying, but the JVM is damn efficient, and you can wrap the crap is something more aesthetic. It's a solid choice no matter what the hipsters say. ;-)
- aba_sababa 13y ago
  Ah! We could, but StatsD provides support for more complex metrics like aggregated sums over time. Not something that lends itself easily to a discrete datapoint. It is. But we use multiprocessing, which is basically the same API. Still, you can't beat the awesome Python stats libraries: Numpy, SciPy, Statsmodels, Pandas..
boothead 13y ago
If you're analyzing streams of data Haskell would be a good choice.* *assumes that you've already put in the requisite couple of years in monastic contemplation to learn Haskell :-)