3 ms·

On the other hand: At Google I worked with the Gmail performance team to decrease server downtime by detecting anomalous behavior before server failure. I desi

by shawn 9y ago

On the other hand: At Google I worked with the Gmail performance team to decrease server downtime by detecting anomalous behavior before server failure. I designed and implemented an original anomaly detection algorithm based on local outlier factors. Does anyone have any info about anything related to this? I want to write a program to detect anomalous behavior in my systems too. (Mostly so I can completely ignore the useful information, but still.)

adtac 9y ago
I can talk about my experience writing such a tool during my internship at a bank that rhymes with Oldman Bachs. Bear in mind, this wasn't my original project; since I had finished my first project, I was allowed to work on something else I find interesting. So it will probably not be very robust, but it worked. The tool would run in trading engines and monitor different metrics of performance. CPU usage, while allowed to burst, shouldn't be sustained at 100% for a long time. Typically, these machines are monsterous and have hundreds of gigs of memory; therefore, they shouldn't come anywhere near 100% memory usage either. There's a bunch of other proprietary internal metrics that I can't really explain (stuff like latency of a particular kind of response). Anyway, since I was most comfortable with writing the tool in Python (also, time was a constraint as I started working on the project towards the end of my internship), I wrote a modular tool with pluggable metrics. The reason why it wasn't a monolithic block: if the company decides to add a new metric they want to measure and follow, they should just have to write the logic necessary to retrieve that metric. The rest should be handled by my framework: anomaly detection, alerting relevant parties, and so on. So banged out the framework and had just enough time to write two modules: CPU usage and memory usage (probably the easiest ones). With a simple moving window average and standard deviation, you can identify outliers: I marked everything outside 2*sigma as an outlier as it's better to have false positives than false negatives (but not excessively many false positives as that would be counter-productive and make humans ignore reports). The alerting part was fairly straightforward as there was a built-in tool to do that. I ran a few tests with historical data and it was pretty good at detecting the anomalies that I would mark myself as a human. It could be better if someone smarter than could come up with a machine learning algo to do the same thing more accurately. (I didn't productionize it as I didn't have the time.)
iandanforth 9y ago
You might also be interested in this older project from etsy https://github.com/etsy/skyline https://github.com/etsy/skyline Has a fairly good mixture of models approach to anomaly detection even if it's not maintained.
jradd 9y ago
First thing that comes to mind is tracing system calls with strace (linux), truss (bsd), or Dtrace (recommended). A common probability theory such as Markov Chains could be used as a reference: Markov Chains http://cucis.ece.northwestern.edu/projects/DMS/publications/AnomalyDetection.pdf http://cucis.ece.northwestern.edu/projects/DMS/publications/... There is no limit of items this model could be applied to, but a few that come to mind; given the context might be; Load Generation or emulation at the filesystem, network or application level. iowait, vmstats, memory leaks (Why am I unable to account for N percent memory), Cache hit vs miss, Network stats, heuristics, buffer overflows, denial of service (listen/accept queue overflow), TCP RTT, NTP Jitter, Response Time, etc…
massaman_yams 9y ago
Netflix has some good posts on outlier / anomaly detection. https://medium.com/netflix-techblog/tracking-down-the-villains-outlier-detection-at-netflix-40360b31732 https://medium.com/netflix-techblog/tracking-down-the-villai... https://medium.com/netflix-techblog/rad-outlier-detection-on-big-data-d6b0494371cc https://medium.com/netflix-techblog/rad-outlier-detection-on...

by shawn 9y ago

adtac 9y ago

I can talk about my experience writing such a tool during my internship at a bank that rhymes with Oldman Bachs. Bear in mind, this wasn't my original project; since I had finished my first project, I was allowed to work on something else I find interesting. So it will probably not be very robust, but it worked. The tool would run in trading engines and monitor different metrics of performance. CPU usage, while allowed to burst, shouldn't be sustained at 100% for a long time. Typically, these machines are monsterous and have hundreds of gigs of memory; therefore, they shouldn't come anywhere near 100% memory usage either. There's a bunch of other proprietary internal metrics that I can't really explain (stuff like latency of a particular kind of response). Anyway, since I was most comfortable with writing the tool in Python (also, time was a constraint as I started working on the project towards the end of my internship), I wrote a modular tool with pluggable metrics. The reason why it wasn't a monolithic block: if the company decides to add a new metric they want to measure and follow, they should just have to write the logic necessary to retrieve that metric. The rest should be handled by my framework: anomaly detection, alerting relevant parties, and so on. So banged out the framework and had just enough time to write two modules: CPU usage and memory usage (probably the easiest ones). With a simple moving window average and standard deviation, you can identify outliers: I marked everything outside 2*sigma as an outlier as it's better to have false positives than false negatives (but not excessively many false positives as that would be counter-productive and make humans ignore reports). The alerting part was fairly straightforward as there was a built-in tool to do that. I ran a few tests with historical data and it was pretty good at detecting the anomalies that I would mark myself as a human. It could be better if someone smarter than could come up with a machine learning algo to do the same thing more accurately. (I didn't productionize it as I didn't have the time.)

iandanforth 9y ago

You might also be interested in this older project from etsy https://github.com/etsy/skyline https://github.com/etsy/skyline Has a fairly good mixture of models approach to anomaly detection even if it's not maintained.

jradd 9y ago

First thing that comes to mind is tracing system calls with strace (linux), truss (bsd), or Dtrace (recommended). A common probability theory such as Markov Chains could be used as a reference: Markov Chains http://cucis.ece.northwestern.edu/projects/DMS/publications/AnomalyDetection.pdf http://cucis.ece.northwestern.edu/projects/DMS/publications/... There is no limit of items this model could be applied to, but a few that come to mind; given the context might be; Load Generation or emulation at the filesystem, network or application level. iowait, vmstats, memory leaks (Why am I unable to account for N percent memory), Cache hit vs miss, Network stats, heuristics, buffer overflows, denial of service (listen/accept queue overflow), TCP RTT, NTP Jitter, Response Time, etc…

massaman_yams 9y ago

Netflix has some good posts on outlier / anomaly detection. https://medium.com/netflix-techblog/tracking-down-the-villains-outlier-detection-at-netflix-40360b31732 https://medium.com/netflix-techblog/tracking-down-the-villai... https://medium.com/netflix-techblog/rad-outlier-detection-on-big-data-d6b0494371cc https://medium.com/netflix-techblog/rad-outlier-detection-on...