4 ms·

Very cool work! You mentioned that in a lot of cases, basic thresholds didn't work very well. I understand that setting a direct cutoff would trigger many false

Very cool work! You mentioned that in a lot of cases, basic thresholds didn't work very well. I understand that setting a direct cutoff would trigger many false positives or not enough true positives, but what about time-based thresholds? I had always figured timed filters like "above X error rate for Y occurrences in Z time" may work. Threshold filters can be much more complex, I'm curious if you tried different types of threshold filters and if they worked or not.

diab0lic 11y ago
One of the more complex situations that this solution attempts to address is the situation where the entire cluster surges upwards. Imagine that service A depends on service B, and service B begins experiencing an issue. The outlier detection system that service A has setup will see the entire cluster surge upwards on errors. This would trip a traditional threshold based approach, but the last thing we'd want to do is terminate servers in service A. In this case the entire cluster moving upwards sets the context that there is a larger scale problem going on (nothing is outlying), whereas if only a few move upwards (become outliers) we have a localized issue we can fix. Meanwhile service B through either outlier or threshold alerts is paged and dealing with their issue. As a side note we do something similar to time sensitive thresholds when doing threshold based alerting on differences between the observed data and a double exponential smoothing fit. Parameter selection is an issue at times in this case, we talk about this a bit in an older blog post on Stream Starts per Second (SPS): http://techblog.netflix.com/2015/02/sps-pulse-of-netflix-streaming.html http://techblog.netflix.com/2015/02/sps-pulse-of-netflix-str... Hope that adds a little context for you. :)
- chrisfosterelli 11y ago
  That makes a lot of sense, thanks for the answer!
  - diab0lic 11y ago
    No problem, thanks for taking the time to read it! :D

diab0lic 11y ago

One of the more complex situations that this solution attempts to address is the situation where the entire cluster surges upwards. Imagine that service A depends on service B, and service B begins experiencing an issue. The outlier detection system that service A has setup will see the entire cluster surge upwards on errors. This would trip a traditional threshold based approach, but the last thing we'd want to do is terminate servers in service A. In this case the entire cluster moving upwards sets the context that there is a larger scale problem going on (nothing is outlying), whereas if only a few move upwards (become outliers) we have a localized issue we can fix. Meanwhile service B through either outlier or threshold alerts is paged and dealing with their issue. As a side note we do something similar to time sensitive thresholds when doing threshold based alerting on differences between the observed data and a double exponential smoothing fit. Parameter selection is an issue at times in this case, we talk about this a bit in an older blog post on Stream Starts per Second (SPS): http://techblog.netflix.com/2015/02/sps-pulse-of-netflix-streaming.html http://techblog.netflix.com/2015/02/sps-pulse-of-netflix-str... Hope that adds a little context for you. :)

chrisfosterelli 11y ago
That makes a lot of sense, thanks for the answer!
- diab0lic 11y ago
  No problem, thanks for taking the time to read it! :D