7 ms·

One of the authors here, I'll be around to answer any questions if anyone has them. I'm sure my colleague will be around as well.

moments 11y ago
How did you pick the window size used in the analysis? Did you try different window sizes? I have seen many time-series analyses posted here lately, and all seem to use arbitrary window sizes. Edit: PS, thanks for posting this and being around for questions.
- diab0lic 11y ago
  Service owners wanted the system to be able to identify these problems as quickly as possible so the initial goal was to go as small as possible, however the initial data source rolled data up into 1 minute windows so we figured 5 minutes was the shortest we could get away with. In practice this seems to work well for service owners as it avoids calling out small spikes as outliers. We've got an experimental streaming version going that hasn't been set loose on any services yet, but it can get much higher granularity metrics ~ 10s (faster if we cared to). edit: Forgot to add that we did try other window sizes, as long as 30 minutes but we found that longer windows allowed the past to influence the decision being made now too much. If it has spiked in the past we were aggressive about calling it an outlier with 30 minute windows, furthermore if it had been in lying and just become an outlier it killed our time to detect which is an important metric for us.
huac 11y ago
I noticed that this (density-based clustering) is conceptually very different from an earlier Netflix approach (RAD - http://techblog.netflix.com/2015/02/rad-outlier-detection-on-big-data.html http://techblog.netflix.com/2015/02/rad-outlier-detection-on...) which performs RPCA and estimates thresholds. Are both systems in place? In which cases would one system outperform another?
- diab0lic 11y ago
  RAD requires at least 2 cycles of data, whereas the real-time analytics team tends to operate on very short time frames. Our approach is less accurate but able to make quick decisions, which is the objective in this case. Both systems are certainly in place, though they serve different business use cases. RAD isn't used in real-time for operational decisions and Kepler isn't used for anything that Data Science uses RAD for.
pealco 11y ago
In what space does the clustering occur? I wasn't able to tell from the post.
- huac 11y ago
  It appears to cluster on a particular server metric (e.g. % CPU used), for readings of that metric across a group of servers. One of the example graphs had 'errors per servers,' and I'd presume each color on that graph represents a different server.
  - diab0lic 11y ago
    Right you are!
- diab0lic 11y ago
  We tend to operate on sever performance metrics, error rates, and networking metrics. We've found through practice that these metrics tend to reveal most issues that we're targeting. If you meant in the more mathematical sense we perform our clustering in a normalized euclidean space.
danso 11y ago
At the risk of sparking off-topic debates about Netflix's content selections...are the same event streaming/data processing frameworks also used to detect trends and make actions on the content/frontpage side? Such as, detecting an anomalous spike of users watching some obscure movie in the last 24 hours, and then updating the "Trending Now" queues for users systemwide? Or is that something that happens (relatively) slow enough not to require an elaborate real-time framework?
- diab0lic 11y ago
  I don't work on the recommendations side of things, but this is actually the ONE technical topic at Netflix that I don't believe I am allowed to discuss. Sorry!
- fred256 11y ago
  The technology behind Trending Now is discussed here: http://techblog.netflix.com/2015/02/whats-trending-on-netflix.html http://techblog.netflix.com/2015/02/whats-trending-on-netfli...
thrownaway2424 11y ago
Are the results fed to automated remediation systems, or raised for operator interventions (or both)? If automatic, what restraints do you have against too many servers being called outliers?
- diab0lic 11y ago
  The answer is most certainly "both". The system can be used to identify problems you don't know you had (generally sending an email / page). Once you've identified a problem you may have it pull instances out of our discovery service but keep them alive so you can ssh in and triage. Once the problem has been identified you can have it mop up by terminating the services displaying the behavior while you work on a fix. We throttle automatic terminations so that it doesn't drop an entire cluster at once. Yet to cause an outage, fingers crossed! We generally run into two classes of errors: 1) Software bugs which follow the process outlined above. 2) Issues with AWS... an example being some virtual servers running on hardware experiencing a network issue. If they're terminated and replaced by the ASG generally the new ones spin up on good hardware and we've avoided the issue. Rare but it does happen at our scale.
chrisfosterelli 11y ago
Very cool work! You mentioned that in a lot of cases, basic thresholds didn't work very well. I understand that setting a direct cutoff would trigger many false positives or not enough true positives, but what about time-based thresholds? I had always figured timed filters like "above X error rate for Y occurrences in Z time" may work. Threshold filters can be much more complex, I'm curious if you tried different types of threshold filters and if they worked or not.
- diab0lic 11y ago
  One of the more complex situations that this solution attempts to address is the situation where the entire cluster surges upwards. Imagine that service A depends on service B, and service B begins experiencing an issue. The outlier detection system that service A has setup will see the entire cluster surge upwards on errors. This would trip a traditional threshold based approach, but the last thing we'd want to do is terminate servers in service A. In this case the entire cluster moving upwards sets the context that there is a larger scale problem going on (nothing is outlying), whereas if only a few move upwards (become outliers) we have a localized issue we can fix. Meanwhile service B through either outlier or threshold alerts is paged and dealing with their issue. As a side note we do something similar to time sensitive thresholds when doing threshold based alerting on differences between the observed data and a double exponential smoothing fit. Parameter selection is an issue at times in this case, we talk about this a bit in an older blog post on Stream Starts per Second (SPS): http://techblog.netflix.com/2015/02/sps-pulse-of-netflix-streaming.html http://techblog.netflix.com/2015/02/sps-pulse-of-netflix-str... Hope that adds a little context for you. :)
  - chrisfosterelli 11y ago
    That makes a lot of sense, thanks for the answer!
    - diab0lic 11y ago
      No problem, thanks for taking the time to read it! :D
jfroma 11y ago
Great article. Do you use this same technique in other areas instead of servers/metrics to detect for instance bad behaving users or applications?
- diab0lic 11y ago
  We most certainly do, one of the more obscure applications is detecting customer service agents who are gaming the system.
cbsmith 11y ago
There are a lot of online methods for reinforcement learning, and one could argue that reinforcement learning is doing exactly what you want for outlier detection: doing a cost benefit analysis of the value of exploring a given context in more or less detail. Have you guys thought at all about adapting a reinforcement learning model for building an online model to detect outliers?
- diab0lic 11y ago
  We have, specifically in the domain of using RL for parameter selection. Part of the reason I was brought on board was background experience in RL. I fully believe it's the key to turning domain (yes/no) decisions into numbers for our algorithms to ingest. Ultimately we would like all of our systems to use RL to interpret user feedback to determine parameters, actions, etc... Brilliant question by the way!
  - cbsmith 11y ago
    Brilliant answer. :-) Sounds like you guys have thought this through really well.
tricolon 11y ago
I enjoyed the subtitles.
deleted 11y ago
[deleted]
sytelus 11y ago
It would be great if you can share your some of your data to play around. I think many of people here might help you break new grounds :).
- diab0lic 11y ago
  It's just a matter of collecting and tagging enough. We've been working on it with our new operational data tagger. Keep an eye out for a future post! :)
mharrison 11y ago
It seems like this is an area where Statistical Process Control would shine. Did you consider using that?
- diab0lic 11y ago
  I didn't consider statistical process control for this task when I did my initial research. It's certainly something I wish I had read about earlier. I have however since started investigating it for a related real-time project that I hope will someday manifest itself on the tech blog as well.
iwwr 11y ago
Is it a worthy goal to also detect outlier good performing servers? Sometimes mutations are beneficial.
- bbrazil 11y ago
  That can mean that the server hasn't got data it needs loaded and is returning empty requests, so it's good to spot those.
daemonk 11y ago
What is considered an "error"? Are there further classifications for different types of errors? Maybe there are certain error types that contribute more to allow you to detect bad servers? Seems like just having a single dimension is a bit heavy-handed. Would multiple dimensions slow it down too much to be useful?
- kefka 11y ago
  The problems are "error". Anything else may be a root cause, which can range from failing hard drives, CPU failure, memory failure, NIC failure, rat chewing on cables, or plenty of other modes. I would say that we don't care about root cause when the event happens; just get the server out of the pool. RCA can be done post mortem. I would reduce dimensionality down to 2d: errors per time. In that case, we have a great deal of statistical tools at hand. They also do not require hand-waving of N dimensional cluster detection, where only the machine has any idea of detecting errors. And having something like 1000 dimensions is just tremendously slow, compared to integral analysis of errors in respect to time.
zniperr 11y ago
In "Parameter Selection" you say "We simplify this by having service owners define the current number of outliers, if there are any, at configuration time." How do service owners get this data? by manual inspection of graphs like the ones shown earlier?
stephen 11y ago
Do you think CloudWatch will ever be "good enough" to replace your custom stack/Atlas?
kefka 11y ago
I think the machine learning aspect is the wrong choice here. I'm seeing another glaring answer, and one that can be done fairly quickly: Integral analysis. The servers that have error accumulation aren't always the ones highest in spikes, as shown in the link. However, they do consistently stay above others in amounts of errors. So the answer is thus: Since your time series is quantized, the integral is simply a sum over the timeframe. I would recommend multiple time windows, like "10 seconds, 3 minutes, 1 hour, 12 hours". Do this for all your servers. Now, you have a normalized 2d graph, with respect for time. We can now calculate the mean of all servers' error areas as well as standard deviation. Now, for reducing CPU load, one can run a modified thompson tau test on +/-2σ for outlier detection. 4σ is around 97.7% of all your data, so you would only be checking ~2.3% of your data. When a server fails an outlier test (in other words, is detected as an outlier), the historical data can also be investigated. Is the machine doing gradually worse? If so, the machine could be removed from service until a memory and CPU test can be run. You can also keep anomaly detection on how many outliers per hour. Passing a threshold could indicate potentially erroneous machines. My answer also assumes that the machines are equal in the amount of load. I'm sure this is not the case, in which you will have to normalize errors/# of requests. This shouldn't be a problem.

moments 11y ago

How did you pick the window size used in the analysis? Did you try different window sizes? I have seen many time-series analyses posted here lately, and all seem to use arbitrary window sizes. Edit: PS, thanks for posting this and being around for questions.

diab0lic 11y ago
Service owners wanted the system to be able to identify these problems as quickly as possible so the initial goal was to go as small as possible, however the initial data source rolled data up into 1 minute windows so we figured 5 minutes was the shortest we could get away with. In practice this seems to work well for service owners as it avoids calling out small spikes as outliers. We've got an experimental streaming version going that hasn't been set loose on any services yet, but it can get much higher granularity metrics ~ 10s (faster if we cared to). edit: Forgot to add that we did try other window sizes, as long as 30 minutes but we found that longer windows allowed the past to influence the decision being made now too much. If it has spiked in the past we were aggressive about calling it an outlier with 30 minute windows, furthermore if it had been in lying and just become an outlier it killed our time to detect which is an important metric for us.

huac 11y ago

I noticed that this (density-based clustering) is conceptually very different from an earlier Netflix approach (RAD - http://techblog.netflix.com/2015/02/rad-outlier-detection-on-big-data.html http://techblog.netflix.com/2015/02/rad-outlier-detection-on...) which performs RPCA and estimates thresholds. Are both systems in place? In which cases would one system outperform another?

diab0lic 11y ago
RAD requires at least 2 cycles of data, whereas the real-time analytics team tends to operate on very short time frames. Our approach is less accurate but able to make quick decisions, which is the objective in this case. Both systems are certainly in place, though they serve different business use cases. RAD isn't used in real-time for operational decisions and Kepler isn't used for anything that Data Science uses RAD for.

pealco 11y ago

In what space does the clustering occur? I wasn't able to tell from the post.

huac 11y ago
It appears to cluster on a particular server metric (e.g. % CPU used), for readings of that metric across a group of servers. One of the example graphs had 'errors per servers,' and I'd presume each color on that graph represents a different server.
- diab0lic 11y ago
  Right you are!
diab0lic 11y ago
We tend to operate on sever performance metrics, error rates, and networking metrics. We've found through practice that these metrics tend to reveal most issues that we're targeting. If you meant in the more mathematical sense we perform our clustering in a normalized euclidean space.

danso 11y ago

At the risk of sparking off-topic debates about Netflix's content selections...are the same event streaming/data processing frameworks also used to detect trends and make actions on the content/frontpage side? Such as, detecting an anomalous spike of users watching some obscure movie in the last 24 hours, and then updating the "Trending Now" queues for users systemwide? Or is that something that happens (relatively) slow enough not to require an elaborate real-time framework?

diab0lic 11y ago
I don't work on the recommendations side of things, but this is actually the ONE technical topic at Netflix that I don't believe I am allowed to discuss. Sorry!
fred256 11y ago
The technology behind Trending Now is discussed here: http://techblog.netflix.com/2015/02/whats-trending-on-netflix.html http://techblog.netflix.com/2015/02/whats-trending-on-netfli...

thrownaway2424 11y ago

Are the results fed to automated remediation systems, or raised for operator interventions (or both)? If automatic, what restraints do you have against too many servers being called outliers?

diab0lic 11y ago
The answer is most certainly "both". The system can be used to identify problems you don't know you had (generally sending an email / page). Once you've identified a problem you may have it pull instances out of our discovery service but keep them alive so you can ssh in and triage. Once the problem has been identified you can have it mop up by terminating the services displaying the behavior while you work on a fix. We throttle automatic terminations so that it doesn't drop an entire cluster at once. Yet to cause an outage, fingers crossed! We generally run into two classes of errors: 1) Software bugs which follow the process outlined above. 2) Issues with AWS... an example being some virtual servers running on hardware experiencing a network issue. If they're terminated and replaced by the ASG generally the new ones spin up on good hardware and we've avoided the issue. Rare but it does happen at our scale.

chrisfosterelli 11y ago

Very cool work! You mentioned that in a lot of cases, basic thresholds didn't work very well. I understand that setting a direct cutoff would trigger many false positives or not enough true positives, but what about time-based thresholds? I had always figured timed filters like "above X error rate for Y occurrences in Z time" may work. Threshold filters can be much more complex, I'm curious if you tried different types of threshold filters and if they worked or not.

diab0lic 11y ago
One of the more complex situations that this solution attempts to address is the situation where the entire cluster surges upwards. Imagine that service A depends on service B, and service B begins experiencing an issue. The outlier detection system that service A has setup will see the entire cluster surge upwards on errors. This would trip a traditional threshold based approach, but the last thing we'd want to do is terminate servers in service A. In this case the entire cluster moving upwards sets the context that there is a larger scale problem going on (nothing is outlying), whereas if only a few move upwards (become outliers) we have a localized issue we can fix. Meanwhile service B through either outlier or threshold alerts is paged and dealing with their issue. As a side note we do something similar to time sensitive thresholds when doing threshold based alerting on differences between the observed data and a double exponential smoothing fit. Parameter selection is an issue at times in this case, we talk about this a bit in an older blog post on Stream Starts per Second (SPS): http://techblog.netflix.com/2015/02/sps-pulse-of-netflix-streaming.html http://techblog.netflix.com/2015/02/sps-pulse-of-netflix-str... Hope that adds a little context for you. :)
- chrisfosterelli 11y ago
  That makes a lot of sense, thanks for the answer!
  - diab0lic 11y ago
    No problem, thanks for taking the time to read it! :D

jfroma 11y ago

Great article. Do you use this same technique in other areas instead of servers/metrics to detect for instance bad behaving users or applications?

diab0lic 11y ago
We most certainly do, one of the more obscure applications is detecting customer service agents who are gaming the system.

cbsmith 11y ago

There are a lot of online methods for reinforcement learning, and one could argue that reinforcement learning is doing exactly what you want for outlier detection: doing a cost benefit analysis of the value of exploring a given context in more or less detail. Have you guys thought at all about adapting a reinforcement learning model for building an online model to detect outliers?

diab0lic 11y ago
We have, specifically in the domain of using RL for parameter selection. Part of the reason I was brought on board was background experience in RL. I fully believe it's the key to turning domain (yes/no) decisions into numbers for our algorithms to ingest. Ultimately we would like all of our systems to use RL to interpret user feedback to determine parameters, actions, etc... Brilliant question by the way!
- cbsmith 11y ago
  Brilliant answer. :-) Sounds like you guys have thought this through really well.

tricolon 11y ago

I enjoyed the subtitles.

deleted 11y ago

[deleted]

sytelus 11y ago

It would be great if you can share your some of your data to play around. I think many of people here might help you break new grounds :).

diab0lic 11y ago
It's just a matter of collecting and tagging enough. We've been working on it with our new operational data tagger. Keep an eye out for a future post! :)

mharrison 11y ago

It seems like this is an area where Statistical Process Control would shine. Did you consider using that?

diab0lic 11y ago
I didn't consider statistical process control for this task when I did my initial research. It's certainly something I wish I had read about earlier. I have however since started investigating it for a related real-time project that I hope will someday manifest itself on the tech blog as well.

iwwr 11y ago

Is it a worthy goal to also detect outlier good performing servers? Sometimes mutations are beneficial.

bbrazil 11y ago
That can mean that the server hasn't got data it needs loaded and is returning empty requests, so it's good to spot those.

daemonk 11y ago

What is considered an "error"? Are there further classifications for different types of errors? Maybe there are certain error types that contribute more to allow you to detect bad servers? Seems like just having a single dimension is a bit heavy-handed. Would multiple dimensions slow it down too much to be useful?

kefka 11y ago
The problems are "error". Anything else may be a root cause, which can range from failing hard drives, CPU failure, memory failure, NIC failure, rat chewing on cables, or plenty of other modes. I would say that we don't care about root cause when the event happens; just get the server out of the pool. RCA can be done post mortem. I would reduce dimensionality down to 2d: errors per time. In that case, we have a great deal of statistical tools at hand. They also do not require hand-waving of N dimensional cluster detection, where only the machine has any idea of detecting errors. And having something like 1000 dimensions is just tremendously slow, compared to integral analysis of errors in respect to time.

zniperr 11y ago

In "Parameter Selection" you say "We simplify this by having service owners define the current number of outliers, if there are any, at configuration time." How do service owners get this data? by manual inspection of graphs like the ones shown earlier?

stephen 11y ago

Do you think CloudWatch will ever be "good enough" to replace your custom stack/Atlas?

kefka 11y ago

I think the machine learning aspect is the wrong choice here. I'm seeing another glaring answer, and one that can be done fairly quickly: Integral analysis. The servers that have error accumulation aren't always the ones highest in spikes, as shown in the link. However, they do consistently stay above others in amounts of errors. So the answer is thus: Since your time series is quantized, the integral is simply a sum over the timeframe. I would recommend multiple time windows, like "10 seconds, 3 minutes, 1 hour, 12 hours". Do this for all your servers. Now, you have a normalized 2d graph, with respect for time. We can now calculate the mean of all servers' error areas as well as standard deviation. Now, for reducing CPU load, one can run a modified thompson tau test on +/-2σ for outlier detection. 4σ is around 97.7% of all your data, so you would only be checking ~2.3% of your data. When a server fails an outlier test (in other words, is detected as an outlier), the historical data can also be investigated. Is the machine doing gradually worse? If so, the machine could be removed from service until a memory and CPU test can be run. You can also keep anomaly detection on how many outliers per hour. Passing a threshold could indicate potentially erroneous machines. My answer also assumes that the machines are equal in the amount of load. I'm sure this is not the case, in which you will have to normalize errors/# of requests. This shouldn't be a problem.