4 ms·

What's the argument for anomaly detection - it's an obvious thing to do that has been tried many times, but doesn't actually seem to provide much value in pract

by lima 6y ago

What's the argument for anomaly detection - it's an obvious thing to do that has been tried many times, but doesn't actually seem to provide much value in practice (especially at large scale, where you'll get spurious correlations). What would you need it for? Once you defined your SLOs, either your service meets them or not. What's the value in alerting someone that "this graph looks funny"?

laminatedsmore 6y ago
Do 4xx responses count against your SLO? For me they don't, but an abnormal increase might still signify that something is actually wrong. (I haven't yet found a useful tool for highlighting this kind of abnormality though)
cakrit 6y ago
It's about troubleshooting. When you have a complex infrastructure, it's not enough to say that your db queries are slower than usual. Ok, so you immediately see that your db server is getting a lot more traffic. What was the root cause though and what can you do about it now? Given enough "funny charts", you can see for example that you have hit a resource limit that you can temporarily raise and also see that a particular component of your infrastructure has an anomalous behavior, e.g. a cron job that was usually utilizing resources for a few seconds, now takes minutes. So you can provide a quick workaround and move on to investigate what changed with that cron job.
mdale 6y ago
I generally agree that well defined SLOs for back end services works as you define service contacts between services and care less about the particular funny graph being surfaced more that a particular service is out of contract. Where automatic anomaly detection was very valuable for us was in the video domain with multi dimensional end user telemetry. I.e what would be lost as noise in top level metics could be surfaced via anomaly detection for specific combination of dimensions that you could not otherwise manually observe. I.e video start time in Mexico is fine ... But an ISP in Mexico City is not failing but when data is sliced and anomaly highlighted we see its newly under preforming and we need to feed this data into our CDN switching to improve video start time there. The data had too many dimensions that were always changing with degraded experience easily lost in the noise when measuring across platform and our software updates, combinations target devices, connection types, geo location, specific content, active ab tests, etc. In such cases automatic anomaly detection was pretty critical.
toomanybeersies 6y ago
> What's the value in alerting someone that "this graph looks funny"? What's the value in mentioning to someone that the chicken they're about to eat looks a bit raw? It stops them from eating it and getting food poisoning. Anomalies are often warnings (harbingers?) of a problem which could lead to a fault and downtime.
andrewm4894 6y ago
I almost think of anomaly detection as a UI/UX type tool to help users navigate the data/systems. So use ML to find "interesting" or "novel" periods of time in your architecture (in the sense that the ML thinks they look novel based on some model), and then enable a user who is ultimately best placed to decide if it's actually of real interest to them or more like a false positive that they can just ignore and move on. So doing it in a way where you can quickly scan such events i think could be useful even if only 1 in 20 actually turn out to be some potential problem that might have been missed by your alarms or maybe could even be a precursor to some impact on SLO's etc. The aim would also be "this collection of graphs look funny at the same time" as opposed to "the individual graph looks funny" as if you have an anomaly score for every chart for sure at any given moment some individual charts would be randomly firing. But when you pool the information across charts and hosts and systems the hope is that then you can use anomaly detection as another way to explore your system and catch when things change unexpectedly.

by lima 6y ago

laminatedsmore 6y ago

Do 4xx responses count against your SLO? For me they don't, but an abnormal increase might still signify that something is actually wrong. (I haven't yet found a useful tool for highlighting this kind of abnormality though)

cakrit 6y ago

It's about troubleshooting. When you have a complex infrastructure, it's not enough to say that your db queries are slower than usual. Ok, so you immediately see that your db server is getting a lot more traffic. What was the root cause though and what can you do about it now? Given enough "funny charts", you can see for example that you have hit a resource limit that you can temporarily raise and also see that a particular component of your infrastructure has an anomalous behavior, e.g. a cron job that was usually utilizing resources for a few seconds, now takes minutes. So you can provide a quick workaround and move on to investigate what changed with that cron job.

mdale 6y ago

I generally agree that well defined SLOs for back end services works as you define service contacts between services and care less about the particular funny graph being surfaced more that a particular service is out of contract. Where automatic anomaly detection was very valuable for us was in the video domain with multi dimensional end user telemetry. I.e what would be lost as noise in top level metics could be surfaced via anomaly detection for specific combination of dimensions that you could not otherwise manually observe. I.e video start time in Mexico is fine ... But an ISP in Mexico City is not failing but when data is sliced and anomaly highlighted we see its newly under preforming and we need to feed this data into our CDN switching to improve video start time there. The data had too many dimensions that were always changing with degraded experience easily lost in the noise when measuring across platform and our software updates, combinations target devices, connection types, geo location, specific content, active ab tests, etc. In such cases automatic anomaly detection was pretty critical.

toomanybeersies 6y ago

> What's the value in alerting someone that "this graph looks funny"? What's the value in mentioning to someone that the chicken they're about to eat looks a bit raw? It stops them from eating it and getting food poisoning. Anomalies are often warnings (harbingers?) of a problem which could lead to a fault and downtime.

andrewm4894 6y ago

I almost think of anomaly detection as a UI/UX type tool to help users navigate the data/systems. So use ML to find "interesting" or "novel" periods of time in your architecture (in the sense that the ML thinks they look novel based on some model), and then enable a user who is ultimately best placed to decide if it's actually of real interest to them or more like a false positive that they can just ignore and move on. So doing it in a way where you can quickly scan such events i think could be useful even if only 1 in 20 actually turn out to be some potential problem that might have been missed by your alarms or maybe could even be a precursor to some impact on SLO's etc. The aim would also be "this collection of graphs look funny at the same time" as opposed to "the individual graph looks funny" as if you have an anomaly score for every chart for sure at any given moment some individual charts would be randomly firing. But when you pool the information across charts and hosts and systems the hope is that then you can use anomaly detection as another way to explore your system and catch when things change unexpectedly.