5 ms·

Disclaimer - i work at Netdata Cloud on ML. This is one of the things i am focusing on most - how to package and then surface up "anomaly events" to the user t

Disclaimer - i work at Netdata Cloud on ML. This is one of the things i am focusing on most - how to package and then surface up "anomaly events" to the user that the user can then quickly digest and decide if they are or are not something that could represent an "incident". So human in the loop sort of ML to help assist and lower the cognitive load of all the charts. We have a first step on this ladder via the python based anomalies collector that you could play around with to see if any use: https://learn.netdata.cloud/docs/agent/collectors/python.d.plugin/anomalies https://learn.netdata.cloud/docs/agent/collectors/python.d.p... That will give two summary aggregation type charts based on anomaly scores built off all the system.* charts by default (but can be configured however you want) - so, at each second, an anomaly probability for each chart and an anomaly flag if the model thinks that chart looked anomalous at that time. We are also working on some other related projects to build on this capability out more and do it at the edge, in C++ or Go as opposed to Python, (or via a parent node) as cheap as possible so minimal, ideally negligible, impact on the agent itself. We should have some more features related to this in coming months as we are just trying to dogfood them internally a little first. Anyone interested i'd love to hear some more feedback here or in this community megathread here: https://community.netdata.cloud/t/anomalies-collector-feedback-megathread/767 https://community.netdata.cloud/t/anomalies-collector-feedba... or just email me at analytics-ml-team@netdata.cloud

lima 5y ago
What's the argument for anomaly detection - it's an obvious thing to do that has been tried many times, but doesn't actually seem to provide much value in practice (especially at large scale, where you'll get spurious correlations). What would you need it for? Once you defined your SLOs, either your service meets them or not. What's the value in alerting someone that "this graph looks funny"?
- laminatedsmore 5y ago
  Do 4xx responses count against your SLO? For me they don't, but an abnormal increase might still signify that something is actually wrong. (I haven't yet found a useful tool for highlighting this kind of abnormality though)
- cakrit 5y ago
  It's about troubleshooting. When you have a complex infrastructure, it's not enough to say that your db queries are slower than usual. Ok, so you immediately see that your db server is getting a lot more traffic. What was the root cause though and what can you do about it now? Given enough "funny charts", you can see for example that you have hit a resource limit that you can temporarily raise and also see that a particular component of your infrastructure has an anomalous behavior, e.g. a cron job that was usually utilizing resources for a few seconds, now takes minutes. So you can provide a quick workaround and move on to investigate what changed with that cron job.
- mdale 5y ago
  I generally agree that well defined SLOs for back end services works as you define service contacts between services and care less about the particular funny graph being surfaced more that a particular service is out of contract. Where automatic anomaly detection was very valuable for us was in the video domain with multi dimensional end user telemetry. I.e what would be lost as noise in top level metics could be surfaced via anomaly detection for specific combination of dimensions that you could not otherwise manually observe. I.e video start time in Mexico is fine ... But an ISP in Mexico City is not failing but when data is sliced and anomaly highlighted we see its newly under preforming and we need to feed this data into our CDN switching to improve video start time there. The data had too many dimensions that were always changing with degraded experience easily lost in the noise when measuring across platform and our software updates, combinations target devices, connection types, geo location, specific content, active ab tests, etc. In such cases automatic anomaly detection was pretty critical.
- toomanybeersies 5y ago
  > What's the value in alerting someone that "this graph looks funny"? What's the value in mentioning to someone that the chicken they're about to eat looks a bit raw? It stops them from eating it and getting food poisoning. Anomalies are often warnings (harbingers?) of a problem which could lead to a fault and downtime.
- andrewm4894 5y ago
  I almost think of anomaly detection as a UI/UX type tool to help users navigate the data/systems. So use ML to find "interesting" or "novel" periods of time in your architecture (in the sense that the ML thinks they look novel based on some model), and then enable a user who is ultimately best placed to decide if it's actually of real interest to them or more like a false positive that they can just ignore and move on. So doing it in a way where you can quickly scan such events i think could be useful even if only 1 in 20 actually turn out to be some potential problem that might have been missed by your alarms or maybe could even be a precursor to some impact on SLO's etc. The aim would also be "this collection of graphs look funny at the same time" as opposed to "the individual graph looks funny" as if you have an anomaly score for every chart for sure at any given moment some individual charts would be randomly firing. But when you pool the information across charts and hosts and systems the hope is that then you can use anomaly detection as another way to explore your system and catch when things change unexpectedly.
antman 5y ago
Since you're creating an interface for python algotithms please make the interface public well documented and easily extendable snd allow it to have different algo per meter.
- andrewm4894 5y ago
  Here is a guide around how to develop your own python based collector. https://learn.netdata.cloud/guides/python-collector https://learn.netdata.cloud/guides/python-collector

lima 5y ago

What's the argument for anomaly detection - it's an obvious thing to do that has been tried many times, but doesn't actually seem to provide much value in practice (especially at large scale, where you'll get spurious correlations). What would you need it for? Once you defined your SLOs, either your service meets them or not. What's the value in alerting someone that "this graph looks funny"?

laminatedsmore 5y ago
Do 4xx responses count against your SLO? For me they don't, but an abnormal increase might still signify that something is actually wrong. (I haven't yet found a useful tool for highlighting this kind of abnormality though)
cakrit 5y ago
It's about troubleshooting. When you have a complex infrastructure, it's not enough to say that your db queries are slower than usual. Ok, so you immediately see that your db server is getting a lot more traffic. What was the root cause though and what can you do about it now? Given enough "funny charts", you can see for example that you have hit a resource limit that you can temporarily raise and also see that a particular component of your infrastructure has an anomalous behavior, e.g. a cron job that was usually utilizing resources for a few seconds, now takes minutes. So you can provide a quick workaround and move on to investigate what changed with that cron job.
mdale 5y ago
I generally agree that well defined SLOs for back end services works as you define service contacts between services and care less about the particular funny graph being surfaced more that a particular service is out of contract. Where automatic anomaly detection was very valuable for us was in the video domain with multi dimensional end user telemetry. I.e what would be lost as noise in top level metics could be surfaced via anomaly detection for specific combination of dimensions that you could not otherwise manually observe. I.e video start time in Mexico is fine ... But an ISP in Mexico City is not failing but when data is sliced and anomaly highlighted we see its newly under preforming and we need to feed this data into our CDN switching to improve video start time there. The data had too many dimensions that were always changing with degraded experience easily lost in the noise when measuring across platform and our software updates, combinations target devices, connection types, geo location, specific content, active ab tests, etc. In such cases automatic anomaly detection was pretty critical.
toomanybeersies 5y ago
> What's the value in alerting someone that "this graph looks funny"? What's the value in mentioning to someone that the chicken they're about to eat looks a bit raw? It stops them from eating it and getting food poisoning. Anomalies are often warnings (harbingers?) of a problem which could lead to a fault and downtime.
andrewm4894 5y ago
I almost think of anomaly detection as a UI/UX type tool to help users navigate the data/systems. So use ML to find "interesting" or "novel" periods of time in your architecture (in the sense that the ML thinks they look novel based on some model), and then enable a user who is ultimately best placed to decide if it's actually of real interest to them or more like a false positive that they can just ignore and move on. So doing it in a way where you can quickly scan such events i think could be useful even if only 1 in 20 actually turn out to be some potential problem that might have been missed by your alarms or maybe could even be a precursor to some impact on SLO's etc. The aim would also be "this collection of graphs look funny at the same time" as opposed to "the individual graph looks funny" as if you have an anomaly score for every chart for sure at any given moment some individual charts would be randomly firing. But when you pool the information across charts and hosts and systems the hope is that then you can use anomaly detection as another way to explore your system and catch when things change unexpectedly.

antman 5y ago

Since you're creating an interface for python algotithms please make the interface public well documented and easily extendable snd allow it to have different algo per meter.

andrewm4894 5y ago
Here is a guide around how to develop your own python based collector. https://learn.netdata.cloud/guides/python-collector https://learn.netdata.cloud/guides/python-collector