4 ms·

I recently tried to homebrew some anomaly detection work for a performance tracking project and was surprised at the absence of any off-the-shelf OSS or Paid so

I recently tried to homebrew some anomaly detection work for a performance tracking project and was surprised at the absence of any off-the-shelf OSS or Paid solutions in this space (that weren’t super basic or way too complex). Lots of fertile ground here!

rad_gruchalski 2y ago
There's a ton of material related to anomaly detection with Prometheus and Grafana stack: https://grafana.com/blog/2024/10/03/how-to-use-prometheus-to-efficiently-detect-anomalies-at-scale/ https://grafana.com/blog/2024/10/03/how-to-use-prometheus-to.... But maybe this is the "way too complex" case you mention.
CubsFan1060 2y ago
I'm still playing around with this one: https://grafana.com/blog/2024/10/03/how-to-use-prometheus-to-efficiently-detect-anomalies-at-scale/ https://grafana.com/blog/2024/10/03/how-to-use-prometheus-to... (there's a github repo for it). So far, it's not terrible, but has some pretty big flaws.
- jcreixell 2y ago
  Hi, co-author of the blog post here. I would love to learn more about the flaws you see and if ideas on how to improve it! We definitely plan to iterate on it and make it as good as we possibly can.
  - nyrikki 2y ago
    Not really related to the above post, but one thing I am not seeing on an initial pass is the advancement of understanding of problems like riddled or wada basins. Especially with time delays this and 3+ attractors this can be problematic. A simple example: https://doi.org/10.21203/rs.3.rs-1088857/v1 https://doi.org/10.21203/rs.3.rs-1088857/v1 There are tools to try and detect these features that were found over the past few decades, and I know I wasted a few years on a project that superficially looked like a FP issue, but ended up being a mix of the wada property and/or porous sets. The complications will describing these worse than traditional chaos indeterminate situations may make it inappropriate for you. But it would be nice if visibility was increased. Funny enough most LLMs corpus is mostly fed from a LSAT question. There has been a lot of movement here when you have n>=3 attractors/exits. Not solutions unfortunately, but tools to help figure out when you hit it.
  - pnathan 2y ago
    The number of manual tweaks required to the approach suggest that it is essentially an ad hoc experimental fitting, rather than a stable theoretical model that can adapt to your time series.
  - CubsFan1060 2y ago
    To be clear "some big flaws" was probably overstating it. I'm going to edit that. Also, thanks for the work on this. I would absolutely love to contribute, but my maths are not good enough for this :) The biggest thing I've run into in my testing is that an anomaly of reasonably short timeframe seems to throw the upper and lower bands off for quite some time. That being said, perhaps changing some of the variables would help with that, and I just don't have enough skill to be able to understand the exact way to adjust that.
    - jcreixell 2y ago
      Thank you for the feedback! There is an issue (https://github.com/grafana/promql-anomaly-detection/issues/7 https://github.com/grafana/promql-anomaly-detection/issues/7) discussing approaches to improve this by introducing a decay function. This is specially relevant for stable series with occasional short large spikes. Nothing conclusive yet, but hopefully something good will come out of it!
- hackernewds 2y ago
  anything in grafana is inherently not exportable to any code though which is rather annoying cuz their UI really sucks
  - davkal 2y ago
    Hi! I work on the Grafana OSS team. We added some more export options recently (dashboards have a big Export button at the top right; panels can export their data via the panel menu / Inspect / Data), try it on our demo page: https://play.grafana.org/d/000000003/graphite3a-sample-website-dashboard https://play.grafana.org/d/000000003/graphite3a-sample-websi... Could you describe your use case around "exportable to any code" a bit more?
ramon156 2y ago
I needed a TS anomaly detection for my internship because we needed to track when a machine/server was doing poorly or had unplanned downtime. I expected Microsoft's C# library to be able to do this, but my god, it's a mess. If someone has the time and will to implement a proper library then that would ve awesome.
- neonsunset 2y ago
  Anomaly detection in time-series data is not a concern of the standard library of all things. Nor is it a concern of "base abstractions" shipped as extensions (think ILogger).
  - Phurist 2y ago
    If only life was as simple as calling .isAnomaly() on anything
    - neonsunset 2y ago
      Hardcoded to return 'false' of course. Because nothing ever happens!
      - sam_bristow 2y ago
        Just advertise it as guaranteed 0% false positive detections and you're good-to-go!
- mr_toad 2y ago
  What you’re probably after is called statistical process control. There are Python libraries like pyspc, but the theory is simple enough that you could write your own pretty easily.
jeffbee 2y ago
The reason there are not off-the-shelf solutions is this is an unsolved problem. There is no approach that is generally useful.
- otterley 2y ago
  Perhaps not, but an efficient, multi-language library of different functions would allow for relatively easy implementation and experimentation.
phirschybar 2y ago
agreed. at my company we ended up rolling our own system. but this area is absolutely ripe for some configurable saas or OS tool with advanced reporting and alerting mechanisms. Datadog has a decent offering, but it's pretty $$$$.
- montereynack 2y ago
  Gonna throw in my hat and say that if you’re working on industrial applications (like energy or manufacturing) give us a holler at www.sentineldevices.com! Plug-and-play time series monitoring for industrial applications is exactly what we do.
hackernewds 2y ago
there's always prophet. forecast the next value and look at the difference

rad_gruchalski 2y ago

There's a ton of material related to anomaly detection with Prometheus and Grafana stack: https://grafana.com/blog/2024/10/03/how-to-use-prometheus-to-efficiently-detect-anomalies-at-scale/ https://grafana.com/blog/2024/10/03/how-to-use-prometheus-to.... But maybe this is the "way too complex" case you mention.

CubsFan1060 2y ago

I'm still playing around with this one: https://grafana.com/blog/2024/10/03/how-to-use-prometheus-to-efficiently-detect-anomalies-at-scale/ https://grafana.com/blog/2024/10/03/how-to-use-prometheus-to... (there's a github repo for it). So far, it's not terrible, but has some pretty big flaws.

jcreixell 2y ago
Hi, co-author of the blog post here. I would love to learn more about the flaws you see and if ideas on how to improve it! We definitely plan to iterate on it and make it as good as we possibly can.
- nyrikki 2y ago
  Not really related to the above post, but one thing I am not seeing on an initial pass is the advancement of understanding of problems like riddled or wada basins. Especially with time delays this and 3+ attractors this can be problematic. A simple example: https://doi.org/10.21203/rs.3.rs-1088857/v1 https://doi.org/10.21203/rs.3.rs-1088857/v1 There are tools to try and detect these features that were found over the past few decades, and I know I wasted a few years on a project that superficially looked like a FP issue, but ended up being a mix of the wada property and/or porous sets. The complications will describing these worse than traditional chaos indeterminate situations may make it inappropriate for you. But it would be nice if visibility was increased. Funny enough most LLMs corpus is mostly fed from a LSAT question. There has been a lot of movement here when you have n>=3 attractors/exits. Not solutions unfortunately, but tools to help figure out when you hit it.
- pnathan 2y ago
  The number of manual tweaks required to the approach suggest that it is essentially an ad hoc experimental fitting, rather than a stable theoretical model that can adapt to your time series.
- CubsFan1060 2y ago
  To be clear "some big flaws" was probably overstating it. I'm going to edit that. Also, thanks for the work on this. I would absolutely love to contribute, but my maths are not good enough for this :) The biggest thing I've run into in my testing is that an anomaly of reasonably short timeframe seems to throw the upper and lower bands off for quite some time. That being said, perhaps changing some of the variables would help with that, and I just don't have enough skill to be able to understand the exact way to adjust that.
  - jcreixell 2y ago
    Thank you for the feedback! There is an issue (https://github.com/grafana/promql-anomaly-detection/issues/7 https://github.com/grafana/promql-anomaly-detection/issues/7) discussing approaches to improve this by introducing a decay function. This is specially relevant for stable series with occasional short large spikes. Nothing conclusive yet, but hopefully something good will come out of it!
hackernewds 2y ago
anything in grafana is inherently not exportable to any code though which is rather annoying cuz their UI really sucks
- davkal 2y ago
  Hi! I work on the Grafana OSS team. We added some more export options recently (dashboards have a big Export button at the top right; panels can export their data via the panel menu / Inspect / Data), try it on our demo page: https://play.grafana.org/d/000000003/graphite3a-sample-website-dashboard https://play.grafana.org/d/000000003/graphite3a-sample-websi... Could you describe your use case around "exportable to any code" a bit more?

ramon156 2y ago

I needed a TS anomaly detection for my internship because we needed to track when a machine/server was doing poorly or had unplanned downtime. I expected Microsoft's C# library to be able to do this, but my god, it's a mess. If someone has the time and will to implement a proper library then that would ve awesome.

neonsunset 2y ago
Anomaly detection in time-series data is not a concern of the standard library of all things. Nor is it a concern of "base abstractions" shipped as extensions (think ILogger).
- Phurist 2y ago
  If only life was as simple as calling .isAnomaly() on anything
  - neonsunset 2y ago
    Hardcoded to return 'false' of course. Because nothing ever happens!
    - sam_bristow 2y ago
      Just advertise it as guaranteed 0% false positive detections and you're good-to-go!
mr_toad 2y ago
What you’re probably after is called statistical process control. There are Python libraries like pyspc, but the theory is simple enough that you could write your own pretty easily.

jeffbee 2y ago

The reason there are not off-the-shelf solutions is this is an unsolved problem. There is no approach that is generally useful.

otterley 2y ago
Perhaps not, but an efficient, multi-language library of different functions would allow for relatively easy implementation and experimentation.

phirschybar 2y ago

agreed. at my company we ended up rolling our own system. but this area is absolutely ripe for some configurable saas or OS tool with advanced reporting and alerting mechanisms. Datadog has a decent offering, but it's pretty $$$$.

montereynack 2y ago
Gonna throw in my hat and say that if you’re working on industrial applications (like energy or manufacturing) give us a holler at www.sentineldevices.com! Plug-and-play time series monitoring for industrial applications is exactly what we do.

hackernewds 2y ago

there's always prophet. forecast the next value and look at the difference