3 ms·

I use offset function in Prometheus to make an average of past weeks as a recording rule. We have a use in our systems that is very "seasonal" as in weekly cycl

I use offset function in Prometheus to make an average of past weeks as a recording rule. We have a use in our systems that is very "seasonal" as in weekly cycles so I make an average of some metric (offset 1 week, 2 week, 3 week , 4 week/4) and I compare it to the current value of that metric. That way the alarms can be set day or night, weekday or weekend, and the thresholds are dynamic. It compares against an average of the day of the week, or time of the day. There is someone in Gitlab that posted a more in depth explanation of this way of working. https://about.gitlab.com/blog/2019/07/23/anomaly-detection-using-prometheus/ https://about.gitlab.com/blog/2019/07/23/anomaly-detection-u... Things get a bit more complicated with holidays, but you can actually programm them into prometheus https://promcon.io/2019-munich/slides/improved-alerting-with-prometheus-and-alertmanager.pdf https://promcon.io/2019-munich/slides/improved-alerting-with...

CubsFan1060 2y ago
Gitlab also has this: https://gitlab.com/gitlab-com/gl-infra/tamland https://gitlab.com/gitlab-com/gl-infra/tamland I'm not really smart in these areas, but it feels like forecasting and anomaly detection are pretty related. I could be wrong though.
- diab0lic 2y ago
  You are not wrong! An entire subclass of anomaly detection can basically be reduced to: forecast the next data point and then measure the forecast error when the data point arrives.
  - deleted 2y ago
    [deleted]
  - fnordpiglet 2y ago
    Well it doesn’t really require a forecast - variance based anomaly detection doesn’t make an assertion of the next point but that its maximum change is within some band. Such models usually can’t be used to make a forecast other than the banding bounds.
    - diab0lic 2y ago
      That would be a different subclass of anomaly detection solutions.
- fraserphysics 2y ago
  If you need to detect anomalies as soon as they occur, that seems right. But if you want to detect them later you can also combine back-casting with forecasting. Like Kalman smoothing.
gr3ml1n 2y ago
Whenever I have a chart in Grafana that isn't too dense, I almost always add a line for the 7d offset value. Super useful to tell what's normal and what isn't.

CubsFan1060 2y ago

Gitlab also has this: https://gitlab.com/gitlab-com/gl-infra/tamland https://gitlab.com/gitlab-com/gl-infra/tamland I'm not really smart in these areas, but it feels like forecasting and anomaly detection are pretty related. I could be wrong though.

diab0lic 2y ago
You are not wrong! An entire subclass of anomaly detection can basically be reduced to: forecast the next data point and then measure the forecast error when the data point arrives.
- deleted 2y ago
  [deleted]
- fnordpiglet 2y ago
  Well it doesn’t really require a forecast - variance based anomaly detection doesn’t make an assertion of the next point but that its maximum change is within some band. Such models usually can’t be used to make a forecast other than the banding bounds.
  - diab0lic 2y ago
    That would be a different subclass of anomaly detection solutions.
fraserphysics 2y ago
If you need to detect anomalies as soon as they occur, that seems right. But if you want to detect them later you can also combine back-casting with forecasting. Like Kalman smoothing.

gr3ml1n 2y ago

Whenever I have a chart in Grafana that isn't too dense, I almost always add a line for the 7d offset value. Super useful to tell what's normal and what isn't.