22 ms·

I don't understand the duplication of alerting with Grafana/Alertmanager. Is it not better to keep all alert configuration central in prometheus alerting rules

by yakata 5y ago

I don't understand the duplication of alerting with Grafana/Alertmanager. Is it not better to keep all alert configuration central in prometheus alerting rules ?

buro9 5y ago
If you use Prometheus and have everything that you want to alert on instrumented with Prometheus then of course that makes sense. But if you want to alert on other data sources, and especially to see and manage all alerts in one place, then this is what Grafana Alerting now enables. Example I encountered a while ago: Alert on billing throughput within a third party (not instrumented with Prometheus) billing system that has a RDBMS backend. The alert can be written in SQL and with the changes in Grafana 8 could be acted upon and routed just like the Prometheus alerts were.
danlimerick 5y ago
Internally (in the Grafana team), we call it unified alerting because it is a unified UI for non-Prometheus data sources together with Prometheus alerting (and Loki/Cortex/Grafana Cloud alerting) and allows you to see all your alerts together. - For Prometheus, can view and search for alerts. - For Loki and Cortex, can create and edit alert rules. See Alertmanager data source: https://grafana.com/docs/grafana/latest/datasources/alertmanager/ https://grafana.com/docs/grafana/latest/datasources/alertman... - For non-Prometheus data sources, uses the Grafana alerting engine but also has an embedded Alertmanager. Can use the same routing, grouping, deduping and silencing features that Alertmanager provides but for data sources like InfluxDB or MySQL.
crad 5y ago
I use a combination of: - Graphite - InfluxDB - PostgreSQL - Timescale - CloudWatch - Prometheus The alerting in Grafana supporting all those variations is a godsend.
elktea 5y ago
Prometheus's language PromQL is pretty inscrutable and the alert manager is very bare-bones. I find the entire project is pretty user-hostile wrt documentation and examples actually.

by yakata 5y ago

I don't understand the duplication of alerting with Grafana/Alertmanager. Is it not better to keep all alert configuration central in prometheus alerting rules ?

buro9 5y ago

If you use Prometheus and have everything that you want to alert on instrumented with Prometheus then of course that makes sense. But if you want to alert on other data sources, and especially to see and manage all alerts in one place, then this is what Grafana Alerting now enables. Example I encountered a while ago: Alert on billing throughput within a third party (not instrumented with Prometheus) billing system that has a RDBMS backend. The alert can be written in SQL and with the changes in Grafana 8 could be acted upon and routed just like the Prometheus alerts were.

danlimerick 5y ago

Internally (in the Grafana team), we call it unified alerting because it is a unified UI for non-Prometheus data sources together with Prometheus alerting (and Loki/Cortex/Grafana Cloud alerting) and allows you to see all your alerts together. - For Prometheus, can view and search for alerts. - For Loki and Cortex, can create and edit alert rules. See Alertmanager data source: https://grafana.com/docs/grafana/latest/datasources/alertmanager/ https://grafana.com/docs/grafana/latest/datasources/alertman... - For non-Prometheus data sources, uses the Grafana alerting engine but also has an embedded Alertmanager. Can use the same routing, grouping, deduping and silencing features that Alertmanager provides but for data sources like InfluxDB or MySQL.

crad 5y ago

I use a combination of: - Graphite - InfluxDB - PostgreSQL - Timescale - CloudWatch - Prometheus The alerting in Grafana supporting all those variations is a godsend.

elktea 5y ago

Prometheus's language PromQL is pretty inscrutable and the alert manager is very bare-bones. I find the entire project is pretty user-hostile wrt documentation and examples actually.