3 ms·

Some thoughts on Prometheus Alertmanager's alert reminders

24 points by zdw 4y ago

dfinninger 4y ago
This is natively supported in PagerDuty, OpsGenie, etc. Routing, time-based suppression, alert reminders, etc. I’ve tried to implement all of that in Alertmanager itself, and it’s much easier to just swipe the card. As for the dashboard, there’s an Alertmanager data source for Grafana that’s super easy to set up. If you have Grafana already I would just go ahead and do it since it takes maybe ten minutes.
- MuffinFlavored 4y ago
  > This is natively supported in PagerDuty, OpsGenie, What is a free self-hosted alternative to these two?
  - djbusby 4y ago
    Alertmanager.
    - bityard 4y ago
      GP's claim was that it was not
  - buro9 4y ago
    Grafana It now uses the Prometheus Alertmanager natively as its alert manager. And if you were using something like Mimir instead of Prometheus then you can seamlessly use Grafana against that Alertmanager too. Grafana is comfortable with any number of Prometheus compatible alert managers since the last major version.
scythmic_waves 4y ago
I found myself nodding along with the author's musings and don't have much to add. I just want to say that I love their site. I'm putting together a personal site of my own and I want to steal some of their ideas, especially the "View Source" page tool at the bottom.
- ugh123 4y ago
  looks like its some form of Markdown
  - scythmic_waves 4y ago
    Agreed. Though I meant more the functionality of being able to click on it.
nullify88 4y ago
Perhaps severity should also influence how often an alert is repeated. And during a holiday break where no one is available to react to incidents, a alert silence should be active until people are back.
knorker 4y ago
The root problem here is not a remind interval, but "email alerts". Either something should file a bug to look at "some time later", or it should start ringing your phone. Email alerts are just spam. For both bugs and pages the reminder is good. A bug should get an update essentially saying "yes, this is still looking bad" (not create a new bug), and if it was important enough to page you the first time, then (unless you silenced the alert) it's important enough to remind you by ringing your phone again.

24 points by zdw 4y ago

This is natively supported in PagerDuty, OpsGenie, etc. Routing, time-based suppression, alert reminders, etc. I’ve tried to implement all of that in Alertmanager itself, and it’s much easier to just swipe the card. As for the dashboard, there’s an Alertmanager data source for Grafana that’s super easy to set up. If you have Grafana already I would just go ahead and do it since it takes maybe ten minutes.

MuffinFlavored 4y ago
> This is natively supported in PagerDuty, OpsGenie, What is a free self-hosted alternative to these two?
- djbusby 4y ago
  Alertmanager.
  - bityard 4y ago
    GP's claim was that it was not
- buro9 4y ago
  Grafana It now uses the Prometheus Alertmanager natively as its alert manager. And if you were using something like Mimir instead of Prometheus then you can seamlessly use Grafana against that Alertmanager too. Grafana is comfortable with any number of Prometheus compatible alert managers since the last major version.

scythmic_waves 4y ago

I found myself nodding along with the author's musings and don't have much to add. I just want to say that I love their site. I'm putting together a personal site of my own and I want to steal some of their ideas, especially the "View Source" page tool at the bottom.

ugh123 4y ago
looks like its some form of Markdown
- scythmic_waves 4y ago
  Agreed. Though I meant more the functionality of being able to click on it.

nullify88 4y ago

Perhaps severity should also influence how often an alert is repeated. And during a holiday break where no one is available to react to incidents, a alert silence should be active until people are back.

knorker 4y ago

The root problem here is not a remind interval, but "email alerts". Either something should file a bug to look at "some time later", or it should start ringing your phone. Email alerts are just spam. For both bugs and pages the reminder is good. A bug should get an update essentially saying "yes, this is still looking bad" (not create a new bug), and if it was important enough to page you the first time, then (unless you silenced the alert) it's important enough to remind you by ringing your phone again.