3 ms·
Yes, we basically have a short lived periodic job on our Alertmangaer server which does a `sync` from the S3 bucket which stores the configs. If something has c
by mr-karan 6y ago
Yes, we basically have a short lived periodic job on our Alertmangaer server which does a `sync` from the S3 bucket which stores the configs. If something has changed, just trigger a `docker restart alertmanager` basically.
Alertmanager doesn't have any solution for this, although if you decide to use the Alertmanager using Prometheus Operator, then you get automatic config reloads. We decided to keep our AM cluster independent and out of K8s, as we pointed out we have a hybrid environment.
Hope that answers your question.
- bogomipz 6y agoThanks. So is your Alertmanager a standalone node then? How do you handle HA on that if do? One other thing I wanted to ask was how many nodes are in your Victoria cluster?
- mr-karan 6y agoAlertmanager can run in a clustered mode with `--cluster*` flags. You can read more here[1]. So basically mutliple nodes can run and all Prometheus instances are configured to send all alerts to all AM instances. The alerts are deduplicated at cluster level. Victoria Metrics right now runs as a single node setup. That works out for now, because as mentioned Prometheus maintains a WAL. So even if the Remote storage is down for sometime, the core functionality of alerts isn't affected. And we've setup alerts on VM health, so if it goes down, we've to act on it and get it back up. Once that happens, all the previous data also is ingested automatically. This is something that we would surely revisit on at a later point of time, and setup a proper HA on it if needed. :) [1]: https://github.com/prometheus/alertmanager/blob/master/README.md#high-availability https://github.com/prometheus/alertmanager/blob/master/READM...