4 ms·

>"GitLab CI pipelines lint and validate the configurations and then upload them to an S3 bucket. There’s a sync server on the Alertmanager cluster to check for

>"GitLab CI pipelines lint and validate the configurations and then upload them to an S3 bucket. There’s a sync server on the Alertmanager cluster to check for new config and automatically reload Alertmanager in case of any config updates." I would be curious is this syncing being done by something custom they wrote or is this something built into Alertmanager now? I had look at the latest Alertmanager docs and nothing jumped out at me regarding this.

mr-karan 7y ago
Yes, we basically have a short lived periodic job on our Alertmangaer server which does a `sync` from the S3 bucket which stores the configs. If something has changed, just trigger a `docker restart alertmanager` basically. Alertmanager doesn't have any solution for this, although if you decide to use the Alertmanager using Prometheus Operator, then you get automatic config reloads. We decided to keep our AM cluster independent and out of K8s, as we pointed out we have a hybrid environment. Hope that answers your question.
- bogomipz 7y ago
  Thanks. So is your Alertmanager a standalone node then? How do you handle HA on that if do? One other thing I wanted to ask was how many nodes are in your Victoria cluster?
  - mr-karan 7y ago
    Alertmanager can run in a clustered mode with `--cluster*` flags. You can read more here[1]. So basically mutliple nodes can run and all Prometheus instances are configured to send all alerts to all AM instances. The alerts are deduplicated at cluster level. Victoria Metrics right now runs as a single node setup. That works out for now, because as mentioned Prometheus maintains a WAL. So even if the Remote storage is down for sometime, the core functionality of alerts isn't affected. And we've setup alerts on VM health, so if it goes down, we've to act on it and get it back up. Once that happens, all the previous data also is ingested automatically. This is something that we would surely revisit on at a later point of time, and setup a proper HA on it if needed. :) [1]: https://github.com/prometheus/alertmanager/blob/master/README.md#high-availability https://github.com/prometheus/alertmanager/blob/master/READM...

mr-karan 7y ago

Yes, we basically have a short lived periodic job on our Alertmangaer server which does a `sync` from the S3 bucket which stores the configs. If something has changed, just trigger a `docker restart alertmanager` basically. Alertmanager doesn't have any solution for this, although if you decide to use the Alertmanager using Prometheus Operator, then you get automatic config reloads. We decided to keep our AM cluster independent and out of K8s, as we pointed out we have a hybrid environment. Hope that answers your question.

bogomipz 7y ago
Thanks. So is your Alertmanager a standalone node then? How do you handle HA on that if do? One other thing I wanted to ask was how many nodes are in your Victoria cluster?
- mr-karan 7y ago
  Alertmanager can run in a clustered mode with `--cluster*` flags. You can read more here[1]. So basically mutliple nodes can run and all Prometheus instances are configured to send all alerts to all AM instances. The alerts are deduplicated at cluster level. Victoria Metrics right now runs as a single node setup. That works out for now, because as mentioned Prometheus maintains a WAL. So even if the Remote storage is down for sometime, the core functionality of alerts isn't affected. And we've setup alerts on VM health, so if it goes down, we've to act on it and get it back up. Once that happens, all the previous data also is ingested automatically. This is something that we would surely revisit on at a later point of time, and setup a proper HA on it if needed. :) [1]: https://github.com/prometheus/alertmanager/blob/master/README.md#high-availability https://github.com/prometheus/alertmanager/blob/master/READM...