3 ms·

No plans currently, but a self hosted option seems reasonable. Although, most people like their emergency tech not hosted on their own tech :)

aalbertson 4y ago
Being not hosted on my same tech is one thing, still being self hosted so I can externalize it for a federal implementation is another. Definitely needs to be self hosted for ALL components. :)
- matryer 4y ago
  Yeah makes sense. I'll add your vote to the list :)
jrockway 4y ago
I dunno, I don't really mind self-hosting monitoring infrastructure. I basically pay for a website uptime checker to check that Alertmanager is working. If Alertmanager is down, obviously you have to manually check to see what else is down, but it doesn't fail open. I wrote a little glue to make this straightforward for anyone else who uses Prometheus/Alertmanager: https://github.com/jrockway/alertmanager-status https://github.com/jrockway/alertmanager-status This ensures that the website check checks the health of the whole alerting pipeline; Prometheus has an always firing alert, Alertmanager is set to send that alert to alertmanager-status, and alertmanager-status starts failing its external health check if it isn't seeing that alert firing at the configured interval. If one of [Prometheus, Alertmanager, alertmanager-status] fails, then your website health check fails.
trog 4y ago
> No plans currently, but a self hosted option seems reasonable. Although, most people like their emergency tech not hosted on their own tech :) FWIW our core application is hosted in AWS but we maintain our own Grafana infrastructure independently. So it's not hosted on our own tech, per se, though we're still responsible for keeping it online. This looks great & would also love to see a self-hosted option. Honestly the more stuff like this that gets rolled into the OSS Grafana it actually makes me both more likely to try it and then more likely to eventually end up on the managed Grafana Cloud, as I will inevitably get sick of trying to maintain our own separate infra & the business case for centralising in Cloud makes more and more sense.

aalbertson 4y ago

Being not hosted on my same tech is one thing, still being self hosted so I can externalize it for a federal implementation is another. Definitely needs to be self hosted for ALL components. :)

matryer 4y ago
Yeah makes sense. I'll add your vote to the list :)

jrockway 4y ago

I dunno, I don't really mind self-hosting monitoring infrastructure. I basically pay for a website uptime checker to check that Alertmanager is working. If Alertmanager is down, obviously you have to manually check to see what else is down, but it doesn't fail open. I wrote a little glue to make this straightforward for anyone else who uses Prometheus/Alertmanager: https://github.com/jrockway/alertmanager-status https://github.com/jrockway/alertmanager-status This ensures that the website check checks the health of the whole alerting pipeline; Prometheus has an always firing alert, Alertmanager is set to send that alert to alertmanager-status, and alertmanager-status starts failing its external health check if it isn't seeing that alert firing at the configured interval. If one of [Prometheus, Alertmanager, alertmanager-status] fails, then your website health check fails.

trog 4y ago

> No plans currently, but a self hosted option seems reasonable. Although, most people like their emergency tech not hosted on their own tech :) FWIW our core application is hosted in AWS but we maintain our own Grafana infrastructure independently. So it's not hosted on our own tech, per se, though we're still responsible for keeping it online. This looks great & would also love to see a self-hosted option. Honestly the more stuff like this that gets rolled into the OSS Grafana it actually makes me both more likely to try it and then more likely to eventually end up on the managed Grafana Cloud, as I will inevitably get sick of trying to maintain our own separate infra & the business case for centralising in Cloud makes more and more sense.