4 ms·
You need to implement a deadman switch. For example if using Prometheus you can configure it to access an HTTP endpoint of a deadman switch service every X seco
by what2 2y ago
You need to implement a deadman switch. For example if using Prometheus you can configure it to access an HTTP endpoint of a deadman switch service every X seconds. When that service detects you have not accessed it in some time it will alert you.
For example: https://blog.ediri.io/how-to-set-up-a-dead-mans-switch-in-prometheus https://blog.ediri.io/how-to-set-up-a-dead-mans-switch-in-pr...
- fullstop 2y agoAnd now you need to monitor the deadman switch service!
- Joker_vD 2y agoOh, you just run two instances of those and point them at each other.
- hinkley 2y agoEven number so that everything is symmetrical.
- Temporary_31337 2y agoYou want odd numbers (n+1) so you can have a quorum in case of network partition.
- hinkley 2y agoI think you're missing the vibe (sarcasm).
- jasongill 2y agoAnd then a third instance in case those both go down at the same time, and a fourth just in case there's a major worldwide outage.... it's monitoring instances all the way down
- Spivak 2y agoTypically that's why people use dead man's switch as a service. You don't assume that they'll never go down, but you're paying for someone who's failures are very likely uncorrelated with your own.