3 ms·
Not if you're monitoring your systems. True engineering means you have a feedback loop into your process. Otherwise you're 'just a programmer'.... :) You shoul
by exabrial 3y ago
Not if you're monitoring your systems. True engineering means you have a feedback loop into your process. Otherwise you're 'just a programmer'.... :)
You should have Telegraph running on every system you have collecting stats and sending it to InfluxDb with sample rate around 15s and +- 5s of jitter.
Each critical process should have an uptime counter. You should create both Deadman alerts (No contact for 5m) and Uptime alerts (uptime falls below 60s).
s/Telgraf|InfluxDb/YourMonitoringSystemOfChoice/
Bonus: If you're running JVMs, you can expose _all_ of the JVMs stats to Telegraf via Jolokia, which can run as a Java Agent and requires 0 code changes. The JVM has very very detailed stats about it's health available by default, making problems very visible (if you look for them).
- glenjamin 3y agoIf you're only monitoring user-facing symptoms, and the application restarts quickly enough to not cause a blip in service, then: a) It's highly likely that your monitoring is not going to spot an occasionally restarting service b) It's probably a good choice not to waste cycles on that sort of monitoring
- exabrial 3y agoBoth of your assumptions are completely incorrect. One should _never_ "monitor" a system like that. Fortunately, the wise Unix wizzards of yesteryear already thought of the solution: uptime counters. An uptime counter counts the number of milliseconds a process is alive. So if it resets to a value lower than the previous recorded one, it's a blue star thats impossible to miss.