3 ms·
Right, but when I get paged... the first thing I do is look at dashboards and logs. At a sufficiently large company, looking at dashboards can be difficult bec
by jamesmishra 9y ago
Right, but when I get paged... the first thing I do is look at dashboards and logs.
At a sufficiently large company, looking at dashboards can be difficult because:
1. there are so many dashboards to look at... for so many different services
2. spurious correlations between two graphs showing unrelated events can lead you down the wrong path, if you don't confirm your dashboard-generated hypotheses with log statements or other information.
#2 can be solved by a stronger reliance on logs, http://opentracing.io/ http://opentracing.io/ style distributed tracing, and other information.
Something like this Show HN would be useful for problem #1 though.
- stephengillie 9y agoSo when paged, your first step is look at something that is difficult to look at, and possibly waste time with false correlations? If the metrics are known as important to your app, monitor specifically on those - that way you're alerted because of low memory or pooled connections or something you're watching, and can lead your team with that info. If the metrics aren't known to be important, why are you wasting your time looking at them on a dashboard? If you're paged - when it's raining is the wrong time to patch the leaky roof - it's the wrong time to debug and fix a problem in code. Those notes and todos should be pulled into next sprint triage. In the moment, just restoring service should be priority. If restoring service requires modifying code, database, routes, etc, then your testing environments and change control policies need improvement.