2 ms·
I agree with collecting data. Monitoring tooling is extremely important. With microservices you need metrics (requests, errors, latency), logs, traces. Every se
by raffraffraff 4y ago
I agree with collecting data. Monitoring tooling is extremely important. With microservices you need metrics (requests, errors, latency), logs, traces. Every service will have specific signals that the service owner should understand.
The SRE book is an OK idea but to be honest, it's hard to get developers to focus on Google-level theory of reliable engineering via error budgets when they're so operationally immature that nobody is oncall (so everybody is oncall). Sounds like the gap is too wide. If these alerts are 50% "customer reported an incident" instead of a SLI, then that SRE book could spawn a cargo cult that creates work for everybody and achieves nothing because the gap is too wide. I've seen clueless managers latch onto theory that's too far ahead of the stone-age. Like "Hey guys, this month I want everybody to record their SLOs", when nobody is even recording metrics because they don't know how, or what to record.
I'd do extremely boring stuff like keep a spreadsheet of incidents with columns for severity, trigger (eg release, operator error, customer load, hidden bug, system error), root cause (eg internal software bug, bad config, system resources etc). If patterns emerge, they should point to gaps in knowledge, testing, automation, monitoring etc. If there's no pattern at all, then your engineering org is in a bad way. When an engineering department is extremely dysfunctional, it needs external help. Or sometimes it needs a clueless manager / lead to quit, allowing suppressed talent too take over and fix stuff.