4 ms·
Dashboards are largely dangerous. They promote a "Star Trek Bridge" theater - a play where one actor looks at a screen, becomes concerned, and takes action. Oft
by stephengillie 9y ago
Dashboards are largely dangerous. They promote a "Star Trek Bridge" theater - a play where one actor looks at a screen, becomes concerned, and takes action. Often this action will include a report to a senior officer.
Business leaders often enjoy seeing their engineers act out this scene. It's gratifying to see someone take expected action, and gratifying to simply look at your direct reports to see if they're working.
But this theater play requires engineers to be idly starting at dashboards first, so they are in the right place to see an issue and take action. This leads to bored engineers, complacency, and delays issue resolution. It's also inefficient to pay people to be bored.
Instead of having a dashboard display a subset of metrics, have alerting configured on these. Pager duty notifies the same if you're in another app, another castle, another room, or another state - you can pay people to do other things instead of staring at a dashboard all day.
A big TV full of metrics is a prop that your actors and engineers will ignore.
- jstanley 9y agoYou're right, it's inefficient to pay people to look at the screens. But when something's gone wrong and you want to know how to fix it, having your key indicators all in one place can be a big time saver.
- r3bl 9y agoAnd let us not even mention that those key indicators could trigger all sorts of alarms (like email notifications and SMS notifications) when something goes wrong.
- stephengillie 9y agoPut monitoring ON your key indicators, so your alerts tell you what's wrong, instead of asking you to figure it out. It can be intellectually gratifying to use brute-intelligence to "save the day" during an urgent incident, but those heroics are part of the "Star Trek Bridge" scene play, and wastes crucial time.
- jamesmishra 9y agoRight, but when I get paged... the first thing I do is look at dashboards and logs. At a sufficiently large company, looking at dashboards can be difficult because: 1. there are so many dashboards to look at... for so many different services 2. spurious correlations between two graphs showing unrelated events can lead you down the wrong path, if you don't confirm your dashboard-generated hypotheses with log statements or other information. #2 can be solved by a stronger reliance on logs, http://opentracing.io/ http://opentracing.io/ style distributed tracing, and other information. Something like this Show HN would be useful for problem #1 though.
- stephengillie 9y agoSo when paged, your first step is look at something that is difficult to look at, and possibly waste time with false correlations? If the metrics are known as important to your app, monitor specifically on those - that way you're alerted because of low memory or pooled connections or something you're watching, and can lead your team with that info. If the metrics aren't known to be important, why are you wasting your time looking at them on a dashboard? If you're paged - when it's raining is the wrong time to patch the leaky roof - it's the wrong time to debug and fix a problem in code. Those notes and todos should be pulled into next sprint triage. In the moment, just restoring service should be priority. If restoring service requires modifying code, database, routes, etc, then your testing environments and change control policies need improvement.