9 ms·

Notes on the Perfidy of Dashboards

93 points by jedixit 5y ago

clipradiowallet 5y ago
From TFA... >every dashboard is a sunk cost >every dashboard is an answer to some long-forgotten question >every dashboard is an invitation to pattern-match the past >instead of interrogate the present >every dashboard gives the illusion of correlation >every dashboard dampens your thinking I disagree with this on all counts. A dashboard is a way to view multiple disparate metrics in a single place. Whether they are correlated isn't important(but it is helpful). And the author doesn't stop there... > They tend to have percentiles like 95th, 99th, 99.9th, 99.99th, etc. Which can cover over a multitude of sins. You really want a tool that allows you to see MAX and MIN, and heatmap distributions. They "tend to"? "You really want"? The author is confusing their own failures/gripes around the concept of dashboards with the world's experience with dashboards. By the end of the article, I was shocked they weren't selling something.
- alexfromapex 5y ago
  I agree with both viewpoints in certain ways, a lot of times visualization surfaces issues that would otherwise go unnoticed but I think the important point is that being able to drill down is important for context.
- _jal 5y ago
  People frequently get dashboards wrong, it is true. I like them for genuinely important things - I want klaxxons when there is a customer-facing issue, for instance. These should be simple and hard to confuse. I also like them for whatever KPIs are considered important this week. A slightly sneaky reason is that time has to be budgeted to modify the dashboard and they are very visible, so dashboards also advertise when the goalposts move. (My current employer is actually not bad about this. This lesson came from $job-1.)
- spullara 5y ago
  I mean, she is selling something. She is the founder (former CEO) of Honeycomb which is all about doing ad hoc queries rather than setting up dashboards.
  - fallat 5y ago
    Get 'em
- cratermoon 5y ago
  > A dashboard is a way to view multiple disparate metrics in a single place. This is technically correct but doesn't approach anywhere near the criticisms the article has. The deeper questions are: how did those metrics come to be collected, and why? What happened that resulted in those particular metrics being aggregated and displayed they way they are? What questions were being asked at the time the dashboards were created? > a way to view multiple disparate metrics So what? Why view them? Pretty graphs? A red/yellow/green? But to what end? This is why the statement is technically correct, but sheds no light at all on the reasons why a developer or customer support troubleshooter would care to look at the disparate metrics gathered in a dashboard. Dashboards are created in response to certain problems and events. Those problems and events may or may not be relevant some time down the road. What happens when someone in the present with a certain set of questions or problems looks at the dashboard full of metrics capturing past questions and forgets that those questions are not today's questions?
  - devchix 5y ago
    That's why good dashboards come with Title, subtitle, legend, the X and Y axis, and units. Count of packets denied from source IP, source port, last 4 hours. Average number of requests forwarded to proxy farm, distributed by server, last 7 days vs. same time last month. Who called 2049, last 24 hours.
    - arrrg 5y ago
      Yeah, but why? You are talking tactics, not strategy. What are the underlying goals?
    - cratermoon 5y ago
      Great, so I know exactly what I'm looking at. But it's not showing me anything I need to know to figure out why half the company is yelling at the team in Slack that something is terribly broken.
      - devchix 5y ago
        You are here, lat=x, long=y, bearing N 45 E at 5km/h, the air temp is -12C, it is 45min to sunset. You don't know where you should be. You are lost. Who's fault is it? Yours, or the dash?
- thebeardisred 5y ago
  Hear hear. At CoreOS I set up a number of dashboards specifically to socialize the normative behavior of our systems. Think of it as the difference between the person who just drives their car versus the person who knows what feels and sounds "normal". The latter can tell when they need an oil change because of different vibrations in the engine and how the car sounds pulling up to a stop light (because of the engine sounds being reflected back into the window by parked cars). Big surprise, it had the desired effect. I'm especially with you on the notion of disparate metrics. While correlation is not causation, it's still a useful diagnostic tool. Let's say someone in marketing walks a dashboard and and sees the following: 1) a new version has been pushed 2) customer tickets are up 20% over the number they're used to seeing Does that mean that the new version caused the tickets? No. Will that allow them to ask? Absolutely. Will that urge behavior to reach out to support and release management to see if there's an interesting story to share internally (or with the world)? Hopefully. You hit the nail on the head by calling out the absolutist / "I'm the authority on this matter" / "there is a single correct perspective" tone. Your lack of shock at the author selling something can be remedied. They have a dog in this fight: https://www.honeycomb.io/teammember/charity-majors/ https://www.honeycomb.io/teammember/charity-majors/
  - slyall 5y ago
    +1 on this. You really need something to tell you what "normal" looks like. Things like what the busy times are each day or what the normal number of requests/errors are. If you have no feel for what normal looks like on you system then it'll take longer to notice problems and longer to fix them.
rocgf 5y ago
I disagree so much with this that I wouldn't even know where to start arguing against it.
- numlock86 5y ago
  Agree. I wonder why this mindless rant even made it to the frontpage.
throwaway_2047 5y ago
The tld is apt, wtf?
SkipperCat 5y ago
Dashboards are invaluable. Humans can intake a lot of data from images and there is not better way to grok data than a graph. We've spent a lot of time building Grafana dashboards and they've been extremely helpful with debugging. It doesn't solve all problems but it certainly helps narrow down where to look. Sure, we still look at log files, use htop and a lot of other tools, but our first stop is always Grafana. I suggest the almost any book by Edward Tufte. There you'll see the beauty and value of visual information.
- tetha 5y ago
  > We've spent a lot of time building Grafana dashboards and they've been extremely helpful with debugging. It doesn't solve all problems but it certainly helps narrow down where to look. This is what I was about to write. Most of our services have 1 or 2 dashboards showing some service KPIs - for example HTTP request throughputh and response time, and also interface metrics to other sub systems - queries to postgres, messages to the message bus and so on. With dashboards like this, you can very quickly build a deduction chain of "The customer opened a ticket, well because our 75%ile of the response time went to 20 seconds, well, because our database response times spiked to an ungodly number". And then you can go to the dashboards about the database, and quickly narrow it down to the subsystem of the database - is something consuming the entire CPU of the database, is the IO blocked, is the network there slow. In the happy cases, you can go from "The cluster is down" to "our database is blocked by a query" within a minute by looking at a few boards. That's very, very powerful and valuable. And sure, at that point, the dashboards aren't useful anymore. But a map doesn't lose value because you can now see your target.
  - ethbr0 5y ago
    This is where dashboard have been most useful to me: extract and expose key metrics of the system, in as unbiased and raw of a form as possible. Or, to put it another way, looking at a dashboard should tell you reliable facts about the system, that lead you to further exploration. As the post puts it, "They’re great for getting a high level sense of system functioning, and tracking important stats over long intervals. They are a good starting point for investigations." Dashboards should not attempt to interpret anything, without being very clear about how, why, and what they're doing. Example: response time statistics vs "responsive green/red light" If it's important enough to have logic built on top of it, that's an alert, and that's something different.
landryraccoon 5y ago
Dashboards aren’t a debugging tool. They’re a QA tool. The point of the dashboard is so someone can say “hey, I’m not an engineer but new user signups sure are taking a nosedive this week. Can we get someone on this asap?” Then you can point at the dashboard and say “this is a problem”.
buremba 5y ago
The main problem is that it's not easy to drill down into a metric in most BI tools because the connection between the dashboard and the source data is usually missing. Looker is one of the first companies that target this specific issue; you transform, model the data, define your metrics before creating your first dashboard. It takes too much effort (also money) to create a (basic?) dashboard because it's not just a "dashboard". Instead, as data analysts, we usually want to write a bunch of SQL queries, create charts from them and expose the data to our business stakeholders. While they can see the underlying SQL queries of the metrics, it's not easy for them to modify these SQL queries, so they often get lost. The dashboards have a long tail. For me, you need to get these four steps done beforehand: 1. Move all the company data into a data warehouse and use it as the single source of truth. 2. Model the data with a transformation tool such as dbt and Airflow. 3. Define metrics in one place on top of your data models in a collaborative way. (This layer is new, and we're tapping it at https://metriql.com https://metriql.com) 4. Use an interactive BI tool that lets you create dashboards on top of these metrics with drill-down capability.
reureu 5y ago
I spent the past four years working as a data scientist for a healthcare company on population health initiatives, and started building out a body of research around how to engage clinicians using data (among other things, through dashboards). That's a bit different than the article, but one of my key learnings was that dashboards are often incredibly ineffective and only promulgated by well-intentioned engineers, based on what they would want to see if they were a clinician. I worked with a behavioral economist, and we started running RCTs looking at different approaches to sharing data, and found that dashboards led to less engagement, when there was engagement it was more likely to drive ineffective interventions, and generally our dashboard groups had worse patient outcomes. Non-dashboard approaches had 20x better engagement and anywhere from 20-100% better patient outcomes (depending on the condition). Unfortunately, both of us left the company when a bunch of engineers moved in, scoffed at our work, and immediately said "doctors need to be able to slice and dice their data" -- which, by every measure we had tested, is simply not true. But the "mission control" style thinking, where you have tons of dials and numbers flashing on a screen, pervaded because it "feels" like the right answer despite being the objectively wrong one.
- lazyasciiart 5y ago
  Sounds interesting, is any of that body of research available publicly?
  - reureu 5y ago
    Some, not all. I'm hesitant to directly post links on here since it will out both me and the company I worked for. If you're interested in this kind of work, you should check out the conference proceedings for AMIA (amia.org)
- kfarr 5y ago
  Curious if you have any examples of "Non-dashboard approaches" to compare and contrast?
  - reureu 5y ago
hbosch 5y ago
I read this and still don't know exactly what the author is asking for. Is an "exploratory, queryable interface" not exactly how you would describe a modern dashboard?
- tcard 5y ago
  The author is asking for, and actually built, https://honeycomb.io https://honeycomb.io.
- lmeyerov 5y ago
  They want you to use their interactive dashboards backed by their database using features their PMs prioritized, like wide columns and drill-downs. But you're right, other systems do support these . Ex: Splunk has done wide columns with index-on-write columnar analytics and point-and-click pivoting/drilldowns since ~day 1. So, as they add features, they get very definitional on each one. Most startup people (myself included!) have a lot to learn from their successful developer/IT marketing style.
mdoms 5y ago
This is a very stupid take from the author just intended to drum up controversy.
tomrod 5y ago
I disagree strongly with the content. But man, why is it designers thing light gray background with mid-gray text is a good idea? Almost unreadable for me.
ziggus 5y ago
Interesting that most of this rant is focused on static dashboards - does anyone really use static dashboards? I can see if you're stuck using 1998 technology like Excel that you might throw together some static charts/graphs/KPIs, but even in the most primitive dashboard tools available now (PowerBI, I'm looking at you) the default is a dashboard that's tied to a 'live' dataset.
- cwillu 5y ago
  The "static" refers to the structure of the data, not the freshness/liveness of it.
iamthepieman 5y ago
The most effective dashboards I've worked on have been glorified interactive spreadsheets with the ability to graph the data in various ways. High density, not pretty. Sometimes there was a map if the data was geospatial. Also, does anyone remember the covid dashboard from John Hopkins. That one is pretty useful.
devchix 5y ago
1. Set up some strawmen about dashboards 1a. Define dashboards in a way that suits your argument 2. Knock em down! 3. Profit? This, https://status.cloud.google.com/ https://status.cloud.google.com/, is a static dashboard. One can make up one's mind about its usefulness and function. A blunt example, but nobody I know "debugs" using dashboards. Dashboard is automation, if you're against dashboards, you're against automation of repetitive, boring, long-winded, laborious, hard-to-remember, tasks. Dashboards aren't sacred, once one outlives its usefulness, get rid of it.
brazzy 5y ago
> That’s not debugging, that’s pattern-matching. That’s … eyeball racing. Um...yes. And that is a very good thing. Because if there is anything the human brain is good at, it's pattern matching. Especially on visual data. It's an extremely quick and efficient way to find out where to start the detailed debugging. And there is a lot of value in that.
- someguydave 5y ago
  the brain is also good at fooling itself
simonw 5y ago
I have trouble trusting any dashboard if it doesn't have the equivalent of a "view source" button that lets me see what data sources were used for it and how they were manipulated. Sadly dashboard systems that encourage this are extremely rare.
jldugger 5y ago
Honestly, the complaint here seems to be less about dashboards and more about the data behind them. Static dashboards sound like timeseries backends, where the data is pre-aggregated (graphite / statsd, prometheus). You can't really drill down into the metrics, or can only drill down into preplanned dimensions. Grafana is a commonly used dashboarding frontend here. Dynamic dashboards, in contrast, are dynamically aggregating data. More akin to structured logging, or maybe splunk / ELK. You have granular data, and write queries to extract, filter, and aggregate on demand. Tableau, PowerBI, Apache Superset all compete in this space. But by focusing on the dashboard angle, the reader doesn't think to hardly about why they're different, and also why you might prefer one over the other. TSDB like Prometheus are very fast, and if you focus on collecting aggregate data, allow you to collect a lot more metrics, or sample much faster. You're probably not logging in the TSDB any labels associated with UserAgent strings, or screen size your mobile app got, etc. By paying the price in dimensionality, you get much faster queries at lower cost. I'll let you guess which type of backend Charity's startup represents. Both have a place. I've been able to build canary dashboards that work quite well using both backends, as a proof of concept that something like Kayenta is feasible for my team. In fact, high dimensionality works against you in release engineering. The more dimensions you can compare across, the higher chance for a false positive, and the more investigations engineers have to do to rule them out. Worse, there are often confounding variables you need to go hunting for, and the dashboard won't find them for you. And execs absolutely don't want to have to care about the complex causality chain you need to model. They want 'a single number' to improve on over time. They don't want a dashboard to dive in and analyze on ten dimensions. They want to see their chosen KPIs going to the right and up. Fundamentally, the dashboard is less important than your audience.
nixpulvis 5y ago
The last paragraph really got me thinking about regression. > raise your hand if you’ve ever been in an incident review where one of the follow up tasks was, “create a dashboard that will help us find this next time” As a disciplined software engineer, I aspire to have each and every user facing bug captured first as an automated test. This helps form trust in the software. Ideally the users themselves can choose to write the tests and submit them for me. This is akin to metrics. I completely agree, system-KPI metrics should be relevant and short. But there's nothing stopping you from collecting an archive of previous data experiment formulas.

93 points by jedixit 5y ago

From TFA... >every dashboard is a sunk cost >every dashboard is an answer to some long-forgotten question >every dashboard is an invitation to pattern-match the past >instead of interrogate the present >every dashboard gives the illusion of correlation >every dashboard dampens your thinking I disagree with this on all counts. A dashboard is a way to view multiple disparate metrics in a single place. Whether they are correlated isn't important(but it is helpful). And the author doesn't stop there... > They tend to have percentiles like 95th, 99th, 99.9th, 99.99th, etc. Which can cover over a multitude of sins. You really want a tool that allows you to see MAX and MIN, and heatmap distributions. They "tend to"? "You really want"? The author is confusing their own failures/gripes around the concept of dashboards with the world's experience with dashboards. By the end of the article, I was shocked they weren't selling something.

alexfromapex 5y ago
I agree with both viewpoints in certain ways, a lot of times visualization surfaces issues that would otherwise go unnoticed but I think the important point is that being able to drill down is important for context.
_jal 5y ago
People frequently get dashboards wrong, it is true. I like them for genuinely important things - I want klaxxons when there is a customer-facing issue, for instance. These should be simple and hard to confuse. I also like them for whatever KPIs are considered important this week. A slightly sneaky reason is that time has to be budgeted to modify the dashboard and they are very visible, so dashboards also advertise when the goalposts move. (My current employer is actually not bad about this. This lesson came from $job-1.)
spullara 5y ago
I mean, she is selling something. She is the founder (former CEO) of Honeycomb which is all about doing ad hoc queries rather than setting up dashboards.
- fallat 5y ago
  Get 'em
cratermoon 5y ago
> A dashboard is a way to view multiple disparate metrics in a single place. This is technically correct but doesn't approach anywhere near the criticisms the article has. The deeper questions are: how did those metrics come to be collected, and why? What happened that resulted in those particular metrics being aggregated and displayed they way they are? What questions were being asked at the time the dashboards were created? > a way to view multiple disparate metrics So what? Why view them? Pretty graphs? A red/yellow/green? But to what end? This is why the statement is technically correct, but sheds no light at all on the reasons why a developer or customer support troubleshooter would care to look at the disparate metrics gathered in a dashboard. Dashboards are created in response to certain problems and events. Those problems and events may or may not be relevant some time down the road. What happens when someone in the present with a certain set of questions or problems looks at the dashboard full of metrics capturing past questions and forgets that those questions are not today's questions?
- devchix 5y ago
  That's why good dashboards come with Title, subtitle, legend, the X and Y axis, and units. Count of packets denied from source IP, source port, last 4 hours. Average number of requests forwarded to proxy farm, distributed by server, last 7 days vs. same time last month. Who called 2049, last 24 hours.
  - arrrg 5y ago
    Yeah, but why? You are talking tactics, not strategy. What are the underlying goals?
  - cratermoon 5y ago
    Great, so I know exactly what I'm looking at. But it's not showing me anything I need to know to figure out why half the company is yelling at the team in Slack that something is terribly broken.
    - devchix 5y ago
      You are here, lat=x, long=y, bearing N 45 E at 5km/h, the air temp is -12C, it is 45min to sunset. You don't know where you should be. You are lost. Who's fault is it? Yours, or the dash?
thebeardisred 5y ago
Hear hear. At CoreOS I set up a number of dashboards specifically to socialize the normative behavior of our systems. Think of it as the difference between the person who just drives their car versus the person who knows what feels and sounds "normal". The latter can tell when they need an oil change because of different vibrations in the engine and how the car sounds pulling up to a stop light (because of the engine sounds being reflected back into the window by parked cars). Big surprise, it had the desired effect. I'm especially with you on the notion of disparate metrics. While correlation is not causation, it's still a useful diagnostic tool. Let's say someone in marketing walks a dashboard and and sees the following: 1) a new version has been pushed 2) customer tickets are up 20% over the number they're used to seeing Does that mean that the new version caused the tickets? No. Will that allow them to ask? Absolutely. Will that urge behavior to reach out to support and release management to see if there's an interesting story to share internally (or with the world)? Hopefully. You hit the nail on the head by calling out the absolutist / "I'm the authority on this matter" / "there is a single correct perspective" tone. Your lack of shock at the author selling something can be remedied. They have a dog in this fight: https://www.honeycomb.io/teammember/charity-majors/ https://www.honeycomb.io/teammember/charity-majors/
- slyall 5y ago
  +1 on this. You really need something to tell you what "normal" looks like. Things like what the busy times are each day or what the normal number of requests/errors are. If you have no feel for what normal looks like on you system then it'll take longer to notice problems and longer to fix them.

rocgf 5y ago

I disagree so much with this that I wouldn't even know where to start arguing against it.

numlock86 5y ago
Agree. I wonder why this mindless rant even made it to the frontpage.

throwaway_2047 5y ago

The tld is apt, wtf?

SkipperCat 5y ago

Dashboards are invaluable. Humans can intake a lot of data from images and there is not better way to grok data than a graph. We've spent a lot of time building Grafana dashboards and they've been extremely helpful with debugging. It doesn't solve all problems but it certainly helps narrow down where to look. Sure, we still look at log files, use htop and a lot of other tools, but our first stop is always Grafana. I suggest the almost any book by Edward Tufte. There you'll see the beauty and value of visual information.

tetha 5y ago
> We've spent a lot of time building Grafana dashboards and they've been extremely helpful with debugging. It doesn't solve all problems but it certainly helps narrow down where to look. This is what I was about to write. Most of our services have 1 or 2 dashboards showing some service KPIs - for example HTTP request throughputh and response time, and also interface metrics to other sub systems - queries to postgres, messages to the message bus and so on. With dashboards like this, you can very quickly build a deduction chain of "The customer opened a ticket, well because our 75%ile of the response time went to 20 seconds, well, because our database response times spiked to an ungodly number". And then you can go to the dashboards about the database, and quickly narrow it down to the subsystem of the database - is something consuming the entire CPU of the database, is the IO blocked, is the network there slow. In the happy cases, you can go from "The cluster is down" to "our database is blocked by a query" within a minute by looking at a few boards. That's very, very powerful and valuable. And sure, at that point, the dashboards aren't useful anymore. But a map doesn't lose value because you can now see your target.
- ethbr0 5y ago
  This is where dashboard have been most useful to me: extract and expose key metrics of the system, in as unbiased and raw of a form as possible. Or, to put it another way, looking at a dashboard should tell you reliable facts about the system, that lead you to further exploration. As the post puts it, "They’re great for getting a high level sense of system functioning, and tracking important stats over long intervals. They are a good starting point for investigations." Dashboards should not attempt to interpret anything, without being very clear about how, why, and what they're doing. Example: response time statistics vs "responsive green/red light" If it's important enough to have logic built on top of it, that's an alert, and that's something different.

landryraccoon 5y ago

Dashboards aren’t a debugging tool. They’re a QA tool. The point of the dashboard is so someone can say “hey, I’m not an engineer but new user signups sure are taking a nosedive this week. Can we get someone on this asap?” Then you can point at the dashboard and say “this is a problem”.

buremba 5y ago

The main problem is that it's not easy to drill down into a metric in most BI tools because the connection between the dashboard and the source data is usually missing. Looker is one of the first companies that target this specific issue; you transform, model the data, define your metrics before creating your first dashboard. It takes too much effort (also money) to create a (basic?) dashboard because it's not just a "dashboard". Instead, as data analysts, we usually want to write a bunch of SQL queries, create charts from them and expose the data to our business stakeholders. While they can see the underlying SQL queries of the metrics, it's not easy for them to modify these SQL queries, so they often get lost. The dashboards have a long tail. For me, you need to get these four steps done beforehand: 1. Move all the company data into a data warehouse and use it as the single source of truth. 2. Model the data with a transformation tool such as dbt and Airflow. 3. Define metrics in one place on top of your data models in a collaborative way. (This layer is new, and we're tapping it at https://metriql.com https://metriql.com) 4. Use an interactive BI tool that lets you create dashboards on top of these metrics with drill-down capability.

reureu 5y ago

I spent the past four years working as a data scientist for a healthcare company on population health initiatives, and started building out a body of research around how to engage clinicians using data (among other things, through dashboards). That's a bit different than the article, but one of my key learnings was that dashboards are often incredibly ineffective and only promulgated by well-intentioned engineers, based on what they would want to see if they were a clinician. I worked with a behavioral economist, and we started running RCTs looking at different approaches to sharing data, and found that dashboards led to less engagement, when there was engagement it was more likely to drive ineffective interventions, and generally our dashboard groups had worse patient outcomes. Non-dashboard approaches had 20x better engagement and anywhere from 20-100% better patient outcomes (depending on the condition). Unfortunately, both of us left the company when a bunch of engineers moved in, scoffed at our work, and immediately said "doctors need to be able to slice and dice their data" -- which, by every measure we had tested, is simply not true. But the "mission control" style thinking, where you have tons of dials and numbers flashing on a screen, pervaded because it "feels" like the right answer despite being the objectively wrong one.

lazyasciiart 5y ago
Sounds interesting, is any of that body of research available publicly?
- reureu 5y ago
  Some, not all. I'm hesitant to directly post links on here since it will out both me and the company I worked for. If you're interested in this kind of work, you should check out the conference proceedings for AMIA (amia.org)
kfarr 5y ago
Curious if you have any examples of "Non-dashboard approaches" to compare and contrast?
- reureu 5y ago

hbosch 5y ago

I read this and still don't know exactly what the author is asking for. Is an "exploratory, queryable interface" not exactly how you would describe a modern dashboard?

tcard 5y ago
The author is asking for, and actually built, https://honeycomb.io https://honeycomb.io.
lmeyerov 5y ago
They want you to use their interactive dashboards backed by their database using features their PMs prioritized, like wide columns and drill-downs. But you're right, other systems do support these . Ex: Splunk has done wide columns with index-on-write columnar analytics and point-and-click pivoting/drilldowns since ~day 1. So, as they add features, they get very definitional on each one. Most startup people (myself included!) have a lot to learn from their successful developer/IT marketing style.

mdoms 5y ago

This is a very stupid take from the author just intended to drum up controversy.

tomrod 5y ago

I disagree strongly with the content. But man, why is it designers thing light gray background with mid-gray text is a good idea? Almost unreadable for me.

ziggus 5y ago

Interesting that most of this rant is focused on static dashboards - does anyone really use static dashboards? I can see if you're stuck using 1998 technology like Excel that you might throw together some static charts/graphs/KPIs, but even in the most primitive dashboard tools available now (PowerBI, I'm looking at you) the default is a dashboard that's tied to a 'live' dataset.

cwillu 5y ago
The "static" refers to the structure of the data, not the freshness/liveness of it.

iamthepieman 5y ago

The most effective dashboards I've worked on have been glorified interactive spreadsheets with the ability to graph the data in various ways. High density, not pretty. Sometimes there was a map if the data was geospatial. Also, does anyone remember the covid dashboard from John Hopkins. That one is pretty useful.

devchix 5y ago

1. Set up some strawmen about dashboards 1a. Define dashboards in a way that suits your argument 2. Knock em down! 3. Profit? This, https://status.cloud.google.com/ https://status.cloud.google.com/, is a static dashboard. One can make up one's mind about its usefulness and function. A blunt example, but nobody I know "debugs" using dashboards. Dashboard is automation, if you're against dashboards, you're against automation of repetitive, boring, long-winded, laborious, hard-to-remember, tasks. Dashboards aren't sacred, once one outlives its usefulness, get rid of it.

brazzy 5y ago

> That’s not debugging, that’s pattern-matching. That’s … eyeball racing. Um...yes. And that is a very good thing. Because if there is anything the human brain is good at, it's pattern matching. Especially on visual data. It's an extremely quick and efficient way to find out where to start the detailed debugging. And there is a lot of value in that.

someguydave 5y ago
the brain is also good at fooling itself

simonw 5y ago

I have trouble trusting any dashboard if it doesn't have the equivalent of a "view source" button that lets me see what data sources were used for it and how they were manipulated. Sadly dashboard systems that encourage this are extremely rare.

jldugger 5y ago

Honestly, the complaint here seems to be less about dashboards and more about the data behind them. Static dashboards sound like timeseries backends, where the data is pre-aggregated (graphite / statsd, prometheus). You can't really drill down into the metrics, or can only drill down into preplanned dimensions. Grafana is a commonly used dashboarding frontend here. Dynamic dashboards, in contrast, are dynamically aggregating data. More akin to structured logging, or maybe splunk / ELK. You have granular data, and write queries to extract, filter, and aggregate on demand. Tableau, PowerBI, Apache Superset all compete in this space. But by focusing on the dashboard angle, the reader doesn't think to hardly about why they're different, and also why you might prefer one over the other. TSDB like Prometheus are very fast, and if you focus on collecting aggregate data, allow you to collect a lot more metrics, or sample much faster. You're probably not logging in the TSDB any labels associated with UserAgent strings, or screen size your mobile app got, etc. By paying the price in dimensionality, you get much faster queries at lower cost. I'll let you guess which type of backend Charity's startup represents. Both have a place. I've been able to build canary dashboards that work quite well using both backends, as a proof of concept that something like Kayenta is feasible for my team. In fact, high dimensionality works against you in release engineering. The more dimensions you can compare across, the higher chance for a false positive, and the more investigations engineers have to do to rule them out. Worse, there are often confounding variables you need to go hunting for, and the dashboard won't find them for you. And execs absolutely don't want to have to care about the complex causality chain you need to model. They want 'a single number' to improve on over time. They don't want a dashboard to dive in and analyze on ten dimensions. They want to see their chosen KPIs going to the right and up. Fundamentally, the dashboard is less important than your audience.

nixpulvis 5y ago

The last paragraph really got me thinking about regression. > raise your hand if you’ve ever been in an incident review where one of the follow up tasks was, “create a dashboard that will help us find this next time” As a disciplined software engineer, I aspire to have each and every user facing bug captured first as an automated test. This helps form trust in the software. Ideally the users themselves can choose to write the tests and submit them for me. This is akin to metrics. I completely agree, system-KPI metrics should be relevant and short. But there's nothing stopping you from collecting an archive of previous data experiment formulas.