4 ms·

I'll enjoy watching this thread evolve. Some thoughts from my experience: - Everyone asks to translate simpler spreadsheets and Excel charts/graphs into dashbo

by grvdrm 3y ago

I'll enjoy watching this thread evolve. Some thoughts from my experience: - Everyone asks to translate simpler spreadsheets and Excel charts/graphs into dashboards in your BI tool of choice. As soon as it's there, they'll ask you why they can't export to manage the data themselves. This vicious cycle can sometimes be stopped but is a slow-motion drag on productivity in lots of orgs. - Build in validations, and/or work on ways to check the dashboard. Dashboards sometimes put their builders and consumers on auto-pilot. The dashboard "must be right" but could easily have a bug or inaccuracy for weeks/months/etc. that isn't obvious without some external validation. - The dashboard never has the "right" metrics - users will continue asking for changes. Be your best advocate and say no as a way of understanding the importance of the ask - Related: always ask why about everything you're building into or modifying in dashboards. Business users often ask for things without an ounce of rationale. - Related: taking away is harder than not doing at all! Finally, I think most dashboards miss one fundamental point. Imagine you're the CEO/COO and you've got this beautiful 3 or 4-chart dashboard in front of you. What should you know about what you're seeing? What's the succinct summary? I like building in spots to write 2-3 sentence executive summaries.

mushufasa 3y ago
> Finally, I think most dashboards miss one fundamental point. Imagine you're the CEO/COO and you've got this beautiful 3 or 4-chart dashboard in front of you. What should you know about what you're seeing? What's the succinct summary? Having been on both sides of this, I think the challenge is that the CEO/COO's job is to figure out "what should we do about this?," which is the right approach to coming up with that summary (it's not just "here's a text version of the chart"). And the corollary challenge is that, in most cases, non-technical people with domain knowledge are the ones who need to produce the analysis: so any feature incomplete dashboard is going to stymie them and any general framework that requires a technical person to step in for code or configuration is going to slow the process to a crawl. It's the rule (not the exception) that (especially if things are going poorly) the next step is asking more questions, which involves investigating something else in more detail. A dashboard, however pretty, is as useless as a doorknob if it doesn't have the needed information. I have found that dashboards per say are always great as the high-level KPI trackers, like the things you would consider hanging on a wall in an office (e.g. "revenue growth this month" or "new customers acquired"). You'll always want to know that information, and many people of unrelated departments need to have that information shared to them. The other helpful area is a deep-dive domain-specific analytics programs, like for example Google Analytics, where it has a very full featureset for non-technical marketing people to go in and drill down to answer questions. The UI/UX designers of that product have spent years honing and A/B testing which types of graphs to show where, and mapping out how to have people click around to find what they're looking for, to the point it is pretty easy for non-technical people. They even have courses and certifications on how to use the system. Organizations that try to internally build a feature-complete system like google analytics for a specific domain need to consider it like building an entire software product (even if there's a general low/no-code BI SaaS to assist) because you'll need collaboration between general technical experts and non-technical stakeholders with changing and vague requirements. It can be done, but likely only with years of investment and UI/UX research, just like any other software product that solves a domain problem well. In practice: millions of dollars. Technologists often forget that Excel *is* a turing complete programming language (and it's a functional programming paradigm too!). If an org is not committed to spend years and millions of dollars on deep dive analytics for a specific domain, the right choice is almost always using a commercial analytics system for that domain that costs less than the internal build, or embracing the trusty spreadsheet.
- itsoktocry 3y ago
  >I think the challenge is that the CEO/COO's job is to figure out "what should we do about this? Totally agree. I'd even go a little further and say the business is in trouble if the CEO doesn't know "what we should do about this". It's the CEOs job to know those things, and it's the data team's job to provide the tools to make those decisions easier, faster and better.
  - grvdrm 3y ago
    > Totally agree. I'd even go a little further and say the business is in trouble if the CEO doesn't know "what we should do about this". It's the CEOs job to know those things, and it's the data team's job to provide the tools to make those decisions easier, faster and better. I agree with one modification. Also the CEO's job to empower folks to pitch what they think the CEO should know too. I've worked in plenty of successful shops where the CEO's answer to "what should we do" is "I'm not sure yet - what do you think?" - and that is a golden opportunity to show your chops if you've got the opportunity
  - yabbs 3y ago
    It's my experience that you get easier, faster or better. Choose any 1, rarely 2, never 3.
  - dgudkov 3y ago
    People frequently overestimate the role of data in decision-making. Metrics, numbers and other quantitative information don't tell the full story. For a CEO to make decisions, the full picture must include qualitative information - risks, opportunities, market events, competitor's actions, etc. Metrics are just part of the full picture. Far less significant than many BI developers and "data teams" tend to think.
clusterhacks 3y ago
"Business users often ask for things without an ounce of rationale." Deserves extra upvotes just for this statement. This has always been painful to me working in the data analysis and reporting space. When I get many requests for dashboards or reports that lack an answer to the question of "how will this be used?", I seem to find the requesting groups are cost-centers in the larger organization and are somewhat obsessed with processes and procedures. This is rarely a good group to build a career with . . .
- proamdev123 3y ago
  As a business user myself, often times we are simply responding to an edict from Leadership that amounts to: “You guys should have some dashboards!” Other times it’s because a supervisor or internal customer likes to see certain things on a chart, and putting the chart in Power BI/Tableau/other tool will make them prettier than Excel charts. Very few people, starting from the top on down, have a good understanding that dashboards mean very little in and of themselves.
conductr 3y ago
I work closely on BI projects but from a finance perspective. The concept I like to explain to the BI teams is that the dashboard is always just a snapshot of “what” is happening. But the underlying base level data is always needed to understand “why” it’s happening. And without the why, there’s no actual intelligence gain. Take a metric like Average Order Value (AOV). It may be ; total sales / order quantity. But as that metric is used it’s often being compared to something like last year, last month, or a plan and anyone interested in that number is really interested in understanding the “why” it has changed from some other point in time/scenario. For that, you actually need to bring in line item details behind orders as each order has multiple products/skus and they likely sold at different prices from a year ago or what was expected in a plan. An analysis of this has a name, price-volume-mix analysis or PVM. I always seem to have to explain this to BI teams when I join a new company and am seeking data. I’m currently going through it with a BI team, that apparently the BI tool wouldn’t store this information. It’s like it only stores aggregate values so it’s not even possible to get base level data for analysis (without major architectural changes). I don’t know if that’s normal in BI or was an implementation decision at some point but I’ve come across this same thing on a handful of companies and as I said I really have to drive this concept for those teams. When I ask of it I’m usually met with a “why would you need that info / give us a use case”. Which means, the don’t even understand how un-intelligent their BI tool is or why the execs likely aren’t feeling like investing in BI has been worthwhile (eg. Ever build a dashboard that then goes unused? I probably wasn’t perceived as useful for some reason like this). This could be more concise put as, understanding your end users needs. Understand the difference of what people ask for is often different than what they need. If they ask for AOV metrics, they’re really saying “I need to understand AOV” and that’s done via PVM analysis.
- grvdrm 3y ago
  Similarly titled towards finance. I specialize in what I'll call decision analytics for insurance underwriters. > Which means, the don’t even understand how un-intelligent their BI tool is or why the execs likely aren’t feeling like investing in BI has been worthwhile And this relates to what I was thinking about in my first comment. I once was conversing the COO of my company (my last job), at a 1000+ person company, and asked him if he thought more concise requests for things would drive productivity. He, point blank, said: "sometimes I don't even know what I'm asking for" I've remembered that moment for years. In so many situations, the actual BI/dashboard is the least important part of the puzzle. Instead it's all of the conversation and discovery to understand the real need(s)
tailspin2019 3y ago
Brilliant summary - mirrors my thoughts and experience quite closely. Validation/testing has always been a challenge, especially given that dashboards are by definition quite “full stack” implementations where testing just the front end or back end is not sufficient and testing both in isolation can also often be challenging due to the huge possible variations in input data. Mocking data is also hard because dashboards may also lean a lot on database-side calculations/filtering. All of this has lead me to take quite a full-fat approach to testing dashboards, by using a real DB populated with test data, and testing the full complete application stack (driven by something like Playwright or Cypress) as well as more granular unit tests where a mocked data layer may be used. I’m also looking at introducing visual regression tests next time I work on this kind of thing. The visual aspects of dashboards can easily drift over time even if the data is correct. You’re often theming charting libraries for example and the compliance of the theme can drift slightly if you update the library without really checking every detail of the visual appearance/layout every time. Or you may not even notice the “visual drift”…
- grvdrm 3y ago
  Interesting - great points > Validation/testing has always been a challenge, especially given that dashboards are by definition quite “full stack” implementations where testing just the front end or back end is not sufficient and testing both in isolation can also often be challenging due to the huge possible variations in input data. Constantly evolving but I've always tried hard to keep calculations away from the display tools. So, I put lots of things in SQL SPs, or in Python, or more broadly in tooling that allows me to recreate the summary data without the front-end. My nightmare is having to check a PowerBI calc that itself is based on an underlying SQL calc. Which one is wrong? Now spend twice as long figuring it out! > The visual aspects of dashboards can easily drift over time even if the data is correct. You’re often theming charting libraries for example and the compliance of the theme can drift slightly if you update the library without really checking every detail of the visual appearance/layout every time. Or you may not even notice the “visual drift”.. Love it, very smart. Why I prefer tables for many things too - one less thing to maintain and check.
  - 3y ago

by grvdrm 3y ago

mushufasa 3y ago

> Finally, I think most dashboards miss one fundamental point. Imagine you're the CEO/COO and you've got this beautiful 3 or 4-chart dashboard in front of you. What should you know about what you're seeing? What's the succinct summary? Having been on both sides of this, I think the challenge is that the CEO/COO's job is to figure out "what should we do about this?," which is the right approach to coming up with that summary (it's not just "here's a text version of the chart"). And the corollary challenge is that, in most cases, non-technical people with domain knowledge are the ones who need to produce the analysis: so any feature incomplete dashboard is going to stymie them and any general framework that requires a technical person to step in for code or configuration is going to slow the process to a crawl. It's the rule (not the exception) that (especially if things are going poorly) the next step is asking more questions, which involves investigating something else in more detail. A dashboard, however pretty, is as useless as a doorknob if it doesn't have the needed information. I have found that dashboards per say are always great as the high-level KPI trackers, like the things you would consider hanging on a wall in an office (e.g. "revenue growth this month" or "new customers acquired"). You'll always want to know that information, and many people of unrelated departments need to have that information shared to them. The other helpful area is a deep-dive domain-specific analytics programs, like for example Google Analytics, where it has a very full featureset for non-technical marketing people to go in and drill down to answer questions. The UI/UX designers of that product have spent years honing and A/B testing which types of graphs to show where, and mapping out how to have people click around to find what they're looking for, to the point it is pretty easy for non-technical people. They even have courses and certifications on how to use the system. Organizations that try to internally build a feature-complete system like google analytics for a specific domain need to consider it like building an entire software product (even if there's a general low/no-code BI SaaS to assist) because you'll need collaboration between general technical experts and non-technical stakeholders with changing and vague requirements. It can be done, but likely only with years of investment and UI/UX research, just like any other software product that solves a domain problem well. In practice: millions of dollars. Technologists often forget that Excel *is* a turing complete programming language (and it's a functional programming paradigm too!). If an org is not committed to spend years and millions of dollars on deep dive analytics for a specific domain, the right choice is almost always using a commercial analytics system for that domain that costs less than the internal build, or embracing the trusty spreadsheet.

itsoktocry 3y ago
>I think the challenge is that the CEO/COO's job is to figure out "what should we do about this? Totally agree. I'd even go a little further and say the business is in trouble if the CEO doesn't know "what we should do about this". It's the CEOs job to know those things, and it's the data team's job to provide the tools to make those decisions easier, faster and better.
- grvdrm 3y ago
  > Totally agree. I'd even go a little further and say the business is in trouble if the CEO doesn't know "what we should do about this". It's the CEOs job to know those things, and it's the data team's job to provide the tools to make those decisions easier, faster and better. I agree with one modification. Also the CEO's job to empower folks to pitch what they think the CEO should know too. I've worked in plenty of successful shops where the CEO's answer to "what should we do" is "I'm not sure yet - what do you think?" - and that is a golden opportunity to show your chops if you've got the opportunity
- yabbs 3y ago
  It's my experience that you get easier, faster or better. Choose any 1, rarely 2, never 3.
- dgudkov 3y ago
  People frequently overestimate the role of data in decision-making. Metrics, numbers and other quantitative information don't tell the full story. For a CEO to make decisions, the full picture must include qualitative information - risks, opportunities, market events, competitor's actions, etc. Metrics are just part of the full picture. Far less significant than many BI developers and "data teams" tend to think.

clusterhacks 3y ago

"Business users often ask for things without an ounce of rationale." Deserves extra upvotes just for this statement. This has always been painful to me working in the data analysis and reporting space. When I get many requests for dashboards or reports that lack an answer to the question of "how will this be used?", I seem to find the requesting groups are cost-centers in the larger organization and are somewhat obsessed with processes and procedures. This is rarely a good group to build a career with . . .

proamdev123 3y ago
As a business user myself, often times we are simply responding to an edict from Leadership that amounts to: “You guys should have some dashboards!” Other times it’s because a supervisor or internal customer likes to see certain things on a chart, and putting the chart in Power BI/Tableau/other tool will make them prettier than Excel charts. Very few people, starting from the top on down, have a good understanding that dashboards mean very little in and of themselves.

conductr 3y ago

I work closely on BI projects but from a finance perspective. The concept I like to explain to the BI teams is that the dashboard is always just a snapshot of “what” is happening. But the underlying base level data is always needed to understand “why” it’s happening. And without the why, there’s no actual intelligence gain. Take a metric like Average Order Value (AOV). It may be ; total sales / order quantity. But as that metric is used it’s often being compared to something like last year, last month, or a plan and anyone interested in that number is really interested in understanding the “why” it has changed from some other point in time/scenario. For that, you actually need to bring in line item details behind orders as each order has multiple products/skus and they likely sold at different prices from a year ago or what was expected in a plan. An analysis of this has a name, price-volume-mix analysis or PVM. I always seem to have to explain this to BI teams when I join a new company and am seeking data. I’m currently going through it with a BI team, that apparently the BI tool wouldn’t store this information. It’s like it only stores aggregate values so it’s not even possible to get base level data for analysis (without major architectural changes). I don’t know if that’s normal in BI or was an implementation decision at some point but I’ve come across this same thing on a handful of companies and as I said I really have to drive this concept for those teams. When I ask of it I’m usually met with a “why would you need that info / give us a use case”. Which means, the don’t even understand how un-intelligent their BI tool is or why the execs likely aren’t feeling like investing in BI has been worthwhile (eg. Ever build a dashboard that then goes unused? I probably wasn’t perceived as useful for some reason like this). This could be more concise put as, understanding your end users needs. Understand the difference of what people ask for is often different than what they need. If they ask for AOV metrics, they’re really saying “I need to understand AOV” and that’s done via PVM analysis.

grvdrm 3y ago
Similarly titled towards finance. I specialize in what I'll call decision analytics for insurance underwriters. > Which means, the don’t even understand how un-intelligent their BI tool is or why the execs likely aren’t feeling like investing in BI has been worthwhile And this relates to what I was thinking about in my first comment. I once was conversing the COO of my company (my last job), at a 1000+ person company, and asked him if he thought more concise requests for things would drive productivity. He, point blank, said: "sometimes I don't even know what I'm asking for" I've remembered that moment for years. In so many situations, the actual BI/dashboard is the least important part of the puzzle. Instead it's all of the conversation and discovery to understand the real need(s)

tailspin2019 3y ago

Brilliant summary - mirrors my thoughts and experience quite closely. Validation/testing has always been a challenge, especially given that dashboards are by definition quite “full stack” implementations where testing just the front end or back end is not sufficient and testing both in isolation can also often be challenging due to the huge possible variations in input data. Mocking data is also hard because dashboards may also lean a lot on database-side calculations/filtering. All of this has lead me to take quite a full-fat approach to testing dashboards, by using a real DB populated with test data, and testing the full complete application stack (driven by something like Playwright or Cypress) as well as more granular unit tests where a mocked data layer may be used. I’m also looking at introducing visual regression tests next time I work on this kind of thing. The visual aspects of dashboards can easily drift over time even if the data is correct. You’re often theming charting libraries for example and the compliance of the theme can drift slightly if you update the library without really checking every detail of the visual appearance/layout every time. Or you may not even notice the “visual drift”…

grvdrm 3y ago
Interesting - great points > Validation/testing has always been a challenge, especially given that dashboards are by definition quite “full stack” implementations where testing just the front end or back end is not sufficient and testing both in isolation can also often be challenging due to the huge possible variations in input data. Constantly evolving but I've always tried hard to keep calculations away from the display tools. So, I put lots of things in SQL SPs, or in Python, or more broadly in tooling that allows me to recreate the summary data without the front-end. My nightmare is having to check a PowerBI calc that itself is based on an underlying SQL calc. Which one is wrong? Now spend twice as long figuring it out! > The visual aspects of dashboards can easily drift over time even if the data is correct. You’re often theming charting libraries for example and the compliance of the theme can drift slightly if you update the library without really checking every detail of the visual appearance/layout every time. Or you may not even notice the “visual drift”.. Love it, very smart. Why I prefer tables for many things too - one less thing to maintain and check.
- 3y ago