8 ms·

The more metrics you track, the less you know

139 points by kiyanwang 4y ago

SanjayMehta 4y ago
There was a paper from Hitachi (I think) decades ago. Their thesis was one should devise a metric to solve a problem, and discard it once the problem was solved. Then on to the next problem.
- Sujan 4y ago
  What prevents the problem from coming back if you stop watching for that?
  - tomrod 4y ago
    If the problem is solved, it does not return due to improvements in process. "Watching for it" bifurcates between actively putting into your view (the problem at hand) or by putting a monitoring daemon in place to inform you when a threshold has been broached.
  - musingsole 4y ago
    Don't overthink it. This advice covers 80-99% of actual bugs. Real bugs don't reappear once fixed -- if they do, it means they were never fixed. Metrics designed for a bug that was fixed may be relevant but specifically ARE NOT TARGETED at any new bugs. Though all of this really highlights the core problem with metrics and dashboards and such in the corporate world: if the problem were a problem and not just a convenient political puppet, we'd have solved it by now instead of talking about what thinking about solving it looks like.
    - pixl97 4y ago
      I would like you to talk to my programming team about regressions.
      - m3047 4y ago
        I think the argument in this (narrow) discussion would be you haven't fixed (or identified / addressed) the problem.
    - eloff 4y ago
      I think applying actual engineering wisdom to software in this case is a mistake. Nobody writes an automated test for a bug and then deletes it once the bug is fixed. You wrote the test in the first place so you get alerted when there is a regression before it gets into production. Regressions happen all the time.
    - Too 4y ago
      In isolation, bugs don't suddenly reappear no. So if you only regression test on unit-level, it's very unlikely the test will ever fire red again. What you should do is write use-case regression tests. This will protect you from major regressions, such as "user-registration is taking several minutes", because this is a major bug that can occur for a million new reasons, even if last time the reason was isolated to "database query x was running very slow", your test should track the former, not the latter.
Sujan 4y ago
Very relevant to me right now. But doesn’t the “ What About the Details?” section acknowledge that you will many of the other metrics later anyway for different people, projects and problems? Isn’t the message then “You should know which metrics are relevant to you and only look at them?”
- ares2012 4y ago
  Yeah, it's hard to generalize these kinds of things since companies can operate so differently. I've seen very large companies where everyone can use the same 5 metrics and others where each team needs their own set of 5. I think the key thing is that no team is using anymore than absolutely necessary.
EdwardDiego 4y ago
Indeed, knowing what metrics need to be monitored is vital. But to do that, you need to understand the system producing the metrics.
whirlwin 4y ago
Metrics can be interpreted in so many ways and is not the solution to the problem itself. A metrics provide some data points, but data points by itself does not solve any problem. That is why metrics is only part of the truth here. Consider everything else NOT taken into consideration by metrics
- chiefalchemist 4y ago
  Put another way: Metrics are simply knowledge. But knowledge isn't power. Understanding is Power. That is, consuming the metrics dots isn't enough. The difference making comes from understanding the broader context, as well as the connections between the dots. Believing in "Knowledge is power" is a trap.
  - ravivyas 4y ago
    Very few folks consume metrics, most want insights and intelligence, but there is lot more focus put on the data pipeline. Edit: I don't mean to say the pipeline is not important, it is required, but the outcome is intelligence
    - ItsMonkk 4y ago
      Using metrics as knowledge instead of applying them for understanding is just another case of cargo-culting.
  - gcanyon 4y ago
    My favorite saying I've seen recently: knowledge is knowing that Frankenstein was the doctor, who created the monster. Understanding is knowing that Frankenstein was the true monster.
- musingsole 4y ago
  It's easy to lie with metrics, but it's MUCH easier to lie without them.
lifeisstillgood 4y ago
Treat metrics like automated tests - if they are not green you have action to take. You can thus have thousands but hardly ever will you have thousands all red. And when you create a new metric, TDD style, the company will not be able / designed / changed to make it green.
- js8 4y ago
  Metrics also share some problems with automated tests - they can be green for years, they don't test anything useful, and the real problems are elsewhere.
- ares2012 4y ago
  But it's not always clear what is "green"? For example, New Users going up can be a good thing but New Users going up by a lot can represent a fraud attack. Metrics always require some interpretation to provide value which is why we have dashboards instead of alerts. There are some metrics that are binary good/bad and I agree that in those cases you should just have an alert.
jaggederest 4y ago
To re-paraphrase a common saying: Half the metrics I track are useless, the trouble is, I don't know which half. Obviously for core business metrics there's a direct cost associated with measuring, validating, and maintaining access, but even for more technical metrics that's true. The issue is, without a comprehensive metric collection and analysis system tracking multiple metrics, troubleshooting reduces down to a red light / green light level, which is not super useful. Is your site "down" because your uplink has gone offline, your servers, or is it your metric collection system throwing a false positive? Same can be said for business metrics - is revenue down from customer acquisition, deal size, or orders per customer? If you don't track more metrics than the minimum, it's hard to tell.
- charcircuit 4y ago
  >Half the metrics I track are useless, the trouble is, I don't know which half. You can measure how correlated metrics are to key metrics. You can ask questions like "Does the rate that my app crashes affect the amount of money people spend when using the app?"
drewcoo 4y ago
I think the framing should be around what makes sense on a dashboard, not how many metrics total that the entire company (or whatever cohort) wants to collect. A dashboard should not distract from what you're trying to do. It should show only the information you really need [1] so that you can focus on actually driving. I worked at one company where we had so many signals on ours that it looked like the dash of a 747 [2] to me. Pilots go through a lot of training and tests to show that they understand those and how to respond to different scenarios! Software startups are a bit more seat-of-pants than that. We can have dashboards for different purposes and roles, for different activities we want to drive. But the dashboards should focus on use. And if there are metrics that don't fit on any of the dashboards, they're not about driving anything - maybe consider scrapping them. [1] https://www.pinterest.com/pin/58054282668632186/ https://www.pinterest.com/pin/58054282668632186/ [2] https://www.reddit.com/r/cockpits/comments/ayr4z7/boeing_747200_cockpit/ https://www.reddit.com/r/cockpits/comments/ayr4z7/boeing_747...
Ozzie_osman 4y ago
There's a difference between evaluative metrics and diagnostic metrics. You have a handful of top-level metrics that tell you whether things are good or bad on a dashboard. When they change, you have to dig in to see why, so you need more metrics. You segment, you cohort, etc. Otherwise, you see GMV is down, and you don't and won't understand why.
- supersrdjan 4y ago
  I heard an analogy that goes like this: As a driver, all I want to see is current speed and RPM. But I like knowing that my car tracks a hundred other things, too, because when it fails, those other metrics will come handy. Or alert me before it fails. I heard it in the context of people analytics[0] but it seems to apply everywhere. [0] https://youtu.be/WE2ePETzzYQ?t=1434 https://youtu.be/WE2ePETzzYQ?t=1434
  - incongruity 4y ago
    Your car is also using metrics to keep you safe, maybe even alive at many moments. Active stability control and ABS are massively data-driven and essential parts of modern driving safety and are always on. I think there’s also an issue of layers of decision making and the requisite metrics at that layer. Abstraction is useful in business just as it is in operating a car.
- ravivyas 4y ago
  > evaluative metrics and diagnostic metrics I am stealing this, I called them "Opportunities & Firefighting" & "Heath / KPI Metrics" in a recent post
  - CyanBird 4y ago
    There is other quote Don't be data-driven, be data-informed
robflaherty 4y ago
The author's framework-"you just start small and only add one at a time as needed"-and given example illustrate exactly the process that leads to too many metrics. Pick a few metrics, add another, add another, add another. If you have 5 teams working on a product you quickly end up in the 50 metrics scenario. Too many metrics can be a problem but it's not the real problem. The real problem is choosing metrics without any regard for the decisions they're supposed to inform. When you understand that the purpose of all measurement in business is to reduce uncertainty for a decision to be made, everything comes into focus, and you'll have a natural constraint on your scope of measurement.
- ares2012 4y ago
  That would be true if you didn't assign a cost to each metric that you add. It was when the cost of tracking metrics plummeted that this started to become a problem, as there was no external friction in collecting more. If you assign a cost, and respect that cost, you shouldn't continuously add more.
deleted 4y ago
[deleted]
_zoltan_ 4y ago
You can never have enough metrics.
fajarsiddiq 4y ago
Priority metrics vs primary and secondary metrics if this is a tool that help us?
not_enoch_wise 4y ago
The beatings will continue until metrics improve
- ares2012 4y ago
  Ha, and I bet there is a metric for that too!
ricardobeat 4y ago
I agree with the sentiment but the example is not very compelling. Tracking averages is its own pit of problems. To be able to understand your funnel in a marketplace you’ll need to at least track profile and product views & click-throughs in addition to the metrics mentioned. How can you possibly tell what’s wrong when all you have is a average GMV metric going down?
- ares2012 4y ago
  I agree, but I think there is a difference between metrics and having data to investigate. A metric is a value you are tracking over time, but when something changes you will likely need to do an investigation. That investigation might be the raw data, it might be some dynamic queries or it might be other metrics. Tracking metrics because you think they might be useful in the future for diagnosing other problems doesn't make sense with modern systems as dynamic queries are so fast.
mydude 4y ago
At <FAANG mega-corp>, I used to work on an internal team which basically did nothing important, certainly nothing users ever think about. The problem that we were trying to solve, well, we had already solved it about 3 years ago, yet the team was still expanding, and our product managers kept spitting out new (mostly useless) project ideas. One of the main areas of focus was metrics. Leadership was obsessed with metrics, we measured everything. Not just plain empirical metrics (like # clicks), but also very complex metrics, theoretical models that we had data scientists work on, etc. At some point maybe 25% of the team worked on metrics related stuff, it was crazy. Why were they doing this? Was it only to keep us busy, so that the team can continue to grow? Or maybe they were desperately trying to find metrics by which they could prove the team was actually still doing meaningful work? I don't know. At some point I left, and moved on to a much smaller team. This team was working on a brand new product, one that customers actually pay money for! One of the first things I said to my new product manager was "so, what metrics do you care about? how do you measure success?". He was taken aback by my question. He seemed genuinely perplexed. He paused for a few seconds, then said "metrics? What do you mean by 'metrics'? I look at revenue, when the line goes up I'm happy". I thought it was... insightful.
- fuzzfactor 4y ago
  Yes, it's the people not the metrics. Problem is solved in the second sentence: >It’s a mistake to ignore the people that make those businesses work, True. >but you need to understand the numbers to know how the business is doing. False. You need people who understand the business even more so without exact numbers, who can hands-down outperform those who rely on metrics instead. That's why they were called "businessmen".
- deleted 4y ago
  [deleted]
- voidfunc 4y ago
  There is an argument to be made that the busy work and good pay of megacorps is basically designed to keep talent in place that could otherwise run off and potentially make a competitor.
gmuslera 4y ago
... but still rely on metrics. Yes, metrics are something easy to compare how things changed, but the system as a whole is more complex than that. There are things that are binary (you get there, you lose), there are things that are multifaceted and is not easy to put an linear number to it. And the Goodhart's Law (https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law) should be taken into account, specially when you lower the number of metrics. They may be a guide, but be careful if you put them as target.
steveBK123 4y ago
My last company lost its mind and started getting into meta-metrics like: X% of systems have metrics. Y% of uncovered systems get added to metrics in Q2. Z% of metrics captured meet their SLO. Meanwhile the SLOs themselves were pulled out of thin air because management only talked to upper management and not actual users.
- eloff 4y ago
  This sounds like a Kafka novel. I can see why you left.
  - steveBK123 4y ago
    Promotion Driven Metrics
FiatLuxDave 4y ago
One thing I see far too rarely in choosing metrics is ensuring that every metric has one or more countervailing metrics to balance the choices made. In other words, if you are optimizing for a metric, you also need to measure what is being sacrificed for it. The classic trio of cheap/fast/good is a great example of this, as are most engineering-style tradeoffs, like capacity vs range. You will often see managers create metrics for the things they want to improve, without realizing that making a metric without measuring the countervailing metric is creating an incentive to throw the countervailing metric in the shitter. A good example of this is call centers which measure the time of call resolution without measuring customer satisfaction with the call. This is how you end up with a low quality, high volume call center. You can do the opposite by measuring only satisfaction and not speed. Now, maybe what your business needs is high volume, low quality, but that should be a choice you make. The problem with not measuring the countervailing metric is that nobody (officially) cares about how bad that metric gets. So, if you have a tradeoff situation, you automatically get an extreme even though that may not be the most profitable place on the spectrum. Perhaps your call center would be better for the company if it was volume 9 quality 2 instead of volume 10 quality 1, but if you only optimize for volume you can't make that choice. This even applies to metrics you really want to go up, things like sales and profit. A manufacturing company that optimizes only for sales while ignoring manufacturing metrics will hire salespeople in preference to manufacturing workers until the inability to deliver product starts affecting sales. That might sound like a nice problem to have, but it can be a real problem, and perhaps sales 9 manufacturing 2 instead of sales 10 manufacturing 1 would result in better sales next year. It's better to have metrics which allow you that choice.
- hbrn 4y ago
  That's basically recognizing that Goodhart's Law exists and fighting it. Unfortunately managers rarely have incentives to do it. Why invest in being more objective if it lowers the perceived value of your successes? Your bullshitting peer that isn't doing this will get promoted instead of you. It can also be hard to even understand what are you sacrificing and why. I recently had an example where product manager was trying to improve conversion while another team was tapping into cheap low-quality traffic. PM always looked at a global conversion and kept wondering why the conversion was dropping despite his best efforts.
deleted 4y ago
[deleted]
efitz 4y ago
I think this article is just wrong, or at best describes the author’s experience in a dysfunctional org. There are a lot of pathological metrics patterns. I used to work at a FAANG/MANGA that prided itself on being metrics driven; the challenge was that most people suck at picking metrics. The most common anti-pattern is choosing metrics based on what is easy to measure. The most valuable metrics I have encountered are metrics that measure directly your teams success in their stated mission. The problem is a lot of teams have bad missions. I tried to tell every team that I worked with, that their mission should read like a problem statement not like a technology statement. For instance, instead of saying that “we are the team that owns the foo service“, the team needs to think about the business problem that inspired the foo service, and make solving that business problem their mission statement. Once you have clarity on the business, problem that your team exists to solve, then you can start thinking about metrics that measure how well you are solving that problem. These are the most valuable kinds of metrics. Now, the thesis of the article was that teams had too many metrics, and that this was bad. Once a team has clarity of mission, they have to implement technology to accomplish the business problem that is their mission. After you have clear metrics that tell you how well you were accomplishing your mission, then you need metrics to tell you how well your technology is functioning. Do not mix the two kinds of metrics. It is a happy coincidence if the technology functioning well directly, corresponds to how well you are accomplishing your business mission. More likely, the business metrics and the technology metrics need to be kept separate. You need metrics around the technology so that you can predict whether you are nearing a problem, detect whether you have a technology problem, And identify the nature of the technology problem that you have. Good metrics around technology will allow you to do all these things. You should not add metrics arbitrarily, but you should analyze your technology for where it is likely to break and prioritize adding metrics in that fashion. My last point is about team just functions. Other anecdotes in this comments thread have described organizations that lacked clarity of mission or that had completely solve. Their mission yet did not pivot to a new mission. In those cases, you end up with a lot of make work, and that make work may consist in part of implementing new metrics. This is just plain org dysfunction and not really a metrics problem.
- ares2012 4y ago
  YMMV but this is based on my experience building some of the largest analytics platforms like Flurry and Outlier.ai which were used by hundreds of thousands of companies. The only dysfunctional company I worked at was Verizon and they... don't really use metrics.
AStellersSeaCow 4y ago
Counterpoint: you don't have enough good metrics.
hbrn 4y ago
It is very easy to see the benefits that metrics provide (typically you start at no metrics at all), but incredibly hard to measure the costs. This bias is ubiquitous: metrics, unit testing, abstractions, startup funding, documentation, security, etc. All of those typically start at zero. All of those are perceived as "good", therefore more is always better. You rarely find people with the balls to say "we need less documentation" or "we invested too much effort in security". Kudos to author for recognizing the same issue with metrics. And he's right: the solution is to recognize our bias and for things where cost is hard to measure just assign some cost to it. Though instead of million dollar value suggested by author you should just treat cost as exponential. Having 3 metrics is better than 0. But having 50 metrics is way worse than 3. In fact, having 50 metrics is likely way worse than 0.
theptip 4y ago
> Most companies have far too many metrics. Companies might have a dozen dashboards, each with 4-5 metrics, leading to 50-100 metrics being tracked at any given time. I think it’s important to divide “internal” / operational metrics, which a team monitors but doesn’t expect anyone else to care about (say, error rate of the DB or some internal ops task latency) vs. “external” metrics which are rolling into OKRs / KPIs or otherwise being reported out as health metrics of the team. I struggle to believe that most companies with 50-100 metrics are actually using most of them as external metrics.
- ares2012 4y ago
  YMMV but this was based on dozens of conversations with companies where they spent hours each week reviewing those dozens of dashboards. I was specifically talking about the metrics used by any given team, as you are right that different teams might use different metrics. "The key is that any given person shouldn’t be using any more metrics than absolutely necessary to do their job well."
deleted 4y ago
[deleted]
userbinator 4y ago
The focus on "data driven decision making" tends to get things stuck in a weird local maximum and result in https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law . "Not everything that counts can be counted, and not everything that can be counted, counts."
cdaringe 4y ago
Not interested, based purely on the title. The article may have insight, but these anti intellectual clickbaitesque headlines are too tired to heed
c3534l 4y ago
I think of metrics like I do accounting. Accounting is all about measuring parts of the business. You track every dollar sent, lost, owed, etc. Every economic transaction is measured. But nobody wants to just stare at a list of raw transactions. You have to summarize those figures: you group them into revenues and expenses, assets and liabitilies, and you break it down by department, by function. You create a picture of what your business is doing. In the end, all some people really care about is profits, some people want a measure of solvency, others want to know how big your company is. Accountants have actually has thought deeply about metrics and written papers about why the measure what they do and I think something close to the golden rule is: does this information have the capacity to influence someone's decision? Information is meant to help you make a decision: should we add more servers, should we end this service, can we sell this feature, are our customers satisfied? Every metric should help you answer a question. Ask yourself, what decisions are you trying to make and what information would help you make that decision?

139 points by kiyanwang 4y ago

There was a paper from Hitachi (I think) decades ago. Their thesis was one should devise a metric to solve a problem, and discard it once the problem was solved. Then on to the next problem.

Sujan 4y ago
What prevents the problem from coming back if you stop watching for that?
- tomrod 4y ago
  If the problem is solved, it does not return due to improvements in process. "Watching for it" bifurcates between actively putting into your view (the problem at hand) or by putting a monitoring daemon in place to inform you when a threshold has been broached.
- musingsole 4y ago
  Don't overthink it. This advice covers 80-99% of actual bugs. Real bugs don't reappear once fixed -- if they do, it means they were never fixed. Metrics designed for a bug that was fixed may be relevant but specifically ARE NOT TARGETED at any new bugs. Though all of this really highlights the core problem with metrics and dashboards and such in the corporate world: if the problem were a problem and not just a convenient political puppet, we'd have solved it by now instead of talking about what thinking about solving it looks like.
  - pixl97 4y ago
    I would like you to talk to my programming team about regressions.
    - m3047 4y ago
      I think the argument in this (narrow) discussion would be you haven't fixed (or identified / addressed) the problem.
  - eloff 4y ago
    I think applying actual engineering wisdom to software in this case is a mistake. Nobody writes an automated test for a bug and then deletes it once the bug is fixed. You wrote the test in the first place so you get alerted when there is a regression before it gets into production. Regressions happen all the time.
  - Too 4y ago
    In isolation, bugs don't suddenly reappear no. So if you only regression test on unit-level, it's very unlikely the test will ever fire red again. What you should do is write use-case regression tests. This will protect you from major regressions, such as "user-registration is taking several minutes", because this is a major bug that can occur for a million new reasons, even if last time the reason was isolated to "database query x was running very slow", your test should track the former, not the latter.

Sujan 4y ago

Very relevant to me right now. But doesn’t the “ What About the Details?” section acknowledge that you will many of the other metrics later anyway for different people, projects and problems? Isn’t the message then “You should know which metrics are relevant to you and only look at them?”

ares2012 4y ago
Yeah, it's hard to generalize these kinds of things since companies can operate so differently. I've seen very large companies where everyone can use the same 5 metrics and others where each team needs their own set of 5. I think the key thing is that no team is using anymore than absolutely necessary.

EdwardDiego 4y ago

Indeed, knowing what metrics need to be monitored is vital. But to do that, you need to understand the system producing the metrics.

whirlwin 4y ago

Metrics can be interpreted in so many ways and is not the solution to the problem itself. A metrics provide some data points, but data points by itself does not solve any problem. That is why metrics is only part of the truth here. Consider everything else NOT taken into consideration by metrics

chiefalchemist 4y ago
Put another way: Metrics are simply knowledge. But knowledge isn't power. Understanding is Power. That is, consuming the metrics dots isn't enough. The difference making comes from understanding the broader context, as well as the connections between the dots. Believing in "Knowledge is power" is a trap.
- ravivyas 4y ago
  Very few folks consume metrics, most want insights and intelligence, but there is lot more focus put on the data pipeline. Edit: I don't mean to say the pipeline is not important, it is required, but the outcome is intelligence
  - ItsMonkk 4y ago
    Using metrics as knowledge instead of applying them for understanding is just another case of cargo-culting.
- gcanyon 4y ago
  My favorite saying I've seen recently: knowledge is knowing that Frankenstein was the doctor, who created the monster. Understanding is knowing that Frankenstein was the true monster.
musingsole 4y ago
It's easy to lie with metrics, but it's MUCH easier to lie without them.

lifeisstillgood 4y ago

Treat metrics like automated tests - if they are not green you have action to take. You can thus have thousands but hardly ever will you have thousands all red. And when you create a new metric, TDD style, the company will not be able / designed / changed to make it green.

js8 4y ago
Metrics also share some problems with automated tests - they can be green for years, they don't test anything useful, and the real problems are elsewhere.
ares2012 4y ago
But it's not always clear what is "green"? For example, New Users going up can be a good thing but New Users going up by a lot can represent a fraud attack. Metrics always require some interpretation to provide value which is why we have dashboards instead of alerts. There are some metrics that are binary good/bad and I agree that in those cases you should just have an alert.

jaggederest 4y ago

To re-paraphrase a common saying: Half the metrics I track are useless, the trouble is, I don't know which half. Obviously for core business metrics there's a direct cost associated with measuring, validating, and maintaining access, but even for more technical metrics that's true. The issue is, without a comprehensive metric collection and analysis system tracking multiple metrics, troubleshooting reduces down to a red light / green light level, which is not super useful. Is your site "down" because your uplink has gone offline, your servers, or is it your metric collection system throwing a false positive? Same can be said for business metrics - is revenue down from customer acquisition, deal size, or orders per customer? If you don't track more metrics than the minimum, it's hard to tell.

charcircuit 4y ago
>Half the metrics I track are useless, the trouble is, I don't know which half. You can measure how correlated metrics are to key metrics. You can ask questions like "Does the rate that my app crashes affect the amount of money people spend when using the app?"

drewcoo 4y ago

I think the framing should be around what makes sense on a dashboard, not how many metrics total that the entire company (or whatever cohort) wants to collect. A dashboard should not distract from what you're trying to do. It should show only the information you really need [1] so that you can focus on actually driving. I worked at one company where we had so many signals on ours that it looked like the dash of a 747 [2] to me. Pilots go through a lot of training and tests to show that they understand those and how to respond to different scenarios! Software startups are a bit more seat-of-pants than that. We can have dashboards for different purposes and roles, for different activities we want to drive. But the dashboards should focus on use. And if there are metrics that don't fit on any of the dashboards, they're not about driving anything - maybe consider scrapping them. [1] https://www.pinterest.com/pin/58054282668632186/ https://www.pinterest.com/pin/58054282668632186/ [2] https://www.reddit.com/r/cockpits/comments/ayr4z7/boeing_747200_cockpit/ https://www.reddit.com/r/cockpits/comments/ayr4z7/boeing_747...

Ozzie_osman 4y ago

There's a difference between evaluative metrics and diagnostic metrics. You have a handful of top-level metrics that tell you whether things are good or bad on a dashboard. When they change, you have to dig in to see why, so you need more metrics. You segment, you cohort, etc. Otherwise, you see GMV is down, and you don't and won't understand why.

supersrdjan 4y ago
I heard an analogy that goes like this: As a driver, all I want to see is current speed and RPM. But I like knowing that my car tracks a hundred other things, too, because when it fails, those other metrics will come handy. Or alert me before it fails. I heard it in the context of people analytics[0] but it seems to apply everywhere. [0] https://youtu.be/WE2ePETzzYQ?t=1434 https://youtu.be/WE2ePETzzYQ?t=1434
- incongruity 4y ago
  Your car is also using metrics to keep you safe, maybe even alive at many moments. Active stability control and ABS are massively data-driven and essential parts of modern driving safety and are always on. I think there’s also an issue of layers of decision making and the requisite metrics at that layer. Abstraction is useful in business just as it is in operating a car.
ravivyas 4y ago
> evaluative metrics and diagnostic metrics I am stealing this, I called them "Opportunities & Firefighting" & "Heath / KPI Metrics" in a recent post
- CyanBird 4y ago
  There is other quote Don't be data-driven, be data-informed

robflaherty 4y ago

The author's framework-"you just start small and only add one at a time as needed"-and given example illustrate exactly the process that leads to too many metrics. Pick a few metrics, add another, add another, add another. If you have 5 teams working on a product you quickly end up in the 50 metrics scenario. Too many metrics can be a problem but it's not the real problem. The real problem is choosing metrics without any regard for the decisions they're supposed to inform. When you understand that the purpose of all measurement in business is to reduce uncertainty for a decision to be made, everything comes into focus, and you'll have a natural constraint on your scope of measurement.

ares2012 4y ago
That would be true if you didn't assign a cost to each metric that you add. It was when the cost of tracking metrics plummeted that this started to become a problem, as there was no external friction in collecting more. If you assign a cost, and respect that cost, you shouldn't continuously add more.

deleted 4y ago

[deleted]

_zoltan_ 4y ago

You can never have enough metrics.

fajarsiddiq 4y ago

Priority metrics vs primary and secondary metrics if this is a tool that help us?

not_enoch_wise 4y ago

The beatings will continue until metrics improve

ares2012 4y ago
Ha, and I bet there is a metric for that too!

ricardobeat 4y ago

I agree with the sentiment but the example is not very compelling. Tracking averages is its own pit of problems. To be able to understand your funnel in a marketplace you’ll need to at least track profile and product views & click-throughs in addition to the metrics mentioned. How can you possibly tell what’s wrong when all you have is a average GMV metric going down?

ares2012 4y ago
I agree, but I think there is a difference between metrics and having data to investigate. A metric is a value you are tracking over time, but when something changes you will likely need to do an investigation. That investigation might be the raw data, it might be some dynamic queries or it might be other metrics. Tracking metrics because you think they might be useful in the future for diagnosing other problems doesn't make sense with modern systems as dynamic queries are so fast.

mydude 4y ago

At <FAANG mega-corp>, I used to work on an internal team which basically did nothing important, certainly nothing users ever think about. The problem that we were trying to solve, well, we had already solved it about 3 years ago, yet the team was still expanding, and our product managers kept spitting out new (mostly useless) project ideas. One of the main areas of focus was metrics. Leadership was obsessed with metrics, we measured everything. Not just plain empirical metrics (like # clicks), but also very complex metrics, theoretical models that we had data scientists work on, etc. At some point maybe 25% of the team worked on metrics related stuff, it was crazy. Why were they doing this? Was it only to keep us busy, so that the team can continue to grow? Or maybe they were desperately trying to find metrics by which they could prove the team was actually still doing meaningful work? I don't know. At some point I left, and moved on to a much smaller team. This team was working on a brand new product, one that customers actually pay money for! One of the first things I said to my new product manager was "so, what metrics do you care about? how do you measure success?". He was taken aback by my question. He seemed genuinely perplexed. He paused for a few seconds, then said "metrics? What do you mean by 'metrics'? I look at revenue, when the line goes up I'm happy". I thought it was... insightful.

fuzzfactor 4y ago
Yes, it's the people not the metrics. Problem is solved in the second sentence: >It’s a mistake to ignore the people that make those businesses work, True. >but you need to understand the numbers to know how the business is doing. False. You need people who understand the business even more so without exact numbers, who can hands-down outperform those who rely on metrics instead. That's why they were called "businessmen".
deleted 4y ago
[deleted]
voidfunc 4y ago
There is an argument to be made that the busy work and good pay of megacorps is basically designed to keep talent in place that could otherwise run off and potentially make a competitor.

gmuslera 4y ago

... but still rely on metrics. Yes, metrics are something easy to compare how things changed, but the system as a whole is more complex than that. There are things that are binary (you get there, you lose), there are things that are multifaceted and is not easy to put an linear number to it. And the Goodhart's Law (https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law) should be taken into account, specially when you lower the number of metrics. They may be a guide, but be careful if you put them as target.

steveBK123 4y ago

My last company lost its mind and started getting into meta-metrics like: X% of systems have metrics. Y% of uncovered systems get added to metrics in Q2. Z% of metrics captured meet their SLO. Meanwhile the SLOs themselves were pulled out of thin air because management only talked to upper management and not actual users.

eloff 4y ago
This sounds like a Kafka novel. I can see why you left.
- steveBK123 4y ago
  Promotion Driven Metrics

FiatLuxDave 4y ago

One thing I see far too rarely in choosing metrics is ensuring that every metric has one or more countervailing metrics to balance the choices made. In other words, if you are optimizing for a metric, you also need to measure what is being sacrificed for it. The classic trio of cheap/fast/good is a great example of this, as are most engineering-style tradeoffs, like capacity vs range. You will often see managers create metrics for the things they want to improve, without realizing that making a metric without measuring the countervailing metric is creating an incentive to throw the countervailing metric in the shitter. A good example of this is call centers which measure the time of call resolution without measuring customer satisfaction with the call. This is how you end up with a low quality, high volume call center. You can do the opposite by measuring only satisfaction and not speed. Now, maybe what your business needs is high volume, low quality, but that should be a choice you make. The problem with not measuring the countervailing metric is that nobody (officially) cares about how bad that metric gets. So, if you have a tradeoff situation, you automatically get an extreme even though that may not be the most profitable place on the spectrum. Perhaps your call center would be better for the company if it was volume 9 quality 2 instead of volume 10 quality 1, but if you only optimize for volume you can't make that choice. This even applies to metrics you really want to go up, things like sales and profit. A manufacturing company that optimizes only for sales while ignoring manufacturing metrics will hire salespeople in preference to manufacturing workers until the inability to deliver product starts affecting sales. That might sound like a nice problem to have, but it can be a real problem, and perhaps sales 9 manufacturing 2 instead of sales 10 manufacturing 1 would result in better sales next year. It's better to have metrics which allow you that choice.

hbrn 4y ago
That's basically recognizing that Goodhart's Law exists and fighting it. Unfortunately managers rarely have incentives to do it. Why invest in being more objective if it lowers the perceived value of your successes? Your bullshitting peer that isn't doing this will get promoted instead of you. It can also be hard to even understand what are you sacrificing and why. I recently had an example where product manager was trying to improve conversion while another team was tapping into cheap low-quality traffic. PM always looked at a global conversion and kept wondering why the conversion was dropping despite his best efforts.

deleted 4y ago

[deleted]

efitz 4y ago

I think this article is just wrong, or at best describes the author’s experience in a dysfunctional org. There are a lot of pathological metrics patterns. I used to work at a FAANG/MANGA that prided itself on being metrics driven; the challenge was that most people suck at picking metrics. The most common anti-pattern is choosing metrics based on what is easy to measure. The most valuable metrics I have encountered are metrics that measure directly your teams success in their stated mission. The problem is a lot of teams have bad missions. I tried to tell every team that I worked with, that their mission should read like a problem statement not like a technology statement. For instance, instead of saying that “we are the team that owns the foo service“, the team needs to think about the business problem that inspired the foo service, and make solving that business problem their mission statement. Once you have clarity on the business, problem that your team exists to solve, then you can start thinking about metrics that measure how well you are solving that problem. These are the most valuable kinds of metrics. Now, the thesis of the article was that teams had too many metrics, and that this was bad. Once a team has clarity of mission, they have to implement technology to accomplish the business problem that is their mission. After you have clear metrics that tell you how well you were accomplishing your mission, then you need metrics to tell you how well your technology is functioning. Do not mix the two kinds of metrics. It is a happy coincidence if the technology functioning well directly, corresponds to how well you are accomplishing your business mission. More likely, the business metrics and the technology metrics need to be kept separate. You need metrics around the technology so that you can predict whether you are nearing a problem, detect whether you have a technology problem, And identify the nature of the technology problem that you have. Good metrics around technology will allow you to do all these things. You should not add metrics arbitrarily, but you should analyze your technology for where it is likely to break and prioritize adding metrics in that fashion. My last point is about team just functions. Other anecdotes in this comments thread have described organizations that lacked clarity of mission or that had completely solve. Their mission yet did not pivot to a new mission. In those cases, you end up with a lot of make work, and that make work may consist in part of implementing new metrics. This is just plain org dysfunction and not really a metrics problem.

ares2012 4y ago
YMMV but this is based on my experience building some of the largest analytics platforms like Flurry and Outlier.ai which were used by hundreds of thousands of companies. The only dysfunctional company I worked at was Verizon and they... don't really use metrics.

AStellersSeaCow 4y ago

Counterpoint: you don't have enough good metrics.

hbrn 4y ago

It is very easy to see the benefits that metrics provide (typically you start at no metrics at all), but incredibly hard to measure the costs. This bias is ubiquitous: metrics, unit testing, abstractions, startup funding, documentation, security, etc. All of those typically start at zero. All of those are perceived as "good", therefore more is always better. You rarely find people with the balls to say "we need less documentation" or "we invested too much effort in security". Kudos to author for recognizing the same issue with metrics. And he's right: the solution is to recognize our bias and for things where cost is hard to measure just assign some cost to it. Though instead of million dollar value suggested by author you should just treat cost as exponential. Having 3 metrics is better than 0. But having 50 metrics is way worse than 3. In fact, having 50 metrics is likely way worse than 0.

theptip 4y ago

> Most companies have far too many metrics. Companies might have a dozen dashboards, each with 4-5 metrics, leading to 50-100 metrics being tracked at any given time. I think it’s important to divide “internal” / operational metrics, which a team monitors but doesn’t expect anyone else to care about (say, error rate of the DB or some internal ops task latency) vs. “external” metrics which are rolling into OKRs / KPIs or otherwise being reported out as health metrics of the team. I struggle to believe that most companies with 50-100 metrics are actually using most of them as external metrics.

ares2012 4y ago
YMMV but this was based on dozens of conversations with companies where they spent hours each week reviewing those dozens of dashboards. I was specifically talking about the metrics used by any given team, as you are right that different teams might use different metrics. "The key is that any given person shouldn’t be using any more metrics than absolutely necessary to do their job well."

deleted 4y ago

[deleted]

userbinator 4y ago

The focus on "data driven decision making" tends to get things stuck in a weird local maximum and result in https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law . "Not everything that counts can be counted, and not everything that can be counted, counts."

cdaringe 4y ago

Not interested, based purely on the title. The article may have insight, but these anti intellectual clickbaitesque headlines are too tired to heed

c3534l 4y ago

I think of metrics like I do accounting. Accounting is all about measuring parts of the business. You track every dollar sent, lost, owed, etc. Every economic transaction is measured. But nobody wants to just stare at a list of raw transactions. You have to summarize those figures: you group them into revenues and expenses, assets and liabitilies, and you break it down by department, by function. You create a picture of what your business is doing. In the end, all some people really care about is profits, some people want a measure of solvency, others want to know how big your company is. Accountants have actually has thought deeply about metrics and written papers about why the measure what they do and I think something close to the golden rule is: does this information have the capacity to influence someone's decision? Information is meant to help you make a decision: should we add more servers, should we end this service, can we sell this feature, are our customers satisfied? Every metric should help you answer a question. Ask yourself, what decisions are you trying to make and what information would help you make that decision?