16 ms·

Management by metrics leads us astray

• tchalla 6y ago
  > Let’s say my goal this year is to get to 10k MRR. Monthly Recurring Revenue, if anyone is wondering.
  • rob74 6y ago
    Thanks! Otherwise it's a great blog post, but I did stumble over that unexplained acronym (and was too lazy to google it)
    • jakobgreenfeld 6y ago
      Thanks for the feedback. Just changed the first sentence.
• helsinkiandrew 6y ago
  It's not the metrics per se - but the metrics not being totally aligned with the required outcome. Good project management means being better at finding the right metrics - or at least recognising that our metrics aren't good enough, often because we haven't all of the information - why are we hiring a new developer, why is this piece of software being developed etc.
  • Xelbair 6y ago
    No matter which metrics you pick, as long as they are known, they will be gamed.
  • hrktb 6y ago
    “We just need to find the appropriate XXX” is a good approach if no one gains from keeping a sub-optimal XXX (whatever it is). A dumb but easy to reach metrics helps if you’re employee that wants money on next evaluation. An ad targeted at the wrong people still brings money to the ad platform, helping find the good targeting mechanism is of low to negative ROI. And so on. As long as it’s an adversarial situation you’ll never reach that appropriate target that needs to be shared among a number of people so they move in the same direction. I think metrics are unavoidable for management, but they shouldn’t be on what you actually strive for, otherwise you’re just sending wishes to a monkey’s paw.
  • madsbuch 6y ago
    The problems with metrics is that they indicate there is a single (multivariate) objective by running a company, running a country, living a life, running a whatever. The thing is just that it is not the case. It in intricate networks of relations, aspirations, missions, ambitions, needs and so forth. Metrics should underlie that fact, not overlie it. Ie. they can be used, just in moderation.
  • niek_pas 6y ago
    There’s usually no such thing as “the right metrics”. Metrics are helpful, yes, but they can only take you so far.
• hikingsimulator 6y ago
  A bit of a segue, but this management by metrics goes beyond just revenue. KPI, KRI, KCI, etc. Truly a heap of metrics that eat out the time of middle management, and consequently of the people down the hierarchical pyramid. It over-allocates resources towards resource management itself rather than towards products themselves. There is a drive to justify your existence, and that of your fellow workers, when your company becomes large enough that it has bureaucratized and, moreover, acquired some form of legacy. David Graeber in Bullshit Jobs mentions that two categories of BS were the box-tickers and the taskmasters. They are evidence of this management by metrics. It's all about budget: gotta spend that hard-earned budget to find ways to save budget on your budget-consuming core products... to make sure your budget allocation is the same next month or year.
  • goatinaboat 6y ago
    Truly a heap of metrics that eat out the time of middle management, and consequently of the people down the hierarchical pyramid. It over-allocates resources towards resource management itself rather than towards products themselves You mean, it provides lots of avenues for the management class to provide cushy jobs for each other, at the expense of the real workers. Most organisations could ditch most of their middle management without skipping a beat.
  • clairity 6y ago
    so you want management without metrics? the problem isn't the metrics (a fundamental attribution error to think so), it's that middle managers are incentivized to appease upper managers (and ultimately owners), and metrics are the most visible available tool. the system is set up this way intentionally by owners and their executive surrogates to insulate them from the callous, negative conqeuences of management that explicitly aims to dehumanize work (this kind of indirection abstraction in human systems is an extremely common source of distorted incentives and unintended consequences; like poor customer service at most companies, google being a preeminent example). metrics that are reported to individual contributors (e.g., toyota production system, agile done appropriately), on the other hand, are fantastic, for managers and workers, as long as it's well-understood that the metrics are not the product, they're simple signposts (indirect/incomplete measures of holistic qualitative desires like efficiency and quality). by the way, (bad) bureaucracies are created by the drive to regiment and order messy human processes that leads to over-reliance on metrics, not people trying to justify their existence. bullshit jobs is a bullshit book written to appeal to our fears and frustrations, not to elucidate us with actionable information.
• roenxi 6y ago
  All employees have radically different ideas about how to organise for success, ranging from free market capitalism to the government literally seizing control of this specific company. The skill range is from "needs constant supervision" to "good at a very limited range of activities". Any specific manager has 40 hours a week, which gets eaten up very quickly (10 reports = 4 hrs each to figure out if you like what they are doing assuming no actual work the manager needs to do or surprises that mean they can't work to schedule). The issue isn't that anyone believes that metric-driven management is a good option, it is that at the scale of 2,000+ people there are not a lot of alternatives. I could easily design a system that works at large scale ... but only by the employees and owners being the same people. Apart from that, the erratic us-them nature of stockholders and employees makes all the good structures untenable. Lay out an org chart for 2k people then ask how long the manager has to work out what their reports are doing. Expand the hierarchy far enough that everyone has time to do a good job and then ask what the odds are that all the people in positions of importance agree on how things should be done by whom. To settle the number of disputes that arise in a big tree-like org structure there isn't have much of a plan B apart from metrics. It would be nice to see more governmental support for limited liability cooperative business models. The organisational opportunities that would open up could allow for so much. Not to mention maybe workers could earning more.
  • sokoloff 6y ago
    Could you give an example of what you mean in the last paragraph? What types of additional government support would help? (Are you talking legal, financing, taxation, reduced paperwork/overhead/fees or other?)
    • roenxi 6y ago
      Seems like a thorough listing. Yes. I'd like to see the major changes in taxation and legal reform to favour encourage bigger risk/reward sharing with workers and reductions in paperwork and regulation to make it easier to set up companies with a diverse range of small, actively involved stakeholders. I think it is notable how well American tech companies did with large equity handouts to their early employees. Maybe the model starts there. I don't have details, I do have a problem - to get great results, large complex organisations need a different model than salaried workers.
• dependsontheq 6y ago
  That's not true and true at the same time. There are myriad instances where management by metrics leads companies totally astray, bit it can easily argued that they are badly managed or have a bad strategy. The core of any kind of measurement effort is a clearly defined objective that sits in balance with other objectives, and the important thing is these objectives are sometimes in conflict with each other, like finance and UX, measuring both gives the company a chance to find the right balance.
• WalterBright 6y ago
  It's the same result when one tries to run an economy through regulation, for the same reasons. A better way would be by incentives. For example, there are regulations that 100 ppm of pollution X is illegal, while 99 ppm of X is legal. Emit 99 ppm, and you're good. Emit 100 ppm and your plant gets shut down. Even worse, you can emit 99 ppm for 10 hours and you're legal, but emit 100 ppm for 1 minute and you're illegal. A far better approach is to tax total X emitted, not the rate, and not binary legal/illegal: 100 ppm for 1 minute times $1/ppm = $100 tax 99 ppm for 10 hours times $1/ppm = $59,400 tax
  • cjfd 6y ago
    At the very least the numbers of 99 ppm and 100ppm are measurement of the problem that one is trying to solve. The more typical braindead form of regulation would mandate one specific kind of mitigation regardless of the current level of pollution. While at the same time it is, of course, a pure coincidence that the more polluting company is friends with a politician while the less polluting company that now has to pay for less necessary or unnecessary mitigation is not friends with said politician.
    • WalterBright 6y ago
      > the numbers of 99 ppm and 100ppm are measurement of the problem that one is trying to solve Not for air pollution. There the total emissions are of interest, not the rate. Car emissions, for example, are regulated for their rate of emission, not total emissions.
      • sokoloff 6y ago
        Even there, ppm is a suboptimal (maybe even terrible) measure. If I can find a way to pump additional air through the exhaust (increasing the denominator), I can lower my tailpipe ppm without lowering the rate of emitted pollutant (the numerator).
        • WalterBright 6y ago
• pratio 6y ago
  I've experienced this first hand so many times. It is so difficult to steer the boat when the management depends heavily on metrics they don't understand the need of. Just because others are doing it. And as the flow of pressure goes from top to bottom, engineers end up working towards achieving a metric which burns out the good ones and they leave. I've spent sometimes hours in talking to clients on how they should first focus on metrics that help them ask the right questions. This is prevalent everywhere, Scrum-masters holding meetings to discuss the scrum velocity.
• melomal 6y ago
  Marketer here and this is so spot on. So many companies that I have worked for have had founders that just didn't seem to understand metrics as well as they thought they did. Huge spread sheets with % and pivot tables and week on week breakdowns BUT no one could tell me much about what all that data meant. Apart from the fact that everyone had their own explanation it was just useless information to be fair. Churn, CAC, amongst various other 'hot' metrics. In the end, most of these companies took a couple of years to realize that these metrics which Unicorns write amazing blog posts on are either BS or just not made for them. So much frustration was felt because a manager read a TechCrunch article about how X (with 100's of millions) did this and achieved something spectacular.
  • pm90 6y ago
    I don’t know if there’s a phrase for this phenomena but I see this a lot and it really bothers me, where leaders get “too” inspired by management or process ideas from blogs. It almost never works out and then the leaders blame teams for incorrectly following whatever the process was rather than accepting that almost every org is somewhat unique so adjustments to strategy need to always be made.
    • melomal 6y ago
      Spot on, it's infuriating to deal with. There's a lot of time spent reading blogs and quoting some flavor of the month founder and then essentially trying to replicate as much as possible. One of my consulting roles got terminated early (1 week before I was about to terminate it myself too) because a Growth Hacking agency with a loud speaking founder turned up, told the founder that my work was not good (an agency saying that there's problems across the board - who would have thought that huh?) only for them to be fired 1 month later and an outcry across social media for 'real growth hackers' and not promising the world from that same founder. Bearing in mind this founder also had the latest Apple products because it was a must...
    • sriku 6y ago
      Survivorship bias clubbed with no true Scotsman?
• tchalla 6y ago
  "Management by reliance ONLY on metrics leads us astray" Gut vs Metrics don't have to be either/or. Gut also has its own limitations - Daniel Kahnemann has documented this very well in Nobel Prize winning research. So what should one do? An organisation has to be self aware that there are no "perfect" metrics. Therefore, one should clearly state the limitations of a metric(s). Then, make a qualitative judgment based on the metrics and their limitations. You make the limitations, assumptions transparent and communicate the risks, uncertainties. Metrics are an input to decision making, they aren't decision makers themselves.
  • disgruntledphd2 6y ago
    Yeah, but it's about avoiding blame, not making the right decision. Unfortunately good decision making is a rare skill, and management is a common profession, so stuff like this will continue to happen. Note that the prerequisite for making good decisions is the space in which to make a bunch of decisions, some of which will inevitably be bad, so without a cultural acceptance of potential failure and the right attitude towards it, things will never improve.
    • tchalla 6y ago
      I have noticed that labelling decisions as binary (right vs wrong, good vs bad) tends to create factions. I recommend turning decisions from binary into options/implications. A "bad" decision is an option with an implication assigned to it. If stakeholders are willing to agree with the implications, that's how it is.
      • disgruntledphd2 6y ago
        Yeah, that's a problem. I kinda meant from the perspective of the person who made the decision in the future (i.e. normally me).
• AnthonBerg 6y ago
  We should do better than to be ignorant of national and global decisionmaking and responsibility in public health and pandemic epidemology. It is a baseless claim that public health is systematically ignored in favor of pushing the pandemic’s reproduction number down. Do we really expect public health officers to be untrained, stupid, and dismissed? Is that what we really are seeing? Do we really think that public officials are ignorant of the metabolic and social implications of having to drive stakes into society? It is a fact that in functioning states, public health decisions are taken within an ethical framework of how to do least harm and preserve the most freedom, considering resource availability and longer-term effects of public health decisions, including the interacting feedback loops of the economy and public health. While overreliance on small metrics is clearly bad, this style of off-hand ignorance of public service is not what we need.
  • ScottBurson 6y ago
    It does seem like a poorly chosen example. In the case of an exponential process, the doubling time is a remarkably important number.
  • logicchains 6y ago
    >It is a baseless claim that public health is systematically ignored in favor of pushing the pandemic’s reproduction number down. It's not baseless when there have been multiple studies now showing lockdowns don't help, and even the WHO now recommends against them, yet authorities still push for them.
    • pjc50 6y ago
      Can you point me at the WHO advice and how they're defining "lockdown"? And what the correct answer is? Because England is currently running at over 1600 COVID deaths a day, or half a 9/11, and we desperately need something that does help.
      • jdsully 6y ago
        Where are you getting this number? NHS published data is less than half this number.
• rolfkip 6y ago
  While I agree with what I read as the premise of the article (that metrics for the sake of metrics is harmful and that qualitative data conversation is a great supplement to quantitative metrics) I wish the post was written with less hyperbole, especially the around big companies and politicians sections. My take is that quantitative metrics are incredibly useful when paired with strong leadership, a clear mission, and vision statement that give human context to the metrics. That way whenever “the numbers” are brought up, the context and focus are around goals that really matter (are we creating a product that is meaningful and making people’s lives better?) versus cherry-picking and gaming the system to meet KPIs. Quantitative metrics are a dangerous tool since numbers are so easily digestible, so special care should always be given to understand exactly what the metrics are holding people accountable to. An example that comes to mind in personal life is fitness. Weight is such an easy metric to get your hands on... doubly so since you can quickly judge weight by looking at yourself in the mirror. It’s really easy to tunnel on trying to lose weight! The activities (both physical and mental!) encouraged by fixation on weight can be incredibly destructive. If instead I first start with a strong vision statement that aligns with my values: My body is my most important possession; I want to keep it healthy so I can live a fulfilling life. In the context of that purpose weight is an important metric to keep in mind as a risk factor, but it is also clear that there are a multitude of other metrics that will allow me to measure, stretch, and grow towards a productive end.
  • mirko22 6y ago
    It comes down to what kind of people are in the positions. If they are kind of people that only care about numbers not amount of metrics will change that, if they are strong leaders with vision they usually have their custom ways of dealing with metrics.
    • pm90 6y ago
      I think the author of the article is trying to illustrate that the result of even a few metric-cheaters is that over time the entire organization or field is dominated by the metric-cheaters. There is just no way to always keep such people out of your organization once you cross a certain size threshold. So the logical result is that with time, everyone is forced to become more like the metric cheaters just to survive so they either leave or become metric cheaters themselves; in either case the product goes to shit as the creative problem solvers leave.
• equitablequal 6y ago
  I very much subscribe to combining this 'Common-sense filter' to metrics when choosing a course of action, and came up with a term for it: inform your gut.
  • jakobgreenfeld 6y ago
    That's a perfect way to phrase it. Thanks!
• waihtis 6y ago
  The core question which should always be asked: when defining or setting a goal, what kind of incentives are you simultaneously creating?
• WalterBright 6y ago
  "Frontline" ran an episode some years back where they investigated some dental clinics that did astonishing numbers of unnecessary major dental operations. It turns out that the government had a program that paid for these operations when performed on poor people. So the clinics would convince their customers that they needed major dental work, then would do the work and bill the government. The clinics were totally specialized just for this. "Frontline" pinned the blame on the clinics, sort of "how could the do that". Of course, they're right. But blaming the clinics doesn't work, because the actual problem was the government set things up so that the rise of clinics to exploit the rules was legal and inevitable. You're now faced with the problem of crafting a bureaucratic rule to define a "necessary" major dental operation. Of course, it's impossible, and the same problem as "management by metrics".
  • deleted 6y ago
    [deleted]
  • sokoloff 6y ago
    I can’t see my way to blaming government for this. Either these operations were indicated but unlikely to happen otherwise (the government program was effective) or not medically-indicated (at which point private clinics are torturing poor people for profit which is abhorrent on so many levels).
    • giantDinosaur 6y ago
      The point isn't that the clinics are not morally culpable, but only that they are a natural response to government incentives over which the government has exclusive control.
      • sgift 6y ago
        Doing something morally wrong is not a natural response. It is a natural response option. To paraphrase Jurassic Park: [Clinics] were so preoccupied with whether or not they could, they didn’t stop to think if they should. Not everyone chooses to do something morally wrong, just because "I'm allowed to! And it makes life easier!"
• l0b0 6y ago
  > A resume will never tell you if a person is genuinely interested in uncovering unknown truths about nature. But a 30-minute conversation will. A 30-minute conversation with a friend might, but this is implicitly talking about an interview setting. How confident are you in your own ability to tell whether "a person is genuinely interested in uncovering unknown truths about nature" knowing that nobody has been able to come up with a similar formula for interviewing programmers, where there are at least in theory easily measurable knowledge levels?
  • TeMPOraL 6y ago
    > knowing that nobody has been able to come up with a similar formula for interviewing programmers, where there are at least in theory easily measurable knowledge levels? There is a technique that's successful and widely used. It's somewhat unfashionable to talk about it in today's political climate, it requires doing some actual work, and it isn't a "formula" - but it works. The technique is exactly what TFA describes: talk to the candidate. Apply a domain-specific bozo filter (like FizzBuzz) if you must to narrow down the pool of candidates, but ultimately what you need to do is to get multiple people from the team for whom you're hiring to talk with the candidate for some hours, in a semi-structured way. Cue in accusations of bias. Sure, it's biased, but what's the alternative? Pretending you can codify worker competency and fit as a set of cheaply testable metrics that are independent of company/team specifics? Such process doesn't exist, and will never exist, because the problem is AI-complete. Instead, why not trust that the people on your team are decent human beings looking for their own interest? And in fact, a lot of companies do just that and happily chug along. But admitting to that doesn't make you an industry thought leader, nor can it be packaged into a service and sold.
• royosherove 6y ago
  Metrics in software is a bag of issues in most companies I see. But there ARE useful ways to use metrics that actually help (software). I did a talk about this at GOTO: Lies, Damned lies, and metrics. https://www.youtube.com/watch?v=goihWvyqRow https://www.youtube.com/watch?v=goihWvyqRow
• StreamBright 6y ago
  >> Who else manages by metrics? Politicians! This is completely false. Politics is the least metrics driven, mostly intention driven. https://www.washingtontimes.com/news/2014/apr/25/sowell-good-intentions-bad-results/ https://www.washingtontimes.com/news/2014/apr/25/sowell-good...
  • tonyedgecombe 6y ago
    I think you are probably right but the article you linked to doesn't demonstrate that.
• ANarrativeApe 6y ago
  'OKRs before KPIs'as it says in our culture deck https://citizenshareholders.com/culture https://citizenshareholders.com/culture If achieving your KPIs compromises your Objectives and Key Requirements you have the wrong KPIs. In the start-up/scale-up world this there can be enormous pressure to meet investor expectations on short term revenue/growth forecasts. Part of the job of founders is to know their market, their company's unfair advantage. 'Don't be Bullied' should probably be the title of a blog post calling on founders to resist investor pressure to follow the norms in achieving early revenue, not because early revenue is a bad thing, far from it, it's amazing. But if the cost of generating that revenue puts OKRs at greater risk, its better to raise more money to extend your runway. If you meet your OKRs, assuming you've chosen the right OKRs, your KPIs will follow. In many ways this is a culture issue: Culture trumps Strategy, OKRs trump KPIs Provided you have the cash. Citizen Shareholders is now looking for that cash. When we're asked why we have no revenue projected for 2 years, it's because being able to survive for 2 years without revenue will derisk us to the early B2B adopters. Having that early revenue in the bank alongside those investor funds is a problem we can live with. Just in time supply chains are so very 2019, we plan to make sure we have enough to succeed by, at times, having more than enough. If you want to find out more: CitizenShareholders.com
  • erikstarck 6y ago
    Unfortunately many people miss the O in OKRs. The Objective is what really matters, but people tend to think it's all about the KRs - then you do end up with optimising for the metric.
• throw8932894 6y ago
  > Their core product got notably worse over time. > Google’s search results are dominated by ads Googles core product are ads, not search engine!
  • brazzy 6y ago
    But their ability to sell ads critically relies on their dominance in the search engine market, which is based on the quality of the search results. There was a time before Google. There were other search engines with large market shares. Google completely destroyed them through superior search results. An alternative search engine that actually works better as a search engine would be an existential threat to Google, and they know that. Because there is very little that prevents people from switching search engines - no network effects, nothing really except default settings and laziness.
    • throw8932894 6y ago
      I remember Altavista. Adsense controls like 80% of ad market. Biggest thread to Google are walled gardens like Facebook or Amazon. Search engines like Bing and Duck are not even on their map. > dominance ... which is based on the quality of the search results Any source for that? Google search already provides inferior results in many areas, but it is still dominant. Google also pays billions to Apple and Mozilla to be default search engine. And if threat emerges and gets like 30% of search market share, Google just dials down ads and make their search experience a bit less crappy.
      • brazzy 6y ago
        > Adsense controls like 80% of ad market. Only about 14% of Google's advertising income is from AdSense, the rest is from ads on Google's own properties, and the vast majority of that is from search. > Biggest thread to Google are walled gardens like Facebook or Amazon. Search engines like Bing and Duck are not even on their map. I guarantee you they are. Facebook and Amazon are threats because they are established with massive userbases so anything they do can quickly have a big impact. But I absolutely guarantee that competition from a better search engine is very much on Google's map as an existential threat, just not an immediate one. The know they cannot stagnate in that field and are continually investing into into research to stay at the bleeding edge. > Any source for that? Google search already provides inferior results in many areas, but it is still dominant. How do you even quantify search result quality? My personal experience from trying out other search engines is that Google is definitely superior, and better search results was most definitely what got them to dominance in the first place - Altavista and Yahoo and Excite weren't even in the same ballpark. > Google also pays billions to Apple and Mozilla to be default search engine. And that should tell you all you need to know about how essential it is for Google to stay the dominant search engine. > And if threat emerges and gets like 30% of search market share, Google just dials down ads and make their search experience a bit less crappy. At that point they would already be fighting desparately for their life, because "just dialing down ads" translates into an immediate, proprotional decrease of revenue from their (at that point already compromised) main cash cow. 30% less ads on 30% less searches would immediately mean Alphabet as a whole is deep in the red. And if a competitor already managed to take substantial market share from them, making the experience "a bit less crappy" is hardly going to be enough to turn things around. No, Google really must be (and is) much more proactive in this: keep investing in research, keep on the lookout for any company that looks like it's doing something that might someday in the future impact search engine technology, and just buy it.
• xgb84j 6y ago
  Great article! I also found that the aim to be objective in everything and use metrics as truth and not just another input is pretty harmful over the long run in business and science. Things that are hard to measure are ignored most of the time. The best example are apples. When you breed apples to be 3% larger each year but taste slightly worse, you can weigh them and see that they got slightly larger. As you cannot accurately measure taste you will notice that they basically taste the same. After years of breeding you have giant apples with bad taste. The author's proposition of using as many metrics as possible and talking to real people is a good solution to make better decisions. I would add that for some decisions you should actually just focus on the few facts that truly matter. The classic example being a potential business partner with a bad reputation. If somebody is not trustworthy everything else probably doesn't matter.
  • flyinglizard 6y ago
    I agree. Anything which only boils down to metrics would have already been automated.
• hasa 6y ago
  Much boils down to the question about the quality of the metrics. In my work they are just numbers that are manipulated to look good so that everyone gets their bonuses and the bosses are happy.
• idid 6y ago
  For those inclined, there's a nice article by Marilyn Strathern, The Tyranny of Transparency [0] that echoes the sentiments in this article. It builds upon Hari Tsoukas' The Tyranny of light: The temptations and the paradoxes of the information society [1], which I heartily recommend if interested in reflections on this topic. To entice, here's a small excerpt from the abstract of [1]: > The overabundance of information in late modernity makes the information society full of temptations. It tempts us into thinking that knowledge-as-information is objective and exists independently of human beings; that everything can be reduced into information; and that generating ever more amounts of information will increase the transparency of society and, thus, lead to the rational management of social problems. However, as argued in this paper, the information society is riddled with paradoxes that prevent it from satisfying the temptations it creates. More information may lead to less understanding; more information may undermine trust; and more information may make society less rationally governable. [0] https://www.tandfonline.com/doi/abs/10.1080/713651562 https://www.tandfonline.com/doi/abs/10.1080/713651562 (alternative: https://sci-hub.se/https://doi.org/10.1080/713651562 https://sci-hub.se/https://doi.org/10.1080/713651562) [1] https://www.sciencedirect.com/science/article/abs/pii/S0016328797000359 https://www.sciencedirect.com/science/article/abs/pii/S00163... (alternative: https://sci-hub.se/https://doi.org/10.1016/S0016-3287(97)00035-9 https://sci-hub.se/https://doi.org/10.1016/S0016-3287(97)000...)
• lordnacho 6y ago
  Great article. My take on it: - Metrics give everyone a false sense of objectivity. If you actually look at any stats in depth, you will end up making decisions about what to measure, and what the numbers mean. These words are actually the most important part of any study, but are always left out in favor of the graph. Want to know how many people died of covid? You need to consider things like population adjusted deaths. And adjustment for age. And a a whole host of other things to do with how the data was collected, and how it was presented. The economic side is even more messy. How much would GDP have been if we'd done this or that? There's no number that's the answer, but we do need numbers. - You'll always care about more metrics than one. There's no generally objective way to decide how to weight those. You might have a utility function that decides this, thus reducing the problem to a simple optimization, but the guns/butter balance has still got to come from some kind of value that you express. - The big question is rules vs discretion. You set up an elaborate measuring system (eg GDP) and you imagine that you have certain levers that you can move in response to whatever situation occurs. Part of this is that actors expect you to behave a certain way. But what happens if something you thought would work mechanically doesn't behave like you thought? Should you move your levers contrary to your previous commitment? - Finally, there are the unmeasurable sensations. You have a team, it feels wrong somehow. You talk to people, get opinions, you get a feel for what is wrong. None of it is something you found in your LOC/user summary, or CI pipeline fails, or whatever.
• artpi 6y ago
  Thank you! Your post hit really hard, because a) Sole focus of my team IS MRR. b) I wrote something very similar. Sorry if my comment is a little maendering, I am a dad as of 7 days and what they tell you about sleep is no joke. I want to plug in one of the less famous Drucker quotes: > Working on the right things is what makes knowledge work effective. This is not capable of being measured by any of the yardsticks for manual work. It's true that "What gets measured gets managed" by the converse is also true - if you focus on one metric only, the rest will fall into disarray. It's good to keep your eyes at the gauges when you are flying, but just as the map is not the territory, the measurement is not the real thing. People love metrics, because they give them an illusion of control over a complicated, real world, messy reality. The core of the problem is unfortunately human bias to simplify. We as a wider tech community are contributing a bit introducing yet-another-dashboard. But as you rightly point, the metrics people pick are the ones that are easiest to measure. NOT the ones worth measuring. In fact I published a post quite similar to yours: https://piszek.com/2019/11/24/metrics/ https://piszek.com/2019/11/24/metrics/ To broaden the theme, I would even argue that Taleb is complaining about roughly a similar set of problems: Gaussian curve is an oversimplification as well and has caused a lot of suffering when misapplied. Despite all of these caveats, there is a benefit to having a focus (like MRR you are mentioning). Having a singular focus can be transformative and can make one developer more effective than 30-people team with no real direction. In that sense, there is benefit to metrics. As with all real-world things, I think the only solution is a dynamic equilibrium, where you change from one approach to the other to cover your bases. As I wrote in the beginning of this comment, the Team I Lead (we build simple ecommerce tools) was tasked with maximizing MRR and the side-effect of having that focus was that we ignored the one-off sales features that our customers wanted desperately.
• gampleman 6y ago
  I'm a but cynical about the recommendation of talking to people. The interviews then turn into its own optimisation game (I think politicians for instance do base their decision making on talking to people, hence lobbying). The same thing happens with job interviews (what is the most useful skill to getting a great SWE salary? Interviewing well).
• usgroup 6y ago
  I don’t think there is an alternative to management by measurement if scalable is what you’re trying to be. That doesn’t mean you don’t have long term goals or nuanced objectives even if your measures are coarse, nor does it mean your measures stay the same. I think optimising for measures focuses activity , which again doesn’t mean you have to optimise for the same thing all the time using the same measures.
• rusk 6y ago
  ”it was a good measurement until it became a target”
• praestigiare 6y ago
  While is is a good (and often discussed) point that the use of metrics causes distortions as the metrics themselves affect behavior, I am not sure that the author's proposed solution of relying instead on intuition is an improvement. While the data may introduce distortions, ignoring it in favor of "going with my gut" is how we get vaccine denialism, conspiracy theories, racism, and so many other negative outcomes. I am also not convinced by the inclusion of the R number in the list of examples - it is unlike the others in an important way. The problem with metrics usually arises when the metric is a proxy for the goal. For example, test scores are not the same as knowledge, and so using test scores as a metric can result in teaching to the test, such that the metric is no longer a good proxy for knowledge. But the R number is not a proxy - it is the actual thing we want to change. (Of course, there are many knock-on issues, such as how we measure that number, which do introduce proxies. Discussion of that would be a much more interesting article.) Which actually points towards the real solution, which is not to simply rely on gut instinct, but to be aware of what you are measuring, and how it relates to what you want to know.
• wizardofmysore 6y ago
  Article shows companies like Google and Facebook. However doesn't talk about big companies managing in a different way. Why? They are all dead. Entropy and decay is part of product cycle. There is no way around it. You push them far away.
• kpgiskpg 6y ago
  My favourite illustration of this is in The Wire. The city wants to crack down on crime and increases the target number of arrests. The police chief sends out the word and police officers begin making a bunch of spurious arrests. Do you blame the politicians for picking crappy metrics and for their poor understanding of human nature? Or do you blame the individuals who choose to exploit the metric out of self-interest? Plenty of blame to go around.
  • ack210 6y ago
    The Reply All episodes on the real life version of this are a really good listen if you haven't heard them already: https://gimletmedia.com/shows/reply-all/o2hx34/ https://gimletmedia.com/shows/reply-all/o2hx34/ (Short interview with one of the hosts on the episode: https://www.fastcompany.com/90249924/reply-all-podcast-nypd-compstat https://www.fastcompany.com/90249924/reply-all-podcast-nypd-...)
  • bonoboTP 6y ago
    The textbook example is when they wanted to reduce the rat population somewhere and paid people for (catching and) bringing in dead rats. Instead, people started to breed them and "sell" the dead specimens to the city.
• baxtr 6y ago
  > You know who manages by metrics? Big companies like Google, Amazon and LinkedIn. And what do they have in common? Their core product got notably worse over time. It’s interesting he doesn’t mention Apple. I have no insight but from the outside of say they have very different things they optimize for than revenue. Money merely pours out of their customer obsession instead.
• BrandoElFollito 6y ago
  I am currently refining the metrics I want to present to my management. I work in information security so we do not exist until a issue arises. I base the value of a metric presented to management on the "and so what?" basis. In other words: how it helps them to better run the business. There is the "annual objectives" kind of report: how much we did and how good we are. Meaningless. There is the "current events" one (nr of incidents, etc.). Meaningless. There is the "these are systemic, company-wise issues I need you to arbitrate over", or "we need to do that, but it means changes and incomfort". Actual decision making data, the worst to present because they show problems and not celebration. I guess that this is the case of these organizations that do not provide green money to the company (aka. "cost centers"). i hate these metrics. (please note that there are metrics that are useful to *operational teams* so that they realize there is a trend, or that some performance upgrades will be needed, but they should stay there)
  • SamuelAdams 6y ago
    Can't you present it as a risk / benefit analysis? Basically at the beginning of the quarter the open / larger risks were XYZ and now those risks have been mitigated or reduced.
    • Terretta 6y ago
      Here’s how that goes before the quarter, by deciders less close to the situation on the ground, or whose own job performance evaluation depends on the situation not being as dire: “You’re being alarmist. Don’t ask us to deploy troops, it’s not like Americans are going to actually storm their own Capitol.” And then after the quarter, “OK, Americans didn’t storm the Inauguration, but are you sure that’s because you so-called mitigated the risk with 20K troops? All this money, no attack, maybe we should spend that money on teachers instead of troops.” In this space, lack of negative outcomes breeds contempt for the mitigation. (Hmmm...) Interestingly, in the very long long run, that might even be right, as better education on civics is solving the problem at the left, earlier in the production line. So now you have a new problem, which is to convince those deciders not close to the problem and disinterested in openly admitting gaps, that they need to both spend on the 20K troops now, and double down on teachers for a payoff 10 years from now, so that after that they don’t have to spend on the troops... best luck.
• HPsquared 6y ago
  The book "Seeing like a State" expounds on this topic significantly, using scientific forestry as an example. The "recipe for failure" [0]: 1. Look at a complex and confusing reality 2. Fail to understand all the subtleties of how the complex reality works 3. Attribute that failure to the irrationality of what you are looking at, rather than your own limitations 4. Come up with an idealized blank-slate vision of what that reality ought to look like 5. Argue that the relative simplicity and platonic orderliness of the vision represents rationality 6. Use authoritarian power to impose that vision, by demolishing the old reality if necessary 7. Watch your rational Utopia fail horribly [0] https://smus.com/books/seeing-like-a-state-by-james-scott/ https://smus.com/books/seeing-like-a-state-by-james-scott/
  • jiggawatts 6y ago
    > 5. Argue that the relative simplicity and platonic orderliness of the vision represents rationality That's an elegant way to articulate a criticism I've had of many technologies that I have been unable to state so clearly. All those file formats, for example, that seek to replace XML. Think of JSON, YAML, and TOML. They all claim to be simple, but end up too simple in practice and cause just as many issues as they solve. Comments, schemas, and namespaces were there for a reason, they weren't just a "messy design" awaiting a visionary to clean up... > 6. Use authoritarian power to impose that vision, by demolishing the old reality if necessary Unfortunately, in the cloud era, we're all faced with trillion dollar behemoth organisations that wield absolute power over their dominions. Us peasants have no hope of changing the vision they choose to impose. Suddenly, the well-established and well-known stable world has become "legacy".
    • HotHotLava 6y ago
      > They all claim to be simple, but end up too simple in practice and cause just as many issues as they solve. The number may be the same, but the set of issues is different, leaving developers the choice if they'd rather work around the lack of namespaces and comments in JSON or around the opaque meta definitions and annoying syntax of XML. And the popular vote was pretty clear on which design is preferred.
    • 6y ago
• sunng 6y ago
  By using metrics we make it a lower dimensional model, that can be managed at scale. However in real world, it often becomes a overfit one. In large scale execution, people only care what metric measures. A talk vs metrics is just higher dimension vs lower one. The information within a talk is complex. Different people received vary information from the same talk. So it's hard to managed by scale. So a metrics system is just a trade-off for its scale.
• anilgulecha 6y ago
  When the situation allows, you want to pick a metric that would not matter if it were gamed. It takes thoughtful leadership to arrive at it.
• nojito 6y ago
  I find this to be very very rich. A guy who lambasts metrics, but whose projects are all based on scraping metrics. https://productexplorer.io/ https://productexplorer.io/ https://newsletterspy.io/ https://newsletterspy.io/ https://gumspy.com/ https://gumspy.com/
• 74d-fe6-2c6 6y ago
  As usual: If you can't measure what's important you declare important what you can measure. That wouldn't be so destructive if management at least was aware of that problem. But they aren't. They'll have their report of metrics which are green or red. Green is good. Red is bad. End of story.
• baryphonic 6y ago
  Greenfeld comes perilously close to independently discovering Hayek. Local, specialized knowledge is essential, and is one of the hardest features to scale, if it's possible at all. Also, this reminds me of stories about Soviet manufacturing quotas[1]: > The Soviet Union was the largest producer of shoes in the world. It was turning out 800 million pairs of shoes a year–twice as many as Italy, three times as many as the United States, four times as many as China. Production amounted to more than three pairs of shoes per year for every Soviet man, woman, and child. > The problem with shoes, it turned out, was not an absolute shortage. It was a far more subtle malfunction. The comfort, the fit, the design, and the size mix of Soviet shoes were so out of sync with what people needed and wanted that they were willing to stand in line for hours to buy the occasional pair, usually imported, that they liked. Or even more ridiculous[2]: > But probabably the most outrageous malfunction of the system was of a factory that fabricated metal products. It had a quota for scrap metal for recycling. The factory was operating more efficiently than expected so it was not generating as much scrap as expected. Since the factory was not meeting its scrap metal quota the higher authorities were going to fine it. So the factor personnel did what to them was rational. They took perfectly good zinc metal sheets and converted them into scrap to fulfill their scrap metal quota. When looking at large organizations through the lens of information problems, one realizes that Amazon, Google, Walmart et al. all exist on continuum along with the Soviet Union or even the US Government. [1]https://www.econlib.org/archives/2009/09/soviet_shoes.html https://www.econlib.org/archives/2009/09/soviet_shoes.html [2]https://www.sjsu.edu/faculty/watkins/stalinmodel.htm https://www.sjsu.edu/faculty/watkins/stalinmodel.htm
  • jimduk 6y ago
    Hi, apart from Hayek (and maybe Ursula Le Guin, and A Hamilton), has anyone got any good pointers on theories for when you favour local over global knowledge or vice versa. Either political theories, or game-theoretic. Transaction cost economics is the closest thing I've seen, and maybe some of Stuart Kauffman's stuff on the adjacent possible and the CEO's job being to 'recut' the company into different divisions. But I've never seen anything along the lines of - if I have a changing fitness landscape L and a set of communicating agents A with different sets of knowledge, and they can use their knowledge to extract value from the local landscape, or they can make tools to increase the value of that knowledge or they can communicate or they can listen, then in what situations should you take different actions. The extremes are easy ( local changing landscape, high comms costs, then don't listen to anyone else; static similar landscape, cheap comms, listen to the cleverest agent); Maybe there is some biology stuff with ants or some agent-based modelling work ?
• dboreham 6y ago
  "What gets measured gets done" , unfortunately.
• 6gvONxR4sf7o 6y ago
  There’s a sweet spot that’s hard to hit. On one end you have people who ignore metrics. They get themselves into trouble breaking things without making the easy measurements to know they’re breaking them. They make obvious mistakes. The next group makes subtle mistakes. They’re the ones who have learned how to use metrics and now they have a hammer, everything looks like a nail. I see this a lot in less senior data scientists. They know how to drive with data, and so everything must be done that way. But they make the mistakes this article gets at. The group I like takes a middle ground approach. We recognize that certain network effects are so coupled that you can power maybe one proper experiment on it at a time, and even those experiments might take too long. Other experiments take too long simply by virtue that the thing you want to measure might be long term behavior. So you experiment on what you can, get everything you can by proxy, and make sure to at least measure health via the metrics that matter, but you don’t try to base every decision in something measurable simply because it’s measurable.
  • theptip 6y ago
    I agree. I think you need to be very careful about giving uni-directional advice like the OP, when in fact you are optimizing a U-shaped curve. Too little AND too much use of metrics is bad. The advice for a given company will depend where they are on that curve.
• 0xbadcafebee 6y ago
  The metrics being discussed here are largely quantitative. OTOH, a qualitative metric is better for measuring if your product is turning to shit over time. Both types are needed to manage big complex things.
• jively 6y ago
  The conclusion of this article is basically "I don't understand how to use common sense and data at the same time, so I'll hark back to old-timey "gut instinct" and good-ol-fashioned selection bias hidden by my own lack of self-understanding to validate my ignorance, oh and here's some BS about lockdown policies." What a joke, all of these posts have the same conclusions: don't deal in absolutes.
• theptip 6y ago
  > You know who manages by metrics? Big companies like Google, Amazon and LinkedIn. This isn’t true through. On this very site’s Startup School you will find lots of experienced entrepreneurs advocating for a metric-driven approach to growing your startup. Maybe not “one metric that matters” all the time, but Lean Startup is about measuring your company’s metrics and running experiments week-by-week to see what moves the needle. Sure, slavish and myopic adherence to a bad metric will lead you astray. But management without metrics will lead you astray too.
• rusty-rust 6y ago
  If anyone is interested in reading about this topic in terms of late stage capitalism I would highly recommend a book by Mark Fisher titled Capitalist Realism. In it he talks about “Stalinist Capitalism” I.e. a capitalist society that is obsessed with metrics which optimize for arbitrary goals at the expense goals which are intangible or hard to measure. It focuses on the metric obsessed cultures that have developed within many private organizations. Two examples come to mind, universities prioritizing publishing count vs publishing impact and lines of code written vs impact of code.
• vannevar 6y ago
  Eric Reis' "The Lean Startup" covers this topic in detail, and makes a useful distinction between "vanity" metrics and "actionable" metrics. Aside from the book, you can read a blog post by him on the subject of metrics here: https://tim.blog/2009/05/19/vanity-metrics-vs-actionable-metrics/ https://tim.blog/2009/05/19/vanity-metrics-vs-actionable-met...
• otavio-coelho 6y ago
  The article fails to acknowledge that there is in fact two very distinct families of metrics. 1. Taylor metrics - The ones invented by management. 2. von Braun metrics - The ones invented by the guy that put men on the Moon. These are everywhere it really matters.