13 ms·
Against metrics: how measuring performance by numbers backfires
- melling 8y ago“But the most dramatic negative effect of metric fixation is its propensity to incentivise gaming: that is, encouraging professionals to maximise the metrics in ways that are at odds with the larger purpose of the organisation”
- nf05papsjfVbc 8y agohttps://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
- navigatesol 8y ago>“But the most dramatic negative effect of metric fixation is its propensity to incentivise gaming "Gaming" the workplace is happening with or without metric fixation. At least with metric fixation it's front and centre when it occurs.
- js8 8y agoYour post reminds me of the famous AI koan: https://news.ycombinator.com/item?id=10970937 https://news.ycombinator.com/item?id=10970937 Maybe there is some truth in it. But I also feel that you might get a better result if you let humans to figure out what is needed to be done to achieve a goal instead of telling them through a metric.
- philipodonnell 8y agoThe feature that distinguishes a well designed metric is exactly that. Specific enough to ensure alignment with broader strategic objectives, general enough not to dictate exactly how.
- js8 8y agoI think the point of the article is summed up in the sentence: "The key components of metric fixation are the belief that it is possible – and desirable – to replace professional judgment (acquired through personal experience and talent) with numerical indicators of comparative performance based upon standardised data (metrics)" Basically, it is a problem of trust. You're trying to replace trust in people with a metric which, in theory, lets you not to have trust in them. Personally, I think it is a foolish goal. You are effectively replacing the trust in people (which, admittedly, is vague) with trust in the metric (which connection to reality is vague). The vagueness has a reason - reality is complicated. Why not just tell people what the "broader strategic objective" is, instead of trying to come up with a metric that is an exact (and so necessarily wrong) description of it?
- philipodonnell 8y agoTrust is the wrong word, its more about alignment. I do trust that employees can exercise their professional judgement. I don't expect that every employee can perfectly align themselves behind a strategic objective without guidance. If my broader strategic objective is to cut costs by 15% allow more competitive pricing, the last thing I want is for every individual to define what that means to them and hope the math works out in the end.
- js8 8y ago"I do trust that employees can exercise their professional judgement. I don't expect that every employee can perfectly align themselves behind a strategic objective without guidance." That's strange to me, because the former surely seems to be much more difficult than the latter.
- jbay808 8y agoCan you provide an example of a metric that meets this criteria?
- philipodonnell 8y ago
- fooblat 8y agoI think this article conflates Management by Metrics and intense focus on short terms results. Management by Metrics is just as compatible with a long term focus or goal. Now on the issue of people gaming metrics, that is a real concern and I typically address it with "counter metrics" intended to protect against gaming. It is not perfect, but it works well enough when you take the time to choose your metrics (and counter metrics) carefully. So instead of "improve A by 20%" we get "improve A by 20% without impacting B"
- AstralStorm 8y agoHow do you measure intangibles like knowledge or customer service or innovation? How do you weigh any number against another? Surely a single number or sign only is insufficient for such broad areas. Much less a predictor based on one of those numbers which is why... measures lie. People lie less often and in a more detectable way. The tangible results of them can be measured but there are multiple variables affecting it, from politics, through skill, talent and finally luck. If any of those elements fail, you get nothing, despite intangible gains that may provide a long term advantage.
- navigatesol 8y ago>How do you measure intangibles like knowledge or customer service or innovation? You figure out proxies as best you can. How do you know "customer service" or "innovation" are important, are improving, exist, or are what you think they are, without figuring out a way to measure them? This is precisely the bane of buzzwords. "We focus on innovation here!" but no one can determine what that is or if the company is any good at it. >The tangible results of them can be measured but there are multiple variables affecting it, from politics, through skill, talent and finally luck. There are a few comments in this thread that seem to come from the same premise: that measuring is hard because "multiple variables" come into play when you measure things. I hate to break it to you, but those factors are in play whether you are measuring things or not. The purpose of measuring is to minimize the unknowns and reduce variance. All of the variety of factors that are in play in a massive organization are precisely why you need to measure things, not a reason to cast measurement aside.
- rubidium 8y agoA good manager will use metrics as information to help in assessment of work, not as the sole measurement. Very important for any manager to coach is the real organizational concern behind the number. If you’ve got more that n=5 things, at a certain point you _need_ metrics to make sense of how things are going. Are we getting our product to customers when they want it (OTD)? Being able to say, yes 90% of the time is very helpful compared to saying “who knows, we don’t do metrics here.”
- jacques_chester 8y agoIt's time again for me to recommend Measuring and Managing Performance in Organizations by Austin[0]. Don't let the underwhelming title fool you: it's largely a book on why it can't be done. Austin takes classical principal-agent models and extends it to add a third participant: the client. He then shows that difficult-to-observe work at first improves and then worsens under any metrics regime. No single metric can be found that will give the desired performance. The basket of metrics that can successfully create the desired outcome is one where the principal has total visibility into the agent's work -- which contradicts the original problem of difficult-to-observe work (ie, anything requiring skill). [0] http://www.dorsethouse.com/books/mmpo.html http://www.dorsethouse.com/books/mmpo.html
- cloogshicer 8y agoThanks for the recommendation, this sounds very interesting.
- Loughla 8y agoThat does look interesting, thank you for the recommendation.
- luiscleto 8y agoReminds me of Goodheart's law[1]. We have known this for a long time, but it is hard to have an alternative system which scales well with huge organizations. For startups and small companies, I could see an informal system working pretty well, but as the company grows to hundreds or thousands of employees, it becomes necessary to standardize and have some kind of metrics used for reporting and evaluations. This will inevitably shift the company's culture towards gaming those metrics. Somewhat like grading systems in education. High grades don't necessarily mean you will be capable of generating more value to society than average grades or even low grades. And students often become good at improving their grades without that actually adding much value. But there is a correlation. And we don't have many better (non-experimental) alternatives that I'm aware of. [1] https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
- asplake 8y ago> ...but as the company grows to hundreds or thousands of employees, it becomes necessary to standardize and have some kind of metrics used for reporting and evaluations. Does it? Is it inconceivable that each part works to its own goals and metrics consistent both with its own values and those of the wider organisation?
- michaelcampbell 8y ago> Is it inconceivable that each part works to its own goals and metrics consistent both with its own values and those of the wider organisation? Beyond a certain number of people, absolutely it is.
- chii 8y ago> Beyond a certain number of people, absolutely it is. that's because of the lack of trust, and the flow of responsibility and control. If you structure a company such that the employees themselves has to be responsible for their output in such a way that higher output leads to more money for them, you'd not have this problem. For example, contractors who work on a results basis.
- dr_dshiv 8y agoIn education, there is a practice known as "backwards design" described in the book "understanding by design". Here, goals are set, behaviours that are evidence of achieving the goal are set (these are assessments), and then instructional activities are designed that "move the needle" on the assessments. But the critical 4th step in this iterative process is continuous alignment between the goals, assessments and instructional activities. This alignment process is intended to prevent the negative outcomes of teaching to the test. From that perspective, any metric oriented organisation might want to have an explicit alignment process that ensures that the metrics continue to capture the desired outcome.
- crispyambulance 8y ago> ...any metric oriented organisation might want to have an explicit alignment process that ensures... Yes, I think many folks would agree that metrics aren't an easy thing to use properly. One huge problem, however, is that organizations that are apt to rely on metrics usually aren't sensitive enough to realize that there's more to their use than coming up with a wish-list and then dreaming up bullshit KPI's which are somehow intended "move the needle" towards correct outcomes. And when it doesn't someone just bullshits their way around the failure instead of doing the hard work needed to honestly address mistakes and problems. That's, I think, what happens a lot in dysfunctional school districts, though many of us might see this comedy of errors show up in orgs with a matrix management structure.
- shikoba 8y agoI'm a developer. I dare you to provide a metric that can measure my outcome. There is no metric that can measure my outcome.
- navigatesol 8y ago>The intelligence analysts who ultimately located Osama bin Laden worked on the problem for years. If measured at any point, the productivity of those analysts would have been zero. Month after month, their failure rate was 100 per cent, until they achieved success. I guess if the metric was "caught Osama Bin Laden or not", then yeah, I agree. But who on earth would measure the analyst's productivity just so? Nothing but a strawman. >The source of the trouble is that when people are judged by performance metrics they are incentivised to do what the metrics measure, and what the metrics measure will be some established goal. But that impedes innovation, which means doing something not yet established, indeed that hasn’t even been tried out. Why? If a (admittedly weak) metric like "# of paying customers" is used as a goal, how does that "impede innovation"? How are people being stopped from coming up with creative ways to get more customers? Why do we assume the ways in which they improve the metric will be bad, and not innovative?
- jimktrains2 8y ago> I guess if the metric was "caught Osama Bin Laden or not", then yeah, I agree. > But who on earth would measure the analyst's productivity just so? Nothing but a strawman. Is that much different than measuring LoC, tickets closed, or features implemented? Some features just take time -- sometimes just the research alone to implement something could take a few weeks where literally nothing measurable happens. That's the point: completion metrics aren't useful for larger or open-ended projects. There was no gnatt chart, no schedule for finding Osama bin Laden; similarly you sometimes cannot plan out a large software project from the very start.
- snovv_crash 8y agoBecause it doesn't take inyo account CAC, which might be higher than the profits made per customer.
- tehjoker 8y agoI really disliked that example. The Pakistanis knew where he was for years and the CIA sent teams of "vaccinators" to conduct a hunt for him, making many people in need of vaccinations distrust modern medicine, a crime that no one will answer for that could easily cost more lives than bin Laden ever did. This is OT, but if you're interested in understanding this epochal event in American media narrative, there's no one better than Sy Hersh to listen to: https://www.lrb.co.uk/v37/n10/seymour-m-hersh/the-killing-of-osama-bin-laden https://www.lrb.co.uk/v37/n10/seymour-m-hersh/the-killing-of... In a way, I guess it is on topic because it shows how "metric fixation" leads to unjust and dangerous outcomes for the people of the world.
- jamessun 8y ago"Show me the incentive and I'll show you the outcome" -Charlie Munger
- deleted 8y ago[deleted]
- peterwwillis 8y ago> Economists [..] report that in recent years the only increase in total-factor productivity in the US economy has been in the information technology-producing industries. And guess how those industries did that? Metrics. Metrics are a simple tool. They are almost worthless by themselves and can even be a detriment. But like any tool, if you pick the right ones and use them right, it's much better than not having the tool. There are tons of ways business processes can improve by analyzing metrics, it's not just an employee productivity stick.
- alexandercrohde 8y agoAll valid points. Of course though, you can't rail against metrics as a whole. We all use them every day and always will. We don't personally verify a financial advisor is a genius, we look at their annual rate of return. We trust that if unit tests fail, code isn't good. We trust the SATs, which are a better and more scalable and fair measure of aptitude than any group of humans I could imagine. The piece rails against metrics, which often backfire, but doesn't even mention the alternative (trust those in power make any choice they desire with no justification) which also is often abused.
- okaleniuk 8y agoI used to work once for a company where the developers were reworded for writing more lines of code than the average. Once.
- deanalevitt 8y ago> Contrary to commonsense belief, attempts to measure productivity through performance metrics discourage initiative, innovation and risk-taking. Is this really contrary to commonsense belief? I think most business leaders are very aware of this, however, the alternative is chaos. That's why certain specialist teams are given leeway.
- acdha 8y agoI think awareness is very bimodal. I've sat in rooms with people who really thought they could transform a business with 5% YOY improvements. (That's not always the wrong view, either — a key part of this is recognizing what kind of situation you're in and whether you really have the trust and resources to bank on a revolutionary change)
- Symmetry 8y agoIs there any reason to think that people are worse at gaming qualitative evaluations than they are at metrics, in general? Things like making friends with one's boss, appearing to work hard, and so forth.
- js8 8y agoThe people who are gaming are probably not. However, the people who are not trying to game the system might decide to do it if there is an objective goal. But perhaps you're right. In capitalism, the most common purpose why companies exist is to game the system (i.e. make profit). I don't see why in that case, employees shouldn't try to game the people who game them (i.e. capitalists). However, in companies that are collectively owned, this becomes less clear.
- SkyBelow 8y ago>However, in companies that are collectively owned, this becomes less clear. Unless it has no management there is still a management class with disproportionate decision making power and thus the antagonistic relationship necessary for there ti be a clear case for gaming continues to exist. Humans are really bad at removing hierarchies, to the extent that I would assume any place without hierarchies has them in a harder to notice format.
- commandlinefan 8y agoNo, but performance metrics actively damage the work of the people who aren't gaming the system. If you can't make things better, you can at least not make them worse.
- polskibus 8y agoReminds me of a great book, about managing the expectations instead of managing the companies for real, and how the system is rigged to select for short-term gains what in most cases backfires. https://www.amazon.com/Fixing-Game-Bubbles-Crashes-Capitalism/dp/1422171647 https://www.amazon.com/Fixing-Game-Bubbles-Crashes-Capitalis...
- agentultra 8y agoI don't have an attribution for this quote, so if someone knows it please share... If you want an opinion let's go with mine. Otherwise let's look at the data. It's true that many quantitative metrics can incentivize people in the wrong way. There's a funny story about a city trying to get rid of rats that incentivizes people to hunt rats and turn in the tails... but the people start farming the rats in order to get the reward instead while the project leaders pat each others' backs. However as an engineering leader you should not discount the value of qualitative metrics or metrics altogether. I try to only make a decision "from the gut" when there is no other choice and decision must be made. Data is key. How do you know how well your code review culture is thriving if you do not see, across the entire team, metrics reporting the number of open pull requests, how long they are staying open on average, how many comments they're getting, etc? How do you know if one of your engineers is struggling if you're not watching what gets checked in, what code is being rewritten frequently, etc? How do you gain leverage for your team to management if their are no facts, metrics, to back you your arguments? Just a reminder that you should use metrics. Maybe avoid using them as reward mechanisms and use them as leverage instead. I don't work for GitPrime but you should definitely check out what they're up to. https://try.gitprime.com/data-driven-engineering-metrics/ https://try.gitprime.com/data-driven-engineering-metrics/
- hopler 8y ago"and if you want data, let's go with mine". -- how to lie with statistics.
- agentultra 8y agoWell if you want to play that game then how do we know the person who's relying on their instincts and intuitions isn't lying either? Who do we believe? I didn't suggest using metrics to make anyone accountable but using them as leverage to empower your team. What would be the point of gaming those metrics? Nobody would profit from lies and if you're checking the providence of your data, in the case of Gitprime your own repository, then identifying a culprit would be rather easy.
- noarchy 8y ago
- edw 8y agoHas it occurred to anyone else that the HN karma system is a performance measurement technique that undermines the goals of HN? I've stopped commenting, largely because for any comment I might make, I can imagine the cynical karma-harvesting comment that someone will make in response, the comment that would never have been made if there were no karma system here. And when I do comment, I waste a lot of time trying to harden my comments to such opportunism.
- vbuwivbiu 8y agoI've largely stopped commenting due to karma also. It's not just that wording the same sentiment differently can result in either up or downvotes, it's that I don't want to have to care about getting points on the internet
- edw 8y agoRight on. I'm a photographer as well as a programmer, and I've largely stopped sharing photos on Instagram because I don't want to measure my work based on the number of likes each receives, and I don't want the people I care about who "follow" me there to feel the need to like my work. I've watched friends go through their various networks' feeds and mechanically like, react, whatever lest their friends think they're giving them the cold shoulder. For the same reason, I've come to find using Uber exhausting. Why do I care about my 4.75 (was 4.77) rating? I take taxis whenever practical because I don't want to endanger my rating by making too much small talk or not making enough small talk.
- diminoten 8y agoIf HN were to revamp how it handled comment sorting, what features would get you to comment again?
- edw 8y agoA thought experiment: What if HN changed nothing except that it never displayed point counts for posts, comments, or users? I think it would tamp down gadfly behavior. It may also, however, do the same to behavior that we want to encourage. There are no easy answers.
- _Codemonkeyism 8y agoRequired reading "Measuring and Managing Performance in Organizations" Has a lot of information about how people react and adapt to setting KPIs, aka "gaming the system".
- wins32767 8y agoManagement via metrics is a really useful shortcut to generating alignment, but it is a shortcut. If you tell people that they need to move a number there isn't a lot of ambiguity and you can just keep beating the same drum over and over. Trying to align people on the whole business context is much harder and takes a lot more time though obviously much more valuable.
- mothsonasloth 8y agoIts happening all over my company, with dashboards for all kinds of useless metrics. The worst part is gaming unit test coverage to get over the 80% mark. I keep trying to say I would rather have 40% well written tests than 80% crappy tests. :(
- zentiggr 8y agoAs I read through the 50-odd comments, I wound up with a kanban / 5 whys level thought. All these points of discussion revolve around pluses and minuses of metrics, but the next level is "Why establish metrics at all?" At a deeper (very generalized) level, we've been infected by production line, Deming style, efficiency focus. If you are handling previously defined routines, great. Optimizing can help you. As soon as you are working with adding value, be it in code, education, product development, metrics are all premature optimization. Promote the idea that you can't apply optimization techniques to creative steps and encourage managers to get off the numerical crutches and make value judgments. Takes a better manager but then that's the point anyway, right?
- hopler 8y agoWhich businesses succeed without a business plan? Without metrics, how do you keep your runway open?
- pathseeker 8y agoPeople manage to survive without planning out the number of breaths they will take every day. A competent engineering team is going to do well even without target lines of code per second goals.
- NyxWulf 8y agoI would argue that's counter to your intended point though. If you go into the E.R. or the hospital, one of the vitals they measure is in fact the number of breaths taken per unit of time. In a normally functioning human we don't have to worry about that metric because it is self regulating. If on the other hand you are in critical condition, measuring your breaths is a very important metric. So as I've said elsewhere, the problem isn't measurement. The problem is as Deming said "The aim defines the system". Well thought out measures in accordance with clear goals and aims can be valuable. Like anything though, it requires careful consideration and may need to be revised based on outcomes.
- lazyasciiart 8y ago
- Mikeb85 8y agoI think metrics can be very useful but as with many things, you need to know how to interpret the data. I'm in the restaurant industry, so my #1 metric is profit (over the long run anyhow). Now, the next metric is sales, since profit is a % of sales. So to get those sales, we're looking at how many new guests are coming through the door, repeat guests, average guest check, etc... After that, it's margins. Sales mixes play into that (relating to COGS), and labour. Now, to determine what we're getting out of our labour, more metrics. How much each server is selling, what products they're selling, how many guests they can serve in a night, etc... For cooks, it's about productivity and consistency. Anyhow, there's a lot of shit restaurants and workers out there (due to low barrier of entry education-wise), lots of people misinterpret the data, leaving to a feedback loop of shit managers and employees. Often restaurants will focus too much on one of those metrics, leading to places that just gauge guests, or others that entertain guests who are fishing for free shit. Now, as for metrics in the programming world which probably don't matter, I always hear about lines of code written. Obviously it's easy to game and hard to discern, as code can be too terse or too verbose. Programmers are also usually so far removed from the sales part of the business that there's no objective sales metric to use either. And that's where you need good managers. I'm sure there's a good set of metrics that give some idea of performance, but they need to be interpreted by someone. And all of this reminds me of stats/economics (what I did in university), where you're bombarded with data and need to interpret it. Like GDP per capita can indicate general well being, but then you adjust it to PPP to get a better metric of quality of life, and add inequality calculations to understand how it affects different segments of the population, and you can go even deeper with specific stats underlining quality of life (amount of disposable income, amount spent on housing, amount spent on entertainment, etc...). Anyhow, tldr here is that metrics matter, but interpreting them is a skill and makes all the difference.
- ratling 8y agoYou get what you measure.
- draw_down 8y agoThe upside is that as an individual contributor you just need to focus on driving the metrics, and it's pretty clear-cut what success looks like. If "professional judgment" determines that things aren't good despite the metrics saying they are, change what's measured or how it's measured. Metrics are also the only way to even get close to something resembling objectivity. Some would claim that objectivity in our hiring will end implicit bias, resulting in a wonderfully diverse workplace which will have all sorts of good effects. I don't think we're at the point of questioning all that just yet, so, hooray for metrics and objectivity and diversity!! :) Ultimately something is being measured, whether that is made explicit or not. Certainly there are issues with being explicit about what's measured, but there are also issues with not being explicit about it.
- jl2718 8y agoCompared to the other sciences, it seems to me that management science, as an entire discipline, is a nearly complete failure. This is a field where startup experiments by complete newbies lead the conversation against a network of business schools and long-standing organizations. There seems to be no consensus whatsoever on the best way to manage people. It’s not even clear, to me at least, that modern management practices at many large corporations are provably better than some kind of naive baseline like a pure democracy or pure dictatorship. Expert texts are nothing more than pontification and hand-picked anecdotes. Does anybody really know what works?
- gabriel34 8y agoEven worse, in the same source basic concepts have multiple definitions, partially overlapping, partially contradictory. This is exactly why I believe C-level officers are grossly overpaid. Previous success does not guarantee future success, especially in different industries, hence they don't know some magic formula to make his salary * 1000 in profits for the shareholders anymore than a mid level manager does.
- KC8ZKF 8y agoThe author was interviewed on the Econtalk podcast last year. http://www.econtalk.org/jerry-muller-on-the-tyranny-of-metrics/ http://www.econtalk.org/jerry-muller-on-the-tyranny-of-metri...
- gabriel34 8y agoThis has been subject of administration theories for the last century: https://en.wikipedia.org/wiki/Management_by_objectives#Arguments_against https://en.wikipedia.org/wiki/Management_by_objectives#Argum...
- hartator 8y ago> to replace professional judgment (acquired through personal experience and talent) The thing is there is no magical way to measure this. How you define personal experience and talent? From the number of years spent in the industry? It's still back to another metric. Metrics do matter. Common sense does also matter. Making decisions in a chaotic world is indeed difficult.
- teahat 8y agoWhat this article misses out is that the metrics will be present and used whether they are articulated or not. By making them explicit they can be improved, validated, debated, and removed. Standards are consistent. When they're implicit, you get none of those benefits and instead promote an insider culture of unspoken biases that get no scrutiny at all. The choice isn't between metrics and human intuition, it's between explicit and implicit metrics. Of course there are bad metrics; to use an example from the article - measuring the output of analysts finding OBL on a binary metric is simply a bad metric. But if we start from there, we can improve it. How many leads do they have at the current time? What stage is each lead at? What intelligence has it generated to this point?
- deleted 8y ago[deleted]
- maxprimer 8y agoThe author in the article focuses only on one-half of the metrics within a given process, the lagging results. There is another whole dimension to performance metrics which is the causal dimension, your leading metrics or leading indicators. You can only effectively use a lagging indicator–such as time to restore–once you understand the activities that make up the world of restoring service. Does the product have an SOP document? Is there monitoring in place to alert technicians quickly? What is the training level of your employees? Are your employees empowered and engaged? All of these things matter far more than lagging outputs. Sure, we can disparage lagging indicators all day, but we can't throw them out just because they can be gamed. You have to push deeper and that requires time, process knowledge, and building a healthy working environment.
- pwncake 8y agoThe Hustle wrote a good article about this a little while ago: https://thehustle.co/Goodharts-Law https://thehustle.co/Goodharts-Law
- avrohom770 8y agoFinding useful performance metrics for evaluation is difficult and assumes somebody knows what is best for the dept/company/division, that the evaluators are well trained, and which metrics show the correct way. There are useful metrics if you are playing Golf, baseball, poker, bridge, chess, but even then individual performance will vary year to year and even the stars have slumps. The Normal Distribution shows 95% of employees are within +3 Std Dev of the mean with only 2.5% excelling and 2.5% needing help in a given year and in the next year everyone will have a different ranking. Performance metrics are mostly a waste of valuable time for both the employee and the management.