3 ms·

I'm currently reading "The Tyranny of Metrics" which provides a decent analysis of how over-measuring things that don't have anything to do with the real goal o

I'm currently reading "The Tyranny of Metrics" which provides a decent analysis of how over-measuring things that don't have anything to do with the real goal often serves as a deterrent to real progress. What managers want to know is how much do their workers contribute to the success of the company. What they measure is how much time do workers spend doing X. They're completely glossing over the middle step of how does the time spent doing X contribute to the success of the company, which is usually taken as some kind of linear equation. Over-measurement is often worse than not measuring anything at all.

padolsey 4y ago
> Over-measurement is often worse than not measuring anything at all. Yes. Not only is "time spent" or "code written" or "features implemented" or even "user engagement" a lossy metric in the best of cases but it can literally end up doing the exact opposite of what you intended. Campbell's & Goodhart's laws are often cited. The trap I see in software companies is a hyperfocus on quantifiable short term wins, entirely glossing over the value created by longer term cultivation of quality. One can see projects where code cleanliness, documentation, testing, CI, and performance have been effectively disincentivised, even though, in the short, mid, and long-term, you will see gains. Such gains, however, are less visible and less quantifiable to the kinds of people who define and foster the metric regimes. A narrative I observe variations of: Universe X: - Programmer A implements feature in one week. They get a admiration and a raise. - One year later: Programmer B implements feature in 3 weeks, because they had to use Programmer A's shoddy and inconsistent API. Universe Y: - Programmer A implements feature in two weeks. Less admiration and no raise. - One year later: Programmer B implements feature in 1 week, because they were able to use Programmer A's better implemented interface. Hopefully it's obvious which universe is preferable and produced a net-win. ... Bad metric regimes are unforuntately not _obviously_ bad when you're inside their machinations. They really need to be dug into to see the fallacies. One has to look no further than Meta/Facebook for the absolute deteriment caused by overzealous performance reviews and a fixation of experimental upticks in user engagement. It is a bad metric regime.
- etothepii 4y ago
  Universe Y is fine if the company is certain to survive for one year but it is quite possible that slow development of features will mean that in Universe Y there is no Programmer B to benefit from the beautiful API.
- IWillForgetThis 4y ago
  The universe X example has some merit too. The developer got a raise, the business was excited enough to get this feature done ahead of schedule that someone actually put in the effort to push through a raise, and the total cost for this was one man-week of extra work a year later - basically a drop in the bucket, budget-wise.
  - lostcolony 4y ago
    You're focusing too much on the specifics of the hypothetical rather than the point of the hypothetical. If you can't get out to the abstraction, then in universe X, programmer B-Z all spend 3 weeks on a feature, and in universe Y, programmer B-Z all spend 1 week on a feature. So we've got an extra man year being burned, all to save a man week upfront. Extend to whatever point short of breaking the analogy you care to.
- deleted 4y ago
  [deleted]
- Raidion 4y ago
  You never want to built anything shoddily, but there is a lot of value in doing minimum scope. Testing, clean code, etc shouldn't be negotiable in most shops, but scope can be really trimmed down in a lot of scenarios. Universe Z: - Programmer X implements feature A in 1 week with tiny scope - Feature B is mothballed and never gets built Universe A: - Programmer X implements feature in 2 weeks - One year later: This functionality is broken apart into a well defined microservice with additional responsibilities. This needs a total rewrite as a v2 API. Universe B: - Programmer implements a feature in 1 week - One month later with that extra week they were able to squeeze in an unrelated enhancement to the system that delivered .5% extra revenue during Black Friday. I find most of the productivity metrics fail because you're squeezing the wrong end of the process. You want to spend the most time making sure you're doing what matters and has value, and then it will almost always be worth the time. This makes people feel engaged, which will lead to a lot more productivity than when you're pushing them to build something they don't care about.
wussboy 4y ago
I think this was covered in "Thinking Fast and Slow", but when our brains encounter a hard problem, we often substitute the answer for an easy question. I think this process is at play in the dynamics you list above. "How does the time spent doing X contribute to the success of the company?" is a hard question. It has incredible value and every company should spend a great deal of time considering it. "How much time do workers spend doing X?" is an easy question, and so we have used its answer as the answer to the hard question.
AyyWS 4y ago
in 2015 I made a change that stopped 90% of the tickets that rolled into my team's queue. The remaining 10% were complicated real customer issues. Our entire department was closed 6 months later, because I assume our ticket completion was so low, and we were so slow to fix issues. :/
- chrisweekly 4y ago
  Ouch.
mrweasel 4y ago
It's not even over-measuring, it's measuring the wrong thing, because the thing you actually want to know isn't (easily) measurable. The whole thing is down to poor management(-culture). Good managers already know who their top performing team members are. They don't need these tools. Measuring LOC or time spend sitting at the desk is in my view attempting to compensate for lack in management skills. It's like advertising agencies that can't sell ads without AdSense. Boss-ware is an attempt by poor managers to make up for a lack of skills by applying something that looks science from a distance.
- deleted 4y ago
  [deleted]
UweSchmidt 4y ago
Workplace surveillance is bad even if it "measures the right thing" and makes us all super-efficient, over-surveilled workers.

padolsey 4y ago

> Over-measurement is often worse than not measuring anything at all. Yes. Not only is "time spent" or "code written" or "features implemented" or even "user engagement" a lossy metric in the best of cases but it can literally end up doing the exact opposite of what you intended. Campbell's & Goodhart's laws are often cited. The trap I see in software companies is a hyperfocus on quantifiable short term wins, entirely glossing over the value created by longer term cultivation of quality. One can see projects where code cleanliness, documentation, testing, CI, and performance have been effectively disincentivised, even though, in the short, mid, and long-term, you will see gains. Such gains, however, are less visible and less quantifiable to the kinds of people who define and foster the metric regimes. A narrative I observe variations of: Universe X: - Programmer A implements feature in one week. They get a admiration and a raise. - One year later: Programmer B implements feature in 3 weeks, because they had to use Programmer A's shoddy and inconsistent API. Universe Y: - Programmer A implements feature in two weeks. Less admiration and no raise. - One year later: Programmer B implements feature in 1 week, because they were able to use Programmer A's better implemented interface. Hopefully it's obvious which universe is preferable and produced a net-win. ... Bad metric regimes are unforuntately not _obviously_ bad when you're inside their machinations. They really need to be dug into to see the fallacies. One has to look no further than Meta/Facebook for the absolute deteriment caused by overzealous performance reviews and a fixation of experimental upticks in user engagement. It is a bad metric regime.

etothepii 4y ago
Universe Y is fine if the company is certain to survive for one year but it is quite possible that slow development of features will mean that in Universe Y there is no Programmer B to benefit from the beautiful API.
IWillForgetThis 4y ago
The universe X example has some merit too. The developer got a raise, the business was excited enough to get this feature done ahead of schedule that someone actually put in the effort to push through a raise, and the total cost for this was one man-week of extra work a year later - basically a drop in the bucket, budget-wise.
- lostcolony 4y ago
  You're focusing too much on the specifics of the hypothetical rather than the point of the hypothetical. If you can't get out to the abstraction, then in universe X, programmer B-Z all spend 3 weeks on a feature, and in universe Y, programmer B-Z all spend 1 week on a feature. So we've got an extra man year being burned, all to save a man week upfront. Extend to whatever point short of breaking the analogy you care to.
deleted 4y ago
[deleted]
Raidion 4y ago
You never want to built anything shoddily, but there is a lot of value in doing minimum scope. Testing, clean code, etc shouldn't be negotiable in most shops, but scope can be really trimmed down in a lot of scenarios. Universe Z: - Programmer X implements feature A in 1 week with tiny scope - Feature B is mothballed and never gets built Universe A: - Programmer X implements feature in 2 weeks - One year later: This functionality is broken apart into a well defined microservice with additional responsibilities. This needs a total rewrite as a v2 API. Universe B: - Programmer implements a feature in 1 week - One month later with that extra week they were able to squeeze in an unrelated enhancement to the system that delivered .5% extra revenue during Black Friday. I find most of the productivity metrics fail because you're squeezing the wrong end of the process. You want to spend the most time making sure you're doing what matters and has value, and then it will almost always be worth the time. This makes people feel engaged, which will lead to a lot more productivity than when you're pushing them to build something they don't care about.

wussboy 4y ago

I think this was covered in "Thinking Fast and Slow", but when our brains encounter a hard problem, we often substitute the answer for an easy question. I think this process is at play in the dynamics you list above. "How does the time spent doing X contribute to the success of the company?" is a hard question. It has incredible value and every company should spend a great deal of time considering it. "How much time do workers spend doing X?" is an easy question, and so we have used its answer as the answer to the hard question.

AyyWS 4y ago

in 2015 I made a change that stopped 90% of the tickets that rolled into my team's queue. The remaining 10% were complicated real customer issues. Our entire department was closed 6 months later, because I assume our ticket completion was so low, and we were so slow to fix issues. :/

chrisweekly 4y ago
Ouch.

mrweasel 4y ago

It's not even over-measuring, it's measuring the wrong thing, because the thing you actually want to know isn't (easily) measurable. The whole thing is down to poor management(-culture). Good managers already know who their top performing team members are. They don't need these tools. Measuring LOC or time spend sitting at the desk is in my view attempting to compensate for lack in management skills. It's like advertising agencies that can't sell ads without AdSense. Boss-ware is an attempt by poor managers to make up for a lack of skills by applying something that looks science from a distance.

deleted 4y ago
[deleted]

UweSchmidt 4y ago

Workplace surveillance is bad even if it "measures the right thing" and makes us all super-efficient, over-surveilled workers.