5 ms·

I think this article conflates Management by Metrics and intense focus on short terms results. Management by Metrics is just as compatible with a long term focu

I think this article conflates Management by Metrics and intense focus on short terms results. Management by Metrics is just as compatible with a long term focus or goal. Now on the issue of people gaming metrics, that is a real concern and I typically address it with "counter metrics" intended to protect against gaming. It is not perfect, but it works well enough when you take the time to choose your metrics (and counter metrics) carefully. So instead of "improve A by 20%" we get "improve A by 20% without impacting B"

AstralStorm 8y ago
How do you measure intangibles like knowledge or customer service or innovation? How do you weigh any number against another? Surely a single number or sign only is insufficient for such broad areas. Much less a predictor based on one of those numbers which is why... measures lie. People lie less often and in a more detectable way. The tangible results of them can be measured but there are multiple variables affecting it, from politics, through skill, talent and finally luck. If any of those elements fail, you get nothing, despite intangible gains that may provide a long term advantage.
- navigatesol 8y ago
  >How do you measure intangibles like knowledge or customer service or innovation? You figure out proxies as best you can. How do you know "customer service" or "innovation" are important, are improving, exist, or are what you think they are, without figuring out a way to measure them? This is precisely the bane of buzzwords. "We focus on innovation here!" but no one can determine what that is or if the company is any good at it. >The tangible results of them can be measured but there are multiple variables affecting it, from politics, through skill, talent and finally luck. There are a few comments in this thread that seem to come from the same premise: that measuring is hard because "multiple variables" come into play when you measure things. I hate to break it to you, but those factors are in play whether you are measuring things or not. The purpose of measuring is to minimize the unknowns and reduce variance. All of the variety of factors that are in play in a massive organization are precisely why you need to measure things, not a reason to cast measurement aside.
mikekchar 8y ago
Every measurement you make that is advertised will change behaviour. It is very important when making measurements to take that into consideration. Similarly, there is a difference between measurements and metrics. A measurement is simply a measure of something. A metric is a measurement that you make that is fed back into the process improvement cycle: in other words it is a measurement you make to drive changes to the official process. Things like targets or published measurements that have no particular purpose in your process improvement cycle are risky because they will change behaviour in undefined ways. You should always avoid them if you can. Measurements should be taken in order to answer a question. You should avoid taking measurements if you have no questions. Similarly, you should try to measure the minimum that you can that will answer your question. This will minimise the change in behaviour. Before you can have metrics you need a defined process (because by definition a metric is a measurement that is used to change your defined process). Avoid taking metrics when you are unsure of your existing process. When you change your process as a result of a metric, use measurements to answer the question, "Did we affect things we didn't expect to affect". If the answer is "Yes", then take more measurements to answer the question, "Is this change something we need to do something about". In this way you will be able to control your process. One last thing. Do not attempt to protect against gaming other than to not measure things. People naturally optimise and will do it whether they are aware of it or not. Simply observe the effect of your measurements and decide if they are what you want. If not, then remove the measurement.
- renholder 8y ago
  >Things like targets or published measurements that have no particular purpose in your process improvement cycle are risky because they will change behaviour in undefined ways. You should always avoid them if you can. The problem is that a lot of businesses focus hard on metrics and this doesn't translate to a process improvement cycle, rather, it's on a "person improvement" cycle - making raises, bonuses, promotions, etc. dependent on that notion. A perfect example how this is problematic is in areas like Customer Support, where a number of handled tickets doesn't equate to a good value of the product provided. In a posit, let's consider there's two Support Agents working on supporting 'X' product for 'Y' company, over the course of a quarter. The product and the company have been around and established for quite some time, so there's no need for semantics on availability pool in this posit. Support Agent 1 works a case that takes 3 months but is a relevantly large product bug that impacts millions of users around the world via the fix. Support Agent 1's workload, of course, has a considerably smaller queue size due to this. Now, Support Agent 2 keeps picking-up the low-hanging fruit because it's a numbers game and they know this. What should happen is that management sees the disproportionate numbers and attempts to find out why that is (and/or attempts to automate the low-hanging fruit out of the equation). They should also look at the impact that both of their work had, in overall user-base. Instead, what more than likely happens, is because Support Agent 1's numbers were so low, this makes Support Agent's Manager's numbers look bad and that goes up the chain. Upper Management, oft, doesn't care for the semantics around numbers, so shit rolls down-hill (as the colloquialism goes) and, now, Support Agent 1 is on "Unsatisfactory/Needs Improvement" for their review cycle, losing raises, bonuses, promotions, etc. I think that would disenfranchise Support Agent 1. Since this is a majority American board, consider the wonderful hell that is known as Wal-Mart. From what I understand, it also utilises metrics across the board. If you think on it, can you name a person "happy" to work there? I'd reckon you couldn't. (Again, this is my understanding of it and not anedotal, first-hand experience with the company or anyone who works for it. Aside from Asda, in the UK, Wal-Mart isn't a "thing" here.) So, more than just being fruitless, if it doesn't go into the process improvement cycle, I'd say it rolls pretty high numbers in damaging the overall morale.
jacques_chester 8y ago
> Now on the issue of people gaming metrics, that is a real concern and I typically address it with "counter metrics" intended to protect against gaming. From the article: > In an attempt to staunch the flow of faulty metrics through gaming, cheating and goal diversion, organisations often institute a cascade of rules, even as complying with them further slows down the institution’s functioning and diminishes its efficiency.
- fooblat 8y ago
  This reads to me like a culture problem. A good engineering culture should help a lot to insulate against people who just want to game the system and aren't really invested in the success of the project. Of course, how you get there is anything but simple.
  - XorNot 8y ago
    The problem is: what is success of the product?
    - renholder 8y ago
      >The problem is: what is success of the product? Agreed. If it's measured in a metric, like bugs for every check-in or number of check-ins or numbers of lines of code, then reducing it down to metrics doesn't do anything to achieve the goal of the success of the product. I'd - cautiously - argue it actually favours success of the individual over success of the product.
  - michaelcampbell 8y ago
    So, "people who do good things well, will do good things well". It always boils down to what to measure, how to measure it, and how to deal with gaming that. "Boils down to culture" is just weasel words.

AstralStorm 8y ago

How do you measure intangibles like knowledge or customer service or innovation? How do you weigh any number against another? Surely a single number or sign only is insufficient for such broad areas. Much less a predictor based on one of those numbers which is why... measures lie. People lie less often and in a more detectable way. The tangible results of them can be measured but there are multiple variables affecting it, from politics, through skill, talent and finally luck. If any of those elements fail, you get nothing, despite intangible gains that may provide a long term advantage.

navigatesol 8y ago
>How do you measure intangibles like knowledge or customer service or innovation? You figure out proxies as best you can. How do you know "customer service" or "innovation" are important, are improving, exist, or are what you think they are, without figuring out a way to measure them? This is precisely the bane of buzzwords. "We focus on innovation here!" but no one can determine what that is or if the company is any good at it. >The tangible results of them can be measured but there are multiple variables affecting it, from politics, through skill, talent and finally luck. There are a few comments in this thread that seem to come from the same premise: that measuring is hard because "multiple variables" come into play when you measure things. I hate to break it to you, but those factors are in play whether you are measuring things or not. The purpose of measuring is to minimize the unknowns and reduce variance. All of the variety of factors that are in play in a massive organization are precisely why you need to measure things, not a reason to cast measurement aside.

mikekchar 8y ago

Every measurement you make that is advertised will change behaviour. It is very important when making measurements to take that into consideration. Similarly, there is a difference between measurements and metrics. A measurement is simply a measure of something. A metric is a measurement that you make that is fed back into the process improvement cycle: in other words it is a measurement you make to drive changes to the official process. Things like targets or published measurements that have no particular purpose in your process improvement cycle are risky because they will change behaviour in undefined ways. You should always avoid them if you can. Measurements should be taken in order to answer a question. You should avoid taking measurements if you have no questions. Similarly, you should try to measure the minimum that you can that will answer your question. This will minimise the change in behaviour. Before you can have metrics you need a defined process (because by definition a metric is a measurement that is used to change your defined process). Avoid taking metrics when you are unsure of your existing process. When you change your process as a result of a metric, use measurements to answer the question, "Did we affect things we didn't expect to affect". If the answer is "Yes", then take more measurements to answer the question, "Is this change something we need to do something about". In this way you will be able to control your process. One last thing. Do not attempt to protect against gaming other than to not measure things. People naturally optimise and will do it whether they are aware of it or not. Simply observe the effect of your measurements and decide if they are what you want. If not, then remove the measurement.

renholder 8y ago
>Things like targets or published measurements that have no particular purpose in your process improvement cycle are risky because they will change behaviour in undefined ways. You should always avoid them if you can. The problem is that a lot of businesses focus hard on metrics and this doesn't translate to a process improvement cycle, rather, it's on a "person improvement" cycle - making raises, bonuses, promotions, etc. dependent on that notion. A perfect example how this is problematic is in areas like Customer Support, where a number of handled tickets doesn't equate to a good value of the product provided. In a posit, let's consider there's two Support Agents working on supporting 'X' product for 'Y' company, over the course of a quarter. The product and the company have been around and established for quite some time, so there's no need for semantics on availability pool in this posit. Support Agent 1 works a case that takes 3 months but is a relevantly large product bug that impacts millions of users around the world via the fix. Support Agent 1's workload, of course, has a considerably smaller queue size due to this. Now, Support Agent 2 keeps picking-up the low-hanging fruit because it's a numbers game and they know this. What should happen is that management sees the disproportionate numbers and attempts to find out why that is (and/or attempts to automate the low-hanging fruit out of the equation). They should also look at the impact that both of their work had, in overall user-base. Instead, what more than likely happens, is because Support Agent 1's numbers were so low, this makes Support Agent's Manager's numbers look bad and that goes up the chain. Upper Management, oft, doesn't care for the semantics around numbers, so shit rolls down-hill (as the colloquialism goes) and, now, Support Agent 1 is on "Unsatisfactory/Needs Improvement" for their review cycle, losing raises, bonuses, promotions, etc. I think that would disenfranchise Support Agent 1. Since this is a majority American board, consider the wonderful hell that is known as Wal-Mart. From what I understand, it also utilises metrics across the board. If you think on it, can you name a person "happy" to work there? I'd reckon you couldn't. (Again, this is my understanding of it and not anedotal, first-hand experience with the company or anyone who works for it. Aside from Asda, in the UK, Wal-Mart isn't a "thing" here.) So, more than just being fruitless, if it doesn't go into the process improvement cycle, I'd say it rolls pretty high numbers in damaging the overall morale.

jacques_chester 8y ago

> Now on the issue of people gaming metrics, that is a real concern and I typically address it with "counter metrics" intended to protect against gaming. From the article: > In an attempt to staunch the flow of faulty metrics through gaming, cheating and goal diversion, organisations often institute a cascade of rules, even as complying with them further slows down the institution’s functioning and diminishes its efficiency.

fooblat 8y ago
This reads to me like a culture problem. A good engineering culture should help a lot to insulate against people who just want to game the system and aren't really invested in the success of the project. Of course, how you get there is anything but simple.
- XorNot 8y ago
  The problem is: what is success of the product?
  - renholder 8y ago
    >The problem is: what is success of the product? Agreed. If it's measured in a metric, like bugs for every check-in or number of check-ins or numbers of lines of code, then reducing it down to metrics doesn't do anything to achieve the goal of the success of the product. I'd - cautiously - argue it actually favours success of the individual over success of the product.
- michaelcampbell 8y ago
  So, "people who do good things well, will do good things well". It always boils down to what to measure, how to measure it, and how to deal with gaming that. "Boils down to culture" is just weasel words.