3 ms·

Measuring outcomes is hard, especially in comparison to outputs which are easier to quantify but are different than outcomes. True measures of outcomes usually

Measuring outcomes is hard, especially in comparison to outputs which are easier to quantify but are different than outcomes. True measures of outcomes usually can only be tracked over a long period of time, whereas outputs will give instant feedback. Look at what has been happening in the schools with standardize testing. Proponents think they are measuring outcomes (effectiveness in preparing children to be skilled & productive members of society) with standardized tests. They are actually measuring outputs (test scores) as a proxy for the outcomes. The result is the system being measured will adapt. Schools "teach to the test" which improves the outputs but we don't know the effect on the desired outcomes. It may result in children being less prepared. This is a well known effect in companies that measure individual performance in terms of outputs (widgets manufactured, # of upsells, etc). The employees will deliver the required metrics but it often comes at a cost to quality and customer satisfaction which is a negative outcome to the company. Which is the main lesson in quantifying outcomes: Measuring the wrong thing can be worse than not measuring at all.

DanielBMarkham 14y ago
"Measuring the wrong thing can be worse than not measuring at all." I'm calling bullshit on that one. Yes, measuring the wrong thing can incentivize poor outcomes, but movement in the wrong direction is visible and can be corrected, whereas lethargic inaction just lazily drifts into a morass. You have to measure outcome. Period. We can have great discussions around what kinds of measurements to have and how to apply them, but money should not go out the door without some indication that something is happening as a result of it -- even the wrong thing. In fact, big visible indicators of the wrong thing happening as a result of measurements is a terrific result. If we're arguing over measurement, the game is already lost. Another way of stating this is that let's say I pick a measurement to implement. It's a terrible measurement, and it causes much pain and suffering. You come to me and say "This is a terrible measurement and causes much pain and suffering!" To which I respond "How do you know this?" So you show me a study -- another form of measurement! Then we have a conversation about including parts of your study in my measurement, scrapping my measurement entirely, and so forth. We cannot have meaningful conversations about management without measuring things. Once we start measuring things, the world is wide open about which things to measure. But if you convince me that the act of measuring itself is broken, we have no further basis for any conversation around performance at all. Management has no function. There is no quantifiable return for any possible amount of money I might invest. Very bad. We don't need to wait around to pick the perfect measurement or do nothing at all. We need to pick measurements -- any measurements -- and rapidly iterate towards finding what works best for us. If the testing system is broken, and I have no doubt that it is, it's broken based on bureaucrats creating a large, brittle, hard-to-change architecture of testing, instead of just trying to continuously evolve what works best. It has failed because we put the same idiots in charge of school testing that were creating the mess the tests were supposed to fix in the first place.
- bryanlarsen 14y ago
  "So you show me a study" No, he shows you an anecdote or two. Which isn't worth very much, but it's more than what you had before. It's not like the original measurement had any studies validating it either. But it looks scientific: it has numbers!
  - JoeAltmaier 14y ago
    But there are so many ways to mismanage a city! I'm sure that makes it hard to prove one is the 'worst'.
marknutter 14y ago
You know what measures outcomes exceptionally well? The free market. Turns out, if you suck at what you do, or a project is mismanaged, it fails. When there isn't a constant source of revenue via taxes, people suddenly start to care whether or not money is being spent wisely. And wouldn't you know it, they come up with all sorts of sophisticated ways to determine whether or not they're likely to fail and adjust accordingly.

DanielBMarkham 14y ago

"Measuring the wrong thing can be worse than not measuring at all." I'm calling bullshit on that one. Yes, measuring the wrong thing can incentivize poor outcomes, but movement in the wrong direction is visible and can be corrected, whereas lethargic inaction just lazily drifts into a morass. You have to measure outcome. Period. We can have great discussions around what kinds of measurements to have and how to apply them, but money should not go out the door without some indication that something is happening as a result of it -- even the wrong thing. In fact, big visible indicators of the wrong thing happening as a result of measurements is a terrific result. If we're arguing over measurement, the game is already lost. Another way of stating this is that let's say I pick a measurement to implement. It's a terrible measurement, and it causes much pain and suffering. You come to me and say "This is a terrible measurement and causes much pain and suffering!" To which I respond "How do you know this?" So you show me a study -- another form of measurement! Then we have a conversation about including parts of your study in my measurement, scrapping my measurement entirely, and so forth. We cannot have meaningful conversations about management without measuring things. Once we start measuring things, the world is wide open about which things to measure. But if you convince me that the act of measuring itself is broken, we have no further basis for any conversation around performance at all. Management has no function. There is no quantifiable return for any possible amount of money I might invest. Very bad. We don't need to wait around to pick the perfect measurement or do nothing at all. We need to pick measurements -- any measurements -- and rapidly iterate towards finding what works best for us. If the testing system is broken, and I have no doubt that it is, it's broken based on bureaucrats creating a large, brittle, hard-to-change architecture of testing, instead of just trying to continuously evolve what works best. It has failed because we put the same idiots in charge of school testing that were creating the mess the tests were supposed to fix in the first place.

bryanlarsen 14y ago
"So you show me a study" No, he shows you an anecdote or two. Which isn't worth very much, but it's more than what you had before. It's not like the original measurement had any studies validating it either. But it looks scientific: it has numbers!
- JoeAltmaier 14y ago
  But there are so many ways to mismanage a city! I'm sure that makes it hard to prove one is the 'worst'.

marknutter 14y ago

You know what measures outcomes exceptionally well? The free market. Turns out, if you suck at what you do, or a project is mismanaged, it fails. When there isn't a constant source of revenue via taxes, people suddenly start to care whether or not money is being spent wisely. And wouldn't you know it, they come up with all sorts of sophisticated ways to determine whether or not they're likely to fail and adjust accordingly.