3 ms·

> Practically any measure in software I can dream up ... will be met with cries of "that doesn't measure my ability" AND "if you promote on those measures, peop

> Practically any measure in software I can dream up ... will be met with cries of "that doesn't measure my ability" AND "if you promote on those measures, people will start gaming the measures". And they will be right. Sure, any measurement system will fail, but there are means to make them more robust. One way is to make it so that what is measured is what is desired: gaming the system results in more of the desired behavior. Consider a support desk measured not by time-to-resolution or call volume, but post-resolution surveys: in order to 'game' the measurement, the agents must actively seek to promote customer satisfaction. Another, more controversial, means for more robust measurements might be an overt, active gamification. Using your example of LoC: instead of unilaterally praising large (or small) work efforts, LoC is used as the measurement in a bid war. I'm not envisioning a particular bid process (silent auction? Liar's dice style round-robin?) but anybody can call BS on the latest bid ... which then holds that bidder to implement. Beat your own estimate for praise, but for punishment you would have to exceed the prior (non-BS) wager. Couple a second bid value (deadline, for example) and the complexities of actively "gaming the system" becomes nigh on impossible. Between this article about the Atlanta cheating scandal and the NPR story you linked, I wholeheartedly believe that the "standardized test" regimen is currently broken. However, I was hopeful that you might have some suggestions on alternative measurements or ways we could better apply the current ones.

bsder 12y ago
Using tests for an "objective measure" is doomed on its face. That's gigantic sign that the people involved are more interested in blame transfer than in progress. You can use tests for trends. That's a good use of tests. You can use tests for feedback to the individuals directly involved. That's also a good use of tests. This corresponds to my "software metrics" points. If I suddenly see an uptick in average lines of code committed, I should take a look. If I start seeing a big increase in bugs filed, I should take a look and see what just happened. Most of the time, there's a good explanation. Sometimes, someone is doing something stupid and I need to go fix that. And, rarely, someone is doing something malicious and needs to be fired. Tests should be used as an indicator to a human to "look here!" However, test results are simply an indicator to a human to exercise judgment--not an objective measure to remove humans from the loop. The problem with education is that people really don't want to use the tests as "indicators". As in the article I pointed out, that particular school now is having a decrease in results--the tests are indicating with a big, red klaxon that something is going wrong. People should be examining the situation with a magnifying glass to figure out the issues and fix them. The problem is that "something going wrong" is very clear--a new principal and new directives from the head bureaucrats. Do you think that they will look at the test results and say "Oh, we need to undo these policies and get a new principal?" Unlikely. Would you like a metric that only ever gets used against you but that your management gets to ignore? Of course not. It's why the complaints by programmers about management are legion. And that's why teachers hate all this stuff. Somehow, it only ever gets used against the teachers. When the evidence points at the administrators, school board or politicians, it gets ignored. This isn't even limited to teachers. Physicians oppose the collection of this kind of data about outcomes for the sames reasons--it will very quickly get turned into a ranking system that gets used against doctors rather than for improving outcomes.

bsder 12y ago

Using tests for an "objective measure" is doomed on its face. That's gigantic sign that the people involved are more interested in blame transfer than in progress. You can use tests for trends. That's a good use of tests. You can use tests for feedback to the individuals directly involved. That's also a good use of tests. This corresponds to my "software metrics" points. If I suddenly see an uptick in average lines of code committed, I should take a look. If I start seeing a big increase in bugs filed, I should take a look and see what just happened. Most of the time, there's a good explanation. Sometimes, someone is doing something stupid and I need to go fix that. And, rarely, someone is doing something malicious and needs to be fired. Tests should be used as an indicator to a human to "look here!" However, test results are simply an indicator to a human to exercise judgment--not an objective measure to remove humans from the loop. The problem with education is that people really don't want to use the tests as "indicators". As in the article I pointed out, that particular school now is having a decrease in results--the tests are indicating with a big, red klaxon that something is going wrong. People should be examining the situation with a magnifying glass to figure out the issues and fix them. The problem is that "something going wrong" is very clear--a new principal and new directives from the head bureaucrats. Do you think that they will look at the test results and say "Oh, we need to undo these policies and get a new principal?" Unlikely. Would you like a metric that only ever gets used against you but that your management gets to ignore? Of course not. It's why the complaints by programmers about management are legion. And that's why teachers hate all this stuff. Somehow, it only ever gets used against the teachers. When the evidence points at the administrators, school board or politicians, it gets ignored. This isn't even limited to teachers. Physicians oppose the collection of this kind of data about outcomes for the sames reasons--it will very quickly get turned into a ranking system that gets used against doctors rather than for improving outcomes.