6 ms·

Your claim makes no sense. If the evaluation method measures what we want teachers to do, then it, by definition, can't be "gamed": if the measurement is high,

Your claim makes no sense. If the evaluation method measures what we want teachers to do, then it, by definition, can't be "gamed": if the measurement is high, then the teacher is doing what we want. Are you claiming that there's something special about teaching that makes teacher performance much more difficult, or even impossible, to quantify? If so, what is the point of teaching if one cannot measure results? Your solution to "ask the students, parents, and staff" is precisely a measurement method (albeit a more qualitative than quantitative one), and moreover, one that can be gamed easily by anyone who's socially shrewd. Qualitative measures like that is exactly why we have so many crappy politicians.

Evbn 14y ago
Do you have any such "evaluation method measures what we want teachers to do" in mind? Point is, no one has found one.
ColinWright 14y ago
I'd like to ask - how do you measure programmer productivity?
WalterBright 14y ago
You can't specify everything in the rules. For example, I know a case where management decided to hand out bonuses to the staff for minimizing inventory, because inventory costs money. The staff minimized inventory, all right, and got their bonuses. Meanwhile, the production line would regularly get halted because they'd run out of things like 5 cent resistors. It was a disaster. The same company decided to rate programmers based on the bug list. A big chart was posted on the wall that showed the daily bug count. It wasn't a week before huge fights erupted over the bug count - over what was and was not a bug. The programmers quickly gamed it. They'd hide bugs, they'd refuse to fix one bug if that fix would produce some other minor bugs (e.g. if the bug was "feature X missing", then feature X is added, but had a couple issues with it, then X would not get fixed). They'd even add "bugs" blamed on other programmers and then "fix" them to get the credit. Management gave up on that after two weeks and pulled the banner down.
- yummyfajitas 14y ago
  For teachers, the metric is (roughly speaking) "% of students capable of multiplying/dividing numbers up to 4 digits in a standardized test setting". How do you game this metric? The fact that one company used a couple of bad objective metrics doesn't mean all objective metrics are bad. They are used with a great deal of success in many fields. Sales people are paid on commission, traders are paid proportionally to (risk adjusted) profits, etc. All it means is that if you set the wrong goals for your company, you'll probably succeed at the wrong thing. So tell us - is "maximize the % of students capable of multiplying/dividing 4 digit numbers" the wrong goal? If so, what is the right goal, and why can't it be measured?
  - WalterBright 14y ago
    One popular way of gaming the metric is systematic cheating on the tests by the teachers. I say popular because it has happened on a large scale, most recently in Atlanta as I recall. Another way to manipulate the test results is to manipulate which students are in your class or your school. The point is, people are endlessly creative in subverting rules to their own benefit - so they conform to the letter of the rules but not the spirit. Consider also the "work to rule" technique used by unions as a bargaining tactic. It's as simple as the workers literally adhering to their job descriptions. It doesn't work out well for the company.
    - yummyfajitas 14y ago
      To prevent cheating, test administration should be handled by someone other than teachers. The fact that the current system has a bunch of cheaters is not an argument against more carefully and objectively measuring the current system. What next - bankers sometimes engage in rogue trading, so we should reduce monitoring of their behavior? Another way to manipulate the test results is to manipulate which students are in your class or your school. This is very difficult with VAM, since the goal is to increase (actual score - statistically predicted score). You need to reliably identify students who will do better than their statistical predictor. I.e., you need to discover students who will improve drastically this year and then pack your student body with them.
WalterBright 14y ago
I don't think it's so easy to manipulate a classroom full of students into thinking you're a great teacher when you're not. Remember, you're in front of them several hours a day, 5 days a week.
- CamperBob2 14y ago
  For what it's worth, my favorite teachers were always the ones who assigned the least work.
  - WalterBright 14y ago
    For some students, sure, but a majority? But I'm not so sure that will impress either the parents or the staff. Also, was that work busywork, or productive work? Students can tell the difference. I had a high school teacher who explicitly wanted to be popular by assigning no work. The students did like him, but they did not respect him (and laughed at him behind his back). Those of us who wanted to go on to college also knew he was shortchanging us.
deleted 14y ago
[deleted]
DataWorker 14y ago
Which is why we have objective measures for programmers.

Evbn 14y ago

Do you have any such "evaluation method measures what we want teachers to do" in mind? Point is, no one has found one.

ColinWright 14y ago

I'd like to ask - how do you measure programmer productivity?

WalterBright 14y ago

You can't specify everything in the rules. For example, I know a case where management decided to hand out bonuses to the staff for minimizing inventory, because inventory costs money. The staff minimized inventory, all right, and got their bonuses. Meanwhile, the production line would regularly get halted because they'd run out of things like 5 cent resistors. It was a disaster. The same company decided to rate programmers based on the bug list. A big chart was posted on the wall that showed the daily bug count. It wasn't a week before huge fights erupted over the bug count - over what was and was not a bug. The programmers quickly gamed it. They'd hide bugs, they'd refuse to fix one bug if that fix would produce some other minor bugs (e.g. if the bug was "feature X missing", then feature X is added, but had a couple issues with it, then X would not get fixed). They'd even add "bugs" blamed on other programmers and then "fix" them to get the credit. Management gave up on that after two weeks and pulled the banner down.

yummyfajitas 14y ago
For teachers, the metric is (roughly speaking) "% of students capable of multiplying/dividing numbers up to 4 digits in a standardized test setting". How do you game this metric? The fact that one company used a couple of bad objective metrics doesn't mean all objective metrics are bad. They are used with a great deal of success in many fields. Sales people are paid on commission, traders are paid proportionally to (risk adjusted) profits, etc. All it means is that if you set the wrong goals for your company, you'll probably succeed at the wrong thing. So tell us - is "maximize the % of students capable of multiplying/dividing 4 digit numbers" the wrong goal? If so, what is the right goal, and why can't it be measured?
- WalterBright 14y ago
  One popular way of gaming the metric is systematic cheating on the tests by the teachers. I say popular because it has happened on a large scale, most recently in Atlanta as I recall. Another way to manipulate the test results is to manipulate which students are in your class or your school. The point is, people are endlessly creative in subverting rules to their own benefit - so they conform to the letter of the rules but not the spirit. Consider also the "work to rule" technique used by unions as a bargaining tactic. It's as simple as the workers literally adhering to their job descriptions. It doesn't work out well for the company.
  - yummyfajitas 14y ago
    To prevent cheating, test administration should be handled by someone other than teachers. The fact that the current system has a bunch of cheaters is not an argument against more carefully and objectively measuring the current system. What next - bankers sometimes engage in rogue trading, so we should reduce monitoring of their behavior? Another way to manipulate the test results is to manipulate which students are in your class or your school. This is very difficult with VAM, since the goal is to increase (actual score - statistically predicted score). You need to reliably identify students who will do better than their statistical predictor. I.e., you need to discover students who will improve drastically this year and then pack your student body with them.

WalterBright 14y ago

I don't think it's so easy to manipulate a classroom full of students into thinking you're a great teacher when you're not. Remember, you're in front of them several hours a day, 5 days a week.

CamperBob2 14y ago
For what it's worth, my favorite teachers were always the ones who assigned the least work.
- WalterBright 14y ago
  For some students, sure, but a majority? But I'm not so sure that will impress either the parents or the staff. Also, was that work busywork, or productive work? Students can tell the difference. I had a high school teacher who explicitly wanted to be popular by assigning no work. The students did like him, but they did not respect him (and laughed at him behind his back). Those of us who wanted to go on to college also knew he was shortchanging us.

deleted 14y ago

[deleted]

DataWorker 14y ago

Which is why we have objective measures for programmers.