4 ms·

When you financially incentivize teachers to hit measurable data targets, don't be surprised when they game the system to hit those targets. The same happens in

When you financially incentivize teachers to hit measurable data targets, don't be surprised when they game the system to hit those targets. The same happens in every industry and job. If I were to tell software developers that their bonus was dependent on hitting story point metrics, they would suddenly be exceeding the story point metrics I desired, despite in reality the same amount of work being completed. The real problem here is that the stakes are the students' education, and in truth their futures. I wish I could think of some easy solution to this problem but it's so damn hard. I believe a lot of it comes down to the parents and their values, but that's a tough problem to fix.

schoen 9y ago
Your observation is related to something called Goodhart's Law: https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law I guess thinking about this increases my sympathy for search engines and spam filters not being transparent about the algorithmic bases for their rankings, even in a week when I've had some personal e-mail blocked as a spam false positive. The resources that are getting put into discovering how to game those scores are staggering. I assume there are also exceptions to Goodhart's Law in terms of metrics that effectively can't be gamed, or that otherwise better manage to align different groups' incentives.
- seibelj 9y ago
  The best case in business is when you tell a sales professional that they will get a percentage of whatever business they bring in, judged the moment when it's in accounts receivable. It's difficult to game a wire transfer from another company. Education is very different than sales. I don't know what the solution is, especially when measuring "learning" itself is very difficult if not impossible.
  - toomuchtodo 9y ago
    > The best case in business is when you tell a sales professional that they will get a percentage of whatever business they bring in, judged the moment when it's in accounts receivable. It's difficult to game a wire transfer from another company. I've seen the result of this. The salesperson oversells, and the engineering team death marches to attempt to build what the salesperson has already received their commission for. Incentives are tricky.
  - jeffdavis 9y ago
    "especially when measuring "learning" itself is very difficult if not impossible." That's ridiculous. Math, science, vocabulary, history, and reading comprehension can all be tested effectively, with little risk of "gaming" it. (at least at the level of a tpyical high scholl curriculum)
    - HarryHirsch 9y ago
      Especially high school math can be gamed effectively. For "word problems" you can drill students to look for keywords and they will arrive at the right result without understanding the problem. My wife teaches at a 4-year state school and tells me that one of her students did just that, she looked for plausible keywords but then didn't know what to do next, this was chemistry, not math. Reading comprehension is harder to game.
      - yorwba 9y ago
        > you can drill students to look for keywords and they will arrive at the right result without understanding the problem > she looked for plausible keywords but then didn't know what to do next That doesn't sound like she arrived at the right result.
    - jcranmer 9y ago
      Test-taking strategies are definitely possible without learning the material. Hunting for keywords is a common one; learning to recognize the kind of answers that show up as confounders is also pretty common. Even free response questions can be gamed, if you learn how to come up with plausible answers that can rack up partial credit. It is difficult to game your way to an A without knowing a fair amount of the material, but it's definitely possible to get a passing grade and possibly as high as a B. And students are often very eager to try to claw as much partial credit on a test as possible.
    - acjohnson55 9y ago
      You might think that. But take it from a former teacher, it's actually very difficult assess learning. It's an entire field of study in itself. [1] https://en.wikipedia.org/wiki/Educational_assessment https://en.wikipedia.org/wiki/Educational_assessment
  - ACow_Adonis 9y ago
    It's not that hard to think about how to game that and the culture and incentives that will develop outside of very limited utopian circumstances. The logical outcome is that the biggest parasites will look to attach themselves to the biggest flows of money with low levels of elasticity relative to their input. And they will do whatever they can through side-channel methods and attacks, whether it be destructive to other employees or the company itself, to stop others getting into the position, including navigating/manipulating the hierarchy to maximize flows through them. Half of finance/executive remuneration is literally built around that strategy (manage big flows of money unrelated to your personal input), and I'd guess a huge number of toxic companies and employees are directly attributable to people who believe they've successfully implemented that incentive scheme.
  - s73v3r_ 9y ago
    Except we all know what happens in that case. Sales promises whatever they need to in order to get the sale, and then engineering is left to pick up the slack, working unpaid overtime to compensate.
- userbinator 9y ago
  Also https://en.wikipedia.org/wiki/Perverse_incentive https://en.wikipedia.org/wiki/Perverse_incentive .
door3 9y ago
Maybe it has something to do with the systematic impoverishment of black communities and the destruction of black families through mass incarceration, and "innovative teaching" will do little to fix these inequalities, which require much deeper and more structural policy change. On a related note, Washington D.C. specifically has a massive lead pipe problem which has been ignored for decades.
- scarface74 9y ago
  A lot starts with criminal justice reform and getting rid of the "War on Drugs". You can't help but to notice that once drugs started affecting rural and suburban American, that it became all about "treatment" and less about locking people up.
nitwit005 9y ago
Suppose you publicly admit you're failing to educate most of the students. Do you see yourself likely to survive the next school board election? But you might coast through those elections on a happy lie. Particularly if you make the papers by getting everyone admitted to university.

schoen 9y ago

Your observation is related to something called Goodhart's Law: https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law I guess thinking about this increases my sympathy for search engines and spam filters not being transparent about the algorithmic bases for their rankings, even in a week when I've had some personal e-mail blocked as a spam false positive. The resources that are getting put into discovering how to game those scores are staggering. I assume there are also exceptions to Goodhart's Law in terms of metrics that effectively can't be gamed, or that otherwise better manage to align different groups' incentives.

seibelj 9y ago
The best case in business is when you tell a sales professional that they will get a percentage of whatever business they bring in, judged the moment when it's in accounts receivable. It's difficult to game a wire transfer from another company. Education is very different than sales. I don't know what the solution is, especially when measuring "learning" itself is very difficult if not impossible.
- toomuchtodo 9y ago
  > The best case in business is when you tell a sales professional that they will get a percentage of whatever business they bring in, judged the moment when it's in accounts receivable. It's difficult to game a wire transfer from another company. I've seen the result of this. The salesperson oversells, and the engineering team death marches to attempt to build what the salesperson has already received their commission for. Incentives are tricky.
- jeffdavis 9y ago
  "especially when measuring "learning" itself is very difficult if not impossible." That's ridiculous. Math, science, vocabulary, history, and reading comprehension can all be tested effectively, with little risk of "gaming" it. (at least at the level of a tpyical high scholl curriculum)
  - HarryHirsch 9y ago
    Especially high school math can be gamed effectively. For "word problems" you can drill students to look for keywords and they will arrive at the right result without understanding the problem. My wife teaches at a 4-year state school and tells me that one of her students did just that, she looked for plausible keywords but then didn't know what to do next, this was chemistry, not math. Reading comprehension is harder to game.
    - yorwba 9y ago
      > you can drill students to look for keywords and they will arrive at the right result without understanding the problem > she looked for plausible keywords but then didn't know what to do next That doesn't sound like she arrived at the right result.
  - jcranmer 9y ago
    Test-taking strategies are definitely possible without learning the material. Hunting for keywords is a common one; learning to recognize the kind of answers that show up as confounders is also pretty common. Even free response questions can be gamed, if you learn how to come up with plausible answers that can rack up partial credit. It is difficult to game your way to an A without knowing a fair amount of the material, but it's definitely possible to get a passing grade and possibly as high as a B. And students are often very eager to try to claw as much partial credit on a test as possible.
  - acjohnson55 9y ago
    You might think that. But take it from a former teacher, it's actually very difficult assess learning. It's an entire field of study in itself. [1] https://en.wikipedia.org/wiki/Educational_assessment https://en.wikipedia.org/wiki/Educational_assessment
- ACow_Adonis 9y ago
  It's not that hard to think about how to game that and the culture and incentives that will develop outside of very limited utopian circumstances. The logical outcome is that the biggest parasites will look to attach themselves to the biggest flows of money with low levels of elasticity relative to their input. And they will do whatever they can through side-channel methods and attacks, whether it be destructive to other employees or the company itself, to stop others getting into the position, including navigating/manipulating the hierarchy to maximize flows through them. Half of finance/executive remuneration is literally built around that strategy (manage big flows of money unrelated to your personal input), and I'd guess a huge number of toxic companies and employees are directly attributable to people who believe they've successfully implemented that incentive scheme.
- s73v3r_ 9y ago
  Except we all know what happens in that case. Sales promises whatever they need to in order to get the sale, and then engineering is left to pick up the slack, working unpaid overtime to compensate.
userbinator 9y ago
Also https://en.wikipedia.org/wiki/Perverse_incentive https://en.wikipedia.org/wiki/Perverse_incentive .

door3 9y ago

Maybe it has something to do with the systematic impoverishment of black communities and the destruction of black families through mass incarceration, and "innovative teaching" will do little to fix these inequalities, which require much deeper and more structural policy change. On a related note, Washington D.C. specifically has a massive lead pipe problem which has been ignored for decades.

scarface74 9y ago
A lot starts with criminal justice reform and getting rid of the "War on Drugs". You can't help but to notice that once drugs started affecting rural and suburban American, that it became all about "treatment" and less about locking people up.

nitwit005 9y ago

Suppose you publicly admit you're failing to educate most of the students. Do you see yourself likely to survive the next school board election? But you might coast through those elections on a happy lie. Particularly if you make the papers by getting everyone admitted to university.