6 ms·

The thing I am always amazed by people who developed metrics is they never seem to stop and think how the metric will be gamed and then take action to prevent t

The thing I am always amazed by people who developed metrics is they never seem to stop and think how the metric will be gamed and then take action to prevent this. It should be part of the development of any new metric to have a "evil" team work out all the ways the metric can be gamed so this can be blocked. For example, it should be impossible to "teach the test" other than by teaching broadly the underlying concepts being tested. A well designed test will touch on so many concepts that only a education that is broad will work.

Spooky23 11y ago
The problem with applying this sort of measuring regime against education or medicine vs. a sales team or IT service operation is that the outcomes are difficult to assess, because he assessments are bogus. A high quality general education should lead to a good outcome on a standardized test, (Say the SAT) right? The reality is the complexity of the assessment requires preparation. Without understanding SAT test strategy, you won't do well. When I took AP US History, the difference between a 4 and a 5 (the highest score) was understanding how to "stage" essays for the graders. Instead of worrying about red teaming standardized tests, we should think log and hard about the cultures being created in these environments.
- thedufer 11y ago
  FWIW, the SAT is a terrible example of a gameable test. Test prep is consistently shown to affect scores by a mere 20-40 points.
SilasX 11y ago
I'm not even sure the voters/parents want the tests to be made harder like that. Then test scores would go down further and schools would have to devote even more time to passing them. And FWIW, I've looked at the Texas tests and they didn't seem game-able to me. If they were, the "don't teach to the test" rabblerousers would cite them in a heartbeat. But no one seems to follow the logical implication of this argument.
- danieltillett 11y ago
  You can make the tests as easy or as hard as you like and still prevent teaching to the test. The pass/fail rate is a function of the effectiveness of the teaching and the difficulty of the exam. If the fail rate is above the politically acceptable rate then make the test easier, just don’t make the test gameable.
  - SilasX 11y ago
    I'm sure the teachers would already prefer a quicker way of teaching that also conveys a more general understand and enables the students to pass the test. But if they already knew this method and could implement it, they would already be doing it.
    - danieltillett 11y ago
      The problem is the teachers don’t control the test difficulty so they are left with gaming. There is no quick way of really teaching. Teaching well is a really hard problem that many think is easy. It is not.
      - SilasX 11y ago
        You're saying the tests are too difficult? Have you looked at them?
        danieltillett 11y ago
        No I am not saying the tests are too difficult, but I am saying that if the failure rate is politically too high then the way to solve this by making the tests easier rather than making the tests gameable. Of course the best solution is to improve the quality of teaching, but this is a much harder problem.
acbart 11y ago
Why do you assume they don't stop to think about that? People who develop assessments usually think quite rigorously about exactly such problems, assuming they're following proper methods. Is it so surprising to you that this is a very difficult problem? I would compare this to asking, "Why don't software developers think about bugs?" They do, and they attempt to handle them, but it's almost impossible to solve all such problems in programs (tests) of non-trivial size.
- drdeca 11y ago
  Well, then the solution there is to only write trivial programs. Haha. Though, with enough resources, time, and effort, I think almost any program one would want in practice could be written with a formal proof of correctness. (I won't say every, because that runs up against halting problem / incompleteness things, but I think for almost all that you would actually want) It's just that there isn't enough demand, and time, for the programs to be proven correct. If there was a program that we could all stand to wait for a very long time for (maybe even a few generations?), and it was important enough that it be completely free of bugs, that could be done, I think. I'm not sure what it would be that important though. Starship stuff?
- deleted 11y ago
  [deleted]
- danieltillett 11y ago
  If they did then it would not be possible to “teach the test” other than by teaching the full subject. If there are any shortcuts a teacher can take with the class then the test is a failure. I used to write tests all the time in my subjects and my tests could not be gamed. I would have half multiple choice questions drawn from every area we covered in the semester and half essay questions where the students had to integrate what they learned into a coherent answer. My students would ask me what they had to learn for the final exam and I would always say everything - if it wasn’t going to be examined there was no point teaching it because the students won’t both learning it :)
NeutronBoy 11y ago
> well designed test will touch on so many concepts that only a education that is broad will work. So you create an education system that's a mile wide and an inch deep.
pps43 11y ago
It's always possible to "teach the standardized test". The only way to avoid it is to make the test subjective and non-standard. But then students who failed will complain that they were treated unfairly. The key point is that unfair treatment of some students should be considered an unavoidable risk, because the alternative is unfair treatment of all students.
- aianus 11y ago
  > The only way to avoid it is to make the test subjective and non-standard. How so? You can make it multiple-choice (objective) difficult, non-algorithmic math problems comprising multiple concepts each and then grade the whole test on a curve. Impossible to game except to actually teach the students a broad range of math skills and how to problem-solve, while still objective and standard (in the sense that everyone gets the same test and is graded the same).
  - iheartmemcache 11y ago
    Ooh yay, a topic I can speak fairly well-informed on. >> You can make it multiple-choice (objective) difficult, non-algorithmic math problems comprising multiple concepts each and then grade the whole test on a curve. You do realize that that's what Olympiads are right? https://en.wikipedia.org/wiki/International_Mathematical_Olympiad https://en.wikipedia.org/wiki/International_Mathematical_Oly... You can be fairly confident the Chinese "teach for the test" to a few handpicked kids who can perform well on those problems (in the same way they select children with an aptitude for gymnastics at an early age). Nearly 20 wins, not one Fields Medal. You can 'teach to the test' at any level up to the 400 levels, at which point you start creating new ways to approach the test, solving problems which weren't solved for a couple hundred years, or founding problems which won't be solved for another couple hundred.
  - mikekchar 11y ago
    As a former English as a Foreign Language teacher, I can assure you that there are always ways to game exams in English. Because I had to help some of my students get into university, I took a look at the entrance exams. Even though I am a native English speaker, I found that I could not finish the Todai (top school in Japan) entrance exam for English in the required time. It required reading and understanding very large amounts of text and then answering difficult multiple choice questions at the end. Sometimes the questions were difficult to understand and required that you were really careful about concentrating on the wording. So instead, I decided to try again (with a different year's test) and instead of reading the material ahead of time, I used the questions to direct my efforts. I would look at all the answers and then scan the text for areas that talked about those things. Having identified those, I then looked at the question and ruled out the answers that didn't seem related. Finally I settled on the answers -- mainly by applying grammar rules mechanically (i.e., a wrong answer would be in the wrong tense compared to the supplied text). I was able to complete the exam in time and get full marks. I had absolutely no idea what the text said. In fact, I didn't even know what half of the questions asked. Now for my dilemma as a teacher. If my student gets accepted to Todai, it will basically make their life. You are almost guaranteed to get a good job that will give you the kind of life that most Japanese people are looking for. If I ask the student to try to do the exam properly, by reading the text, they will fail. Even I would fail. I can teach them to pass it in a different way. What should I do? The test is too difficult. There are always going to be ways to game it and the successful students do. Therefore the test gets more difficult and the students are forced to game it even more. As a teacher, should I tell my student that they can't go to Todai, or should I teach them how to game the exam? As far as I was concerned, I decided that teaching English and passing entrance exams were 2 different things. I taught English for those students who were interested. I gave up a lot of my free time (a few hours every day) to help them do that. When exam time came, though, I taught no English at all. I taught them how to do what they needed to do to be successful. Every one of my colleagues did the same thing. Probably every teacher in the country does the same thing. Standardized tests that grade you and decide your future are insane.
briffle 11y ago
20 years ago, a certain chain of fast food restaurants pushed through new metrics to compare different locations by. One of those was which shifts kept drive through average times under 60 seconds.. We got all sorts of recognition and praise.. Nobody noticed that first thing when opening was we ran through 20 or orders through drive through for water (no cost) and closed within 5 seconds. Then intermixed all day between orders, when it was slow.
- danieltillett 11y ago
  This is a excellent example of a poor metric resulting from the test developers not thinking about how the metric could be gamed. If they had they could have made sure that they only counted orders over a certain value or even better counted size adjusted orders.
deleted 11y ago
[deleted]
iheartmemcache 11y ago
How do you define "genius" or "satisfactory"? Education should be taught in a way such that one is exposed to the methodologies by which one should approach problems, rather than solving the problem itself[1]. "Teaching to the test" is awful, I agree. How can one, on a national basis, ensure that every school is affording as equal an opportunity as possible? I'm not being sarcastic - what is the better alternative, the better metric by which we can gauge the performance of both our students and our teachers? I certainly don't think 8th grade math teachers should be penalized for the poor performance of their students if those 7th graders who matriculated into your class on September didn't have a functional understanding of fractions. (Conversely, I think teachers who reached 'tenure'[2] and have clocked out until they've hit that pension-year _should_ be punished). How should those teachers be evaluated? One could suggest "grade the teachers on the improvement of performance of their students compared to their prior years performance" but that brings us back to the same problem -- you have a test. I ask this without snark - how do you test as many concepts at that broad a scope of subject, at that grand a scale (let's start small - 3 districts in NY state chosen at random)? How do you even choose what concepts are important at that broad a level? [1] This gets into a whole meta-discussion about what the appropriate methodologies are, which I'll leave as an exercise to those doctoral candidates finishing up their dissertations. [2] Tenure as I understand it serves to allow researchers could undergo extensive, perhaps subversive, study to enrich the human body of knowledge while retaining job security. While educators at a grade/secondary school level perform a great service, I fail (heh) to see the purpose it serves at that level.
- danieltillett 11y ago
  Well each year should have a syllabus and what the students are expected to know and be able to do (we used to call them rubrics). You just test that the students perform in all the things in the syllabus. The only way to “teach the test” if testing was done like this would be to teach the syllabus which is exactly what the teachers are supposed to be doing anyway.
bsder 11y ago
If gaming the metric is a useful behavior, the metric is useless. Secondary infections in hospitals is a good metric. If you game it, it's still a net win for everybody. Pass rates in schools is a bad metric. I can improve my metric by getting my lowest students to quit. Rather than trying to "anti-game" a metric, simply observe if the metric is being gamed. If it is, the metric is a priori bad and should be removed.
- danieltillett 11y ago
  Yes this is 100% true, but the point I was making is if the test writes think first about how the metric can be gamed then they can take action to prevent the gaming including changing the metric and test.
- ptaipale 11y ago
  > If gaming the metric is a useful behavior, the metric is useless. That is exactly why you should have a review of how a metric might be gamed, before you accept to use a this metric.

Spooky23 11y ago

The problem with applying this sort of measuring regime against education or medicine vs. a sales team or IT service operation is that the outcomes are difficult to assess, because he assessments are bogus. A high quality general education should lead to a good outcome on a standardized test, (Say the SAT) right? The reality is the complexity of the assessment requires preparation. Without understanding SAT test strategy, you won't do well. When I took AP US History, the difference between a 4 and a 5 (the highest score) was understanding how to "stage" essays for the graders. Instead of worrying about red teaming standardized tests, we should think log and hard about the cultures being created in these environments.

thedufer 11y ago
FWIW, the SAT is a terrible example of a gameable test. Test prep is consistently shown to affect scores by a mere 20-40 points.

SilasX 11y ago

I'm not even sure the voters/parents want the tests to be made harder like that. Then test scores would go down further and schools would have to devote even more time to passing them. And FWIW, I've looked at the Texas tests and they didn't seem game-able to me. If they were, the "don't teach to the test" rabblerousers would cite them in a heartbeat. But no one seems to follow the logical implication of this argument.

danieltillett 11y ago
You can make the tests as easy or as hard as you like and still prevent teaching to the test. The pass/fail rate is a function of the effectiveness of the teaching and the difficulty of the exam. If the fail rate is above the politically acceptable rate then make the test easier, just don’t make the test gameable.
- SilasX 11y ago
  I'm sure the teachers would already prefer a quicker way of teaching that also conveys a more general understand and enables the students to pass the test. But if they already knew this method and could implement it, they would already be doing it.
  - danieltillett 11y ago
    The problem is the teachers don’t control the test difficulty so they are left with gaming. There is no quick way of really teaching. Teaching well is a really hard problem that many think is easy. It is not.
    - SilasX 11y ago
      You're saying the tests are too difficult? Have you looked at them?
      - danieltillett 11y ago
        No I am not saying the tests are too difficult, but I am saying that if the failure rate is politically too high then the way to solve this by making the tests easier rather than making the tests gameable. Of course the best solution is to improve the quality of teaching, but this is a much harder problem.

acbart 11y ago

Why do you assume they don't stop to think about that? People who develop assessments usually think quite rigorously about exactly such problems, assuming they're following proper methods. Is it so surprising to you that this is a very difficult problem? I would compare this to asking, "Why don't software developers think about bugs?" They do, and they attempt to handle them, but it's almost impossible to solve all such problems in programs (tests) of non-trivial size.

drdeca 11y ago
Well, then the solution there is to only write trivial programs. Haha. Though, with enough resources, time, and effort, I think almost any program one would want in practice could be written with a formal proof of correctness. (I won't say every, because that runs up against halting problem / incompleteness things, but I think for almost all that you would actually want) It's just that there isn't enough demand, and time, for the programs to be proven correct. If there was a program that we could all stand to wait for a very long time for (maybe even a few generations?), and it was important enough that it be completely free of bugs, that could be done, I think. I'm not sure what it would be that important though. Starship stuff?
deleted 11y ago
[deleted]
danieltillett 11y ago
If they did then it would not be possible to “teach the test” other than by teaching the full subject. If there are any shortcuts a teacher can take with the class then the test is a failure. I used to write tests all the time in my subjects and my tests could not be gamed. I would have half multiple choice questions drawn from every area we covered in the semester and half essay questions where the students had to integrate what they learned into a coherent answer. My students would ask me what they had to learn for the final exam and I would always say everything - if it wasn’t going to be examined there was no point teaching it because the students won’t both learning it :)

NeutronBoy 11y ago

> well designed test will touch on so many concepts that only a education that is broad will work. So you create an education system that's a mile wide and an inch deep.

pps43 11y ago

It's always possible to "teach the standardized test". The only way to avoid it is to make the test subjective and non-standard. But then students who failed will complain that they were treated unfairly. The key point is that unfair treatment of some students should be considered an unavoidable risk, because the alternative is unfair treatment of all students.

aianus 11y ago
> The only way to avoid it is to make the test subjective and non-standard. How so? You can make it multiple-choice (objective) difficult, non-algorithmic math problems comprising multiple concepts each and then grade the whole test on a curve. Impossible to game except to actually teach the students a broad range of math skills and how to problem-solve, while still objective and standard (in the sense that everyone gets the same test and is graded the same).
- iheartmemcache 11y ago
  Ooh yay, a topic I can speak fairly well-informed on. >> You can make it multiple-choice (objective) difficult, non-algorithmic math problems comprising multiple concepts each and then grade the whole test on a curve. You do realize that that's what Olympiads are right? https://en.wikipedia.org/wiki/International_Mathematical_Olympiad https://en.wikipedia.org/wiki/International_Mathematical_Oly... You can be fairly confident the Chinese "teach for the test" to a few handpicked kids who can perform well on those problems (in the same way they select children with an aptitude for gymnastics at an early age). Nearly 20 wins, not one Fields Medal. You can 'teach to the test' at any level up to the 400 levels, at which point you start creating new ways to approach the test, solving problems which weren't solved for a couple hundred years, or founding problems which won't be solved for another couple hundred.
- mikekchar 11y ago
  As a former English as a Foreign Language teacher, I can assure you that there are always ways to game exams in English. Because I had to help some of my students get into university, I took a look at the entrance exams. Even though I am a native English speaker, I found that I could not finish the Todai (top school in Japan) entrance exam for English in the required time. It required reading and understanding very large amounts of text and then answering difficult multiple choice questions at the end. Sometimes the questions were difficult to understand and required that you were really careful about concentrating on the wording. So instead, I decided to try again (with a different year's test) and instead of reading the material ahead of time, I used the questions to direct my efforts. I would look at all the answers and then scan the text for areas that talked about those things. Having identified those, I then looked at the question and ruled out the answers that didn't seem related. Finally I settled on the answers -- mainly by applying grammar rules mechanically (i.e., a wrong answer would be in the wrong tense compared to the supplied text). I was able to complete the exam in time and get full marks. I had absolutely no idea what the text said. In fact, I didn't even know what half of the questions asked. Now for my dilemma as a teacher. If my student gets accepted to Todai, it will basically make their life. You are almost guaranteed to get a good job that will give you the kind of life that most Japanese people are looking for. If I ask the student to try to do the exam properly, by reading the text, they will fail. Even I would fail. I can teach them to pass it in a different way. What should I do? The test is too difficult. There are always going to be ways to game it and the successful students do. Therefore the test gets more difficult and the students are forced to game it even more. As a teacher, should I tell my student that they can't go to Todai, or should I teach them how to game the exam? As far as I was concerned, I decided that teaching English and passing entrance exams were 2 different things. I taught English for those students who were interested. I gave up a lot of my free time (a few hours every day) to help them do that. When exam time came, though, I taught no English at all. I taught them how to do what they needed to do to be successful. Every one of my colleagues did the same thing. Probably every teacher in the country does the same thing. Standardized tests that grade you and decide your future are insane.

briffle 11y ago

20 years ago, a certain chain of fast food restaurants pushed through new metrics to compare different locations by. One of those was which shifts kept drive through average times under 60 seconds.. We got all sorts of recognition and praise.. Nobody noticed that first thing when opening was we ran through 20 or orders through drive through for water (no cost) and closed within 5 seconds. Then intermixed all day between orders, when it was slow.

danieltillett 11y ago
This is a excellent example of a poor metric resulting from the test developers not thinking about how the metric could be gamed. If they had they could have made sure that they only counted orders over a certain value or even better counted size adjusted orders.

deleted 11y ago

[deleted]

iheartmemcache 11y ago

How do you define "genius" or "satisfactory"? Education should be taught in a way such that one is exposed to the methodologies by which one should approach problems, rather than solving the problem itself[1]. "Teaching to the test" is awful, I agree. How can one, on a national basis, ensure that every school is affording as equal an opportunity as possible? I'm not being sarcastic - what is the better alternative, the better metric by which we can gauge the performance of both our students and our teachers? I certainly don't think 8th grade math teachers should be penalized for the poor performance of their students if those 7th graders who matriculated into your class on September didn't have a functional understanding of fractions. (Conversely, I think teachers who reached 'tenure'[2] and have clocked out until they've hit that pension-year _should_ be punished). How should those teachers be evaluated? One could suggest "grade the teachers on the improvement of performance of their students compared to their prior years performance" but that brings us back to the same problem -- you have a test. I ask this without snark - how do you test as many concepts at that broad a scope of subject, at that grand a scale (let's start small - 3 districts in NY state chosen at random)? How do you even choose what concepts are important at that broad a level? [1] This gets into a whole meta-discussion about what the appropriate methodologies are, which I'll leave as an exercise to those doctoral candidates finishing up their dissertations. [2] Tenure as I understand it serves to allow researchers could undergo extensive, perhaps subversive, study to enrich the human body of knowledge while retaining job security. While educators at a grade/secondary school level perform a great service, I fail (heh) to see the purpose it serves at that level.

danieltillett 11y ago
Well each year should have a syllabus and what the students are expected to know and be able to do (we used to call them rubrics). You just test that the students perform in all the things in the syllabus. The only way to “teach the test” if testing was done like this would be to teach the syllabus which is exactly what the teachers are supposed to be doing anyway.

bsder 11y ago

If gaming the metric is a useful behavior, the metric is useless. Secondary infections in hospitals is a good metric. If you game it, it's still a net win for everybody. Pass rates in schools is a bad metric. I can improve my metric by getting my lowest students to quit. Rather than trying to "anti-game" a metric, simply observe if the metric is being gamed. If it is, the metric is a priori bad and should be removed.

danieltillett 11y ago
Yes this is 100% true, but the point I was making is if the test writes think first about how the metric can be gamed then they can take action to prevent the gaming including changing the metric and test.
ptaipale 11y ago
> If gaming the metric is a useful behavior, the metric is useless. That is exactly why you should have a review of how a metric might be gamed, before you accept to use a this metric.