7 ms·

Under Pressure, Teachers Tamper With Test Scores

28 points by asnyder 16y ago

RiderOfGiraffes 16y ago
That cheating occurs is no surprise and is closely related to the question of assessing a programmer's output. Every time you instigate a method of measurement, the programmer (or teacher) finds some way to increase the result without achieving the actual aim. In the case of a programmer, the number of lines of code output, or the number of functions written, or the number of bugs detected and fixed, can all be achieved with less "true value output" (whatever that means). The measurement can be gamed. Teachers in the UK have been accused of "teaching to the test" rather than "teaching for understanding" - but what more can be expected? When exams test for understanding rather than rote skills, only then will understanding be the aim of the teachers looking for the easy way out. Let me add that I know many dedicated teachers who do the right thing regardless of the threats posed. They encourage and enthuse their students, ranging widely over their subject to get the students engaged and interested. They know that this serves them better, and will, mostly, improve the test scores, even if not as much as "teaching to the test" would. They know they're doing the right thing, but come under enormous pressure to work more obviously towards improving test scores. I'm not surprised that many of them buckle under the strain and take less honorable courses. That's part of the tragedy - the destruction of otherwise excellent teachers.
- alexandros 16y ago
  I've written extensively on this phenomenon, take a look: http://lesswrong.com/lw/28r/is_google_paperclipping_the_web_the_perils_of/ http://lesswrong.com/lw/28r/is_google_paperclipping_the_web_...
- guelo 16y ago
  No, the tragedy is uneducated kids.
  - RiderOfGiraffes 16y ago
    That's why I said "part" - but each gifted teacher lost to the profession leads to many, perhaps thousands, of less well educated children.
- liedra 16y ago
  See I think that these sorts of tests are looking at the wrong sorts of things. What they should be looking for is relative improvement rather than reaching a particular level. Especially in underprivileged areas, where the kids might be doing much more poorly than more privileged areas. Don't get me wrong, I think that if there are poor teachers then they should be dealt with, but I think that these standardised tests of kids across the board are not the way to go about it. I'm thinking maybe something like a "peer review" type system could be put in place that takes more into account than just the output of the kids on one test (the results of continuous assessment across the whole year, for example; teaching reviews, etc.). Of course this would come with its own issues, but it would certainly cut down on the pressure that comes from that one test!
  - RiderOfGiraffes 16y ago
    Most teacher and school assessments in the UK are based exactly on that - achieved improvement versus expected improvement. Schools in underpriveleged areas and which perform poorly on standardised tests still themselves score well if the results are "good given the context and circumstances." The tragedy is that teachers who don't teach to the test and provide an excellent education that is tailored to the children they have are starting to worry about the security of their jobs, whereas poor teachers who have no real ability except to regurgitate the texts and drill the kids on process and recitation are relatively safe.
    - tokenadult 16y ago
      The tragedy is that teachers who don't teach to the test and provide an excellent education that is tailored to the children they have are starting to worry about the security of their jobs, whereas poor teachers who have no real ability except to regurgitate the texts and drill the kids on process and recitation are relatively safe. Evidence for this empirical statement? I have my doubts about this, because the best teachers I had in childhood (too few, but more than one) were able both to engage the pupils from the dullest to the brightest with deep, thoughtful questions and able to cover the fundamental facts of the lessons with efficiency.
patio11 16y ago
“Educators feel that their schools’ reputation, their livelihoods, their psychic meaning in life is at stake,” said Robert Schaeffer, public education director for FairTest, a nonprofit group critical of standardized testing. “That ends up pushing more and more of them over the line.” There is a certain strain of teacher who never felt their reputation, livelihood, or psychic essence was at danger when they were merely failing to educate, for decades. After all, to the extent that anyone noticed, it was poor black kids. Who the heck cares what they think. That strain of teacher is now motivated by fear of exposure. Which is as it should be.
- Natsu 16y ago
  This is one place where cheaters can really ruin a system. Once cheaters invade, they can hurt everyone by raising expectations of what can be done. Worse, too often, the natural tendency is to raise the standards if some cheating is uncovered. If you raise your standards high enough, those who were formerly honest will be forced into cheating while those who remain honest are forced out. If you raise standards high enough, you end up with only cheaters, because past some threshold, it's no longer possible to meet expectations honestly.
  - tokenadult 16y ago
    If you raise standards high enough, you end up with only cheaters, because past some threshold, it's no longer possible to meet expectations honestly. Evidence for this empirical statement?
    - deleted 16y ago
      [deleted]
    - jerf 16y ago
      That statement as written is trivially true. The question is whether it is what is occurring, which isn't going to be possible to ascertain. I don't think anyone really has a firm grasp on what it is possible for an average child to learn, which would be a prerequisite to answering the question. (There is some idea what an average child can learn in our current system but that's such a small part of the possible state space of educational systems I don't pay much attention to it.)
karzeem 16y ago
This shouldn't be a surprise. When you create strong incentives for someone to do something, they'll probably do it.
- patio11 16y ago
  Like embezzlement shouldn't be a surprise. I agree, this is predictable, and in a just world our response would be predictable, too: if you falsify test scores to get a $2,970 bonus, you get kicked out of the profession, just like if you took $2,970 out of the drawer as a bank teller and tried to pocket it. (n.b. While acknowledging that 95% of bank tellers are morally upright individuals, we devise our systems under the assumption that they need to be caught if they are attempting to cheat. That assumption is instructive.) In the world we actually live in, you'll have a union rep testify as to how dedicated of a teacher you are, and you'll probably not miss a day of class.
  - gjm11 16y ago
    There's an interesting difference between the situations. If you're a bank teller, your whole job is to manage the flow of money and information to and from customers of the bank, the ultimate goal being to make more money for the bank. If you take $3k out of the drawer, then you have failed at your immediate task (money has gone to the wrong place) and worked against the overall purpose of your job (the bank has less money, rather than more). If you're a teacher, the heart of your job is (or should be) teaching your pupils: enabling them to know more and think better and so forth. A secondary purpose (the real purpose, according to some cynics) is to keep them out of their parents' hair and off the streets for a few hours every day. Another (subsidiary to the first, in an ideal world) is enabling them to do as well as possible in whatever external examinations they take, in the hope that it'll help them in later life. Arranging for your pupils' standardized test scores not to be too high comes very low down that list -- and if you get away with it, it benefits your pupils and your school. (While harming other pupils and other schools and other people generally, by making the test results less useful. Tragedy of the commons.) That's not to say that it's not very bad for teachers to cheat to get better test results for their pupils. But it's bad in a different way from a bank teller's embezzlement: peripheral to their job rather than central, and immediately beneficial rather than harmful to the people you're supposed to be working for. A better analogy (though perhaps an overdramatic one, given recent economic events) would be a mortgage salesperson, working for a mortgage broker, who sells people mortgages they are too likely to default on. When that happens, the salesperson gets more commission; his employer, the broker, gets more commission; the person who buys the mortgage gets a mortgage they want and otherwise couldn't have had (and, let's say, does manage to keep up the payments in 90% of cases, fails to do so but not so badly that they lose their home in 5% of cases, and crashes hard in 5% of cases -- which is probably kinda parallel to the outcomes for pupils whose test scores are inflated); other people, less directly connected with the broker, are the ones who get screwed. Anyone expect mortgage salespeople who sell a bit too enthusiastically to get kicked out of their profession en masse? ... No, nor me. > In the world we actually live in ... you'll probably not miss a day of class. That doesn't appear to be what happened in the cases in the NYT article.
samratjp 16y ago
Obligatory-long-yet-relevant-animation(?) on our educational system:http://www.youtube.com/watch?v=A3oIiH7BLmg http://www.youtube.com/watch?v=A3oIiH7BLmg When are we (loosely speaking) going to realize that we're not teaching in the right medium.
politicalist 16y ago
I hope some teachers "cheated" on the tests for principled reasons, not just for personal reward, like this UMass teacher: http://books.google.com/books?id=x3ertj1IcaAC&lpg=PA62&ots=R-V61uVgj4&pg=PA215#v=onepage&q&f=false http://books.google.com/books?id=x3ertj1IcaAC&lpg=PA62&#...
- count 16y ago
  To clarify, MOST school districts are not giving financial or other rewards for passing the exams at an acceptable level, it's simply expected to occur.
count 16y ago
Cheating is wrong - I don't see any justification otherwise. With that said, the NCLB and standardized tests are destroying school - the only focus now is to learn to answer those multiple choice questions at the end of the year. In Norfolk (one of the cities in the NYT article), the 'SOL' annual standardized test is a sole determining factor in pupil promotion. If you come out of 4th grade with a 100% average, but you fail your SOL for whatever reason (you're nervous and didn't do well, you were sick that day, etc.) - YOU FAIL 4TH GRADE. That seems ridiculous.
- ktsmith 16y ago
  In schools on the watch list in Nevada where the state has stepped in teachers are no longer really allowed to teach. In 4th grade students receive grades in Reading, Writing, Math, Art, Social Studies, Physical Education and a few other categories. At these schools the teachers are only allowed to teach reading, writing and math. Every lesson plan must be geared towards meeting a standard in one of those three categories. The teachers are required to give out grades in the areas they are not allowed to teach. When pressed on "what" grade the children should receive the answer is typically "whatever you feel is appropriate." These children aren't receiving an education, they are being taught how to memorize information for their CRT's then for their benchmarks, then for whatever test follows that. This isn't what teachers want, this is what teachers are forced to do by their states in order to achieve the largest number of passing schools and keep federal funding levels high.
tansey 16y ago
Seems like there is a need for an independent third-party group to monitor the tests. Since that is likely prohibitively expensive, here's an idea: 1) Have schools partner with each other, where each school is matched to another school of similar size. 2) Extend the testing into twice as long periods (e.g., if it was a 1-day test, now the test period is 2 days). 3) Have partner schools test on opposite period halves. 4) Have partner schools give students the day(s) off on the non-test half. 5) Have schools send their teachers to their partner schools to monitor all the tests. Now, this isn't fool-proof. It's possible that teachers across multiple schools could collude. It would be crucial to mix teacher pairs up so that if A monitors B then B does not monitor A, thus requiring a multi-way collusion that involves many participants.

28 points by asnyder 16y ago

That cheating occurs is no surprise and is closely related to the question of assessing a programmer's output. Every time you instigate a method of measurement, the programmer (or teacher) finds some way to increase the result without achieving the actual aim. In the case of a programmer, the number of lines of code output, or the number of functions written, or the number of bugs detected and fixed, can all be achieved with less "true value output" (whatever that means). The measurement can be gamed. Teachers in the UK have been accused of "teaching to the test" rather than "teaching for understanding" - but what more can be expected? When exams test for understanding rather than rote skills, only then will understanding be the aim of the teachers looking for the easy way out. Let me add that I know many dedicated teachers who do the right thing regardless of the threats posed. They encourage and enthuse their students, ranging widely over their subject to get the students engaged and interested. They know that this serves them better, and will, mostly, improve the test scores, even if not as much as "teaching to the test" would. They know they're doing the right thing, but come under enormous pressure to work more obviously towards improving test scores. I'm not surprised that many of them buckle under the strain and take less honorable courses. That's part of the tragedy - the destruction of otherwise excellent teachers.

alexandros 16y ago
I've written extensively on this phenomenon, take a look: http://lesswrong.com/lw/28r/is_google_paperclipping_the_web_the_perils_of/ http://lesswrong.com/lw/28r/is_google_paperclipping_the_web_...
guelo 16y ago
No, the tragedy is uneducated kids.
- RiderOfGiraffes 16y ago
  That's why I said "part" - but each gifted teacher lost to the profession leads to many, perhaps thousands, of less well educated children.
liedra 16y ago
See I think that these sorts of tests are looking at the wrong sorts of things. What they should be looking for is relative improvement rather than reaching a particular level. Especially in underprivileged areas, where the kids might be doing much more poorly than more privileged areas. Don't get me wrong, I think that if there are poor teachers then they should be dealt with, but I think that these standardised tests of kids across the board are not the way to go about it. I'm thinking maybe something like a "peer review" type system could be put in place that takes more into account than just the output of the kids on one test (the results of continuous assessment across the whole year, for example; teaching reviews, etc.). Of course this would come with its own issues, but it would certainly cut down on the pressure that comes from that one test!
- RiderOfGiraffes 16y ago
  Most teacher and school assessments in the UK are based exactly on that - achieved improvement versus expected improvement. Schools in underpriveleged areas and which perform poorly on standardised tests still themselves score well if the results are "good given the context and circumstances." The tragedy is that teachers who don't teach to the test and provide an excellent education that is tailored to the children they have are starting to worry about the security of their jobs, whereas poor teachers who have no real ability except to regurgitate the texts and drill the kids on process and recitation are relatively safe.
  - tokenadult 16y ago
    The tragedy is that teachers who don't teach to the test and provide an excellent education that is tailored to the children they have are starting to worry about the security of their jobs, whereas poor teachers who have no real ability except to regurgitate the texts and drill the kids on process and recitation are relatively safe. Evidence for this empirical statement? I have my doubts about this, because the best teachers I had in childhood (too few, but more than one) were able both to engage the pupils from the dullest to the brightest with deep, thoughtful questions and able to cover the fundamental facts of the lessons with efficiency.

patio11 16y ago

“Educators feel that their schools’ reputation, their livelihoods, their psychic meaning in life is at stake,” said Robert Schaeffer, public education director for FairTest, a nonprofit group critical of standardized testing. “That ends up pushing more and more of them over the line.” There is a certain strain of teacher who never felt their reputation, livelihood, or psychic essence was at danger when they were merely failing to educate, for decades. After all, to the extent that anyone noticed, it was poor black kids. Who the heck cares what they think. That strain of teacher is now motivated by fear of exposure. Which is as it should be.

Natsu 16y ago
This is one place where cheaters can really ruin a system. Once cheaters invade, they can hurt everyone by raising expectations of what can be done. Worse, too often, the natural tendency is to raise the standards if some cheating is uncovered. If you raise your standards high enough, those who were formerly honest will be forced into cheating while those who remain honest are forced out. If you raise standards high enough, you end up with only cheaters, because past some threshold, it's no longer possible to meet expectations honestly.
- tokenadult 16y ago
  If you raise standards high enough, you end up with only cheaters, because past some threshold, it's no longer possible to meet expectations honestly. Evidence for this empirical statement?
  - deleted 16y ago
    [deleted]
  - jerf 16y ago
    That statement as written is trivially true. The question is whether it is what is occurring, which isn't going to be possible to ascertain. I don't think anyone really has a firm grasp on what it is possible for an average child to learn, which would be a prerequisite to answering the question. (There is some idea what an average child can learn in our current system but that's such a small part of the possible state space of educational systems I don't pay much attention to it.)

karzeem 16y ago

This shouldn't be a surprise. When you create strong incentives for someone to do something, they'll probably do it.

patio11 16y ago
Like embezzlement shouldn't be a surprise. I agree, this is predictable, and in a just world our response would be predictable, too: if you falsify test scores to get a $2,970 bonus, you get kicked out of the profession, just like if you took $2,970 out of the drawer as a bank teller and tried to pocket it. (n.b. While acknowledging that 95% of bank tellers are morally upright individuals, we devise our systems under the assumption that they need to be caught if they are attempting to cheat. That assumption is instructive.) In the world we actually live in, you'll have a union rep testify as to how dedicated of a teacher you are, and you'll probably not miss a day of class.
- gjm11 16y ago
  There's an interesting difference between the situations. If you're a bank teller, your whole job is to manage the flow of money and information to and from customers of the bank, the ultimate goal being to make more money for the bank. If you take $3k out of the drawer, then you have failed at your immediate task (money has gone to the wrong place) and worked against the overall purpose of your job (the bank has less money, rather than more). If you're a teacher, the heart of your job is (or should be) teaching your pupils: enabling them to know more and think better and so forth. A secondary purpose (the real purpose, according to some cynics) is to keep them out of their parents' hair and off the streets for a few hours every day. Another (subsidiary to the first, in an ideal world) is enabling them to do as well as possible in whatever external examinations they take, in the hope that it'll help them in later life. Arranging for your pupils' standardized test scores not to be too high comes very low down that list -- and if you get away with it, it benefits your pupils and your school. (While harming other pupils and other schools and other people generally, by making the test results less useful. Tragedy of the commons.) That's not to say that it's not very bad for teachers to cheat to get better test results for their pupils. But it's bad in a different way from a bank teller's embezzlement: peripheral to their job rather than central, and immediately beneficial rather than harmful to the people you're supposed to be working for. A better analogy (though perhaps an overdramatic one, given recent economic events) would be a mortgage salesperson, working for a mortgage broker, who sells people mortgages they are too likely to default on. When that happens, the salesperson gets more commission; his employer, the broker, gets more commission; the person who buys the mortgage gets a mortgage they want and otherwise couldn't have had (and, let's say, does manage to keep up the payments in 90% of cases, fails to do so but not so badly that they lose their home in 5% of cases, and crashes hard in 5% of cases -- which is probably kinda parallel to the outcomes for pupils whose test scores are inflated); other people, less directly connected with the broker, are the ones who get screwed. Anyone expect mortgage salespeople who sell a bit too enthusiastically to get kicked out of their profession en masse? ... No, nor me. > In the world we actually live in ... you'll probably not miss a day of class. That doesn't appear to be what happened in the cases in the NYT article.

samratjp 16y ago

Obligatory-long-yet-relevant-animation(?) on our educational system:http://www.youtube.com/watch?v=A3oIiH7BLmg http://www.youtube.com/watch?v=A3oIiH7BLmg When are we (loosely speaking) going to realize that we're not teaching in the right medium.

politicalist 16y ago

I hope some teachers "cheated" on the tests for principled reasons, not just for personal reward, like this UMass teacher: http://books.google.com/books?id=x3ertj1IcaAC&lpg=PA62&ots=R-V61uVgj4&pg=PA215#v=onepage&q&f=false http://books.google.com/books?id=x3ertj1IcaAC&lpg=PA62&#...

count 16y ago
To clarify, MOST school districts are not giving financial or other rewards for passing the exams at an acceptable level, it's simply expected to occur.

count 16y ago

Cheating is wrong - I don't see any justification otherwise. With that said, the NCLB and standardized tests are destroying school - the only focus now is to learn to answer those multiple choice questions at the end of the year. In Norfolk (one of the cities in the NYT article), the 'SOL' annual standardized test is a sole determining factor in pupil promotion. If you come out of 4th grade with a 100% average, but you fail your SOL for whatever reason (you're nervous and didn't do well, you were sick that day, etc.) - YOU FAIL 4TH GRADE. That seems ridiculous.

ktsmith 16y ago
In schools on the watch list in Nevada where the state has stepped in teachers are no longer really allowed to teach. In 4th grade students receive grades in Reading, Writing, Math, Art, Social Studies, Physical Education and a few other categories. At these schools the teachers are only allowed to teach reading, writing and math. Every lesson plan must be geared towards meeting a standard in one of those three categories. The teachers are required to give out grades in the areas they are not allowed to teach. When pressed on "what" grade the children should receive the answer is typically "whatever you feel is appropriate." These children aren't receiving an education, they are being taught how to memorize information for their CRT's then for their benchmarks, then for whatever test follows that. This isn't what teachers want, this is what teachers are forced to do by their states in order to achieve the largest number of passing schools and keep federal funding levels high.

tansey 16y ago

Seems like there is a need for an independent third-party group to monitor the tests. Since that is likely prohibitively expensive, here's an idea: 1) Have schools partner with each other, where each school is matched to another school of similar size. 2) Extend the testing into twice as long periods (e.g., if it was a 1-day test, now the test period is 2 days). 3) Have partner schools test on opposite period halves. 4) Have partner schools give students the day(s) off on the non-test half. 5) Have schools send their teachers to their partner schools to monitor all the tests. Now, this isn't fool-proof. It's possible that teachers across multiple schools could collude. It would be crucial to mix teacher pairs up so that if A monitors B then B does not monitor A, thus requiring a multi-way collusion that involves many participants.