3 ms·

Okay, let's have that "actual conversation" -- if testing is just a giant boondoggle, what method(s) do you favor for measuring education and achievement in a s

Okay, let's have that "actual conversation" -- if testing is just a giant boondoggle, what method(s) do you favor for measuring education and achievement in a socioeconomic-neutral fashion?

mhuffman 12y ago
I have never seen or heard of a test that is SES-neutral. Even simple "G-Loaded" activities like counting backwards shows both racial and SES bias in younger children. I personally think giving principals more power over hiring and firing and salaries of teachers would go a lot further than testing to improve academic achievement at the school level.
bsder 12y ago
The problem isn't testing. Testing is fine and is actually necessary. Even testing with socioeconomic bias is okay since you can norm to historic values to see trends. The problem occurs at two points: 1) when you then use the results of those tests more broadly than applicable. Comparing results in the same school year on year is generally valid (although you have to watch out for events like a big employer closing). Comparing two different schools close in location requires a bit of finesse. Comparing schools in completely different locations with completely different demographics is generally useless--yet NCLB effectively tries to lump all schools together with the same tests. 2) when you apply those tests to provide punitive measures Why don't people get that doing this is going distort the system? Software people wail continuously about "you can't judge my craft with objective measures." Practically any measure in software I can dream up: number of bugs fixed, number of bugs found, number of commits, number of lines of code written, etc. will be met with cries of "that doesn't measure my ability" AND "if you promote on those measures, people will start gaming the measures". And they will be right. Why can't people see that the exact same thing will occur in teaching? The standard complaint is that only programmers can judge programmers and that non-programmer managers can't judge correctly. Yet, with respect to teaching, we magically think that any moron can judge teaching ability and that non-teaching managers are well equipped to judge teachers. This is one of my hot buttons. Tech folks are so very quick to defend their own area with "you can't objectively judge and punish me" and so quick to apply "but I can objectively judge and punish you." And, even when a school is doing things right, it's really hard to keep it going. Positive results take years to show up unequivocally while negative results can appear almost immediately. See: http://www.thisamericanlife.org/radio-archives/episode/275/transcript http://www.thisamericanlife.org/radio-archives/episode/275/t... The bit about how quickly things dropped one year when they didn't have parents come in 3-times-a-year to review report cards is particularly enlightening. Achievement is getting a lot of little things right, and only the teachers can see when they're right and when they're wrong. By contrast, it took 10 years to move from 15% passing reading and math to 66%. That's roughly 5% a year, and I suspect the first years were slower than that. 5% is statistical noise to a test--which means that all of the things that were occurring in the first years would have been filtered out by a test. Not the result you wanted. So, given that achievement at that school is now dropping, what does the test tell us to do? Where does the test say: "Whoops. The teachers are the same, so A) either the children got worse or B) we need to unroll every single change made by administrators in the last year." Where is THAT application of the test? Now do you see the problem with testing?
- AKrumbach 12y ago
  > Practically any measure in software I can dream up ... will be met with cries of "that doesn't measure my ability" AND "if you promote on those measures, people will start gaming the measures". And they will be right. Sure, any measurement system will fail, but there are means to make them more robust. One way is to make it so that what is measured is what is desired: gaming the system results in more of the desired behavior. Consider a support desk measured not by time-to-resolution or call volume, but post-resolution surveys: in order to 'game' the measurement, the agents must actively seek to promote customer satisfaction. Another, more controversial, means for more robust measurements might be an overt, active gamification. Using your example of LoC: instead of unilaterally praising large (or small) work efforts, LoC is used as the measurement in a bid war. I'm not envisioning a particular bid process (silent auction? Liar's dice style round-robin?) but anybody can call BS on the latest bid ... which then holds that bidder to implement. Beat your own estimate for praise, but for punishment you would have to exceed the prior (non-BS) wager. Couple a second bid value (deadline, for example) and the complexities of actively "gaming the system" becomes nigh on impossible. Between this article about the Atlanta cheating scandal and the NPR story you linked, I wholeheartedly believe that the "standardized test" regimen is currently broken. However, I was hopeful that you might have some suggestions on alternative measurements or ways we could better apply the current ones.
  - bsder 12y ago
    Using tests for an "objective measure" is doomed on its face. That's gigantic sign that the people involved are more interested in blame transfer than in progress. You can use tests for trends. That's a good use of tests. You can use tests for feedback to the individuals directly involved. That's also a good use of tests. This corresponds to my "software metrics" points. If I suddenly see an uptick in average lines of code committed, I should take a look. If I start seeing a big increase in bugs filed, I should take a look and see what just happened. Most of the time, there's a good explanation. Sometimes, someone is doing something stupid and I need to go fix that. And, rarely, someone is doing something malicious and needs to be fired. Tests should be used as an indicator to a human to "look here!" However, test results are simply an indicator to a human to exercise judgment--not an objective measure to remove humans from the loop. The problem with education is that people really don't want to use the tests as "indicators". As in the article I pointed out, that particular school now is having a decrease in results--the tests are indicating with a big, red klaxon that something is going wrong. People should be examining the situation with a magnifying glass to figure out the issues and fix them. The problem is that "something going wrong" is very clear--a new principal and new directives from the head bureaucrats. Do you think that they will look at the test results and say "Oh, we need to undo these policies and get a new principal?" Unlikely. Would you like a metric that only ever gets used against you but that your management gets to ignore? Of course not. It's why the complaints by programmers about management are legion. And that's why teachers hate all this stuff. Somehow, it only ever gets used against the teachers. When the evidence points at the administrators, school board or politicians, it gets ignored. This isn't even limited to teachers. Physicians oppose the collection of this kind of data about outcomes for the sames reasons--it will very quickly get turned into a ranking system that gets used against doctors rather than for improving outcomes.

Okay, let's have that "actual conversation" -- if testing is just a giant boondoggle, what method(s) do you favor for measuring education and achievement in a socioeconomic-neutral fashion?

mhuffman 12y ago

I have never seen or heard of a test that is SES-neutral. Even simple "G-Loaded" activities like counting backwards shows both racial and SES bias in younger children. I personally think giving principals more power over hiring and firing and salaries of teachers would go a lot further than testing to improve academic achievement at the school level.

bsder 12y ago

The problem isn't testing. Testing is fine and is actually necessary. Even testing with socioeconomic bias is okay since you can norm to historic values to see trends. The problem occurs at two points: 1) when you then use the results of those tests more broadly than applicable. Comparing results in the same school year on year is generally valid (although you have to watch out for events like a big employer closing). Comparing two different schools close in location requires a bit of finesse. Comparing schools in completely different locations with completely different demographics is generally useless--yet NCLB effectively tries to lump all schools together with the same tests. 2) when you apply those tests to provide punitive measures Why don't people get that doing this is going distort the system? Software people wail continuously about "you can't judge my craft with objective measures." Practically any measure in software I can dream up: number of bugs fixed, number of bugs found, number of commits, number of lines of code written, etc. will be met with cries of "that doesn't measure my ability" AND "if you promote on those measures, people will start gaming the measures". And they will be right. Why can't people see that the exact same thing will occur in teaching? The standard complaint is that only programmers can judge programmers and that non-programmer managers can't judge correctly. Yet, with respect to teaching, we magically think that any moron can judge teaching ability and that non-teaching managers are well equipped to judge teachers. This is one of my hot buttons. Tech folks are so very quick to defend their own area with "you can't objectively judge and punish me" and so quick to apply "but I can objectively judge and punish you." And, even when a school is doing things right, it's really hard to keep it going. Positive results take years to show up unequivocally while negative results can appear almost immediately. See: http://www.thisamericanlife.org/radio-archives/episode/275/transcript http://www.thisamericanlife.org/radio-archives/episode/275/t... The bit about how quickly things dropped one year when they didn't have parents come in 3-times-a-year to review report cards is particularly enlightening. Achievement is getting a lot of little things right, and only the teachers can see when they're right and when they're wrong. By contrast, it took 10 years to move from 15% passing reading and math to 66%. That's roughly 5% a year, and I suspect the first years were slower than that. 5% is statistical noise to a test--which means that all of the things that were occurring in the first years would have been filtered out by a test. Not the result you wanted. So, given that achievement at that school is now dropping, what does the test tell us to do? Where does the test say: "Whoops. The teachers are the same, so A) either the children got worse or B) we need to unroll every single change made by administrators in the last year." Where is THAT application of the test? Now do you see the problem with testing?

AKrumbach 12y ago
> Practically any measure in software I can dream up ... will be met with cries of "that doesn't measure my ability" AND "if you promote on those measures, people will start gaming the measures". And they will be right. Sure, any measurement system will fail, but there are means to make them more robust. One way is to make it so that what is measured is what is desired: gaming the system results in more of the desired behavior. Consider a support desk measured not by time-to-resolution or call volume, but post-resolution surveys: in order to 'game' the measurement, the agents must actively seek to promote customer satisfaction. Another, more controversial, means for more robust measurements might be an overt, active gamification. Using your example of LoC: instead of unilaterally praising large (or small) work efforts, LoC is used as the measurement in a bid war. I'm not envisioning a particular bid process (silent auction? Liar's dice style round-robin?) but anybody can call BS on the latest bid ... which then holds that bidder to implement. Beat your own estimate for praise, but for punishment you would have to exceed the prior (non-BS) wager. Couple a second bid value (deadline, for example) and the complexities of actively "gaming the system" becomes nigh on impossible. Between this article about the Atlanta cheating scandal and the NPR story you linked, I wholeheartedly believe that the "standardized test" regimen is currently broken. However, I was hopeful that you might have some suggestions on alternative measurements or ways we could better apply the current ones.
- bsder 12y ago
  Using tests for an "objective measure" is doomed on its face. That's gigantic sign that the people involved are more interested in blame transfer than in progress. You can use tests for trends. That's a good use of tests. You can use tests for feedback to the individuals directly involved. That's also a good use of tests. This corresponds to my "software metrics" points. If I suddenly see an uptick in average lines of code committed, I should take a look. If I start seeing a big increase in bugs filed, I should take a look and see what just happened. Most of the time, there's a good explanation. Sometimes, someone is doing something stupid and I need to go fix that. And, rarely, someone is doing something malicious and needs to be fired. Tests should be used as an indicator to a human to "look here!" However, test results are simply an indicator to a human to exercise judgment--not an objective measure to remove humans from the loop. The problem with education is that people really don't want to use the tests as "indicators". As in the article I pointed out, that particular school now is having a decrease in results--the tests are indicating with a big, red klaxon that something is going wrong. People should be examining the situation with a magnifying glass to figure out the issues and fix them. The problem is that "something going wrong" is very clear--a new principal and new directives from the head bureaucrats. Do you think that they will look at the test results and say "Oh, we need to undo these policies and get a new principal?" Unlikely. Would you like a metric that only ever gets used against you but that your management gets to ignore? Of course not. It's why the complaints by programmers about management are legion. And that's why teachers hate all this stuff. Somehow, it only ever gets used against the teachers. When the evidence points at the administrators, school board or politicians, it gets ignored. This isn't even limited to teachers. Physicians oppose the collection of this kind of data about outcomes for the sames reasons--it will very quickly get turned into a ranking system that gets used against doctors rather than for improving outcomes.