6 ms·

Should the U.S. Make Standardized Tests Harder?

30 points by tokenadult 12y ago

logfromblammo 12y ago
Before addressing the difficulty factor, the testing companies should be refining the testing methods to better judge student mastery of the subject matter rather than their test-taking ability. This involves automated production of questions unique to a particular test-taker, and automated scoring of the answers. Once that has been done, the difficulty of the test can even be adjusted on the fly to better determine the exact ability level of the student. Rather than having separate static tests for an expected range of ability levels, everything from preschool readiness to SAT and GRE equivalents can be tested by the same software. If a child takes state-mandated standardized tests, and it shows a perfect score in math, the teachers may put that student in the most advanced math class taught at that grade level. But what if the test could show that the student's ability exceeded that of the typical student four or five years older, without becoming less precise for typical students the same age? It's really a shame that standardized tests haven't yet evolved beyond multiple choice answers for human-written questions and scored by machines with optical scanners. But in general, I do think that test should be harder. They should be so hard that no one can achieve a perfect score, even by chance. And they should also be so easy that no one can score zero. It's the only way to get a good idea of the population distribution of the people you are testing.
- Torgo 12y ago
  >It's really a shame that standardized tests haven't yet evolved beyond multiple choice answers for human-written questions and scored by machines with optical scanners. They have somewhat, it's called adaptive, or "branched" testing. It is a computer-based test that gives you progressively harder questions to find the limit of your abilities and when you get them wrong it drops the difficulty again in an attempt to try to find out exactly where your knowledge drops off. That's an extremely oversimplified explanation, but you can read the Wikipedia article, which is OK: https://en.wikipedia.org/wiki/Computerized_adaptive_testing https://en.wikipedia.org/wiki/Computerized_adaptive_testing The problem is money, and to be honest, the motivations of who is buying the test (for some clients the testing is just meant to be a professional barrier to entry or to do the bare minimum to comply with a government requirement.) I worked in standardized testing for almost a decade on nearly every kind of test, helping psychometricians and test administrators, so I saw a lot of bad and good tests.
danieltillett 12y ago
Yes if means you can't teach to the test. The problem with standardised testing is not testing, it is the design of the tests. Any test that can be gamed is a bad test.
- trynumber9 12y ago
  While a good sentiment, what kind of tests do not have a slight gaming effect when you have previous exposure to a similar test?
  - danieltillett 12y ago
    Tests that require the student draws on a wide variety of skills and knowledge to be able to answer. When I was a professor I used to design my exams in two parts. The first were pure multiple choice questions that the students could game. The second were complex question that required integrating knowledge from multiple areas. Answering these question required understanding - these can't be gamed other than by learning. You might be wondering why have the first part? The reason was because the administration set a quota on the pass rate. If I had only used questions of the second part then too many of my students would have failed. The bargain with my students was if you could fog a mirror you would pass, but to get an A you had to show understanding.
chroma 12y ago
It's important to consider what we want a standardized test to measure. Is it to check that students have learned the curriculum? Is it to evaluate abilities for future prospects? Current tests seem to skew toward the former, and the article suggests they should do more of the latter. One wonders what a pure test for abilities would look like. The biggest factors in predicting long-term success are intelligence, knowledge, and conscientiousness. A hypothetical "how good you are at life" test could look something like this: 1. For intelligence, a straight-up IQ test. Raven's Progressive Matrices[1] is an excellent candidate. It can measure almost the entire human range of IQ, and it has little cultural bias. Heck, it doesn't even require literacy. 2. For knowledge, the format would be similar to existing tests, with one caveat: it must be very broad; so broad that no one person could know everything on it. To keep the length reasonable, one could use random sampling. Just as a vocabulary test need not ask the definition for every word in the dictionary, so too could a knowledge test remain accurate without asking for a copy of Wikipedia. To prevent cheating, it would be crucial to keep the exact questions secret each year. 3. Conscientiousness is the hardest to test, because it requires measuring one's ability to work hard over a long time. Besides self-reports, the only short test for conscientiousness involves telling the test taker that the task is unimportant. If they know the truth, the jig is up and they'll work hard. This sort of test would be extremely hard to game. Training helps RPM very little. Studying for the knowledge section would help one's score, but it would have the side-effect of increasing one's overall knowledge. The biggest wrinkle is the conscientiousness evaluation. It either requires honesty on the part of the test-taker or trickery on the part of the test-giver. While interesting to ponder, I doubt such a test will ever exist. It's not just students and parents who want to be able to game tests. School boards and other organizations want to be able to show improvement over time. If they can't teach the test, their true effectiveness (or lack thereof) might be revealed. 1. http://en.wikipedia.org/wiki/Raven's_Progressive_Matrices http://en.wikipedia.org/wiki/Raven's_Progressive_Matrices
- shutupalready 12y ago
  > The biggest factors in predicting long-term success are intelligence, knowledge, and conscientiousness I wouldn't dispute that those are important factors, but what about: - initiative and/or risk-taking (which is very different from conscientiousness) - attractiveness (either physical or non-physical things like charm or schmoozing) - starting capital (it's easier turn $1 million into $2 million than starting with $0 and getting to $1 million) - general health and fitness (successful people are rarely sickly or unfit) - leadership - number and quality of connections Looking at the tech sector, it seems that a high level of intelligence, knowledge, and conscientiousness will get you a good job. But the most successful people in tech are not those with even higher levels of intelligence, knowledge, and conscientiousness -- for those people it's the other factors that dominate.
exDM69 12y ago
I find it very strange that one of the most important tests that US students take is a bombardment of multiple choice questions. The equivalent where I am from is written essays and/or math, physics, chemistry exercises (ie. calculation) where you have to have all intermediate steps visible (which means you have an chance of scoring a few points even if the final answer is incorrect). You take four to eight exams during a period of two weeks (opportunity to participate is twice a year), six hours per exam. In language exams, there's a listening comprehension part (in addition to the six hours of written exam) where you listen to a recording and answer multiple choice or with a few sentences. The method used here is not entirely problem-free either but it's a whole lot more practical than having a long string of multiple choice questions where you have a 25% chance by guessing and you get to know all the possible answers. This is not really representative of what you need in life or what you'll encounter in later studies in College/University.
- rtpg 12y ago
  A lot of countries do this, and the biggest issue is a practical one: it takes a long time to correct. Multiple choice you can just stick it into a machine. Just another example of how most guiding principles in US school systems are due to a lack of resources. We can only hope that some innovative people will be able to get around this (big advantage of US system is experimentation at the local level)
  - danieltillett 12y ago
    It is not a lack of resources, it is a resource allocation problem. The usa is one of the highest spenders on education in the world, but for some reason can't manage to find the money to test correctly.
  - illumen 12y ago
    Efficient use of resources is a thing as well. Many places use their resources better, and prioritise areas like education. Education, which people pay the state for, is failing in the USA. With 49% functional illiteracy, it is a catastrophic failure.
  - 12y ago
higherpurpose 12y ago
US should give up on the "leave no one behind" mentality - or at least seriously overhaul that system. Because it seems to have pushed everyone to the lowest common denominator. From what I hear, schools even consider everyone a "winner" now, just for passing the year. That seems a bit much...What's the incentive for excellence then?
- ausjke 12y ago
  Exactly, race-to-the-top is more beneficial absolutely, the C-grade president Bush's no-child-left-behind policy is making the whole country's students a C-level group. The bar is set way too low these days, even communism society does not have dumb policy like that to make everyone "happy". The no-child-left-behind policy could drag the whole nation to new lows in the hi-tech era.
  - Dewie 12y ago
    > , the C-grade president Bush's no-child-left-behind policy is making the whole country's students a C-level group. The fact that "C" is an insult is a sign of grade inflation, which might be another symptom of the bar being set too low (and that has to be compensated for by using grades like A+, etc. to distinguish the top performers).
analog31 12y ago
The present day regime of standardized testing was pitched as a way to make school more like "business" by providing a numerical metric for driving things like funding and teacher promotions. But just like in business, we know that people will figure out how to game the metric. Speaking out of fairly narrow self interest, the effect of the tests on my kids is that it dumbs down the curriculum, because drilling the test material becomes more important than moving on to more advanced topics or more complex types of assignments. They are both consistently in the 98th or 99th percentile on every test, and it seems like the testing is constant. For my kids, more sophisticated and varied tests would mean a more sophisticated and varied curriculum.
- spacecowboy_lon 12y ago
  In the UK Schools agressivly game the system to get sligtly dim kids to the required Grade C which is what they are scored on. Of course thise means ingnoring realy bright kids and dumping the ones needing help.

30 points by tokenadult 12y ago

Before addressing the difficulty factor, the testing companies should be refining the testing methods to better judge student mastery of the subject matter rather than their test-taking ability. This involves automated production of questions unique to a particular test-taker, and automated scoring of the answers. Once that has been done, the difficulty of the test can even be adjusted on the fly to better determine the exact ability level of the student. Rather than having separate static tests for an expected range of ability levels, everything from preschool readiness to SAT and GRE equivalents can be tested by the same software. If a child takes state-mandated standardized tests, and it shows a perfect score in math, the teachers may put that student in the most advanced math class taught at that grade level. But what if the test could show that the student's ability exceeded that of the typical student four or five years older, without becoming less precise for typical students the same age? It's really a shame that standardized tests haven't yet evolved beyond multiple choice answers for human-written questions and scored by machines with optical scanners. But in general, I do think that test should be harder. They should be so hard that no one can achieve a perfect score, even by chance. And they should also be so easy that no one can score zero. It's the only way to get a good idea of the population distribution of the people you are testing.

Torgo 12y ago
>It's really a shame that standardized tests haven't yet evolved beyond multiple choice answers for human-written questions and scored by machines with optical scanners. They have somewhat, it's called adaptive, or "branched" testing. It is a computer-based test that gives you progressively harder questions to find the limit of your abilities and when you get them wrong it drops the difficulty again in an attempt to try to find out exactly where your knowledge drops off. That's an extremely oversimplified explanation, but you can read the Wikipedia article, which is OK: https://en.wikipedia.org/wiki/Computerized_adaptive_testing https://en.wikipedia.org/wiki/Computerized_adaptive_testing The problem is money, and to be honest, the motivations of who is buying the test (for some clients the testing is just meant to be a professional barrier to entry or to do the bare minimum to comply with a government requirement.) I worked in standardized testing for almost a decade on nearly every kind of test, helping psychometricians and test administrators, so I saw a lot of bad and good tests.

danieltillett 12y ago

Yes if means you can't teach to the test. The problem with standardised testing is not testing, it is the design of the tests. Any test that can be gamed is a bad test.

trynumber9 12y ago
While a good sentiment, what kind of tests do not have a slight gaming effect when you have previous exposure to a similar test?
- danieltillett 12y ago
  Tests that require the student draws on a wide variety of skills and knowledge to be able to answer. When I was a professor I used to design my exams in two parts. The first were pure multiple choice questions that the students could game. The second were complex question that required integrating knowledge from multiple areas. Answering these question required understanding - these can't be gamed other than by learning. You might be wondering why have the first part? The reason was because the administration set a quota on the pass rate. If I had only used questions of the second part then too many of my students would have failed. The bargain with my students was if you could fog a mirror you would pass, but to get an A you had to show understanding.

chroma 12y ago

It's important to consider what we want a standardized test to measure. Is it to check that students have learned the curriculum? Is it to evaluate abilities for future prospects? Current tests seem to skew toward the former, and the article suggests they should do more of the latter. One wonders what a pure test for abilities would look like. The biggest factors in predicting long-term success are intelligence, knowledge, and conscientiousness. A hypothetical "how good you are at life" test could look something like this: 1. For intelligence, a straight-up IQ test. Raven's Progressive Matrices[1] is an excellent candidate. It can measure almost the entire human range of IQ, and it has little cultural bias. Heck, it doesn't even require literacy. 2. For knowledge, the format would be similar to existing tests, with one caveat: it must be very broad; so broad that no one person could know everything on it. To keep the length reasonable, one could use random sampling. Just as a vocabulary test need not ask the definition for every word in the dictionary, so too could a knowledge test remain accurate without asking for a copy of Wikipedia. To prevent cheating, it would be crucial to keep the exact questions secret each year. 3. Conscientiousness is the hardest to test, because it requires measuring one's ability to work hard over a long time. Besides self-reports, the only short test for conscientiousness involves telling the test taker that the task is unimportant. If they know the truth, the jig is up and they'll work hard. This sort of test would be extremely hard to game. Training helps RPM very little. Studying for the knowledge section would help one's score, but it would have the side-effect of increasing one's overall knowledge. The biggest wrinkle is the conscientiousness evaluation. It either requires honesty on the part of the test-taker or trickery on the part of the test-giver. While interesting to ponder, I doubt such a test will ever exist. It's not just students and parents who want to be able to game tests. School boards and other organizations want to be able to show improvement over time. If they can't teach the test, their true effectiveness (or lack thereof) might be revealed. 1. http://en.wikipedia.org/wiki/Raven's_Progressive_Matrices http://en.wikipedia.org/wiki/Raven's_Progressive_Matrices

shutupalready 12y ago
> The biggest factors in predicting long-term success are intelligence, knowledge, and conscientiousness I wouldn't dispute that those are important factors, but what about: - initiative and/or risk-taking (which is very different from conscientiousness) - attractiveness (either physical or non-physical things like charm or schmoozing) - starting capital (it's easier turn $1 million into $2 million than starting with $0 and getting to $1 million) - general health and fitness (successful people are rarely sickly or unfit) - leadership - number and quality of connections Looking at the tech sector, it seems that a high level of intelligence, knowledge, and conscientiousness will get you a good job. But the most successful people in tech are not those with even higher levels of intelligence, knowledge, and conscientiousness -- for those people it's the other factors that dominate.

exDM69 12y ago

I find it very strange that one of the most important tests that US students take is a bombardment of multiple choice questions. The equivalent where I am from is written essays and/or math, physics, chemistry exercises (ie. calculation) where you have to have all intermediate steps visible (which means you have an chance of scoring a few points even if the final answer is incorrect). You take four to eight exams during a period of two weeks (opportunity to participate is twice a year), six hours per exam. In language exams, there's a listening comprehension part (in addition to the six hours of written exam) where you listen to a recording and answer multiple choice or with a few sentences. The method used here is not entirely problem-free either but it's a whole lot more practical than having a long string of multiple choice questions where you have a 25% chance by guessing and you get to know all the possible answers. This is not really representative of what you need in life or what you'll encounter in later studies in College/University.

rtpg 12y ago
A lot of countries do this, and the biggest issue is a practical one: it takes a long time to correct. Multiple choice you can just stick it into a machine. Just another example of how most guiding principles in US school systems are due to a lack of resources. We can only hope that some innovative people will be able to get around this (big advantage of US system is experimentation at the local level)
- danieltillett 12y ago
  It is not a lack of resources, it is a resource allocation problem. The usa is one of the highest spenders on education in the world, but for some reason can't manage to find the money to test correctly.
- illumen 12y ago
  Efficient use of resources is a thing as well. Many places use their resources better, and prioritise areas like education. Education, which people pay the state for, is failing in the USA. With 49% functional illiteracy, it is a catastrophic failure.
- 12y ago

higherpurpose 12y ago

US should give up on the "leave no one behind" mentality - or at least seriously overhaul that system. Because it seems to have pushed everyone to the lowest common denominator. From what I hear, schools even consider everyone a "winner" now, just for passing the year. That seems a bit much...What's the incentive for excellence then?

ausjke 12y ago
Exactly, race-to-the-top is more beneficial absolutely, the C-grade president Bush's no-child-left-behind policy is making the whole country's students a C-level group. The bar is set way too low these days, even communism society does not have dumb policy like that to make everyone "happy". The no-child-left-behind policy could drag the whole nation to new lows in the hi-tech era.
- Dewie 12y ago
  > , the C-grade president Bush's no-child-left-behind policy is making the whole country's students a C-level group. The fact that "C" is an insult is a sign of grade inflation, which might be another symptom of the bar being set too low (and that has to be compensated for by using grades like A+, etc. to distinguish the top performers).

analog31 12y ago

The present day regime of standardized testing was pitched as a way to make school more like "business" by providing a numerical metric for driving things like funding and teacher promotions. But just like in business, we know that people will figure out how to game the metric. Speaking out of fairly narrow self interest, the effect of the tests on my kids is that it dumbs down the curriculum, because drilling the test material becomes more important than moving on to more advanced topics or more complex types of assignments. They are both consistently in the 98th or 99th percentile on every test, and it seems like the testing is constant. For my kids, more sophisticated and varied tests would mean a more sophisticated and varied curriculum.

spacecowboy_lon 12y ago
In the UK Schools agressivly game the system to get sligtly dim kids to the required Grade C which is what they are scored on. Of course thise means ingnoring realy bright kids and dumping the ones needing help.