4 ms·

The SAT used to have a wrong answer penalty to discourage random guessing. This seems to me a simple, effective way to reflect a student's certainty in an answ

The SAT used to have a wrong answer penalty to discourage random guessing. This seems to me a simple, effective way to reflect a student's certainty in an answer.

mikeash 10y ago
That works too, but it's much less granular. You only have two levels of certainty there, with only three possible results: certain and correct, certain and incorrect, and uncertain with unknown correctness. That means you can distinguish between, say, 50% certainty and 10% certainty, but you can't distinguish between 50% certainty and 99% certainty.
- ChicagoBoy11 10y ago
  True, but how useful is that granularity, really? If you combine the fact that the test has multiple items assessing the same skill, and the fact that they employ (reasonably) sophisticated statistics (Item-response theory) to come up with a probabilistic assessment of the student's true ability given the observed answer pattern, would something like this really buy you a whole lot more?
  - mikeash 10y ago
    It's probably not too useful. Ideally it would reduce variance a bit, but probably not enormously. People don't really understand their own level of certainty, anyway, so that would be a confounding factor. I wonder if it would be a better compromise to allow multiple answers. If I've narrowed it down to two, I guess one and then I have a 50% chance of getting it right. Instead, let me just put down both answers, and if I got it right, give me half a point. I think you'd get the same ultimate outcome as penalizing wrong answers, but with a bit less randomness, while maintaining comprehension. Maybe!
daxfohl 10y ago
Funny thing is the weights they assigned didn't discourage random guessing at all; they merely made your expected gain from guessing equal with the expected gain from not answering (i.e. zero). So even if you had the very slightest idea, you should absolutely guess. If you took the test multiple times, then probabilistically you'd be better off if you always guessed even when you had no clue: some of your scores would be higher and some would be lower than if you hadn't guessed, and usually only your highest score is considered. Most guides overemphasized the penalty. That said, it can screw you of course too if you happen to be unlucky in your guessing. This is what Terry's scheme overcomes.
- tamana 10y ago
  Terry's scheme punishes well educated people who are conservative in their claims of confidence
  - nilkn 10y ago
    If one were to actually use this scheme for a class, I think it'd make sense to compute both the classical grade and the probabilistic grade and isolate students whose classical and probabilistic grades differed greatly (as in your example of someone who answers everything correctly, but with low confidence). The professor could then exercise manual judgment to assign the appropriate grade and also probably discuss the discrepancy with the student (i.e., they might find it interesting that they consistently underestimated their knowledge of the subject; on the other hand, a student who was overly confident but did poorly might find it interesting that they were consistently overestimating their knowledge).
  - Jabbles 10y ago
    Why should being conservative in your confidence be rewarded above having an accurate assessment of your confidence?
    - Jtsummers 10y ago
      In the case of an exam, students are under stress and pressure that will often lead them to underestimate themselves, especially knowing that if they're wrong that 100% probability will net them a -infinity score for the exam, even if they get everything else right. This also means questions have to be very carefully worded. I recall a number of exam questions (over many years of school, not one class in particular) where the "wrong" answers were arguably right depending on how the sentence was parsed just based on poor use of punctuation or other ambiguous wording.
  - arjie 10y ago
    As it should. If you don't know that you know, do you really know?
  - 10y ago

The SAT used to have a wrong answer penalty to discourage random guessing. This seems to me a simple, effective way to reflect a student's certainty in an answer.

mikeash 10y ago

That works too, but it's much less granular. You only have two levels of certainty there, with only three possible results: certain and correct, certain and incorrect, and uncertain with unknown correctness. That means you can distinguish between, say, 50% certainty and 10% certainty, but you can't distinguish between 50% certainty and 99% certainty.

ChicagoBoy11 10y ago
True, but how useful is that granularity, really? If you combine the fact that the test has multiple items assessing the same skill, and the fact that they employ (reasonably) sophisticated statistics (Item-response theory) to come up with a probabilistic assessment of the student's true ability given the observed answer pattern, would something like this really buy you a whole lot more?
- mikeash 10y ago
  It's probably not too useful. Ideally it would reduce variance a bit, but probably not enormously. People don't really understand their own level of certainty, anyway, so that would be a confounding factor. I wonder if it would be a better compromise to allow multiple answers. If I've narrowed it down to two, I guess one and then I have a 50% chance of getting it right. Instead, let me just put down both answers, and if I got it right, give me half a point. I think you'd get the same ultimate outcome as penalizing wrong answers, but with a bit less randomness, while maintaining comprehension. Maybe!

daxfohl 10y ago

Funny thing is the weights they assigned didn't discourage random guessing at all; they merely made your expected gain from guessing equal with the expected gain from not answering (i.e. zero). So even if you had the very slightest idea, you should absolutely guess. If you took the test multiple times, then probabilistically you'd be better off if you always guessed even when you had no clue: some of your scores would be higher and some would be lower than if you hadn't guessed, and usually only your highest score is considered. Most guides overemphasized the penalty. That said, it can screw you of course too if you happen to be unlucky in your guessing. This is what Terry's scheme overcomes.

tamana 10y ago
Terry's scheme punishes well educated people who are conservative in their claims of confidence
- nilkn 10y ago
  If one were to actually use this scheme for a class, I think it'd make sense to compute both the classical grade and the probabilistic grade and isolate students whose classical and probabilistic grades differed greatly (as in your example of someone who answers everything correctly, but with low confidence). The professor could then exercise manual judgment to assign the appropriate grade and also probably discuss the discrepancy with the student (i.e., they might find it interesting that they consistently underestimated their knowledge of the subject; on the other hand, a student who was overly confident but did poorly might find it interesting that they were consistently overestimating their knowledge).
- Jabbles 10y ago
  Why should being conservative in your confidence be rewarded above having an accurate assessment of your confidence?
  - Jtsummers 10y ago
    In the case of an exam, students are under stress and pressure that will often lead them to underestimate themselves, especially knowing that if they're wrong that 100% probability will net them a -infinity score for the exam, even if they get everything else right. This also means questions have to be very carefully worded. I recall a number of exam questions (over many years of school, not one class in particular) where the "wrong" answers were arguably right depending on how the sentence was parsed just based on poor use of punctuation or other ambiguous wording.
- arjie 10y ago
  As it should. If you don't know that you know, do you really know?
- 10y ago