4 ms·
A comment on the reddit discussion of this said that a similar scheme was used at CMU for midterm and final exams in the "Decision Systems and Decision Support
by tzs 10y ago
A comment on the reddit discussion of this said that a similar scheme was used at CMU for midterm and final exams in the "Decision Systems and Decision Support Analysis" class offered by the decision theory department. The exams were multiple choice with four choices per question. The commentator linked to this handout describing the grading for the midterm: http://www.contrib.andrew.cmu.edu/%7Esbaugh/midterm_grading_function.pdf http://www.contrib.andrew.cmu.edu/%7Esbaugh/midterm_grading_...
If you assigned probability p to the correct choice, your score for that question was 1 + ln(p)/ln(4). You were not allowed to assign a probability of 0 or 1 to a choice.
The handout points out the importance of thinking about how to approach these tests ahead of time, and also explains the benefits of using this scoring system:
---- begin quote ----
I cannot stress strongly enough the need for each of you to sit down and think about different strategies for answering the questions. This grading technique completely removes any benefit of random guessing. Such a guess could be disastrous. You're much better off admitting that you don't know the answer to a question. (Placing a 0.25 probability by every option indicates that you have no idea which answer is correct, and your score will be 0 for that question). Assessments of probability 0 (0%) or 1 (100%) are not allowed. These answers will be interpreted as probability 0.001 (0.1%) and 0.997 (99.7%) respectively. Your probability assessments must sum to 1 (100%). A probability of 0.001 by the correct answer will result in a score of -4. In contrast, a probability of 0.997 on the correct answer only earns a score of 1. Think about the implications of this before the day of the test.
I strongly recommend that you analyze the grading problem from a decision analysis perspective. Calculate expected values (or expected utilities) for various levels of personal uncertainty. Notice what happens if you are overconfident or underconfident.
This grading scheme makes the midterm harder then a standard multiple choice test, but this is the point. It has many benefits from a teaching/learning perspective.
1) It teaches you to apply the techniques that have been discussed in class. You have to assess your own personal probabilities and apply them to problems that have very real (and potentially) important payoffs. It is impossible to get these points across with the few simple lotteries demonstrated during class.
2) It helps to removes the element of chance from the test. Because of the severe penalty for guessing, the test will more accurately measure your knowledge.
3) The test will also measure what you know about your knowledge.
4) By analyzing how you answer the questions, I will be able to determine which questions are hard and which questions you believe are hard. (A "flatter" distribution for a question indicates a lower confidence in the question, and therefore, a belief that it is hard.)
5) It will allow you to appreciate how hard it is to assess probabilities.
---- end quote ----