6 ms·
In the traditional formulation, the Brier score is 0 for correct guesses with p=1, 1 for incorrect guesses with p=1 and 0.25 for guesses with p=0.5: https://en.
by shoyer 10y ago
In the traditional formulation, the Brier score is 0 for correct guesses with p=1, 1 for incorrect guesses with p=1 and 0.25 for guesses with p=0.5:
https://en.wikipedia.org/wiki/Brier_score#Example https://en.wikipedia.org/wiki/Brier_score#Example
When translated into a grading rule, this gives us a score range from -3 to +1, with 0 for uncertain guesses. So it's not as harsh as logarithmic scoring, but still penalizes confident wrong guesses more harshly than it rewards confident correct guesses.
- deleted 10y ago[deleted]
- Retric 10y agoSounds interesting. The problem with logarithmic scoring is your penalized for picking actual percentages vs gaming the system. EX: If your 99% sure of each question and there are 100 questions then getting 1 wrong gives the best points at 99%. But, randomly you can only add +1 point from 100 correct but randomly there is a long tail of missing several. Further, scores are not linearly valuable trading a lower max score but higher chance of getting an A is a positive result. Ideally you should set things up so accurate estimates give the best results.
- openasocket 10y agoI don't see how this follows. From Tao's derivation, if a student believes the probability that an answer is right is p and they put down q, then their expected score is pf(q) - (1-p)f(1-q). If the scoring function is a logarithm, the maximum of this expectation occurs at q=p.
- Retric 10y agoSuppose you rolled a 100 sided die 100 times. What are the odds it hits 99 (0,1,2,3,4...) times. Now average each of those to find your expected score. Second, suppose you only care if your final score is an A. If you say your 98% positive and get 2 wrong that's > saying 99% and getting 2 wrong. Sure, you can get a higher max score by saying 99%, but if you only care about an A then you might as well hedge.
- gjm11 10y agoSome students will want to maximize their score. Some will want to maximize their probability of getting an A. Some will want to maximize their probability of not failing. There's obviously no scoring rule you can use that will make all of these be optimized by giving accurate probabilities. The logarithmic scoring rule (or any other proper scoring rule) means that a student wanting to maximize their expected score is incentivized to give accurate probabilities. I don't think it's reasonable to ask for more.
- dllthomas 10y agoI agree that it's probably (let's say 60% confidence) impossible to satisfy all of these at once. That said, if we have to pick just one, I think it's worth asking what information is conveyed by the test. For instance, if all that is reported is "pass/fail", we should pick a scoring rule that maximizes a test taker's chance of passing when they are well calibrated. Most interesting to me is how different these scoring rules wind up being...
- zardeh 10y agoSimply state "The test will be (bell)curved". Now all students have the same goal: perform better than as many other students as possible.
- shoyer 10y ago> Ideally you should set things up so accurate estimates give the best results. This is true for both the logarithmic score and the Brier score. These are both "strictly proper scoring rules", which is a formal way of stating your requirement that it should not be possible to game the system. In both cases, you get the highest score (on average) if you guess is true distribution.
- Retric 10y agoThe problem is if your X% sure on each of a set of questions and you want to get an A, you want to minimize your chances of getting a bad grade more than you want to maximizes your score. Alternatively, there is no point in setting your odds for every question below the point that takes the minimum number of correct answers to pass. EX: If you need to get X% correct to pass, then don't set your odds below X%. For a similar example consider what you bet on the final question in Jeopardy does not just depend on your estimate of your odds, but other factors. Of course even thinking about this stuff is distracting, you may be better off picking a very small number of odds. Say 99%, 90%, 70%, and 50%.
- dllthomas 10y agoRetric was distinguishing between "the highest score" and "the best result". A higher score on an exam doesn't always turn into a higher grade in the class. If your outcomes are bucketed, you may be able to increase the odds you fall in the highest bucket by decreasing your probable final score.