4 ms·

As one of the commenters points out, Tao has rediscovered the notion of a proper scoring rule -- https://en.wikipedia.org/wiki/Scoring_rule#Proper_scoring_rules

by gjm11 10y ago

As one of the commenters points out, Tao has rediscovered the notion of a proper scoring rule -- https://en.wikipedia.org/wiki/Scoring_rule#Proper_scoring_rules https://en.wikipedia.org/wiki/Scoring_rule#Proper_scoring_ru... -- and the specific (rather nice, if you don't mind all the scores being negative and the infinite penalty when something happens that you said definitely wouldn't) logarithmic scoring rule -- https://en.wikipedia.org/wiki/Scoring_rule#Logarithmic_scoring_rule https://en.wikipedia.org/wiki/Scoring_rule#Logarithmic_scori.... The Brier score -- you score minus the average squared error between your prediction and [1 for the right outcome, 0 for the others] -- is also a proper scoring rule (i.e., incentivizes you to report your probabilities accurately) and doesn't penalize maximally-wrong answers infinitely. For some purposes it's a better choice than the logarithmic score.

shoyer 10y ago
In the traditional formulation, the Brier score is 0 for correct guesses with p=1, 1 for incorrect guesses with p=1 and 0.25 for guesses with p=0.5: https://en.wikipedia.org/wiki/Brier_score#Example https://en.wikipedia.org/wiki/Brier_score#Example When translated into a grading rule, this gives us a score range from -3 to +1, with 0 for uncertain guesses. So it's not as harsh as logarithmic scoring, but still penalizes confident wrong guesses more harshly than it rewards confident correct guesses.
- deleted 10y ago
  [deleted]
- Retric 10y ago
  Sounds interesting. The problem with logarithmic scoring is your penalized for picking actual percentages vs gaming the system. EX: If your 99% sure of each question and there are 100 questions then getting 1 wrong gives the best points at 99%. But, randomly you can only add +1 point from 100 correct but randomly there is a long tail of missing several. Further, scores are not linearly valuable trading a lower max score but higher chance of getting an A is a positive result. Ideally you should set things up so accurate estimates give the best results.
  - openasocket 10y ago
    I don't see how this follows. From Tao's derivation, if a student believes the probability that an answer is right is p and they put down q, then their expected score is pf(q) - (1-p)f(1-q). If the scoring function is a logarithm, the maximum of this expectation occurs at q=p.
    - Retric 10y ago
      Suppose you rolled a 100 sided die 100 times. What are the odds it hits 99 (0,1,2,3,4...) times. Now average each of those to find your expected score. Second, suppose you only care if your final score is an A. If you say your 98% positive and get 2 wrong that's > saying 99% and getting 2 wrong. Sure, you can get a higher max score by saying 99%, but if you only care about an A then you might as well hedge.
      - gjm11 10y ago
        Some students will want to maximize their score. Some will want to maximize their probability of getting an A. Some will want to maximize their probability of not failing. There's obviously no scoring rule you can use that will make all of these be optimized by giving accurate probabilities. The logarithmic scoring rule (or any other proper scoring rule) means that a student wanting to maximize their expected score is incentivized to give accurate probabilities. I don't think it's reasonable to ask for more.
        dllthomas 10y ago
        I agree that it's probably (let's say 60% confidence) impossible to satisfy all of these at once. That said, if we have to pick just one, I think it's worth asking what information is conveyed by the test. For instance, if all that is reported is "pass/fail", we should pick a scoring rule that maximizes a test taker's chance of passing when they are well calibrated. Most interesting to me is how different these scoring rules wind up being...
      - zardeh 10y ago
Houshalter 10y ago
The Brier score isn't very principled mathematically though. Squared error was developed for findinding models over real numbers, and doesn't fit very naturally with probabilities. Log likelihood does allow infinitely negative scores, but maybe it should. If you assign a probability of 0, you are saying it's literally impossible. Not just unlikely, or even really really unlikely. I prefer geometric mean of probability. It represents the "average probability" you gave to the correct answer, and the number is interpretable. But it's isomorphic to log likelihood, so has all the same mathematical properties.
- contravariant 10y ago
  There's one problem with considering the geometric mean to be the "average probability" which is that the geometric mean of p and q isn't 1 minus the mean of 1 - p and 1 - q. Ideally you'd like the average probability of something happening and the average probability of something not happening to add up to one. Depending on what you do with it this may or may not be a problem.
  - Houshalter 10y ago
    That's true, but it's not supposed to add up to 1. Likelihood represents the probability you assigned to the actual outcome. So the probability of every possible outcome needs to sum to 1. But because each question increases the number of possible outcomes exponentially, the probability you mass you assign to any specific outcome must become very very small. And it also becomes uninterpretable, because it is such a small number. You can recover the likelihood from the geometric mean of likelihood, by exponentiating it to the power of the number of questions. So if you were predicting 10 coin flips, and the geometric mean was 0.5, you can find the original likelihood by taking 0.5^10. Then if you do that for every possible outcome, the probability will sum to 1.

by gjm11 10y ago

shoyer 10y ago

In the traditional formulation, the Brier score is 0 for correct guesses with p=1, 1 for incorrect guesses with p=1 and 0.25 for guesses with p=0.5: https://en.wikipedia.org/wiki/Brier_score#Example https://en.wikipedia.org/wiki/Brier_score#Example When translated into a grading rule, this gives us a score range from -3 to +1, with 0 for uncertain guesses. So it's not as harsh as logarithmic scoring, but still penalizes confident wrong guesses more harshly than it rewards confident correct guesses.

deleted 10y ago
[deleted]
Retric 10y ago
Sounds interesting. The problem with logarithmic scoring is your penalized for picking actual percentages vs gaming the system. EX: If your 99% sure of each question and there are 100 questions then getting 1 wrong gives the best points at 99%. But, randomly you can only add +1 point from 100 correct but randomly there is a long tail of missing several. Further, scores are not linearly valuable trading a lower max score but higher chance of getting an A is a positive result. Ideally you should set things up so accurate estimates give the best results.
- openasocket 10y ago
  I don't see how this follows. From Tao's derivation, if a student believes the probability that an answer is right is p and they put down q, then their expected score is pf(q) - (1-p)f(1-q). If the scoring function is a logarithm, the maximum of this expectation occurs at q=p.
  - Retric 10y ago
    Suppose you rolled a 100 sided die 100 times. What are the odds it hits 99 (0,1,2,3,4...) times. Now average each of those to find your expected score. Second, suppose you only care if your final score is an A. If you say your 98% positive and get 2 wrong that's > saying 99% and getting 2 wrong. Sure, you can get a higher max score by saying 99%, but if you only care about an A then you might as well hedge.
    - gjm11 10y ago
      Some students will want to maximize their score. Some will want to maximize their probability of getting an A. Some will want to maximize their probability of not failing. There's obviously no scoring rule you can use that will make all of these be optimized by giving accurate probabilities. The logarithmic scoring rule (or any other proper scoring rule) means that a student wanting to maximize their expected score is incentivized to give accurate probabilities. I don't think it's reasonable to ask for more.
      - dllthomas 10y ago
        I agree that it's probably (let's say 60% confidence) impossible to satisfy all of these at once. That said, if we have to pick just one, I think it's worth asking what information is conveyed by the test. For instance, if all that is reported is "pass/fail", we should pick a scoring rule that maximizes a test taker's chance of passing when they are well calibrated. Most interesting to me is how different these scoring rules wind up being...
    - zardeh 10y ago

Houshalter 10y ago

The Brier score isn't very principled mathematically though. Squared error was developed for findinding models over real numbers, and doesn't fit very naturally with probabilities. Log likelihood does allow infinitely negative scores, but maybe it should. If you assign a probability of 0, you are saying it's literally impossible. Not just unlikely, or even really really unlikely. I prefer geometric mean of probability. It represents the "average probability" you gave to the correct answer, and the number is interpretable. But it's isomorphic to log likelihood, so has all the same mathematical properties.

contravariant 10y ago
There's one problem with considering the geometric mean to be the "average probability" which is that the geometric mean of p and q isn't 1 minus the mean of 1 - p and 1 - q. Ideally you'd like the average probability of something happening and the average probability of something not happening to add up to one. Depending on what you do with it this may or may not be a problem.
- Houshalter 10y ago
  That's true, but it's not supposed to add up to 1. Likelihood represents the probability you assigned to the actual outcome. So the probability of every possible outcome needs to sum to 1. But because each question increases the number of possible outcomes exponentially, the probability you mass you assign to any specific outcome must become very very small. And it also becomes uninterpretable, because it is such a small number. You can recover the likelihood from the geometric mean of likelihood, by exponentiating it to the power of the number of questions. So if you were predicting 10 coin flips, and the geometric mean was 0.5, you can find the original likelihood by taking 0.5^10. Then if you do that for every possible outcome, the probability will sum to 1.