4 ms·

I don't see how this follows. From Tao's derivation, if a student believes the probability that an answer is right is p and they put down q, then their expected

I don't see how this follows. From Tao's derivation, if a student believes the probability that an answer is right is p and they put down q, then their expected score is pf(q) - (1-p)f(1-q). If the scoring function is a logarithm, the maximum of this expectation occurs at q=p.

Retric 10y ago
Suppose you rolled a 100 sided die 100 times. What are the odds it hits 99 (0,1,2,3,4...) times. Now average each of those to find your expected score. Second, suppose you only care if your final score is an A. If you say your 98% positive and get 2 wrong that's > saying 99% and getting 2 wrong. Sure, you can get a higher max score by saying 99%, but if you only care about an A then you might as well hedge.
- gjm11 10y ago
  Some students will want to maximize their score. Some will want to maximize their probability of getting an A. Some will want to maximize their probability of not failing. There's obviously no scoring rule you can use that will make all of these be optimized by giving accurate probabilities. The logarithmic scoring rule (or any other proper scoring rule) means that a student wanting to maximize their expected score is incentivized to give accurate probabilities. I don't think it's reasonable to ask for more.
  - dllthomas 10y ago
    I agree that it's probably (let's say 60% confidence) impossible to satisfy all of these at once. That said, if we have to pick just one, I think it's worth asking what information is conveyed by the test. For instance, if all that is reported is "pass/fail", we should pick a scoring rule that maximizes a test taker's chance of passing when they are well calibrated. Most interesting to me is how different these scoring rules wind up being...
- zardeh 10y ago
  Simply state "The test will be (bell)curved". Now all students have the same goal: perform better than as many other students as possible.

Retric 10y ago

Suppose you rolled a 100 sided die 100 times. What are the odds it hits 99 (0,1,2,3,4...) times. Now average each of those to find your expected score. Second, suppose you only care if your final score is an A. If you say your 98% positive and get 2 wrong that's > saying 99% and getting 2 wrong. Sure, you can get a higher max score by saying 99%, but if you only care about an A then you might as well hedge.

gjm11 10y ago
Some students will want to maximize their score. Some will want to maximize their probability of getting an A. Some will want to maximize their probability of not failing. There's obviously no scoring rule you can use that will make all of these be optimized by giving accurate probabilities. The logarithmic scoring rule (or any other proper scoring rule) means that a student wanting to maximize their expected score is incentivized to give accurate probabilities. I don't think it's reasonable to ask for more.
- dllthomas 10y ago
  I agree that it's probably (let's say 60% confidence) impossible to satisfy all of these at once. That said, if we have to pick just one, I think it's worth asking what information is conveyed by the test. For instance, if all that is reported is "pass/fail", we should pick a scoring rule that maximizes a test taker's chance of passing when they are well calibrated. Most interesting to me is how different these scoring rules wind up being...
zardeh 10y ago
Simply state "The test will be (bell)curved". Now all students have the same goal: perform better than as many other students as possible.