3 ms·
> Ideally you should set things up so accurate estimates give the best results. This is true for both the logarithmic score and the Brier score. These are both
by shoyer 10y ago
> Ideally you should set things up so accurate estimates give the best results.
This is true for both the logarithmic score and the Brier score. These are both "strictly proper scoring rules", which is a formal way of stating your requirement that it should not be possible to game the system. In both cases, you get the highest score (on average) if you guess is true distribution.
- Retric 10y agoThe problem is if your X% sure on each of a set of questions and you want to get an A, you want to minimize your chances of getting a bad grade more than you want to maximizes your score. Alternatively, there is no point in setting your odds for every question below the point that takes the minimum number of correct answers to pass. EX: If you need to get X% correct to pass, then don't set your odds below X%. For a similar example consider what you bet on the final question in Jeopardy does not just depend on your estimate of your odds, but other factors. Of course even thinking about this stuff is distracting, you may be better off picking a very small number of odds. Say 99%, 90%, 70%, and 50%.
- dllthomas 10y agoRetric was distinguishing between "the highest score" and "the best result". A higher score on an exam doesn't always turn into a higher grade in the class. If your outcomes are bucketed, you may be able to increase the odds you fall in the highest bucket by decreasing your probable final score.