4 ms·
Probability doesn't work like that. Likelihood has an upper bound at 1, and log likelihood has an upper bound at 0. Likelihood just represents the probability y
by Houshalter 10y ago
Probability doesn't work like that. Likelihood has an upper bound at 1, and log likelihood has an upper bound at 0. Likelihood just represents the probability you assigned to the actual outcome. You can't have a probability higher than 1.
- mikeash 10y agoYou can't have a probability lower than zero, but you're still able to obtain negative infinity points. Why is there an upper bound but not a lower bound?
- Jabbles 10y agoBecause it takes a lot of skill to be "always wrong" (you would have to know the right answer in order to answer incorrectly). No skill is represented by getting 50% right/wrong, and indeed it scores you 0 points.
- scott_s 10y agoI don't think that answers mikeash's question. He is not talking about always being wrong. He is pointing out that the incentives are not symmetric. If you say that you are 100% certain, and you are wrong, then you fail the class. If the incentives were symmetric, if you said you were 100% certain and you were correct, then you would automatically pass the class. A potential answer is: this asymmetry is similar to incentives outside the classroom. Claiming 100% certainty and being wrong can be disastrous to your reputation, and likely much more negative than the positive benefits of claiming 100% certainty and being correct.
- mikeash 10y agoGetting 50% right/wrong while expressing 100% confidence in your answers scores you negative infinity points. Indeed, getting 99.9% right/wrong while expressing 100% confidence scores you negative infinity points. But getting 100% right with 100% confidence only scores you a finite positive number of points.
- Houshalter 10y agoThat's just the way log probabilities work. Log(0) is negative infinity, but log(1) is just 0. You can have infinitely high scores if you use log odds though: http://lesswrong.com/lw/mp/0_and_1_are_not_probabilities/ http://lesswrong.com/lw/mp/0_and_1_are_not_probabilities/ But that can only happen if you get every question 100% right. Whereas getting a single question 100% wrong is easy to do.
- mikeash 10y agoWhy is log probability the correct way to evaluate this?
- Houshalter 10y agoIt's convenient, because multiplying probabilities together quickly gets very very small. Typically smaller than can be represented in floating point even. Whereas logarithms make multiplication just addition, and they get small very very slowly. Personally I prefer geometric mean. Which is equivalent to the average log likelihood. It has an upper bound at 1 and lower bound at 0, and represents the "average" probability you assigned the correct answer.
- ryanmonroe 10y agoI think I read a blog post about that somewehre.
- scott_s 10y agomikeash was not advocating for a probability higher than 1. Rather, he was advocating for an infinite reward for indicating a probability of 1. We could reframe his suggestion to be, why are the incentives not symmetric? (I think there are good reasons for this, but it's a valid question.)
- Houshalter 10y agoI was just saying that's the way the math works out. It's not an arbitrary decision, it just happens that the natural way to score probabilities allows negative infinity from a single question. You could make an arbitrary scoring metric that does whatever you want. But it wouldn't be principled, or have nice mathematical properties like this.
- Retric 10y agoThe math was chosen arbitrarily so every outcome is also arbitrary.