4 ms·

This curve is not enough to evaluate the value of a weather forecasting service: if it rains, say 30% of the days in a specific area, you could forecast a 30% c

This curve is not enough to evaluate the value of a weather forecasting service: if it rains, say 30% of the days in a specific area, you could forecast a 30% chance of rain every day and have good "accuracy". And yet that would be of no practical value. I think a better metric would probably be something from information theory like mutual information, but I'm not sure which one exactly.

rhiever 12y ago
I think you're right, and in that sense, maybe "accuracy" isn't the best word. "Reliability" might be the better word here.
ivank 12y ago
If someone just predicted 30% every day, you'd notice the lack of samples for any other predicted probability, and no one would judge such a discontinuous calibration graph as well-calibrated. Calibration graphs can still be gamed - you can predict dishonestly to "dent" the graph. For example, if your 70% is happening 60% of the time, predict 70% even if you're 100% sure an event will happen.
bo1024 12y ago
Right. The curve demonstrates "calibration". A separate question is daily accuracy, or what might be called "sharpness". One way to measure accuracy is to use a "scoring rule" each day. The scoring rule takes the prediction (a probability of rain), e.g. p=0.3, and the actual outcome, e.g. "rain" or "no rain", and returns a numerical score, where higher is better. You'd expect a reasonable scoring rule to have properties like, the highest score is for predicting p=1 and it rains (or p=0 and it doesn't), the lowest score is for p=1 and it doesn't (or p=0 and it does), and so on. The log scoring rule is a popular choice: If it rains, your score is log(p); if it doesn't, your score is log(1-p). (Don't let it bother you that these are negative numbers for p in [0,1]. Closer to zero is still better.) Notice that your expected score for predicting p is precisely plog(p) + (1-p)log(1-p), which is the negative of the entropy of the distribution, that is, -H(Bernoulli(p)). Now, what we could do to measure accuracy is to average, over all the days, the log scoring rule applied to your prediction and the outcome. A perfect score would be if you predicted 1 every time it rained and 0 every time it didn't, which would give you a score of zero; if you're not perfect, you'll have a negative score, with more negative being worse. So the average score will be (the negative of) the average entropy of your predictions, which makes sense as entropy corresponds to "uncertainty". (This is assuming your predictions are calibrated. If you are uncalibrated, i.e. you make predictions with little uncertainty but they are wrong, then you will pay a penalty in proportion.) We can connect this to your idea of mutual information, if we had a more detailed setup. For instance, suppose that each day, nature picks a true probability of rain p', then you observe some information related to p' and make a prediction p; then it rains with probability p'. Here, the best you could possibly do is to observe all the information and predict p'. Then your average score will be the average entropy H(Bernoulli(p')). But if you are making poor predictions, then the theory of proper scoring rules tells us that your average score will be the average mutual information between the distributions Bernoulli(p') and Bernoulli(p). (If we are using some scoring rule other than the log scoring rule, this mutual information, which comes from KL-divergence, is replaced with a "Bregman divergence".) Unfortunately, we don't know this true p' each day, so we can't actually use mutual information to evaluate forecasts, but we can still use the entropy, or log scoring rule. For a survey on proper scoring rules (not very fun for the layperson I'm afraid), see http://www.eecs.harvard.edu/cs286r/courses/fall12/papers/Gneiting07.pdf http://www.eecs.harvard.edu/cs286r/courses/fall12/papers/Gne...
jmount 12y ago
Deviance or log-liklihood are good measures (and when properly normalized give you one of the cross-entropies). Roughly you do -sum(ifelse(rained_i,log(p_i),log(1-p_i)). So you git a high (bad) penalty score if you get wrong probabilities and a nice score of 0 if you put 1's on days that rained and 0's elsewhere.
baddox 12y ago
> if it rains, say 30% of the days in a specific area, you could forecast a 30% chance of rain every day and have good "accuracy". And yet that would be of no practical value. That's not exactly true. There is practical value to knowing the long-term probability of rain. Assuming that carrying an umbrella itself imposes a "cost" (such that it's not acceptable to simply carry one every single day), you're statistically better off randomly carrying an umbrella 30% of the time than carrying an umbrella any other percent of the time (including 0 or 100).

rhiever 12y ago

I think you're right, and in that sense, maybe "accuracy" isn't the best word. "Reliability" might be the better word here.

ivank 12y ago

If someone just predicted 30% every day, you'd notice the lack of samples for any other predicted probability, and no one would judge such a discontinuous calibration graph as well-calibrated. Calibration graphs can still be gamed - you can predict dishonestly to "dent" the graph. For example, if your 70% is happening 60% of the time, predict 70% even if you're 100% sure an event will happen.

bo1024 12y ago

Right. The curve demonstrates "calibration". A separate question is daily accuracy, or what might be called "sharpness". One way to measure accuracy is to use a "scoring rule" each day. The scoring rule takes the prediction (a probability of rain), e.g. p=0.3, and the actual outcome, e.g. "rain" or "no rain", and returns a numerical score, where higher is better. You'd expect a reasonable scoring rule to have properties like, the highest score is for predicting p=1 and it rains (or p=0 and it doesn't), the lowest score is for p=1 and it doesn't (or p=0 and it does), and so on. The log scoring rule is a popular choice: If it rains, your score is log(p); if it doesn't, your score is log(1-p). (Don't let it bother you that these are negative numbers for p in [0,1]. Closer to zero is still better.) Notice that your expected score for predicting p is precisely plog(p) + (1-p)log(1-p), which is the negative of the entropy of the distribution, that is, -H(Bernoulli(p)). Now, what we could do to measure accuracy is to average, over all the days, the log scoring rule applied to your prediction and the outcome. A perfect score would be if you predicted 1 every time it rained and 0 every time it didn't, which would give you a score of zero; if you're not perfect, you'll have a negative score, with more negative being worse. So the average score will be (the negative of) the average entropy of your predictions, which makes sense as entropy corresponds to "uncertainty". (This is assuming your predictions are calibrated. If you are uncalibrated, i.e. you make predictions with little uncertainty but they are wrong, then you will pay a penalty in proportion.) We can connect this to your idea of mutual information, if we had a more detailed setup. For instance, suppose that each day, nature picks a true probability of rain p', then you observe some information related to p' and make a prediction p; then it rains with probability p'. Here, the best you could possibly do is to observe all the information and predict p'. Then your average score will be the average entropy H(Bernoulli(p')). But if you are making poor predictions, then the theory of proper scoring rules tells us that your average score will be the average mutual information between the distributions Bernoulli(p') and Bernoulli(p). (If we are using some scoring rule other than the log scoring rule, this mutual information, which comes from KL-divergence, is replaced with a "Bregman divergence".) Unfortunately, we don't know this true p' each day, so we can't actually use mutual information to evaluate forecasts, but we can still use the entropy, or log scoring rule. For a survey on proper scoring rules (not very fun for the layperson I'm afraid), see http://www.eecs.harvard.edu/cs286r/courses/fall12/papers/Gneiting07.pdf http://www.eecs.harvard.edu/cs286r/courses/fall12/papers/Gne...

jmount 12y ago

Deviance or log-liklihood are good measures (and when properly normalized give you one of the cross-entropies). Roughly you do -sum(ifelse(rained_i,log(p_i),log(1-p_i)). So you git a high (bad) penalty score if you get wrong probabilities and a nice score of 0 if you put 1's on days that rained and 0's elsewhere.

baddox 12y ago

> if it rains, say 30% of the days in a specific area, you could forecast a 30% chance of rain every day and have good "accuracy". And yet that would be of no practical value. That's not exactly true. There is practical value to knowing the long-term probability of rain. Assuming that carrying an umbrella itself imposes a "cost" (such that it's not acceptable to simply carry one every single day), you're statistically better off randomly carrying an umbrella 30% of the time than carrying an umbrella any other percent of the time (including 0 or 100).