6 ms·

In my experience I’ve never found an instance where you would use Brier scores over cross entropy/Bernoulli/Binomial log likelihoods. Does anybody know a concre

In my experience I’ve never found an instance where you would use Brier scores over cross entropy/Bernoulli/Binomial log likelihoods. Does anybody know a concrete example when you would prefer Brier??

closed 5y ago
The article mentions Brier score is just mean squared error, so it's connected to binomial through that (e.g. where correct prediction is 1, incorrect is 0, it is the mean of the binomial).
tlb 5y ago
It's appropriate when the cost of false-positive and false-negative errors are the same. Which isn't common in the real world.
- kqr 5y ago
  More generally, if one views probability as separate from the utility of the outcome it's attached to, one is bound to make bad decisions. Real decision problems contain a lot of nonlinearities if decomposed the wrong way. The only way to decompose it is as a linear combination of probability and utility (because the utility swallows the nonlinearities). But for each component both probability and utility matters in determining the overall value of the decision.
datarecipes 5y ago
Both the Brier score and log loss are proper scoring rules (i.e. optimized when the predicted probabilities are the true outcome probabilities), and the choice between the two seems to have minimal impact on the conclusions that can be drawn (https://pubsonline.informs.org/doi/abs/10.1287/deca.2013.0280 https://pubsonline.informs.org/doi/abs/10.1287/deca.2013.028...). I covered the Brier score in the post as I thought it would be easier to digest for a general audience. As Frank Harrell wrote on his blog (https://www.fharrell.com/post/class-damage/ https://www.fharrell.com/post/class-damage/), one advantage of the Brier score could be its interpretability and the ability to break it decompose it into discrimination and calibration components.
- srean 5y ago
  Indeed. Note though that proper scoring rules form a large class and it can matter which one you choose. For example, for logistic regression, things become a lot simpler if one chooses log loss (equivalently KL divergence) because one ends up with a convex minimization problem. Had one chosen Brier score here the problem is no longer convex and where one starts the training iteration will determine where the updates converge to. Sometimes this indeterminacy is a problem -- am getting poor results, is it because the data has changed, or is it that my initial seed has changed and the udates have converged to a worse solution.

closed 5y ago

The article mentions Brier score is just mean squared error, so it's connected to binomial through that (e.g. where correct prediction is 1, incorrect is 0, it is the mean of the binomial).

tlb 5y ago

It's appropriate when the cost of false-positive and false-negative errors are the same. Which isn't common in the real world.

kqr 5y ago
More generally, if one views probability as separate from the utility of the outcome it's attached to, one is bound to make bad decisions. Real decision problems contain a lot of nonlinearities if decomposed the wrong way. The only way to decompose it is as a linear combination of probability and utility (because the utility swallows the nonlinearities). But for each component both probability and utility matters in determining the overall value of the decision.

datarecipes 5y ago

Both the Brier score and log loss are proper scoring rules (i.e. optimized when the predicted probabilities are the true outcome probabilities), and the choice between the two seems to have minimal impact on the conclusions that can be drawn (https://pubsonline.informs.org/doi/abs/10.1287/deca.2013.0280 https://pubsonline.informs.org/doi/abs/10.1287/deca.2013.028...). I covered the Brier score in the post as I thought it would be easier to digest for a general audience. As Frank Harrell wrote on his blog (https://www.fharrell.com/post/class-damage/ https://www.fharrell.com/post/class-damage/), one advantage of the Brier score could be its interpretability and the ability to break it decompose it into discrimination and calibration components.

srean 5y ago
Indeed. Note though that proper scoring rules form a large class and it can matter which one you choose. For example, for logistic regression, things become a lot simpler if one chooses log loss (equivalently KL divergence) because one ends up with a convex minimization problem. Had one chosen Brier score here the problem is no longer convex and where one starts the training iteration will determine where the updates converge to. Sometimes this indeterminacy is a problem -- am getting poor results, is it because the data has changed, or is it that my initial seed has changed and the udates have converged to a worse solution.