3 ms·
Right, I think the general point is that log-loss is a bit unintuitive as a scoring mechanism since it really penalizes overconfident wrong answers, much more s
by theptip 3y ago
Right, I think the general point is that log-loss is a bit unintuitive as a scoring mechanism since it really penalizes overconfident wrong answers, much more strongly than underconfident right answers (at least in the domain where you are not asking questions with very high probability answers).
There is absolutely a metagame to this game. Those who have spent time forecasting on Metaculus will do much better, for example.
- H8crilA 3y agoThe sad thing about it is that log loss is optimal when gambling (mental shortcut - see Kelly Criterion). This partly explains why such a large proportion of people lose much more than they win. Not only the bets are suboptimal, the ruin is achieved earlier than necessary. Note that this applies also outside of a casino, gambling (making wagers with imperfect information) is inherent to life.
- mif 3y ago+1 for the Metaculus [1] reference. I came across the concept of calibration while reading “How to measure anything” (see the _equivalent bet test_ in this review [2]) and find it really powerful in theory and really hard in practice. [1] https://www.metaculus.com/home/ https://www.metaculus.com/home/ [2] https://www.lesswrong.com/posts/ybYBCK9D7MZCcdArB/how-to-measure-anything https://www.lesswrong.com/posts/ybYBCK9D7MZCcdArB/how-to-mea...