4 ms·

Just my take, but I think that if you care how you scored or what the scoring criteria are you are missing the point. This "quiz" is just a guided meditation on

Just my take, but I think that if you care how you scored or what the scoring criteria are you are missing the point. This "quiz" is just a guided meditation on what LLMs are better and worse at and how that interacts with our expectations. I found it to be very thought-provoking and I learned a few things. I have no idea how I "scored".

H8crilA 3y ago
It's also a good reminder of how absolutely terrible most people are at gambling (judged by log-loss, like in the Kelly criterion). At the end you find out that almost 80% of people did worse than leaving the guess at 50%.
- theptip 3y ago
  Right, I think the general point is that log-loss is a bit unintuitive as a scoring mechanism since it really penalizes overconfident wrong answers, much more strongly than underconfident right answers (at least in the domain where you are not asking questions with very high probability answers). There is absolutely a metagame to this game. Those who have spent time forecasting on Metaculus will do much better, for example.
  - H8crilA 3y ago
    The sad thing about it is that log loss is optimal when gambling (mental shortcut - see Kelly Criterion). This partly explains why such a large proportion of people lose much more than they win. Not only the bets are suboptimal, the ruin is achieved earlier than necessary. Note that this applies also outside of a casino, gambling (making wagers with imperfect information) is inherent to life.
  - mif 3y ago
    +1 for the Metaculus [1] reference. I came across the concept of calibration while reading “How to measure anything” (see the _equivalent bet test_ in this review [2]) and find it really powerful in theory and really hard in practice. [1] https://www.metaculus.com/home/ https://www.metaculus.com/home/ [2] https://www.lesswrong.com/posts/ybYBCK9D7MZCcdArB/how-to-measure-anything https://www.lesswrong.com/posts/ybYBCK9D7MZCcdArB/how-to-mea...

H8crilA 3y ago

It's also a good reminder of how absolutely terrible most people are at gambling (judged by log-loss, like in the Kelly criterion). At the end you find out that almost 80% of people did worse than leaving the guess at 50%.

theptip 3y ago
Right, I think the general point is that log-loss is a bit unintuitive as a scoring mechanism since it really penalizes overconfident wrong answers, much more strongly than underconfident right answers (at least in the domain where you are not asking questions with very high probability answers). There is absolutely a metagame to this game. Those who have spent time forecasting on Metaculus will do much better, for example.
- H8crilA 3y ago
  The sad thing about it is that log loss is optimal when gambling (mental shortcut - see Kelly Criterion). This partly explains why such a large proportion of people lose much more than they win. Not only the bets are suboptimal, the ruin is achieved earlier than necessary. Note that this applies also outside of a casino, gambling (making wagers with imperfect information) is inherent to life.
- mif 3y ago
  +1 for the Metaculus [1] reference. I came across the concept of calibration while reading “How to measure anything” (see the _equivalent bet test_ in this review [2]) and find it really powerful in theory and really hard in practice. [1] https://www.metaculus.com/home/ https://www.metaculus.com/home/ [2] https://www.lesswrong.com/posts/ybYBCK9D7MZCcdArB/how-to-measure-anything https://www.lesswrong.com/posts/ybYBCK9D7MZCcdArB/how-to-mea...