2 ms·
Wait, so roughly is it rewarding being confident when correct, and penalizing being confident when wrong? Meaning that the highest score is only achievable if y
by gcanyon 7mo ago
Wait, so roughly is it rewarding being confident when correct, and penalizing being confident when wrong? Meaning that the highest score is only achievable if you answer fully confident true or false, and get all 10 correct?
If so, isn't that conflating knowledge with over/under confidence?
- convexly 7mo agoYour point on scoring is correct, if you're 100% confident and right on everything you would score a perfect 0. The calibration insight is in how you handle the questions where you don't know the answer. Say you're highly knowledgeable and 95% confident on everything, but get 2 wrong scores compared to someone that says they are 70% confident on those same two questions. That would indicate that you are overconfident compared to the other person!
- gcanyon 7mo agoI think the 100% certain and always right scenario invalidates the calculation. In that outcome you know nothing about my (over) confidence level when I am wrong. You should either return NA in that circumstance, or keep asking questions until you have actual data to work with.
- macleginn 7mo agoHow are they different? If you "know" something, you are 100% confident in it, which gives you an easy 0 for this question (or a surprising 1). Philosophically, the problem is more that there is no difference between confidently and modestly wrong in terms of consequences of binary decisions.