I'm working on something like this right now. We asked 8 trivia questions on a survey of users of our website, and had people assign probability estimates to th

I'm working on something like this right now. We asked 8 trivia questions on a survey of users of our website, and had people assign probability estimates to the probability they got the answer right. First of all it seems like everyone had exactly the same probability of getting any random question right. Some people got every question right, and some got none right. But in exactly the proportion you would expect by random chance - that some were just particularly lucky or unlucky. The second is that probability estimates did not vary much based on questions gotten right. Everyone expected to get about 44% of questions right, regardless how many questions they actually got right. People who only got 1 right assigned the same probability as people who got 5 right. Likewise people who estimated higher probabilities of getting more questions right, got the same amount of questions right. And a decent percent of people were underconfident too, and assigned probabilities too low (but got the same amount of questions right.) Lastly, people are really uncalibrated. Some people are just bad at estimating probability. When they say "80% chance of something" they mean that thing will only happen 58% of the time. You can be trained, in a relatively short time, to become calibrated. By estimating probabilities and seeing how many you actually got right. But most people aren't trained, so it would be a bit unfair to put this on a real test.