3 ms·

If 10,000 people guess 10,000 fair coin flips each one of them will get more guesses right than any of the others, one of them will get fewer guesses right than

If 10,000 people guess 10,000 fair coin flips each one of them will get more guesses right than any of the others, one of them will get fewer guesses right than any of the others, and the gulf between the two is likely to be over 4 standard deviations wide. I'm certain that I, being an untutored schmuck from Pittsburgh and having thought of this almost immediately after reading about this contest, cannot be the first person to realize this is a potential problem for a forecasting contest. But I can't find anything they've done to mitigate that problem. Can anyone clue me in?

cman1444 15d ago
I don't understand your analogy. Are you just suggesting that luck plays too large a role in this contest? Clearly there is some "skill" or ability factor because AI's have been scoring higher and higher each year. Also, they make reference to superforecaster humans, who are presumably consistently better at forecasting than their peers.
- ratelimitsteve 14d ago
  i'm not looking at the guesses, i'm looking at the distribution of guesser success rates. assuming random distribution, counterintuitively enough, you would expect some guessers to appear much better or much worse than others. i know i'm not the first person to think of this, so i'm asking what's been done to mitigate it because i can't find anything. if you expect x% of guesses to be within two std devs of the mean that means you can expect 100-x% to be outside that, even without any guessers actually being better at guessing than any of the others.
  - chumzygood 13d ago
    [flagged]
adleyjulian 15d ago
They aren't guessing heads or tails, they give odds for each event. It's more like eyeballing a thousand coins to guess how fair they are, and then flipping each one just once. Some are weighted to be 99% heads, others are 10% heads etc. You could have 1,000,000 people guess random percentages for each coin, but suppose 10 of the coins are weighted 100% heads. To guess within 25% of the true value for all 10 of those coins would be roughly 1 in a million. So a lucky guy guesses within 25% for all 10, he'd have another 990 coins he's being judged on.

cman1444 15d ago

I don't understand your analogy. Are you just suggesting that luck plays too large a role in this contest? Clearly there is some "skill" or ability factor because AI's have been scoring higher and higher each year. Also, they make reference to superforecaster humans, who are presumably consistently better at forecasting than their peers.

ratelimitsteve 14d ago
i'm not looking at the guesses, i'm looking at the distribution of guesser success rates. assuming random distribution, counterintuitively enough, you would expect some guessers to appear much better or much worse than others. i know i'm not the first person to think of this, so i'm asking what's been done to mitigate it because i can't find anything. if you expect x% of guesses to be within two std devs of the mean that means you can expect 100-x% to be outside that, even without any guessers actually being better at guessing than any of the others.
- chumzygood 13d ago
  [flagged]

adleyjulian 15d ago

They aren't guessing heads or tails, they give odds for each event. It's more like eyeballing a thousand coins to guess how fair they are, and then flipping each one just once. Some are weighted to be 99% heads, others are 10% heads etc. You could have 1,000,000 people guess random percentages for each coin, but suppose 10 of the coins are weighted 100% heads. To guess within 25% of the true value for all 10 of those coins would be roughly 1 in a million. So a lucky guy guesses within 25% for all 10, he'd have another 990 coins he's being judged on.