4 ms·

Have you heard the one about the Emperor of China's nose? https://imaginatorium.org/stuff/nose.htm https://imaginatorium.org/stuff/nose.htm Basically, making

Have you heard the one about the Emperor of China's nose? https://imaginatorium.org/stuff/nose.htm https://imaginatorium.org/stuff/nose.htm Basically, making an average of a large number of estimates of an unknown value will (of course) fail if most/none of the estimators have any idea of the actual value being estimated.

ethbr0 3y ago
> most/none of the estimators have any idea of the actual value being estimated. A subtle distinction is who is allowed to participate in a prediction market. "Everyone with $1" is a terrible answer, and produces the bad results people are pointing to. Financial markets avoid this because of their scale, where there's enough smart money to (usually) punish stupid money. Absent that scale, it's just stupid money muddling the decisions of smart money. Prediction markets with a knowledge barrier to entry would produce better results.
- gilleain 3y ago
  How would you construct such a knowledge barrier? Another prediction market? Also, suggesting that there is such a thing as 'smart' money - presumably due to having more of it? - is amusing. As pointed out elsewhere in this discussion, there has been a lot of smart money acting particularly dumb over the last few years.
  - ethbr0 3y ago
    A "Do you understand what superconductivity is?" or "Do you have a physics or engineering degree?" barrier? And the relevant question isn't whether 'smart' money does dumb things: it's whether 'smart' money does dumb things less frequently than a random sample of money. No one is an oracle, and there are absolutely outlier events that specifically confound experts, but I can't believe that increased expertise is uncorrelated with increased accuracy.
    - kritiko 3y ago
      Tetlock’s Superforecasters performed better than experts, though: >In the Good Judgment Project, "the top forecasters... performed about 30 percent better than the average for intelligence community analysts who could read intercepts and other secret data"
tinco 3y ago
I don't think I've seen that before. But the article you linked doesn't make the conclusion you suggest at all, instead they pose a corrected value. If indeed no one estimating had no information at all, the average length of Chinese person's nose would be close to that corrected value. It's the same with this topic. You won't get an answer to the question "Is this particular paper true or not?" but you'll get an answer to the question "Are papers submitted under these circumstances making claims like this likely to be true?". The crowd will only answer the question they can answer. I think that's from "Thinking fast and slow".
- gilleain 3y ago
  It's not well explained in the version I linked (apologies, I should have looked for a clearer version). The point of the story when I originally heard it is that no one has SEEN the Emperor's nose. So any statistical function (like averaging) of estimates is totally useless as they are all guesses. No one has seen 'papers submitted under these circumstances' so no amount of 'crowd wisdom' will make any difference. Also, as an aside the idea that 'the crowd will only answer the question they can answer' is ... bizarre. People will answer anything you ask them, and you have no way to know if they are just making it up.

ethbr0 3y ago

> most/none of the estimators have any idea of the actual value being estimated. A subtle distinction is who is allowed to participate in a prediction market. "Everyone with $1" is a terrible answer, and produces the bad results people are pointing to. Financial markets avoid this because of their scale, where there's enough smart money to (usually) punish stupid money. Absent that scale, it's just stupid money muddling the decisions of smart money. Prediction markets with a knowledge barrier to entry would produce better results.

gilleain 3y ago
How would you construct such a knowledge barrier? Another prediction market? Also, suggesting that there is such a thing as 'smart' money - presumably due to having more of it? - is amusing. As pointed out elsewhere in this discussion, there has been a lot of smart money acting particularly dumb over the last few years.
- ethbr0 3y ago
  A "Do you understand what superconductivity is?" or "Do you have a physics or engineering degree?" barrier? And the relevant question isn't whether 'smart' money does dumb things: it's whether 'smart' money does dumb things less frequently than a random sample of money. No one is an oracle, and there are absolutely outlier events that specifically confound experts, but I can't believe that increased expertise is uncorrelated with increased accuracy.
  - kritiko 3y ago
    Tetlock’s Superforecasters performed better than experts, though: >In the Good Judgment Project, "the top forecasters... performed about 30 percent better than the average for intelligence community analysts who could read intercepts and other secret data"

tinco 3y ago

I don't think I've seen that before. But the article you linked doesn't make the conclusion you suggest at all, instead they pose a corrected value. If indeed no one estimating had no information at all, the average length of Chinese person's nose would be close to that corrected value. It's the same with this topic. You won't get an answer to the question "Is this particular paper true or not?" but you'll get an answer to the question "Are papers submitted under these circumstances making claims like this likely to be true?". The crowd will only answer the question they can answer. I think that's from "Thinking fast and slow".

gilleain 3y ago
It's not well explained in the version I linked (apologies, I should have looked for a clearer version). The point of the story when I originally heard it is that no one has SEEN the Emperor's nose. So any statistical function (like averaging) of estimates is totally useless as they are all guesses. No one has seen 'papers submitted under these circumstances' so no amount of 'crowd wisdom' will make any difference. Also, as an aside the idea that 'the crowd will only answer the question they can answer' is ... bizarre. People will answer anything you ask them, and you have no way to know if they are just making it up.