3 ms·

1. Find 20,000 people to participate in Coin Predicting. 2. Flip a coin. Sort out all participants that were wrong. 3. Repeat step 2 until there are 10 people

by ntaso 12y ago

1. Find 20,000 people to participate in Coin Predicting. 2. Flip a coin. Sort out all participants that were wrong. 3. Repeat step 2 until there are 10 people left. These 10 people now have a track record of predicting the future. Some of them even will continue to predict the future for a number of times. Eventually, all will be wrong. The thing is, you don't know who will be the expert forecaster in the first place and the same is true for the Good Judgement Project.

danboarder 12y ago
The Good Judgement Project does encourage forecasters to make informed predictions. In fact there is an RSS feed of relevant news stories on the topic up for trade. Your coin example is a game of pure chance while trading on current events have much more context to inform predictions. It's really quite a difference.
- lutusp 12y ago
  > Your coin example is a game of pure chance while trading on current events have much more context to inform predictions. It's really quite a difference. The only difference is that, in a coin toss, the statistics are much clearer. In real event forecasting, the perversities of language can make a failed forecast seem successful. Here's an example: A seer in the Greek city of Troy issues a warning: "Beware of Greeks bearing gifts." The citizens don't know what to make of this saying, so they proceed as before. After the now-famous Trojan Horse incident, the seer repairs her reputation by revising her earlier warning -- "I actually meant to say, 'Beware of gifts bearing Greeks'." The bottom line is that, whether flipping coins or evaluating real-world predictions, one can separate successes from failures with equal efficiency, and an equal lack of meaning.
  - danboarder 12y ago
    In a prediction marketplace there are predictions that are more successful or less successful, resulting in more or less profit for the predictor. While the result may be binary, the forecast intelligence is useful in that it contains trend data that provides better predictions than any one forecaster can provide. Further, the outcomes of many individual prediction questions are multifaceted rather than binary, with four, five or more possible outcomes.
    - lutusp 12y ago
      Yes, and without rigorous scientific controls, predictions can be made to look much, much better than they really are. The study being discussed didn't even perform a statistical analysis based on the null hypothesis or produce a p-factor -- the possibility that the result came about because of chance. These are basic to science, and in their absence, the result is anecdote, not science. http://en.wikipedia.org/wiki/P-value http://en.wikipedia.org/wiki/P-value Quote: "In statistical significance testing, the p-value is the probability of obtaining a test statistic result at least as extreme or as close to the one that was actually observed, assuming that the null hypothesis is true." Chance occurrences can be made to look very convincing. Consider the "Miracle Man" scheme used in investing -- on first hearing about it, many people assume it's not possible: http://www.arachnoid.com/equities_myths/index.html#Miracle_Man http://www.arachnoid.com/equities_myths/index.html#Miracle_M...
davidw 12y ago
> The thing is, you don't know who will be the expert forecaster in the first place and the same is true for the Good Judgement Project. They should turn it on itself - make anonymous profiles of the participants, and see if they can predict who will make a good forecaster!
Houshalter 12y ago
I don't know why everyone seems to think the researchers are incompetent. From here (http://www.economist.com/news/21589145-how-sort-best-rest-whos-good-forecasts http://www.economist.com/news/21589145-how-sort-best-rest-wh...): > The big surprise has been the support for the unabashedly elitist “super-forecaster” hypothesis. The top 2% of forecasters in Year 1 showed that there is more than luck at play. If it were just luck, the “supers” would regress to the mean: yesterday’s champs would be today’s chumps. But they actually got better. When we randomly assigned “supers” into elite teams, they blew the lid off IARPA’s performance goals. They beat the unweighted average (wisdom-of-overall-crowd) by 65%; beat the best algorithms of four competitor institutions by 35-60%; and beat two prediction markets by 20-35%
onion2k 12y ago
Given that it's a random event, a good coin flip forecaster would be predicting a 0.5 probability for each side rather than a 1 for either heads or tails. Consequently, of the 20,000 coin flippers, you can tell who'll be a good predictor before a single coin has been thrown - the good ones are the ones who think it through and are open-minded enough to say their prediction is "I don't know." If you insist on a heads or a tails then your experiment is broken by design and tells you nothing.
sgdesign 12y ago
I don't know if your assessment of the study is valid (it seems a little simplistic to think that they wouldn't account for this to be honest), but what you describe is classic Survivorship Bias. Here's an entertaining video from Derren Brown that shows how to use this technique to convince people you have psychic powers: https://www.youtube.com/watch?v=9R5OWh7luL4 https://www.youtube.com/watch?v=9R5OWh7luL4 Edit: also see the comment about the Miracle Man scam in this thread.

by ntaso 12y ago

danboarder 12y ago

The Good Judgement Project does encourage forecasters to make informed predictions. In fact there is an RSS feed of relevant news stories on the topic up for trade. Your coin example is a game of pure chance while trading on current events have much more context to inform predictions. It's really quite a difference.

lutusp 12y ago
> Your coin example is a game of pure chance while trading on current events have much more context to inform predictions. It's really quite a difference. The only difference is that, in a coin toss, the statistics are much clearer. In real event forecasting, the perversities of language can make a failed forecast seem successful. Here's an example: A seer in the Greek city of Troy issues a warning: "Beware of Greeks bearing gifts." The citizens don't know what to make of this saying, so they proceed as before. After the now-famous Trojan Horse incident, the seer repairs her reputation by revising her earlier warning -- "I actually meant to say, 'Beware of gifts bearing Greeks'." The bottom line is that, whether flipping coins or evaluating real-world predictions, one can separate successes from failures with equal efficiency, and an equal lack of meaning.
- danboarder 12y ago
  In a prediction marketplace there are predictions that are more successful or less successful, resulting in more or less profit for the predictor. While the result may be binary, the forecast intelligence is useful in that it contains trend data that provides better predictions than any one forecaster can provide. Further, the outcomes of many individual prediction questions are multifaceted rather than binary, with four, five or more possible outcomes.
  - lutusp 12y ago
    Yes, and without rigorous scientific controls, predictions can be made to look much, much better than they really are. The study being discussed didn't even perform a statistical analysis based on the null hypothesis or produce a p-factor -- the possibility that the result came about because of chance. These are basic to science, and in their absence, the result is anecdote, not science. http://en.wikipedia.org/wiki/P-value http://en.wikipedia.org/wiki/P-value Quote: "In statistical significance testing, the p-value is the probability of obtaining a test statistic result at least as extreme or as close to the one that was actually observed, assuming that the null hypothesis is true." Chance occurrences can be made to look very convincing. Consider the "Miracle Man" scheme used in investing -- on first hearing about it, many people assume it's not possible: http://www.arachnoid.com/equities_myths/index.html#Miracle_Man http://www.arachnoid.com/equities_myths/index.html#Miracle_M...

davidw 12y ago

> The thing is, you don't know who will be the expert forecaster in the first place and the same is true for the Good Judgement Project. They should turn it on itself - make anonymous profiles of the participants, and see if they can predict who will make a good forecaster!

Houshalter 12y ago

I don't know why everyone seems to think the researchers are incompetent. From here (http://www.economist.com/news/21589145-how-sort-best-rest-whos-good-forecasts http://www.economist.com/news/21589145-how-sort-best-rest-wh...): > The big surprise has been the support for the unabashedly elitist “super-forecaster” hypothesis. The top 2% of forecasters in Year 1 showed that there is more than luck at play. If it were just luck, the “supers” would regress to the mean: yesterday’s champs would be today’s chumps. But they actually got better. When we randomly assigned “supers” into elite teams, they blew the lid off IARPA’s performance goals. They beat the unweighted average (wisdom-of-overall-crowd) by 65%; beat the best algorithms of four competitor institutions by 35-60%; and beat two prediction markets by 20-35%

onion2k 12y ago

Given that it's a random event, a good coin flip forecaster would be predicting a 0.5 probability for each side rather than a 1 for either heads or tails. Consequently, of the 20,000 coin flippers, you can tell who'll be a good predictor before a single coin has been thrown - the good ones are the ones who think it through and are open-minded enough to say their prediction is "I don't know." If you insist on a heads or a tails then your experiment is broken by design and tells you nothing.

sgdesign 12y ago

I don't know if your assessment of the study is valid (it seems a little simplistic to think that they wouldn't account for this to be honest), but what you describe is classic Survivorship Bias. Here's an entertaining video from Derren Brown that shows how to use this technique to convince people you have psychic powers: https://www.youtube.com/watch?v=9R5OWh7luL4 https://www.youtube.com/watch?v=9R5OWh7luL4 Edit: also see the comment about the Miracle Man scam in this thread.