4 ms·

This whole concept reminds me very much of a character in an Iain M Banks novel (I forget which) whereby this character had--historically--shown an incredible k

by Osmium 12y ago

This whole concept reminds me very much of a character in an Iain M Banks novel (I forget which) whereby this character had--historically--shown an incredible knack for predicting/analysing future events, but there was always doubt whether they were just a statistical anomaly (given enough people, someone has to get it right all the time--until, suddenly, they don't) or a real effect. Interesting idea. Presumably they try and control for this.

lutusp 12y ago
> Presumably they try and control for this. How could they? They're using a retrospective design, which means they wait until someone's past forecast turns out to be correct, then they tally that as a data point in favor of their thesis. In actual science, one must use a prospective design, to avoid the perversities and statistical pitfalls of a retrospective one, and one must craft and then test a falsifiable theory -- an explanation -- about the forecasting ability. In a prospective design, one being run by people with even a tiny bit of common sense, on the first day of the study the experimenters would say, "Okay, if there were really an ability to forecast the future, anyone possessing the gift would be busy making a billion dollars in equities, and coincidentally they would refuse to talk to us." In a retrospective study, the falsifiability criterion on which science depends is easy to circumvent -- all one needs to do is tally successful predictions and discard failed ones. The originally huge experimental group will keep getting smaller, but unless everyone has made a bad prediction, the theory still has merit. Locating an astonishing record of predictions is trivial -- for a group of X participants, after Y binary predictions, there will likely remain X / 2^Y subjects with perfect prediction records. That means for a million participants making binary predictions (the stock market will rise/fall in the coming month), after a run of 16 predictions, 15 subjects are very likely to remain standing with perfect records. How the designers of this bogus study don't understand this basic statistical fact is beyond me. It's pseudoscience. If it were not pseudoscience, someone would become richer than Bill Gates or Carlos Slim by making equities calls from his penthouse overlooking the drab world occupied by those who understand logic.
- Houshalter 12y ago
  You are completely wrong. I don't know why everyone seems to think the researchers are incompetent. From [here](http://www.economist.com/news/21589145-how-sort-best-rest-whos-good-forecasts http://www.economist.com/news/21589145-how-sort-best-rest-wh...): > The big surprise has been the support for the unabashedly elitist “super-forecaster” hypothesis. The top 2% of forecasters in Year 1 showed that there is more than luck at play. If it were just luck, the “supers” would regress to the mean: yesterday’s champs would be today’s chumps. But they actually got better. When we randomly assigned “supers” into elite teams, they blew the lid off IARPA’s performance goals. They beat the unweighted average (wisdom-of-overall-crowd) by 65%; beat the best algorithms of four competitor institutions by 35-60%; and beat two prediction markets by 20-35%
  - lutusp 12y ago
    > You are completely wrong. Citation needed. There has never been a study of forecasting ability that has stood the test of time -- ever, anywhere. > I don't know why everyone seems to think the researchers are incompetent. The null hypothesis, the gold standard of statistical science, offers this as the default explanation, and the researchers have the burden of evidence to contradict it. They will fail, indeed they don't seem to realize they have the burden of evidence. Have you even considered that the positive results arise from chance? What is the p-factor for their result? Do you know what I am referring to? Your quotation doesn't have a single reliable word in it, or any content recognizable as science. If it were a scientific paper, it would contain the standard phrase, "Here is our p-factor, the statistical assessment that our result arises from chance." But it's not science, it's public relations. The thesis is assumed to be correct until incontrovertible evidence proves it false -- the opposite of the scientific outlook, which assumes an idea to be false until incontrovertible evidence supports it.
    - Houshalter 12y ago
      I gave a citation. The quote is more than enough evidence to disprove your naive "the researchers didn't bother to test for regression to mean" hypothesis. They didn't give a p-value, but the reported effect size is quite huge to be chance, and the study included thousands of people.

by Osmium 12y ago

lutusp 12y ago

> Presumably they try and control for this. How could they? They're using a retrospective design, which means they wait until someone's past forecast turns out to be correct, then they tally that as a data point in favor of their thesis. In actual science, one must use a prospective design, to avoid the perversities and statistical pitfalls of a retrospective one, and one must craft and then test a falsifiable theory -- an explanation -- about the forecasting ability. In a prospective design, one being run by people with even a tiny bit of common sense, on the first day of the study the experimenters would say, "Okay, if there were really an ability to forecast the future, anyone possessing the gift would be busy making a billion dollars in equities, and coincidentally they would refuse to talk to us." In a retrospective study, the falsifiability criterion on which science depends is easy to circumvent -- all one needs to do is tally successful predictions and discard failed ones. The originally huge experimental group will keep getting smaller, but unless everyone has made a bad prediction, the theory still has merit. Locating an astonishing record of predictions is trivial -- for a group of X participants, after Y binary predictions, there will likely remain X / 2^Y subjects with perfect prediction records. That means for a million participants making binary predictions (the stock market will rise/fall in the coming month), after a run of 16 predictions, 15 subjects are very likely to remain standing with perfect records. How the designers of this bogus study don't understand this basic statistical fact is beyond me. It's pseudoscience. If it were not pseudoscience, someone would become richer than Bill Gates or Carlos Slim by making equities calls from his penthouse overlooking the drab world occupied by those who understand logic.

Houshalter 12y ago
You are completely wrong. I don't know why everyone seems to think the researchers are incompetent. From [here](http://www.economist.com/news/21589145-how-sort-best-rest-whos-good-forecasts http://www.economist.com/news/21589145-how-sort-best-rest-wh...): > The big surprise has been the support for the unabashedly elitist “super-forecaster” hypothesis. The top 2% of forecasters in Year 1 showed that there is more than luck at play. If it were just luck, the “supers” would regress to the mean: yesterday’s champs would be today’s chumps. But they actually got better. When we randomly assigned “supers” into elite teams, they blew the lid off IARPA’s performance goals. They beat the unweighted average (wisdom-of-overall-crowd) by 65%; beat the best algorithms of four competitor institutions by 35-60%; and beat two prediction markets by 20-35%
- lutusp 12y ago
  > You are completely wrong. Citation needed. There has never been a study of forecasting ability that has stood the test of time -- ever, anywhere. > I don't know why everyone seems to think the researchers are incompetent. The null hypothesis, the gold standard of statistical science, offers this as the default explanation, and the researchers have the burden of evidence to contradict it. They will fail, indeed they don't seem to realize they have the burden of evidence. Have you even considered that the positive results arise from chance? What is the p-factor for their result? Do you know what I am referring to? Your quotation doesn't have a single reliable word in it, or any content recognizable as science. If it were a scientific paper, it would contain the standard phrase, "Here is our p-factor, the statistical assessment that our result arises from chance." But it's not science, it's public relations. The thesis is assumed to be correct until incontrovertible evidence proves it false -- the opposite of the scientific outlook, which assumes an idea to be false until incontrovertible evidence supports it.
  - Houshalter 12y ago
    I gave a citation. The quote is more than enough evidence to disprove your naive "the researchers didn't bother to test for regression to mean" hypothesis. They didn't give a p-value, but the reported effect size is quite huge to be chance, and the study included thousands of people.