6 ms·

I need to know more. If the questions are all yes/no questions ("Will there be a significant attack on Israeli territory before May 10, 2014?"), and the sample

I need to know more. If the questions are all yes/no questions ("Will there be a significant attack on Israeli territory before May 10, 2014?"), and the sample set is large enough, you would -expect- there to be some outliers who mostly got things right, by pure chance. And even outside of that, I want to know what sort of average the CIA agents were batting; if they were hitting just 50%, I would -expect- nearly any sample size to have an outlier who did better. That is, to borrow the old get rich quick scheme, take a list of 1024 addresses. Send to half of them "Stock X will be higher on (date) than today", and to the other half "Stock X will be lower on (date) than today". Repeat with the set you sent the correct 'predictions' to, and continue repeating until you're down to four people who you've sent the correct prediction to every time. Suggest they sign up for your premium stock tips newsletter. All you're doing is playing statistics, you're not actually demonstrating any predictive ability.

joosters 13y ago
Agreed. How do we know that the top 1% predictors are there through skill rather than luck? Knowing more about the questions, numbers etc would help us to come to a good idea about how many people should make these predictions by chance alone, and so on.
- chiph 13y ago
  Or, she could be a "Teela Brown" -- someone who is naturally (or unnaturally...) lucky. http://en.wikipedia.org/wiki/Teela_Brown http://en.wikipedia.org/wiki/Teela_Brown
  - Noelkd 13y ago
    I don't see why this is being down voted as someone who has read the Ringworld novels I couldn't help but think of Teela Brown and her luck.
    - anigbrowl 13y ago
      Because chiph is offering a fictional explanation for a real world phenomenon. There have been lots of scientific studies involving randomness in fields as diverse as economics, game theory, gambling behavior, and parapsychology; if 'luckiness' was a measurable characteristic of people then you'd expect it to reliably show up in such contexts. Invoking it as an explanation where a simpler alternative exists (the idea that some people are more skilled at analyzing current affairs) is illogical.
    - malka 13y ago
      I cant downvote, but i expected the link to point to someone real, not a fictional character.
      - chiph 13y ago
        Ok, here you go. http://www.cnn.com/2014/04/01/us/powerball-lottery-winner/index.html?iref=allsearch http://www.cnn.com/2014/04/01/us/powerball-lottery-winner/in...
        malka 13y ago
        And on the opposite side of the spectrum : http://en.wikipedia.org/wiki/Roy_Sullivan http://en.wikipedia.org/wiki/Roy_Sullivan
jerf 13y ago
"and the sample set is large enough, you would -expect- there to be some outliers who mostly got things right, by pure chance." According to the article, "In fact, she's so good she's been put on a special team with other super forecasters whose predictions are reportedly 30 percent better than intelligence officers with access to actual classified information." If they mean that after the team was formed, the predictions are still 30% better, than random chance ceases to be an acceptable explanation. If they mean that the team that they formed was 30% better on the predictions used to grant entrance into the team, then your objection still holds. Unfortunately, I don't think the article makes it clear which is the case. That said, there are definitely some mechanisms that could lead an outside group to higher accuracy on these sorts of things. The two biggest are: 1. The insiders are simply that bad, for whatever reason, and random chance can easily beat them, and 2. The failures of the insiders are correlated and overwhelm their theoretically superior information, and I'd especially finger potential ideological biases there, which could affect both information filtering and the analysis. (It is one thing to believe the world should be a certain way as a result of one's ideology... it is quite another to believe the world must already be a certain way as a result of one's ideology, even when one is staring evidence in the face that it is not. One is noble and laudable... the other somewhat less so.)
- JoeAltmaier 13y ago
  Its not clear they continue to be 30% better - just that they started out that way, and got put on a team. I'm still not convinced these folks aren't just outliers. People in all walks of life give far too much credence to lucky outliers - CEOs are hired that way, ball players, stock portfolios, sports predictors. If you make enough predictions and cherry-pick the best, you can always find a 'miracle man'.
  - jerf 13y ago
    "Its not clear they continue to be 30% better" Yes, that was my point. The article failed to make it clear. It is worded such that it could go either way. I agree they may be outliers. However, I can't dismiss out-of-hand the possibility that they are actually better. Outliers are common... but so are systematic/correlated weaknesses within large organizations.
edgarallenbro 13y ago
>was given access to a website that listed dozens of carefully worded questions on events of interest to the intelligence community, along with a place for her to enter her numerical estimate of their likelihood. If you need to know more, try reading the article? It says that they're not yes/no questions.
- waterlesscloud 13y ago
  I'm participating in this project, as are at least a couple other HNers (surprising to no one, huh?). I won't go into details about it because it's not my place to do so at this point (maybe after). But I will confirm that for every response you give, you're required to enter a percentage estimate of likelihood. For example, you'd enter 90% or 72.212% or whatever on whichever question you're responding to. So there's a potential mechanism for further ranking of participants beyond the binary. The voting mechanism itself is more complicated, but again, I'll leave discussion for when it's over.
  - notahacker 13y ago
    The interesting thing about this is it raises the [theoretical, in the absence of information about weighting/ranking systems] possibility professional intelligence analysts' relative underperformance against the measure is less due to inaccurately identifying high/low probability more frequently than amateur "superforecasters", and more due to professionals making the systematic error of overconfidence in the evidence they have when weighting their estimates - e.g. if the amateurs are a lot more likely to either pick 50% on events where there genuinely isn't enough information to forecast and less likely to assign single digit probabilities to events which no available evidence suggest are likely which nevertheless happen. In other words, it sounds highly plausible that if you asked simple binary questions about expected outcomes both groups would give almost identical answers and usually be correct, but the professionals are more confident when both groups are wrong. If this is the case then it's reasonable to assume CIA's statisticians would have done the analysis and know that's the reason these "superforecasters" are better: doubt I guess the reverse is also possible: professional intelligence analysts are systematically tending towards being overcautious and tend to pick numbers towards the middle of the range, either out of a desire not to look silly or because they're more aware of policy implications. But subjectively I'd assign that a lower probability.
Houshalter 13y ago
I'm assuming their smart enough to know that and that the article was just glossing over the details.
nzp 13y ago
In general these are valid remarks. But we're not talking about random clueless people deciding to play statistics, the research is obviously conducted by people who know how this kind of stuff (statistics) works. This reminds me of laypeople trying to poke holes in theories of physics (and I'm not talking about quacks and lunatics, just well intentioned people who think they have a better explanation based on rudimentary high school or even undergraduate science and math). It's such a charming, cute naivete. Do they really think their simple ideas never occurred to a single professional physicist who immediately saw it was wrong for a number of reasons? So, do you really think the researchers in this case hadn't thought of this? I mean yeah, it's OK to doubt and inquire, but come on, this is so rudimentary... > If the questions are all yes/no questions The article says (or at least implies) that the subjects give probabilities of "yes" to each question. EDIT: Oh, and to add, the article clearly says they're not just making shit up and randomly coming up with probabilities. They collect open source info on events and base their estimates on that. So it's not just blind guessing.
- lostcolony 13y ago
  As to the researchers probably know what they're doing, see my original comment and the very first thing I said, "I need to know more". It's not charming cute naiveté, I simply really think that the article tells me nothing about the methodology. See response here as to why they're still yes/no questions (TLDR; the outcomes are yes/no) - https://news.ycombinator.com/item?id=7515824 https://news.ycombinator.com/item?id=7515824
  - nzp 13y ago
    Well, I didn't mean your comment was a case of naivete, because, TBH, the analogy with my layman physics example isn't perfect, but it reminded me, it's a similar thing: people who are or should be aware that they have only superficial info on something, act as though what they know is all there is to it. The point is, we can't possibly come up with a meaningful critique of this research based on the given article, and, I think, the most reasonable course of action in this particular case is to assume the researchers know what they're doing (considering who's involved).
- 13y ago

joosters 13y ago

Agreed. How do we know that the top 1% predictors are there through skill rather than luck? Knowing more about the questions, numbers etc would help us to come to a good idea about how many people should make these predictions by chance alone, and so on.

chiph 13y ago
Or, she could be a "Teela Brown" -- someone who is naturally (or unnaturally...) lucky. http://en.wikipedia.org/wiki/Teela_Brown http://en.wikipedia.org/wiki/Teela_Brown
- Noelkd 13y ago
  I don't see why this is being down voted as someone who has read the Ringworld novels I couldn't help but think of Teela Brown and her luck.
  - anigbrowl 13y ago
    Because chiph is offering a fictional explanation for a real world phenomenon. There have been lots of scientific studies involving randomness in fields as diverse as economics, game theory, gambling behavior, and parapsychology; if 'luckiness' was a measurable characteristic of people then you'd expect it to reliably show up in such contexts. Invoking it as an explanation where a simpler alternative exists (the idea that some people are more skilled at analyzing current affairs) is illogical.
  - malka 13y ago
    I cant downvote, but i expected the link to point to someone real, not a fictional character.
    - chiph 13y ago
      Ok, here you go. http://www.cnn.com/2014/04/01/us/powerball-lottery-winner/index.html?iref=allsearch http://www.cnn.com/2014/04/01/us/powerball-lottery-winner/in...
      - malka 13y ago
        And on the opposite side of the spectrum : http://en.wikipedia.org/wiki/Roy_Sullivan http://en.wikipedia.org/wiki/Roy_Sullivan

jerf 13y ago

"and the sample set is large enough, you would -expect- there to be some outliers who mostly got things right, by pure chance." According to the article, "In fact, she's so good she's been put on a special team with other super forecasters whose predictions are reportedly 30 percent better than intelligence officers with access to actual classified information." If they mean that after the team was formed, the predictions are still 30% better, than random chance ceases to be an acceptable explanation. If they mean that the team that they formed was 30% better on the predictions used to grant entrance into the team, then your objection still holds. Unfortunately, I don't think the article makes it clear which is the case. That said, there are definitely some mechanisms that could lead an outside group to higher accuracy on these sorts of things. The two biggest are: 1. The insiders are simply that bad, for whatever reason, and random chance can easily beat them, and 2. The failures of the insiders are correlated and overwhelm their theoretically superior information, and I'd especially finger potential ideological biases there, which could affect both information filtering and the analysis. (It is one thing to believe the world should be a certain way as a result of one's ideology... it is quite another to believe the world must already be a certain way as a result of one's ideology, even when one is staring evidence in the face that it is not. One is noble and laudable... the other somewhat less so.)

JoeAltmaier 13y ago
Its not clear they continue to be 30% better - just that they started out that way, and got put on a team. I'm still not convinced these folks aren't just outliers. People in all walks of life give far too much credence to lucky outliers - CEOs are hired that way, ball players, stock portfolios, sports predictors. If you make enough predictions and cherry-pick the best, you can always find a 'miracle man'.
- jerf 13y ago
  "Its not clear they continue to be 30% better" Yes, that was my point. The article failed to make it clear. It is worded such that it could go either way. I agree they may be outliers. However, I can't dismiss out-of-hand the possibility that they are actually better. Outliers are common... but so are systematic/correlated weaknesses within large organizations.

edgarallenbro 13y ago

>was given access to a website that listed dozens of carefully worded questions on events of interest to the intelligence community, along with a place for her to enter her numerical estimate of their likelihood. If you need to know more, try reading the article? It says that they're not yes/no questions.

waterlesscloud 13y ago
I'm participating in this project, as are at least a couple other HNers (surprising to no one, huh?). I won't go into details about it because it's not my place to do so at this point (maybe after). But I will confirm that for every response you give, you're required to enter a percentage estimate of likelihood. For example, you'd enter 90% or 72.212% or whatever on whichever question you're responding to. So there's a potential mechanism for further ranking of participants beyond the binary. The voting mechanism itself is more complicated, but again, I'll leave discussion for when it's over.
- notahacker 13y ago
  The interesting thing about this is it raises the [theoretical, in the absence of information about weighting/ranking systems] possibility professional intelligence analysts' relative underperformance against the measure is less due to inaccurately identifying high/low probability more frequently than amateur "superforecasters", and more due to professionals making the systematic error of overconfidence in the evidence they have when weighting their estimates - e.g. if the amateurs are a lot more likely to either pick 50% on events where there genuinely isn't enough information to forecast and less likely to assign single digit probabilities to events which no available evidence suggest are likely which nevertheless happen. In other words, it sounds highly plausible that if you asked simple binary questions about expected outcomes both groups would give almost identical answers and usually be correct, but the professionals are more confident when both groups are wrong. If this is the case then it's reasonable to assume CIA's statisticians would have done the analysis and know that's the reason these "superforecasters" are better: doubt I guess the reverse is also possible: professional intelligence analysts are systematically tending towards being overcautious and tend to pick numbers towards the middle of the range, either out of a desire not to look silly or because they're more aware of policy implications. But subjectively I'd assign that a lower probability.

Houshalter 13y ago

I'm assuming their smart enough to know that and that the article was just glossing over the details.

nzp 13y ago

In general these are valid remarks. But we're not talking about random clueless people deciding to play statistics, the research is obviously conducted by people who know how this kind of stuff (statistics) works. This reminds me of laypeople trying to poke holes in theories of physics (and I'm not talking about quacks and lunatics, just well intentioned people who think they have a better explanation based on rudimentary high school or even undergraduate science and math). It's such a charming, cute naivete. Do they really think their simple ideas never occurred to a single professional physicist who immediately saw it was wrong for a number of reasons? So, do you really think the researchers in this case hadn't thought of this? I mean yeah, it's OK to doubt and inquire, but come on, this is so rudimentary... > If the questions are all yes/no questions The article says (or at least implies) that the subjects give probabilities of "yes" to each question. EDIT: Oh, and to add, the article clearly says they're not just making shit up and randomly coming up with probabilities. They collect open source info on events and base their estimates on that. So it's not just blind guessing.

lostcolony 13y ago
As to the researchers probably know what they're doing, see my original comment and the very first thing I said, "I need to know more". It's not charming cute naiveté, I simply really think that the article tells me nothing about the methodology. See response here as to why they're still yes/no questions (TLDR; the outcomes are yes/no) - https://news.ycombinator.com/item?id=7515824 https://news.ycombinator.com/item?id=7515824
- nzp 13y ago
  Well, I didn't mean your comment was a case of naivete, because, TBH, the analogy with my layman physics example isn't perfect, but it reminded me, it's a similar thing: people who are or should be aware that they have only superficial info on something, act as though what they know is all there is to it. The point is, we can't possibly come up with a meaningful critique of this research based on the given article, and, I think, the most reasonable course of action in this particular case is to assume the researchers know what they're doing (considering who's involved).
13y ago