3 ms·

This Economist post[0] addresses some of the many comments about statistical outliers: > The big surprise has been the support for the unabashedly elitist “sup

This Economist post[0] addresses some of the many comments about statistical outliers: > The big surprise has been the support for the unabashedly elitist “super-forecaster” hypothesis. The top 2% of forecasters in Year 1 showed that there is more than luck at play. If it were just luck, the “supers” would regress to the mean: yesterday’s champs would be today’s chumps. But they actually got better. When we randomly assigned “supers” into elite teams, they blew the lid off IARPA’s performance goals. They beat the unweighted average (wisdom-of-overall-crowd) by 65%; beat the best algorithms of four competitor institutions by 35-60%; and beat two prediction markets by 20-35% [0]http://www.economist.com/news/21589145-how-sort-best-rest-whos-good-forecasts http://www.economist.com/news/21589145-how-sort-best-rest-wh...

lostcolony 13y ago
Very nice. Thanks for finding that.
ctlaltdefeat 13y ago
>If it were just luck, the “supers” would regress to the mean: yesterday’s champs would be today’s chumps. huh?
- epi8 13y ago
  If I just got really freaking lucky with my predictions yesterday, then there's no reason to imagine I would get really freaking lucky again. Over time, my average accuracy would become the mean of the distribution. On the other hand, if I actually _am_ good at making predictions, then I will continue to make good predictions.
blake8086 13y ago
I would bet that the only reason they're beating prediction markets is because the markets aren't "real money" or they aren't able to participate in them. If they were, they would go make money in the prediction markets, and the prediction markets would become more accurate.
TrainedMonkey 13y ago
Could this be simple selectivity? I do not know how CIA selects analysts, but I doubt it is by having large pool of applicants that are evaluated on their predictions. On the other hand we have group of 30 people, selected for their guessing from amongst 3000 population. When put that way, I would expect 30 people to be consistently better.
- akira2501 13y ago
  These people are also isolated from the internal agency narrative, I imagine that can help quite a bit.
  - negativity 13y ago
    In other words, what if the "taxpayer-funded" forecasts we're permitted to know about are "on the books", so to speak, and so much more is "off the books" and memory-holed beyond even the typical "black magic marker, and white out" form of redacted classification that some times surfaces into the public eye? Truths so brutal, and predictions so harsh that no one dares write them down. But now we enter the information age, and certain realities are essentially undeniable, and can no longer concealed from the civilian panopticons like Google. Sort of like how TV Networks (and even basic cable channels to a degree), with their Standards & Practices departments, waste time sanitizing television destined for the public air waves and radio spectrum, while media on the internet operates with an almost total lack of restraint.
- wisty 13y ago
  You can find ads online. Basically, get a degree in a related field (international politics, security, languages, demographics, economics, etc), and have decent marks. Apply. Don't tell anyone (until they have briefed you on what you can say). Don't have any skeletons in your closet. It's basically like getting a USAID job, but probably not as competitive (lots of funding = lots of positions, and most people doing these kinds of degrees tend to not be very pro-defence) and with slightly higher security requirements.
standeven 13y ago
Predictions are all about discerning signals from noise, and some people seem to be naturally better at discernment. I'm sure it can be learned as well (to a degree), but I'm not at all surprised that some people are exceptionally better than others.

lostcolony 13y ago

Very nice. Thanks for finding that.

ctlaltdefeat 13y ago

>If it were just luck, the “supers” would regress to the mean: yesterday’s champs would be today’s chumps. huh?

epi8 13y ago
If I just got really freaking lucky with my predictions yesterday, then there's no reason to imagine I would get really freaking lucky again. Over time, my average accuracy would become the mean of the distribution. On the other hand, if I actually _am_ good at making predictions, then I will continue to make good predictions.

blake8086 13y ago

I would bet that the only reason they're beating prediction markets is because the markets aren't "real money" or they aren't able to participate in them. If they were, they would go make money in the prediction markets, and the prediction markets would become more accurate.

TrainedMonkey 13y ago

Could this be simple selectivity? I do not know how CIA selects analysts, but I doubt it is by having large pool of applicants that are evaluated on their predictions. On the other hand we have group of 30 people, selected for their guessing from amongst 3000 population. When put that way, I would expect 30 people to be consistently better.

akira2501 13y ago
These people are also isolated from the internal agency narrative, I imagine that can help quite a bit.
- negativity 13y ago
  In other words, what if the "taxpayer-funded" forecasts we're permitted to know about are "on the books", so to speak, and so much more is "off the books" and memory-holed beyond even the typical "black magic marker, and white out" form of redacted classification that some times surfaces into the public eye? Truths so brutal, and predictions so harsh that no one dares write them down. But now we enter the information age, and certain realities are essentially undeniable, and can no longer concealed from the civilian panopticons like Google. Sort of like how TV Networks (and even basic cable channels to a degree), with their Standards & Practices departments, waste time sanitizing television destined for the public air waves and radio spectrum, while media on the internet operates with an almost total lack of restraint.
wisty 13y ago
You can find ads online. Basically, get a degree in a related field (international politics, security, languages, demographics, economics, etc), and have decent marks. Apply. Don't tell anyone (until they have briefed you on what you can say). Don't have any skeletons in your closet. It's basically like getting a USAID job, but probably not as competitive (lots of funding = lots of positions, and most people doing these kinds of degrees tend to not be very pro-defence) and with slightly higher security requirements.

standeven 13y ago

Predictions are all about discerning signals from noise, and some people seem to be naturally better at discernment. I'm sure it can be learned as well (to a degree), but I'm not at all surprised that some people are exceptionally better than others.