29 ms·

A group of 3,000 citizens is making better forecasts than CIA analysts

297 points by sizzle 13y ago

epeterson19 13y ago
"The wisdom of crowds is a very important part of this project, and it's an important driver of accuracy," Tetlock said This part threw me off. But overall, crowdsourced estimates could be a useful tool, provided everyone is actually trying to guess correctly (ie national pride, monetary reward, etc)
nnq 13y ago
Now actually selecting the outliers, the people that, on average, made really good predictions despite access to limited information, and data-mining their features set (psychological traits, information diet, background etc.), THAT would be an interesting study...
- nbouscal 13y ago
  They are doing a bit of that too. They have found correlations with IQ and other metrics. http://www.cis.upenn.edu/~ungar/papers/forecast_AAAI_MAGG.pdf http://www.cis.upenn.edu/~ungar/papers/forecast_AAAI_MAGG.pd...
lukasm 13y ago
Same rules apply to other institutions e.g. investment banks and recommendations. Everybody sticks close to the herd.
matznerd 13y ago
How does this forecasting market compare to something like Intrade?
- LanceH 13y ago
  Intrade is a futures market where people with insider knowledge can buy into a position. So if you know of a scandal for a political candidate that will cause him to withdraw, you can go buy up all the other candidates and make some money when he does. This will alter his price, becoming a predictor before the news actually comes out. Additionally, there is the "put your money where your mouth is" aspect of it. While every candidate in the primaries, is announced as, "the next president of the United States", you can see who the people really think will win based on which futures they purchase. The CIA effort is more like taking a whole lot of people, asking them to predict something and seeing who is good at it. There appears to be some manipulation of those guesses across the breadth of the crowd but it is not mentioned. In contrast to futures markets, the players in the CIA effort have no stake in the game.
  - llamataboot 13y ago
    Right, so one would expect something like Intrade or other prediction markets to leverage both the statistical "wisdom of crowds" and insider information to be even more accurate.
- gwern 13y ago
  The GJP prediction market is pretty similar to how Intrade operated, but a head to head comparison is going to be very difficult inasmuch as a) Intrade has been shut down for like a year now; and b) Intrade's contracts overlapped only a little bit with the IARPA questions.
probablyfiction 13y ago
One thing that I really enjoyed about this article was its description of "the wisdom of crowds." Typically that phrase shrouded in marketing bullshit that just boils down to the idea that groups are always smarter than individuals because magic. That idea on its own is really counterintuitive and hard to accept. It was a good call to talk about the scientific basis for the idea; I wish more places that relied on crowdsourcing would do the same thing.
- Noxchi 13y ago
  > That idea on its own is really counterintuitive and hard to accept. Not really... For example, have you heard of the Cicada puzzle? That is solved by groups of people, and probably wouldn't be solved if it was just one person. Multiple people with specialized knowledge > one person with specialized knowledge. We didn't evolve to be social animals because it would make us stupid...
lostcolony 13y ago
I need to know more. If the questions are all yes/no questions ("Will there be a significant attack on Israeli territory before May 10, 2014?"), and the sample set is large enough, you would -expect- there to be some outliers who mostly got things right, by pure chance. And even outside of that, I want to know what sort of average the CIA agents were batting; if they were hitting just 50%, I would -expect- nearly any sample size to have an outlier who did better. That is, to borrow the old get rich quick scheme, take a list of 1024 addresses. Send to half of them "Stock X will be higher on (date) than today", and to the other half "Stock X will be lower on (date) than today". Repeat with the set you sent the correct 'predictions' to, and continue repeating until you're down to four people who you've sent the correct prediction to every time. Suggest they sign up for your premium stock tips newsletter. All you're doing is playing statistics, you're not actually demonstrating any predictive ability.
- joosters 13y ago
  Agreed. How do we know that the top 1% predictors are there through skill rather than luck? Knowing more about the questions, numbers etc would help us to come to a good idea about how many people should make these predictions by chance alone, and so on.
  - chiph 13y ago
    Or, she could be a "Teela Brown" -- someone who is naturally (or unnaturally...) lucky. http://en.wikipedia.org/wiki/Teela_Brown http://en.wikipedia.org/wiki/Teela_Brown
    - Noelkd 13y ago
      I don't see why this is being down voted as someone who has read the Ringworld novels I couldn't help but think of Teela Brown and her luck.
      - anigbrowl 13y ago
        Because chiph is offering a fictional explanation for a real world phenomenon. There have been lots of scientific studies involving randomness in fields as diverse as economics, game theory, gambling behavior, and parapsychology; if 'luckiness' was a measurable characteristic of people then you'd expect it to reliably show up in such contexts. Invoking it as an explanation where a simpler alternative exists (the idea that some people are more skilled at analyzing current affairs) is illogical.
happyhappy 13y ago
"In fact, Tetlock and his team have even engineered ways to significantly improve the wisdom of the crowd" Does anyone know what these methods might be?
- morganherlocker 13y ago
  One example briefly mentioned was providing feedback to respondents to let them know when they were getting things right or wrong.
  - joosters 13y ago
    And how would that work? Take their example question - 'Will North Korea launch a new multi-stage missile before May 1, 2014?' That's a yes/no question. But none of the participants knows the answer. They are just trying to forecast the future based upon the balance of probabilities. So if NK does launch a missile, the people who answered 'no' were wrong, but the reasons upon which they came to pick 'no' could still have been right. So giving feedback to these people and telling them that they were wrong does not magically aid/improve predictions. Perhaps a simpler example could be used. 'Will the next roll of this dice score 3?' Well, it's obviously more likely that some other result will happen. But if the 3 does come up, you can't say that all those people who said 'no' are worse at predictions...
    - jbattle 13y ago
      Even trickier - it sounds like the participants are giving probabilities. So if you guessed some possibility was 42% likely and then it did in fact happen - are you right or wrong? I don't think there is an answer to that question.
      - Houshalter 13y ago
        PredictionBook [1] does it that way. You get a chart of how many times you assigned a certain probability and what percent of times [1] http://predictionbook.com/ http://predictionbook.com/
Robin_Message 13y ago
It's totally unclear what they are actually doing. Are they averaging everyone's answer (wisdom of the crowd)? Are they looking back at who got lucky and declaring them "super forecasters" (and then seeing large regressions to the mean)? Did they have separate periods of testing to try and control for this? Did they have a control group of similar size just flipping coins? The way this is written it sounds like someone used a Gladwell book as their entire science education.
- nbouscal 13y ago
  There are multiple experimental groups in the project, which use different methods. One of the methods is a prediction market, where you can buy or sell shares of any given outcome, with the market price being the group estimate of the probability. For example, if the group thinks something will happen with 35% probability, you can buy shares for $35 (fake money). When the questions are eventually resolved, correct shares are valued at $100, and incorrect shares are valued at $0.
  - tedsanders 13y ago
    There were multiple experiment groups at the beginning, but I'm not sure if any are left besides Good Judgment. DAGGRE, the prediction market group, is out. Edit: Oops, I misunderstood. Apparently Good Judgment also has a prediction market group inside of it. Thanks nbouscal.
    - deleted 13y ago
      [deleted]
    - deleted 13y ago
      [deleted]
babs474 13y ago
For a prediction market anyone can join checkout https://scicast.org/ https://scicast.org/ Whats neat about SciCast is you can make predictions based on the assumption of how other questions will turn out. eg I think that if the price of bitcoin exceeds 1000 by 2015 at least one presidential candidate will accept bitcoin donations. If btc ends up being below 1000 I make no prediction.
- tedsanders 13y ago
  Just to add, the SciCast project came out of the DAGGRE project, which was a competitor to Good Judgment in the IARPA prediction challenge.
joosters 13y ago
How does the 'crowd' adjust for bias in sources? e.g. if they are all picking the answers to the surveys based on similar methods, e.g. google searches or from listening to the news, how does the crowd avoid the biases from news sources? You can't just 'average them out' if there are systematic skews. e.g. just because half of the people are watching (say) CNN and the other half are watching (say) Fox News doesn't magically mean that the average of their opinions will be correct. What if both news sources overplayed the likelihood of terrorist attacks? How would the crowd adjust to this?
yiedyie 13y ago
Crowdsource prophet. This is not new, that happened before with the "peak oil community".
cottonseed 13y ago
Pure speculation: random, untrained and disinterested parties might be better than CIA analysts at making geopolitical predictions because their results aren't politicized. "Are there WMDs in Iraq?"
- chflamplighter 13y ago
  I was having the same thought, at what point/how do you factor in the, "if I make this predication, which is contrary to my boss's expectations, will I still have a job". Surely there is a huge amount of pressure on "rowing in the same direction" towards a predetermined group think (which is terrible, of course).
  - zeidrich 13y ago
    There's also the "if I make this prediction and I'm wrong, will it affect me?" bias too. The more involved you are, and especially the more accountable you are held for your decisions, the more you're likely to second guess yourself. There's normally a reason your second guess wasn't your first guess.
- graylights 13y ago
  I actually thought the opposite. A lot of these federal positions are military. So I assumed it was some enlisted guy out of high school that didn't want a desk job but wants to be firing guns. He's been doing it years and wants out. He's not invested in any of this analysis. Compared to the guy that self-selected to sign up for an analysis website as a hobby. He finds it fun to think and analyze. He gets points for being right and has a score to improve. The other factor is workloads. If a CIA analyst has to make X decisions in Y time, then they may be rushing. I read this story as a failure of federal services to reward successful performers and remove failing performers.
- michaelt 13y ago
  And there's always going to be a political aspect to recruiting. If you believe this terrorism stuff is overblown and most other countries keep themselves to themselves and aren't out to get us, you probably won't be signing up to be a CIA analyst. But if you think terrorists and communists are a big threat to the country, signing up with the CIA would seem much more important.
thewarrior 13y ago
If you have 3000 people predicting the outcomes of events with say a probablity of 0.3 after 6 trials there would be 2 people who got all of them correct and many others who would have a very good record out of pure chance alone. Unless they publish the detailed numbers we can't be sure of whats going on here. If some person with access only to normal sources is able to perform so well against people with access to lots of classified sources and information , what are we to conclude ? Either they are just lucky or the people at the CIA aren't as competent as we think they are. If we are unwilling to accept any of these theories then we need to know what makes these people good at it. I don't see how just plain simple analysis can beat people at the CIA.
- adsr 13y ago
  But the point here seems to be to use the group average, not to find outliers with good prediction ability per se.
  - logfromblammo 13y ago
    That's exactly it, which is why I was so confused about the "team" of "superpredictors". While you might learn something by comparing their methods with everybody else, you still have to do the same amount of scrutiny on everyone in the crowd to learn anything useful. With enough data, you can apply an automatic adjustment to even the most horrible predictors, to make their contribution to the final result more useful. If, for instance, someone always predicts exactly 10% more favorable results for Israel, independently of the facts, you can just reverse that for that one person on every question mentioning Israel, to produce a bias-adjusted result. That is tremenddously difficult, though, and not nearly as useful as just increasing the size of the crowd. Which is the whole point. With a big enough crowd, you don't need to do that at all, because the stupid little biases unrelated to the facts tend to be noise, not signal.
damian2000 13y ago
Highly recommend this book, which is on the subject ... http://en.wikipedia.org/wiki/The_Wisdom_of_Crowds http://en.wikipedia.org/wiki/The_Wisdom_of_Crowds
- nswanberg 13y ago
  This is a fantastic book and explains most of the questions currently in the thread about the circumstances where large groups of people produce more effective answers than any one of its members and the cases where they do not. If this article was interesting it is worth reading. The book also briefly covers the Policy Analysis Market, a program to experiment with using a futures market to help predict terrorist attacks. As this program used real money, and offered the opportunity to make money off of terrorist attacks, some members of Congress were outraged and the program was immediately cancelled (http://news.bbc.co.uk/2/hi/americas/3106559.stm http://news.bbc.co.uk/2/hi/americas/3106559.stm). This version discussed in this NPR article seems more publicly palatable. In fact, it looks like the similarity was already reported: http://articles.latimes.com/2012/aug/21/nation/la-na-cia-crowds-20120821 http://articles.latimes.com/2012/aug/21/nation/la-na-cia-cro... By the way, can anyone recommend a more rigorous book than Wisdom of Crowds on this subject?
  - jaynos 13y ago
    This one [1] is recommended by the guys behind the Radio Lab podcast (specific episode [2]). I left the referral tag on it since it'll go to help them produce more great radio. [1] http://www.amazon.com/exec/obidos/ASIN/0385503865/wnycorg-20 http://www.amazon.com/exec/obidos/ASIN/0385503865/wnycorg-20 [2] http://www.radiolab.org/story/91502-the-invisible-hand/ http://www.radiolab.org/story/91502-the-invisible-hand/
- pico 13y ago
  John Cook has a recent post that gets to the point: http://www.johndcook.com/blog/2014/03/21/independent-decision-making/ http://www.johndcook.com/blog/2014/03/21/independent-decisio... Crowds (as in mobs) aren’t wise; large groups of independent decision makers are wise.
stefantalpalaru 13y ago
False classified information might actually work against the analyst who has access to it. The same piece of info, if public, will have to stand on its own, without the bias introduced by restricted access / multiple levels of filtering / the attached opinions of people wearing suits.
- ahi 13y ago
  Even true classified information can cause problems if overvalued. The questions seem to be rather general predictions so publicly available geopolitical context likely matters more than any individual bit of inside information.
joshuahedlund 13y ago
This Economist post[0] addresses some of the many comments about statistical outliers: > The big surprise has been the support for the unabashedly elitist “super-forecaster” hypothesis. The top 2% of forecasters in Year 1 showed that there is more than luck at play. If it were just luck, the “supers” would regress to the mean: yesterday’s champs would be today’s chumps. But they actually got better. When we randomly assigned “supers” into elite teams, they blew the lid off IARPA’s performance goals. They beat the unweighted average (wisdom-of-overall-crowd) by 65%; beat the best algorithms of four competitor institutions by 35-60%; and beat two prediction markets by 20-35% [0]http://www.economist.com/news/21589145-how-sort-best-rest-whos-good-forecasts http://www.economist.com/news/21589145-how-sort-best-rest-wh...
- lostcolony 13y ago
  Very nice. Thanks for finding that.
- ctlaltdefeat 13y ago
  >If it were just luck, the “supers” would regress to the mean: yesterday’s champs would be today’s chumps. huh?
  - epi8 13y ago
    If I just got really freaking lucky with my predictions yesterday, then there's no reason to imagine I would get really freaking lucky again. Over time, my average accuracy would become the mean of the distribution. On the other hand, if I actually _am_ good at making predictions, then I will continue to make good predictions.
- blake8086 13y ago
  I would bet that the only reason they're beating prediction markets is because the markets aren't "real money" or they aren't able to participate in them. If they were, they would go make money in the prediction markets, and the prediction markets would become more accurate.
- TrainedMonkey 13y ago
user24 13y ago
That article felt like it ended far too early. I was left with a "is that it?" feeling. Not that I was underwhelmed by the message of the article, but that I couldn't find a message in it. If it had a point beyond "crowdsourcing works", I missed it.
- dredmorbius 13y ago
  True for far, far too many articles IMO.
sailfast 13y ago
I am a big fan / proponent of prediction "markets" or similar otherwise incentivized crowd prediction tools. Happy to see someone execute it within the government while avoiding the terrible branding and public relations faux pas of past "terrorism market" endeavors. Hopefully this will scale more broadly, continue to improve its incentives, and increase our capability to prepare for future events.
nswanberg 13y ago
This is a great topic for a site that uses some "wisdom of crowds" mechanisms as part of the story and comment ranking. Though HN is different in some significant ways: in the experiment, subjects are asked carefully worded questions and give answers without being shown any information from others, whereas here there are several signals available before voting on a story or comment, like prior vote count and previous comments, which can bias a vote.
ahi 13y ago
I went to undergrad with quite a few future CIA analysts. You should not overestimate the average intelligence of CIA analysts. No doubt they have some brilliant people, but you shouldn't be surprised that amateurs with Google News outperform careerists inside a giant bureaucracy.
speeder 13y ago
Assuming you can do that (ie: forecast world events), anyone know how you can use that for your own advantage, beside investing in stocks (even if not necessarily with the stock exchange, OTC or classic VC/angel investing enters this too) or commodities?
deleted 13y ago
[deleted]
logfromblammo 13y ago
They started off reasonable. Ask ordinary people to assign probabilities to potential world events that can be verified. The next step is also reasonable. With a large enough crowd, the estimation noise around the "true" probability will cancel out. That's how the crowd mean estimated the ox weight so closely. But then they turned left onto stupid street. They took the people who, perhaps by chance alone, predicted results closest to the crowd mean, and put them on a "team". What the heck is the point of that? It just reduces the size of the crowd and shows a fundamental misunderstanding of statistics. Past performance does not necessarily predict future results. Start with 10000 coins, and flip them all. Keep the ones that land heads. Flip those, and keep the ones that land heads. Repeat until all coins land on tails, or one coin remains as the "champion of heads." What is the probability that when it is flipped again, it comes up heads? Still 50%. It isn't a supercoin. You just managed to select it from a giant group by making it the sole outlier from a series of independent trials. Though you should probably check to make sure it actually has a tails side, just in case.
- nzp 13y ago
  > It just reduces the size of the crowd and shows a fundamental misunderstanding of statistics. Why do you assume that what's in the NPR article is all there is to it, that there's no info, technical or other, left out? Of course they thought of that, and checked. There's no regression to the mean, the elite group further improved, their estimates were 65 % better than the overall crowd.[0] [0] http://www.economist.com/news/21589145-how-sort-best-rest-whos-good-forecasts http://www.economist.com/news/21589145-how-sort-best-rest-wh...
  - logfromblammo 13y ago
    That article is much more informative. Kudos to you, nzp.
    - nzp 13y ago
      Heh, actually, kudos to joshuahedlund[0], I found it in his comment. [0] https://news.ycombinator.com/item?id=7515263 https://news.ycombinator.com/item?id=7515263
canvia 13y ago
The formatting of the questions and answers could contribute to a candidate's success as well. Unlikely events could be phrased differently than likely events with the intention of skewing the success rate to the positive. I would hate for data from this organization to be used for any sort of real decision making process. You might as well hire a psychic.
kriro 13y ago
Most people in the world only have access to public information so if you think most geopolitical events are brought about by "the masses" instead of by some Platoesque elites (which is of course very much debatable) it may well make sense to base your predictions on the data that is available to most players.
avaku 13y ago
Was the project shut down? (the website is not working, sorry if I haven't missed that in the article)
- gwern 13y ago
  GJP is still running and just added some new IARPA questions today. Registration for the next season may not be open yet, though.
GarvielLoken 13y ago
Intuitors.
lostcolony 13y ago
Now the fun question; are there any anti-forecasters? People who guess wrong so consistently and powerfully they're as good as super forecasters?
- tedsanders 13y ago
  No, there were not.
  - jff 13y ago
    Well, people claiming to be psychic have sometimes managed to get worse-than-random results when tested...
- cryoshon 13y ago
  This is a really interesting question, actually. In my undergrad ecology class, we were assigned a scent-based laboratory experiment which involved all the male and female members of the class wearing the same white t-shirt for 48 hours in order to accrue bodily odors-- heavy workouts, perfume, showering, etc were all prohibited. At the end, everyone in the class sniffed all of the t-shirts and guessed which t shirt had been worn by what sex. Most people had a 50-50 chance, and a few guessed slightly better than chance. I was far worse than chance, guessing only 10% correctly. In a way, anti-forecasting is a type of forecasting, too. If someone had asked me for my guess and then guessed the opposite, they'd have had superior guessing power. This paradigm falls apart quickly when the chances of events aren't so clear, however.
  - maxerickson 13y ago
    I wonder how practice/training would impact those results.
  - adambard 13y ago
    Ah, the old Costanza maneuver.
- bane 13y ago
  That's an excellent question I'd like to know the answer to. I've personally known lots of people who were ideologically predisposed towards certain biases who would consistently predict outcomes of political events incorrectly. e.g. "if <insert presidential candidate> is elected, that's gonna be the end of America" with similarly failed predictions over the next four years when that candidate is elected.
jljljl 13y ago
They may be good at prediction, but who writes their questions? It's probably much easier to reach a successful prediction when a lot of the framing has already been done for them. If you take the Ukraine question, for example, there's already a lot of intelligence encapsulated, such as the City and Date when the CIA is expecting a Russian invasion. As a post mentions below, the questions may even be phrased/biased towards the right answer. I wonder if they would be as good at prediction if they didn't have the resources and intelligence of the CIA providing them with the correct focus, prioritization, and framing.
- kazagistar 13y ago
  The CIA is also providing focus, prioritization, and framing for themselves, and is being outperformed.
njharman 13y ago
Maybe the founding father's "fear" of the ignorant masses and choice of representative government needs to be reevaluated. Go back to direct actual democracy. But with ranges instead of yes/no. So as to capture "average is true signal". Maybe first step is chance voting of representatives from yes/no to 1-100 how much you wan this person to win.
joosters 13y ago
Doug Stanhope has a good rant about these opinion polls: http://www.youtube.com/watch?feature=player_detailpage&v=RycwYRcm3Lc#t=84 http://www.youtube.com/watch?feature=player_detailpage&v=Ryc...
psikorsky 13y ago
Obviously there are some very smart people working in government intelligence agencies, including the CIA, MI6, etc. However, in my experience working in the private intelligence sector, I found that many former government employees lacked creativity and relied too heavily on 'classified sources' and their network. The best analysts were extremely lateral thinkers who used Google and public databases, but would never have passed a government test on account of their quirks.
doctorpangloss 13y ago
>First, if you want people to get better at making predictions, you need to keep score of how accurate their predictions turn out to be, so they have concrete feedback. >But also, if you take a large crowd of different people with access to different information and pool their predictions, you will be in much better shape than if you rely on a single very smart person, or even a small group of very smart people." Both points are clearly true. But it's obvious that the CIA, and probably anything political, has problems "keeping score," not getting more voices.
dang 13y ago
A note on the title change: When the story title is linkbait and there are no good subtitles, we sometimes draw on the first sentence of the story. In this case I used the description under the photo. The key thing is to use only language that's already there.
tghw 13y ago
Now I'm really confused. The title of this post was originally "So You Think You're Smarter Than A CIA Agent ", which is consistent with the title of the article. Now it's been changed to editorialize the content? Isn't that the opposite of HN policies?
- kissickas 13y ago
  Is this policy written somewhere? The only trend I've noticed is that I always agree with anyone complaining about a changed submission title. (I don't think you're complaining, are you? The NPR title is pretty cheap IMHO.)
  - Noelkd 13y ago
    You can read the guidelines here: http://ycombinator.com/newsguidelines.html http://ycombinator.com/newsguidelines.html This may fall under: " Otherwise please use the original title, unless it is misleading or linkbait. "
    - kissickas 13y ago
      I would call it linkbait, cool. Sad to see coming from NPR. Thanks for the link, it's been a few years since I read those rules.
- Houshalter 13y ago
  I had the same thing happen to one of my posts. It wasn't changed to the title of the article or to something that made it more descriptive, it was just given a random new title that made it more vague.
- ama729 13y ago
  HN mod explained here: https://news.ycombinator.com/item?id=7516890 https://news.ycombinator.com/item?id=7516890
syncsynchalt 13y ago
John Brunner's book "The Shockwave Rider" (1975), which was heavily inspired by Toffler's "Future Shock", had this concept (called a Delphi pool) as a major and successful component of the future society.
- gwern 13y ago
  Delphi pools were invented by RAND, not Brunner, IIRC, have been studied over the years (Armstrong's _Principles of Forecasting_ has some papers on them), and work differently from prediction markets: Delphi pools collect a small number of experts, asks for written analyses and breakdowns and qualitative assessments (not necessarily the precise numerical probabilities of prediction markets), feeds each expert's analysis to the others, asks them for a new analysis incorporating the others, and repeats this for 2 or 3 rounds until a rough consensus has been reached. This is not very similar to prediction markets.
diminoten 13y ago
> In other words, there are errors on every side of the mark, but there is a truth at the center that people are responding to, and if you average a large number of predictions together, the errors will end up canceling each other out, and you are left with a more accurate guess. This just reminds me all too much of the Twitch Plays Pokémon phenomenon that took place last month. The fact that they actually are completing games is, frankly, astounding.
Justen 13y ago
Are the underpinnings of this experiment in line with the idea behind the HyperLogLog story[0] I read about the other day? [0]http://antirez.com/news/75 http://antirez.com/news/75
jmzbond 13y ago
Doesn't this article seem to contradict itself? It cites superforecasters' amazing ability, then talks about how better forecasting is the result of averaging the noise from a crowd of diverse opinions...
- dean 13y ago
  That's what jumped out at me as well. First the article extols how the Wisdom of Crowds is smarter than the experts. Then it goes on to say how some of the crowd are superforecasters, (essentially experts), and the rest of the crowd makes no difference. It would be interesting to see how small the group of superforecasters would need to get before their accuracy falls below that of the CIA analysts.
  - jmzbond 13y ago
    Good test!
anigbrowl 13y ago
Inaccurate headline. Citizenship is not required for participation.
lifeisstillgood 13y ago
So could I suggest that CIA forecasts are affected by internal politics, perceived needs of political establishment, and afflicted by biases in top secret material that cannot be easily counter-balanced. Whereas if they put these super-forecasters in the CIA and said "if you dont predict what i want its off to Afghanistan with you" they will soon regress to the mean.
ballard 13y ago
The Wikipedia page on prediction markets storied history and interesting bit. Curious that USM had a prototype that was shutdown for being a "terrorism futures market." http://en.wikipedia.org/wiki/Prediction_market http://en.wikipedia.org/wiki/Prediction_market
easy_rider 13y ago
I sometimes have good runs in blackjack.
SolarNet 13y ago
This reminds me of the future predicted in Rainbows End by Vernor Vinge. Where he explores the ability to spin up pools of analysts like a cloud server.
ronaldx 13y ago
I did exactly this in high school: I set up an experiment to find the people best at guessing Zener cards picked at random. I surveryed 100 people and the best 5 did notably better than the others. I dubbed these people "clairvoyant". (Spoiler: on retesting, they weren't, but then again I only followed this process once...). Wisdom of the crowds will work just fine for anything modestly obvious, and it will fail for any black swan events that intelligence actually needs to be aware of - if the predictors even get asked about those.
prestadige 13y ago
It seems to me that all professional experts are subject to groupthink. However, could anonymous lay forecasters still make such excellent predictions without those groups in place? Anyway, I think all this argues in favor of Robin Hanson's concept of "Futarchy": http://hanson.gmu.edu/futarchy.html http://hanson.gmu.edu/futarchy.html
pserwylo 13y ago
The issue of crowd sourcing answers to questions is a very interesting one. I've recently been reading research in this area with a colleague, and it is far more involved than the article makes it sound. If you are interested, one of the first people to work on the problem were Dawid & Skene [0] back in 1979. They were trying to solve the issue of clinicians talking to patients and asking how they are. Each time they asked, the patient may give a slightly different answer. The goal was to estimate the "true" answer from the number of various "noisy" answers (sound familiar after reading the OP?) The general intuition for most future research into the topic is that if you tend to agree with others more often than not, your score goes up. If your score is higher, your answers are trusted more. Then there are a whole bunch of interesting additions to this principle, such as factoring in question difficulty [1] (e.g. questions which have the most disagreement can be considered more difficult, and hence you may want to ask more people about these questions to get a clearer picture). It is also helpful to seed certain questions for which the answer is "known" (may or may not be possible, depending on the project). Then you can use the answers to these questions to help score people even more accurately. An even more interesting approach is to take into account people who answer adversarially, that is, they always answer incorrectly (either due to malicious intent, or they consistently misinterpret the questions in a backward fasion). If you have somebody who answers incorrectly 90% of the time, then they provide almost exactly the same information as someone who answers correctly 90% of the time. All you need to do is flip their vote once you've identified this. The whole field is very facinating, and there seems to be more and more publications coming out with interesting approaches to the problem. [0] http://scholar.google.com/scholar?cites=15953840037809938294&as_sdt=5,39&sciodt=0,39&hl=en http://scholar.google.com/scholar?cites=15953840037809938294... [1] http://scholar.google.com/scholar?cluster=14409115865258228521&hl=en&as_sdt=0,39 http://scholar.google.com/scholar?cluster=144091158652582285...
mtrn 13y ago
Hearing the article, I felt thrown into PKD novel. One of the recurring theme in his work are precogs, normal people with special abilities to forecast the future. Or also to mispredict the future very accurately. Large corporations use groups of precogs to plan marketing campaigns, product designs, etc.
blazespin 13y ago
Where's the API?
- allochthon 13y ago
  It might be possible to put something together using Amazon's Mechanical Turk.
grn 13y ago
See also http://www.johndcook.com/blog/2014/03/21/independent-decision-making/ http://www.johndcook.com/blog/2014/03/21/independent-decisio...
secfirstmd 13y ago
I noticed this phenomenon alot when I used to work at Intrade.

297 points by sizzle 13y ago

"The wisdom of crowds is a very important part of this project, and it's an important driver of accuracy," Tetlock said This part threw me off. But overall, crowdsourced estimates could be a useful tool, provided everyone is actually trying to guess correctly (ie national pride, monetary reward, etc)

nnq 13y ago

Now actually selecting the outliers, the people that, on average, made really good predictions despite access to limited information, and data-mining their features set (psychological traits, information diet, background etc.), THAT would be an interesting study...

nbouscal 13y ago
They are doing a bit of that too. They have found correlations with IQ and other metrics. http://www.cis.upenn.edu/~ungar/papers/forecast_AAAI_MAGG.pdf http://www.cis.upenn.edu/~ungar/papers/forecast_AAAI_MAGG.pd...

lukasm 13y ago

Same rules apply to other institutions e.g. investment banks and recommendations. Everybody sticks close to the herd.

matznerd 13y ago

How does this forecasting market compare to something like Intrade?

LanceH 13y ago
Intrade is a futures market where people with insider knowledge can buy into a position. So if you know of a scandal for a political candidate that will cause him to withdraw, you can go buy up all the other candidates and make some money when he does. This will alter his price, becoming a predictor before the news actually comes out. Additionally, there is the "put your money where your mouth is" aspect of it. While every candidate in the primaries, is announced as, "the next president of the United States", you can see who the people really think will win based on which futures they purchase. The CIA effort is more like taking a whole lot of people, asking them to predict something and seeing who is good at it. There appears to be some manipulation of those guesses across the breadth of the crowd but it is not mentioned. In contrast to futures markets, the players in the CIA effort have no stake in the game.
- llamataboot 13y ago
  Right, so one would expect something like Intrade or other prediction markets to leverage both the statistical "wisdom of crowds" and insider information to be even more accurate.
gwern 13y ago
The GJP prediction market is pretty similar to how Intrade operated, but a head to head comparison is going to be very difficult inasmuch as a) Intrade has been shut down for like a year now; and b) Intrade's contracts overlapped only a little bit with the IARPA questions.

probablyfiction 13y ago

One thing that I really enjoyed about this article was its description of "the wisdom of crowds." Typically that phrase shrouded in marketing bullshit that just boils down to the idea that groups are always smarter than individuals because magic. That idea on its own is really counterintuitive and hard to accept. It was a good call to talk about the scientific basis for the idea; I wish more places that relied on crowdsourcing would do the same thing.

Noxchi 13y ago
> That idea on its own is really counterintuitive and hard to accept. Not really... For example, have you heard of the Cicada puzzle? That is solved by groups of people, and probably wouldn't be solved if it was just one person. Multiple people with specialized knowledge > one person with specialized knowledge. We didn't evolve to be social animals because it would make us stupid...

lostcolony 13y ago

I need to know more. If the questions are all yes/no questions ("Will there be a significant attack on Israeli territory before May 10, 2014?"), and the sample set is large enough, you would -expect- there to be some outliers who mostly got things right, by pure chance. And even outside of that, I want to know what sort of average the CIA agents were batting; if they were hitting just 50%, I would -expect- nearly any sample size to have an outlier who did better. That is, to borrow the old get rich quick scheme, take a list of 1024 addresses. Send to half of them "Stock X will be higher on (date) than today", and to the other half "Stock X will be lower on (date) than today". Repeat with the set you sent the correct 'predictions' to, and continue repeating until you're down to four people who you've sent the correct prediction to every time. Suggest they sign up for your premium stock tips newsletter. All you're doing is playing statistics, you're not actually demonstrating any predictive ability.

joosters 13y ago
Agreed. How do we know that the top 1% predictors are there through skill rather than luck? Knowing more about the questions, numbers etc would help us to come to a good idea about how many people should make these predictions by chance alone, and so on.
- chiph 13y ago
  Or, she could be a "Teela Brown" -- someone who is naturally (or unnaturally...) lucky. http://en.wikipedia.org/wiki/Teela_Brown http://en.wikipedia.org/wiki/Teela_Brown
  - Noelkd 13y ago
    I don't see why this is being down voted as someone who has read the Ringworld novels I couldn't help but think of Teela Brown and her luck.
    - anigbrowl 13y ago
      Because chiph is offering a fictional explanation for a real world phenomenon. There have been lots of scientific studies involving randomness in fields as diverse as economics, game theory, gambling behavior, and parapsychology; if 'luckiness' was a measurable characteristic of people then you'd expect it to reliably show up in such contexts. Invoking it as an explanation where a simpler alternative exists (the idea that some people are more skilled at analyzing current affairs) is illogical.

happyhappy 13y ago

"In fact, Tetlock and his team have even engineered ways to significantly improve the wisdom of the crowd" Does anyone know what these methods might be?

morganherlocker 13y ago
One example briefly mentioned was providing feedback to respondents to let them know when they were getting things right or wrong.
- joosters 13y ago
  And how would that work? Take their example question - 'Will North Korea launch a new multi-stage missile before May 1, 2014?' That's a yes/no question. But none of the participants knows the answer. They are just trying to forecast the future based upon the balance of probabilities. So if NK does launch a missile, the people who answered 'no' were wrong, but the reasons upon which they came to pick 'no' could still have been right. So giving feedback to these people and telling them that they were wrong does not magically aid/improve predictions. Perhaps a simpler example could be used. 'Will the next roll of this dice score 3?' Well, it's obviously more likely that some other result will happen. But if the 3 does come up, you can't say that all those people who said 'no' are worse at predictions...
  - jbattle 13y ago
    Even trickier - it sounds like the participants are giving probabilities. So if you guessed some possibility was 42% likely and then it did in fact happen - are you right or wrong? I don't think there is an answer to that question.
    - Houshalter 13y ago
      PredictionBook [1] does it that way. You get a chart of how many times you assigned a certain probability and what percent of times [1] http://predictionbook.com/ http://predictionbook.com/

Robin_Message 13y ago

It's totally unclear what they are actually doing. Are they averaging everyone's answer (wisdom of the crowd)? Are they looking back at who got lucky and declaring them "super forecasters" (and then seeing large regressions to the mean)? Did they have separate periods of testing to try and control for this? Did they have a control group of similar size just flipping coins? The way this is written it sounds like someone used a Gladwell book as their entire science education.

nbouscal 13y ago
There are multiple experimental groups in the project, which use different methods. One of the methods is a prediction market, where you can buy or sell shares of any given outcome, with the market price being the group estimate of the probability. For example, if the group thinks something will happen with 35% probability, you can buy shares for $35 (fake money). When the questions are eventually resolved, correct shares are valued at $100, and incorrect shares are valued at $0.
- tedsanders 13y ago
  There were multiple experiment groups at the beginning, but I'm not sure if any are left besides Good Judgment. DAGGRE, the prediction market group, is out. Edit: Oops, I misunderstood. Apparently Good Judgment also has a prediction market group inside of it. Thanks nbouscal.
  - deleted 13y ago
    [deleted]
  - deleted 13y ago
    [deleted]

babs474 13y ago

For a prediction market anyone can join checkout https://scicast.org/ https://scicast.org/ Whats neat about SciCast is you can make predictions based on the assumption of how other questions will turn out. eg I think that if the price of bitcoin exceeds 1000 by 2015 at least one presidential candidate will accept bitcoin donations. If btc ends up being below 1000 I make no prediction.

tedsanders 13y ago
Just to add, the SciCast project came out of the DAGGRE project, which was a competitor to Good Judgment in the IARPA prediction challenge.

joosters 13y ago

How does the 'crowd' adjust for bias in sources? e.g. if they are all picking the answers to the surveys based on similar methods, e.g. google searches or from listening to the news, how does the crowd avoid the biases from news sources? You can't just 'average them out' if there are systematic skews. e.g. just because half of the people are watching (say) CNN and the other half are watching (say) Fox News doesn't magically mean that the average of their opinions will be correct. What if both news sources overplayed the likelihood of terrorist attacks? How would the crowd adjust to this?

yiedyie 13y ago

Crowdsource prophet. This is not new, that happened before with the "peak oil community".

cottonseed 13y ago

Pure speculation: random, untrained and disinterested parties might be better than CIA analysts at making geopolitical predictions because their results aren't politicized. "Are there WMDs in Iraq?"

chflamplighter 13y ago
I was having the same thought, at what point/how do you factor in the, "if I make this predication, which is contrary to my boss's expectations, will I still have a job". Surely there is a huge amount of pressure on "rowing in the same direction" towards a predetermined group think (which is terrible, of course).
- zeidrich 13y ago
  There's also the "if I make this prediction and I'm wrong, will it affect me?" bias too. The more involved you are, and especially the more accountable you are held for your decisions, the more you're likely to second guess yourself. There's normally a reason your second guess wasn't your first guess.
graylights 13y ago
I actually thought the opposite. A lot of these federal positions are military. So I assumed it was some enlisted guy out of high school that didn't want a desk job but wants to be firing guns. He's been doing it years and wants out. He's not invested in any of this analysis. Compared to the guy that self-selected to sign up for an analysis website as a hobby. He finds it fun to think and analyze. He gets points for being right and has a score to improve. The other factor is workloads. If a CIA analyst has to make X decisions in Y time, then they may be rushing. I read this story as a failure of federal services to reward successful performers and remove failing performers.
michaelt 13y ago
And there's always going to be a political aspect to recruiting. If you believe this terrorism stuff is overblown and most other countries keep themselves to themselves and aren't out to get us, you probably won't be signing up to be a CIA analyst. But if you think terrorists and communists are a big threat to the country, signing up with the CIA would seem much more important.

thewarrior 13y ago

If you have 3000 people predicting the outcomes of events with say a probablity of 0.3 after 6 trials there would be 2 people who got all of them correct and many others who would have a very good record out of pure chance alone. Unless they publish the detailed numbers we can't be sure of whats going on here. If some person with access only to normal sources is able to perform so well against people with access to lots of classified sources and information , what are we to conclude ? Either they are just lucky or the people at the CIA aren't as competent as we think they are. If we are unwilling to accept any of these theories then we need to know what makes these people good at it. I don't see how just plain simple analysis can beat people at the CIA.

adsr 13y ago
But the point here seems to be to use the group average, not to find outliers with good prediction ability per se.
- logfromblammo 13y ago
  That's exactly it, which is why I was so confused about the "team" of "superpredictors". While you might learn something by comparing their methods with everybody else, you still have to do the same amount of scrutiny on everyone in the crowd to learn anything useful. With enough data, you can apply an automatic adjustment to even the most horrible predictors, to make their contribution to the final result more useful. If, for instance, someone always predicts exactly 10% more favorable results for Israel, independently of the facts, you can just reverse that for that one person on every question mentioning Israel, to produce a bias-adjusted result. That is tremenddously difficult, though, and not nearly as useful as just increasing the size of the crowd. Which is the whole point. With a big enough crowd, you don't need to do that at all, because the stupid little biases unrelated to the facts tend to be noise, not signal.

damian2000 13y ago

Highly recommend this book, which is on the subject ... http://en.wikipedia.org/wiki/The_Wisdom_of_Crowds http://en.wikipedia.org/wiki/The_Wisdom_of_Crowds

nswanberg 13y ago
This is a fantastic book and explains most of the questions currently in the thread about the circumstances where large groups of people produce more effective answers than any one of its members and the cases where they do not. If this article was interesting it is worth reading. The book also briefly covers the Policy Analysis Market, a program to experiment with using a futures market to help predict terrorist attacks. As this program used real money, and offered the opportunity to make money off of terrorist attacks, some members of Congress were outraged and the program was immediately cancelled (http://news.bbc.co.uk/2/hi/americas/3106559.stm http://news.bbc.co.uk/2/hi/americas/3106559.stm). This version discussed in this NPR article seems more publicly palatable. In fact, it looks like the similarity was already reported: http://articles.latimes.com/2012/aug/21/nation/la-na-cia-crowds-20120821 http://articles.latimes.com/2012/aug/21/nation/la-na-cia-cro... By the way, can anyone recommend a more rigorous book than Wisdom of Crowds on this subject?
- jaynos 13y ago
  This one [1] is recommended by the guys behind the Radio Lab podcast (specific episode [2]). I left the referral tag on it since it'll go to help them produce more great radio. [1] http://www.amazon.com/exec/obidos/ASIN/0385503865/wnycorg-20 http://www.amazon.com/exec/obidos/ASIN/0385503865/wnycorg-20 [2] http://www.radiolab.org/story/91502-the-invisible-hand/ http://www.radiolab.org/story/91502-the-invisible-hand/
pico 13y ago
John Cook has a recent post that gets to the point: http://www.johndcook.com/blog/2014/03/21/independent-decision-making/ http://www.johndcook.com/blog/2014/03/21/independent-decisio... Crowds (as in mobs) aren’t wise; large groups of independent decision makers are wise.

stefantalpalaru 13y ago

False classified information might actually work against the analyst who has access to it. The same piece of info, if public, will have to stand on its own, without the bias introduced by restricted access / multiple levels of filtering / the attached opinions of people wearing suits.

ahi 13y ago
Even true classified information can cause problems if overvalued. The questions seem to be rather general predictions so publicly available geopolitical context likely matters more than any individual bit of inside information.

joshuahedlund 13y ago

This Economist post[0] addresses some of the many comments about statistical outliers: > The big surprise has been the support for the unabashedly elitist “super-forecaster” hypothesis. The top 2% of forecasters in Year 1 showed that there is more than luck at play. If it were just luck, the “supers” would regress to the mean: yesterday’s champs would be today’s chumps. But they actually got better. When we randomly assigned “supers” into elite teams, they blew the lid off IARPA’s performance goals. They beat the unweighted average (wisdom-of-overall-crowd) by 65%; beat the best algorithms of four competitor institutions by 35-60%; and beat two prediction markets by 20-35% [0]http://www.economist.com/news/21589145-how-sort-best-rest-whos-good-forecasts http://www.economist.com/news/21589145-how-sort-best-rest-wh...

lostcolony 13y ago
Very nice. Thanks for finding that.
ctlaltdefeat 13y ago
>If it were just luck, the “supers” would regress to the mean: yesterday’s champs would be today’s chumps. huh?
- epi8 13y ago
  If I just got really freaking lucky with my predictions yesterday, then there's no reason to imagine I would get really freaking lucky again. Over time, my average accuracy would become the mean of the distribution. On the other hand, if I actually _am_ good at making predictions, then I will continue to make good predictions.
blake8086 13y ago
I would bet that the only reason they're beating prediction markets is because the markets aren't "real money" or they aren't able to participate in them. If they were, they would go make money in the prediction markets, and the prediction markets would become more accurate.
TrainedMonkey 13y ago

user24 13y ago

That article felt like it ended far too early. I was left with a "is that it?" feeling. Not that I was underwhelmed by the message of the article, but that I couldn't find a message in it. If it had a point beyond "crowdsourcing works", I missed it.

dredmorbius 13y ago
True for far, far too many articles IMO.

sailfast 13y ago

I am a big fan / proponent of prediction "markets" or similar otherwise incentivized crowd prediction tools. Happy to see someone execute it within the government while avoiding the terrible branding and public relations faux pas of past "terrorism market" endeavors. Hopefully this will scale more broadly, continue to improve its incentives, and increase our capability to prepare for future events.

nswanberg 13y ago

This is a great topic for a site that uses some "wisdom of crowds" mechanisms as part of the story and comment ranking. Though HN is different in some significant ways: in the experiment, subjects are asked carefully worded questions and give answers without being shown any information from others, whereas here there are several signals available before voting on a story or comment, like prior vote count and previous comments, which can bias a vote.

ahi 13y ago

I went to undergrad with quite a few future CIA analysts. You should not overestimate the average intelligence of CIA analysts. No doubt they have some brilliant people, but you shouldn't be surprised that amateurs with Google News outperform careerists inside a giant bureaucracy.

speeder 13y ago

Assuming you can do that (ie: forecast world events), anyone know how you can use that for your own advantage, beside investing in stocks (even if not necessarily with the stock exchange, OTC or classic VC/angel investing enters this too) or commodities?

deleted 13y ago

[deleted]

logfromblammo 13y ago

They started off reasonable. Ask ordinary people to assign probabilities to potential world events that can be verified. The next step is also reasonable. With a large enough crowd, the estimation noise around the "true" probability will cancel out. That's how the crowd mean estimated the ox weight so closely. But then they turned left onto stupid street. They took the people who, perhaps by chance alone, predicted results closest to the crowd mean, and put them on a "team". What the heck is the point of that? It just reduces the size of the crowd and shows a fundamental misunderstanding of statistics. Past performance does not necessarily predict future results. Start with 10000 coins, and flip them all. Keep the ones that land heads. Flip those, and keep the ones that land heads. Repeat until all coins land on tails, or one coin remains as the "champion of heads." What is the probability that when it is flipped again, it comes up heads? Still 50%. It isn't a supercoin. You just managed to select it from a giant group by making it the sole outlier from a series of independent trials. Though you should probably check to make sure it actually has a tails side, just in case.

nzp 13y ago
> It just reduces the size of the crowd and shows a fundamental misunderstanding of statistics. Why do you assume that what's in the NPR article is all there is to it, that there's no info, technical or other, left out? Of course they thought of that, and checked. There's no regression to the mean, the elite group further improved, their estimates were 65 % better than the overall crowd.[0] [0] http://www.economist.com/news/21589145-how-sort-best-rest-whos-good-forecasts http://www.economist.com/news/21589145-how-sort-best-rest-wh...
- logfromblammo 13y ago
  That article is much more informative. Kudos to you, nzp.
  - nzp 13y ago
    Heh, actually, kudos to joshuahedlund[0], I found it in his comment. [0] https://news.ycombinator.com/item?id=7515263 https://news.ycombinator.com/item?id=7515263

canvia 13y ago

The formatting of the questions and answers could contribute to a candidate's success as well. Unlikely events could be phrased differently than likely events with the intention of skewing the success rate to the positive. I would hate for data from this organization to be used for any sort of real decision making process. You might as well hire a psychic.

kriro 13y ago

Most people in the world only have access to public information so if you think most geopolitical events are brought about by "the masses" instead of by some Platoesque elites (which is of course very much debatable) it may well make sense to base your predictions on the data that is available to most players.

avaku 13y ago

Was the project shut down? (the website is not working, sorry if I haven't missed that in the article)

gwern 13y ago
GJP is still running and just added some new IARPA questions today. Registration for the next season may not be open yet, though.

GarvielLoken 13y ago

Intuitors.

lostcolony 13y ago

Now the fun question; are there any anti-forecasters? People who guess wrong so consistently and powerfully they're as good as super forecasters?

tedsanders 13y ago
No, there were not.
- jff 13y ago
  Well, people claiming to be psychic have sometimes managed to get worse-than-random results when tested...
cryoshon 13y ago
This is a really interesting question, actually. In my undergrad ecology class, we were assigned a scent-based laboratory experiment which involved all the male and female members of the class wearing the same white t-shirt for 48 hours in order to accrue bodily odors-- heavy workouts, perfume, showering, etc were all prohibited. At the end, everyone in the class sniffed all of the t-shirts and guessed which t shirt had been worn by what sex. Most people had a 50-50 chance, and a few guessed slightly better than chance. I was far worse than chance, guessing only 10% correctly. In a way, anti-forecasting is a type of forecasting, too. If someone had asked me for my guess and then guessed the opposite, they'd have had superior guessing power. This paradigm falls apart quickly when the chances of events aren't so clear, however.
- maxerickson 13y ago
  I wonder how practice/training would impact those results.
- adambard 13y ago
  Ah, the old Costanza maneuver.
bane 13y ago
That's an excellent question I'd like to know the answer to. I've personally known lots of people who were ideologically predisposed towards certain biases who would consistently predict outcomes of political events incorrectly. e.g. "if <insert presidential candidate> is elected, that's gonna be the end of America" with similarly failed predictions over the next four years when that candidate is elected.

jljljl 13y ago

They may be good at prediction, but who writes their questions? It's probably much easier to reach a successful prediction when a lot of the framing has already been done for them. If you take the Ukraine question, for example, there's already a lot of intelligence encapsulated, such as the City and Date when the CIA is expecting a Russian invasion. As a post mentions below, the questions may even be phrased/biased towards the right answer. I wonder if they would be as good at prediction if they didn't have the resources and intelligence of the CIA providing them with the correct focus, prioritization, and framing.

kazagistar 13y ago
The CIA is also providing focus, prioritization, and framing for themselves, and is being outperformed.

njharman 13y ago

Maybe the founding father's "fear" of the ignorant masses and choice of representative government needs to be reevaluated. Go back to direct actual democracy. But with ranges instead of yes/no. So as to capture "average is true signal". Maybe first step is chance voting of representatives from yes/no to 1-100 how much you wan this person to win.

joosters 13y ago

Doug Stanhope has a good rant about these opinion polls: http://www.youtube.com/watch?feature=player_detailpage&v=RycwYRcm3Lc#t=84 http://www.youtube.com/watch?feature=player_detailpage&v=Ryc...

psikorsky 13y ago

Obviously there are some very smart people working in government intelligence agencies, including the CIA, MI6, etc. However, in my experience working in the private intelligence sector, I found that many former government employees lacked creativity and relied too heavily on 'classified sources' and their network. The best analysts were extremely lateral thinkers who used Google and public databases, but would never have passed a government test on account of their quirks.

doctorpangloss 13y ago

>First, if you want people to get better at making predictions, you need to keep score of how accurate their predictions turn out to be, so they have concrete feedback. >But also, if you take a large crowd of different people with access to different information and pool their predictions, you will be in much better shape than if you rely on a single very smart person, or even a small group of very smart people." Both points are clearly true. But it's obvious that the CIA, and probably anything political, has problems "keeping score," not getting more voices.

dang 13y ago

A note on the title change: When the story title is linkbait and there are no good subtitles, we sometimes draw on the first sentence of the story. In this case I used the description under the photo. The key thing is to use only language that's already there.

tghw 13y ago

Now I'm really confused. The title of this post was originally "So You Think You're Smarter Than A CIA Agent ", which is consistent with the title of the article. Now it's been changed to editorialize the content? Isn't that the opposite of HN policies?

kissickas 13y ago
Is this policy written somewhere? The only trend I've noticed is that I always agree with anyone complaining about a changed submission title. (I don't think you're complaining, are you? The NPR title is pretty cheap IMHO.)
- Noelkd 13y ago
  You can read the guidelines here: http://ycombinator.com/newsguidelines.html http://ycombinator.com/newsguidelines.html This may fall under: " Otherwise please use the original title, unless it is misleading or linkbait. "
  - kissickas 13y ago
    I would call it linkbait, cool. Sad to see coming from NPR. Thanks for the link, it's been a few years since I read those rules.
Houshalter 13y ago
I had the same thing happen to one of my posts. It wasn't changed to the title of the article or to something that made it more descriptive, it was just given a random new title that made it more vague.
ama729 13y ago
HN mod explained here: https://news.ycombinator.com/item?id=7516890 https://news.ycombinator.com/item?id=7516890

syncsynchalt 13y ago

John Brunner's book "The Shockwave Rider" (1975), which was heavily inspired by Toffler's "Future Shock", had this concept (called a Delphi pool) as a major and successful component of the future society.

gwern 13y ago
Delphi pools were invented by RAND, not Brunner, IIRC, have been studied over the years (Armstrong's _Principles of Forecasting_ has some papers on them), and work differently from prediction markets: Delphi pools collect a small number of experts, asks for written analyses and breakdowns and qualitative assessments (not necessarily the precise numerical probabilities of prediction markets), feeds each expert's analysis to the others, asks them for a new analysis incorporating the others, and repeats this for 2 or 3 rounds until a rough consensus has been reached. This is not very similar to prediction markets.

diminoten 13y ago

> In other words, there are errors on every side of the mark, but there is a truth at the center that people are responding to, and if you average a large number of predictions together, the errors will end up canceling each other out, and you are left with a more accurate guess. This just reminds me all too much of the Twitch Plays Pokémon phenomenon that took place last month. The fact that they actually are completing games is, frankly, astounding.

Justen 13y ago

Are the underpinnings of this experiment in line with the idea behind the HyperLogLog story[0] I read about the other day? [0]http://antirez.com/news/75 http://antirez.com/news/75

jmzbond 13y ago

Doesn't this article seem to contradict itself? It cites superforecasters' amazing ability, then talks about how better forecasting is the result of averaging the noise from a crowd of diverse opinions...

dean 13y ago
That's what jumped out at me as well. First the article extols how the Wisdom of Crowds is smarter than the experts. Then it goes on to say how some of the crowd are superforecasters, (essentially experts), and the rest of the crowd makes no difference. It would be interesting to see how small the group of superforecasters would need to get before their accuracy falls below that of the CIA analysts.
- jmzbond 13y ago
  Good test!

anigbrowl 13y ago

Inaccurate headline. Citizenship is not required for participation.

lifeisstillgood 13y ago

So could I suggest that CIA forecasts are affected by internal politics, perceived needs of political establishment, and afflicted by biases in top secret material that cannot be easily counter-balanced. Whereas if they put these super-forecasters in the CIA and said "if you dont predict what i want its off to Afghanistan with you" they will soon regress to the mean.

ballard 13y ago

The Wikipedia page on prediction markets storied history and interesting bit. Curious that USM had a prototype that was shutdown for being a "terrorism futures market." http://en.wikipedia.org/wiki/Prediction_market http://en.wikipedia.org/wiki/Prediction_market

easy_rider 13y ago

I sometimes have good runs in blackjack.

SolarNet 13y ago

This reminds me of the future predicted in Rainbows End by Vernor Vinge. Where he explores the ability to spin up pools of analysts like a cloud server.

ronaldx 13y ago

I did exactly this in high school: I set up an experiment to find the people best at guessing Zener cards picked at random. I surveryed 100 people and the best 5 did notably better than the others. I dubbed these people "clairvoyant". (Spoiler: on retesting, they weren't, but then again I only followed this process once...). Wisdom of the crowds will work just fine for anything modestly obvious, and it will fail for any black swan events that intelligence actually needs to be aware of - if the predictors even get asked about those.

prestadige 13y ago

It seems to me that all professional experts are subject to groupthink. However, could anonymous lay forecasters still make such excellent predictions without those groups in place? Anyway, I think all this argues in favor of Robin Hanson's concept of "Futarchy": http://hanson.gmu.edu/futarchy.html http://hanson.gmu.edu/futarchy.html

pserwylo 13y ago

The issue of crowd sourcing answers to questions is a very interesting one. I've recently been reading research in this area with a colleague, and it is far more involved than the article makes it sound. If you are interested, one of the first people to work on the problem were Dawid & Skene [0] back in 1979. They were trying to solve the issue of clinicians talking to patients and asking how they are. Each time they asked, the patient may give a slightly different answer. The goal was to estimate the "true" answer from the number of various "noisy" answers (sound familiar after reading the OP?) The general intuition for most future research into the topic is that if you tend to agree with others more often than not, your score goes up. If your score is higher, your answers are trusted more. Then there are a whole bunch of interesting additions to this principle, such as factoring in question difficulty [1] (e.g. questions which have the most disagreement can be considered more difficult, and hence you may want to ask more people about these questions to get a clearer picture). It is also helpful to seed certain questions for which the answer is "known" (may or may not be possible, depending on the project). Then you can use the answers to these questions to help score people even more accurately. An even more interesting approach is to take into account people who answer adversarially, that is, they always answer incorrectly (either due to malicious intent, or they consistently misinterpret the questions in a backward fasion). If you have somebody who answers incorrectly 90% of the time, then they provide almost exactly the same information as someone who answers correctly 90% of the time. All you need to do is flip their vote once you've identified this. The whole field is very facinating, and there seems to be more and more publications coming out with interesting approaches to the problem. [0] http://scholar.google.com/scholar?cites=15953840037809938294&as_sdt=5,39&sciodt=0,39&hl=en http://scholar.google.com/scholar?cites=15953840037809938294... [1] http://scholar.google.com/scholar?cluster=14409115865258228521&hl=en&as_sdt=0,39 http://scholar.google.com/scholar?cluster=144091158652582285...

mtrn 13y ago

Hearing the article, I felt thrown into PKD novel. One of the recurring theme in his work are precogs, normal people with special abilities to forecast the future. Or also to mispredict the future very accurately. Large corporations use groups of precogs to plan marketing campaigns, product designs, etc.

blazespin 13y ago

Where's the API?

allochthon 13y ago
It might be possible to put something together using Amazon's Mechanical Turk.

grn 13y ago

See also http://www.johndcook.com/blog/2014/03/21/independent-decision-making/ http://www.johndcook.com/blog/2014/03/21/independent-decisio...

secfirstmd 13y ago

I noticed this phenomenon alot when I used to work at Intrade.