6 ms·

Except that we don't know how well this 'focus group' forecasting performs relative to voluminous reporting by analysts at State, CIA, RAND, Congressional Resea

by rrggrr 12y ago

Except that we don't know how well this 'focus group' forecasting performs relative to voluminous reporting by analysts at State, CIA, RAND, Congressional Research, think tanks, etc. Often the difference between good analysis and bad is simply how successful the analyst is at getting their projections read by the 'right' people. Along the same lines, if policy makers could stop shopping for analysis that confirms their biases, much progress could finally be made in international and public policy.

jacques_chester 12y ago
> Except that we don't know how well this 'focus group' forecasting performs relative to voluminous reporting by analysts at State, CIA, RAND, Congressional Research, think tanks, etc. Actually, we do. Philip Tetlock, who is mentioned in the article, spent more than a decade collecting tens of thousands of predictions from geopolitical experts. He found that humans are terrible at making predictions about complex phenomena; typically we are stomped by simple linear models or probablistic random walks. Some humans were slightly better at predicting outcomes, but compared to simple models, they still sucked. It didn't matter what other variables were involved. Education, access to classified information, experience, seniority, nationality, profession ... none of them mattered. A Russian politics expert is about as likely to pick Putin's next move as a financial economist who reads the New York Times. Tetlock's book, Expert Political Judgement, is comprehensive. I reviewed it here: http://chester.id.au/2012/07/29/review-expert-political-judgement/ http://chester.id.au/2012/07/29/review-expert-political-judg...
- rrggrr 12y ago
  You don't understand how DC works. On a given issue analysis and projections are produced by numerous agencies. For example a Country desk at State may deliver an accurate estimate, but another group at state may water it down, or it may be stuffed by a competing estimate from CIA. Since its politically safer to bury your head in the sand than it is to fight for an estimate, the process is ripe for manipulation. All this concept does is pass the buck without - apparently - any comparison to all the analysis produced. Tl;dr - This solution speaks to the failings of the intel process and not the people.
  - jacques_chester 12y ago
    Tetlock's work was based on individuals giving unfiltered predictions.
  - deleted 12y ago
    [deleted]
pmorici 12y ago
I was thinking the same thing. It sounds like they just did a really good job of rooting out the people that were bad at the work and keeping the ones that were good around. In government I would be skeptical that they actually promote or fire people based on how good they are at the job.
- jrapdx3 12y ago
  It seems logical that keeping the people who did well at the task and dismissing those who didn't would be the way to find people with forecasting talent in a specific domain. It's the same principle as evolutionary selection. If I want to have a stand of only unusual purple flowers whereas the usual ones are red, then save seeds of the purple ones and plant those seeds and keep selecting. Maybe it could be called a purple flower meritocracy. It sure seems to be the case that many employers aren't distilling the most capable employees among those employed so an effective workforce doesn't evolve. One thing that bugged me a little was the hantavirus prediction. Seems the incidence hadn't changed much over time, I wonder how hard it was to predict accurately?
  - drunkpotato 12y ago
    Define a metric for "did well at the task". Whatever you define will be, at best, a close proxy to what you want. The downfall of an organization lies in that gap. Evolution fails spectacularly badly at matching our intuitions for what we want. A naive optimization process will optimize exactly for the cost function, at the expense of everything else in the universe. This is to say that what you propose is not that easy to accomplish when you get beyond the domain of words and into the implementation. I found this an entertaining read on why allowing an evolutionary process to optimize for you does not always lead to the results you expect: http://lesswrong.com/lw/l8/conjuring_an_evolution_to_serve_you/ http://lesswrong.com/lw/l8/conjuring_an_evolution_to_serve_y...
  - astazangasta 12y ago
    I'm not sure selection is a good way to think this particular problem, or management in general. Selection operates by replicating or destroying an immutable unit (a genetic variant). People cannot be replicated, only fired. Furthermore, people are not immutable; most importantly, they can learn from their mistakes, and they can be taught. Someone who fails at forecasting something correctly the first time might be someone who goes back, looks deeply at why they were wrong, and improves their method. Someone who succeeds might merely have gotten lucky. These people will not be identified correctly by a selection mechanism. Time and judgment, collaboration and the inculcation of expertise seem better tools in this regard.

by rrggrr 12y ago

jacques_chester 12y ago

> Except that we don't know how well this 'focus group' forecasting performs relative to voluminous reporting by analysts at State, CIA, RAND, Congressional Research, think tanks, etc. Actually, we do. Philip Tetlock, who is mentioned in the article, spent more than a decade collecting tens of thousands of predictions from geopolitical experts. He found that humans are terrible at making predictions about complex phenomena; typically we are stomped by simple linear models or probablistic random walks. Some humans were slightly better at predicting outcomes, but compared to simple models, they still sucked. It didn't matter what other variables were involved. Education, access to classified information, experience, seniority, nationality, profession ... none of them mattered. A Russian politics expert is about as likely to pick Putin's next move as a financial economist who reads the New York Times. Tetlock's book, Expert Political Judgement, is comprehensive. I reviewed it here: http://chester.id.au/2012/07/29/review-expert-political-judgement/ http://chester.id.au/2012/07/29/review-expert-political-judg...

rrggrr 12y ago
You don't understand how DC works. On a given issue analysis and projections are produced by numerous agencies. For example a Country desk at State may deliver an accurate estimate, but another group at state may water it down, or it may be stuffed by a competing estimate from CIA. Since its politically safer to bury your head in the sand than it is to fight for an estimate, the process is ripe for manipulation. All this concept does is pass the buck without - apparently - any comparison to all the analysis produced. Tl;dr - This solution speaks to the failings of the intel process and not the people.
- jacques_chester 12y ago
  Tetlock's work was based on individuals giving unfiltered predictions.
- deleted 12y ago
  [deleted]

pmorici 12y ago

I was thinking the same thing. It sounds like they just did a really good job of rooting out the people that were bad at the work and keeping the ones that were good around. In government I would be skeptical that they actually promote or fire people based on how good they are at the job.

jrapdx3 12y ago
It seems logical that keeping the people who did well at the task and dismissing those who didn't would be the way to find people with forecasting talent in a specific domain. It's the same principle as evolutionary selection. If I want to have a stand of only unusual purple flowers whereas the usual ones are red, then save seeds of the purple ones and plant those seeds and keep selecting. Maybe it could be called a purple flower meritocracy. It sure seems to be the case that many employers aren't distilling the most capable employees among those employed so an effective workforce doesn't evolve. One thing that bugged me a little was the hantavirus prediction. Seems the incidence hadn't changed much over time, I wonder how hard it was to predict accurately?
- drunkpotato 12y ago
  Define a metric for "did well at the task". Whatever you define will be, at best, a close proxy to what you want. The downfall of an organization lies in that gap. Evolution fails spectacularly badly at matching our intuitions for what we want. A naive optimization process will optimize exactly for the cost function, at the expense of everything else in the universe. This is to say that what you propose is not that easy to accomplish when you get beyond the domain of words and into the implementation. I found this an entertaining read on why allowing an evolutionary process to optimize for you does not always lead to the results you expect: http://lesswrong.com/lw/l8/conjuring_an_evolution_to_serve_you/ http://lesswrong.com/lw/l8/conjuring_an_evolution_to_serve_y...
- astazangasta 12y ago
  I'm not sure selection is a good way to think this particular problem, or management in general. Selection operates by replicating or destroying an immutable unit (a genetic variant). People cannot be replicated, only fired. Furthermore, people are not immutable; most importantly, they can learn from their mistakes, and they can be taught. Someone who fails at forecasting something correctly the first time might be someone who goes back, looks deeply at why they were wrong, and improves their method. Someone who succeeds might merely have gotten lucky. These people will not be identified correctly by a selection mechanism. Time and judgment, collaboration and the inculcation of expertise seem better tools in this regard.