3 ms·

I am a little disappointed to see that the results of “super forecasters” in the economist and on the underlying Good Judgement open website does not present a

I am a little disappointed to see that the results of “super forecasters” in the economist and on the underlying Good Judgement open website does not present a 95% credible interval, or even a good old fashioned confidence interval. Would love to see results presented with the uncertainty quantified. Especially given that the yes/no questions are aggregated binarized predictions from what is almost certainly a collection of continuous models. A lot of information is lost between the people performing the analysis and either of these pages.

fddr 4y ago
They are giving probabilities for discrete events, which already captures their level of uncertainty. Probabilities of probabilities (i.e., a probability distribution of a probability) are not very useful concepts.
- alexilliamson 4y ago
  This would imply that the confidence interval around the coefficient in a logistic regression is not a very useful concept, which I don't think is true.
  - fddr 4y ago
    That it is a little different. There you are estimating a continuous parameter (which happens to be interpretable as a probability) and it makes sense to have a probability distribution over that. But if you are talking about whether a single discrete events will happen or not, a single number (the probability) already fully captures the uncertainty about it.
    - diab0lic 4y ago
      There is a lot of missing information in their probability of binary events presented. Presumably they polled N forecasters and are presenting an x/N prediction. The fact that each forecaster is estimating in a continuous space and then binarizing their result means that a lot of information has been lost. To look at an extreme example… were all the “yes” votes 95%+ certain and the “no” votes just under the line 49%? Or was it more like a bunch of no votes at 49% and a bunch of yes votes at 51%? Binarizing forecasts necessarily discards information. Aggregating a bunch of binary predictions into a percentage does no recapture said information, unfortunately.
- subroutine 4y ago
  It looks like they are simply providing the summary data to their multiple choice survey questions. Still, CI or SEM would not apply (there is no SEM for 80% of forecasters said 'yes'). These graphs are literally just telling you the percent of responders who picked a given answer to the corresponding question. edit- the longer I browse their website for the exact methodology, the less impressed I am with this group. The "Introducing the Superforecasters" section is so cringe. https://goodjudgment.com/about/ https://goodjudgment.com/about/
operator-name 4y ago
It's definitely an odd emission since the original research project used such a calculation. Metaculus, which uses a similar technique provides such a confidence interval, along with a nice history graph. As some wild speculation, I suspect that since the GJP only employs a handful of Superforcasters, the initial confidence intervals for these broad questions may be quite large. That's to be expected when predicting a year in advance, but publically admitting to have such a broad confidence interval is probably not very good for marketing.
- tedsanders 4y ago
  These events are binary. They will happen or not. A percentage prediction already incorporates the uncertainty. In what way would a 45-55% prediction be any different than a 50% prediction? The domain of outcomes is a set of size 2: (0, 1). There are no intermediate outcomes. Both predictions are functionally identical: {0: 50%, 1: 50%}. (Where confidence intervals over percentages makes more sense is estimating a parameter rather than a single event. E.g., if we flip this coin a bunch of times in series, I predict that the percentage of flips landing heads will be between 45% and 55%. Or next year I predict GDP will be between 1%-3%. Or I estimate the effect size of this ad campaign was -$2-$5 earned per dollar spent.) ((One sense in which a 45%-55% prediction on a binary event might have some semantic meaning is that it could signal a higher willingness to adjust if new evidence or information is brought to light. But that's quite different than a confidence interval.))

fddr 4y ago

They are giving probabilities for discrete events, which already captures their level of uncertainty. Probabilities of probabilities (i.e., a probability distribution of a probability) are not very useful concepts.

alexilliamson 4y ago
This would imply that the confidence interval around the coefficient in a logistic regression is not a very useful concept, which I don't think is true.
- fddr 4y ago
  That it is a little different. There you are estimating a continuous parameter (which happens to be interpretable as a probability) and it makes sense to have a probability distribution over that. But if you are talking about whether a single discrete events will happen or not, a single number (the probability) already fully captures the uncertainty about it.
  - diab0lic 4y ago
    There is a lot of missing information in their probability of binary events presented. Presumably they polled N forecasters and are presenting an x/N prediction. The fact that each forecaster is estimating in a continuous space and then binarizing their result means that a lot of information has been lost. To look at an extreme example… were all the “yes” votes 95%+ certain and the “no” votes just under the line 49%? Or was it more like a bunch of no votes at 49% and a bunch of yes votes at 51%? Binarizing forecasts necessarily discards information. Aggregating a bunch of binary predictions into a percentage does no recapture said information, unfortunately.
subroutine 4y ago
It looks like they are simply providing the summary data to their multiple choice survey questions. Still, CI or SEM would not apply (there is no SEM for 80% of forecasters said 'yes'). These graphs are literally just telling you the percent of responders who picked a given answer to the corresponding question. edit- the longer I browse their website for the exact methodology, the less impressed I am with this group. The "Introducing the Superforecasters" section is so cringe. https://goodjudgment.com/about/ https://goodjudgment.com/about/

operator-name 4y ago

It's definitely an odd emission since the original research project used such a calculation. Metaculus, which uses a similar technique provides such a confidence interval, along with a nice history graph. As some wild speculation, I suspect that since the GJP only employs a handful of Superforcasters, the initial confidence intervals for these broad questions may be quite large. That's to be expected when predicting a year in advance, but publically admitting to have such a broad confidence interval is probably not very good for marketing.

tedsanders 4y ago
These events are binary. They will happen or not. A percentage prediction already incorporates the uncertainty. In what way would a 45-55% prediction be any different than a 50% prediction? The domain of outcomes is a set of size 2: (0, 1). There are no intermediate outcomes. Both predictions are functionally identical: {0: 50%, 1: 50%}. (Where confidence intervals over percentages makes more sense is estimating a parameter rather than a single event. E.g., if we flip this coin a bunch of times in series, I predict that the percentage of flips landing heads will be between 45% and 55%. Or next year I predict GDP will be between 1%-3%. Or I estimate the effect size of this ad campaign was -$2-$5 earned per dollar spent.) ((One sense in which a 45%-55% prediction on a binary event might have some semantic meaning is that it could signal a higher willingness to adjust if new evidence or information is brought to light. But that's quite different than a confidence interval.))