5 ms·

> experts also participated in an online “prediction market,” trading shares that corresponded to studies, which paid out only if the given study was replicated

by _rpd 8y ago

> experts also participated in an online “prediction market,” trading shares that corresponded to studies, which paid out only if the given study was replicated. > Both approaches did well at predicting the outcome for individual studies, and they predicted an overall replication rate very close to the actual figure of 62%. I think the real headline here is that we should supplement peer review with replication prediction markets.

timr 8y ago
The real headline is that this is a more formalized version of what science already does. Science, as a process, naturally replicates studies that are worth replicating, and drops those that are not. The idea of "prediction markets" is a new-tech way of doing what scientists already do: decide where to invest their (limited) budgets to maximize long-term career success.
philipodonnell 8y ago
Is there any information on these prediction markets? Is there a reason only "experts" can participate? This would be an interesting application for ML side projects that are monetarily beneficial while also benefiting society as a whole.
- creaghpatr 8y ago
  The experts aren't necessarily credentialed experts as we would typically think of them. You can participate in forecasting tournaments like Good Judgement Project and if you score high enough you can participate in invite-only events that are in partnership with entities like DARPA. But the people come from all backgrounds- I got all this from the book Superforecasting by Philip Tetlock. Edit: got the title wrong
- njarboe 8y ago
  It is too bad that society does not support a robust system of prediction markets on everything that would be interested in knowing the truth about. I think the excuse is that it is too much like gambling. I would say there would definitely be some gambling going on in such markets, but we already allow them for much the more mundane reason of predicting the future time-discounted profits of a business. The whole country participates and has a good deal of its wealth tied up in them (ie. stock markets).

by _rpd 8y ago

timr 8y ago

The real headline is that this is a more formalized version of what science already does. Science, as a process, naturally replicates studies that are worth replicating, and drops those that are not. The idea of "prediction markets" is a new-tech way of doing what scientists already do: decide where to invest their (limited) budgets to maximize long-term career success.

philipodonnell 8y ago

Is there any information on these prediction markets? Is there a reason only "experts" can participate? This would be an interesting application for ML side projects that are monetarily beneficial while also benefiting society as a whole.

creaghpatr 8y ago
The experts aren't necessarily credentialed experts as we would typically think of them. You can participate in forecasting tournaments like Good Judgement Project and if you score high enough you can participate in invite-only events that are in partnership with entities like DARPA. But the people come from all backgrounds- I got all this from the book Superforecasting by Philip Tetlock. Edit: got the title wrong
njarboe 8y ago
It is too bad that society does not support a robust system of prediction markets on everything that would be interested in knowing the truth about. I think the excuse is that it is too much like gambling. I would say there would definitely be some gambling going on in such markets, but we already allow them for much the more mundane reason of predicting the future time-discounted profits of a business. The whole country participates and has a good deal of its wealth tied up in them (ie. stock markets).