4 ms·

A database that list all pundits' specific prediction and give them a rating based on if they succeeded or not?

by kiba 16y ago

papa 16y ago
That could be a component. I can't put my finger on the other ones, and it was just a slapdash thought, but I'd just love a scorecard that maybe considered: predictive success, factual error rate/truthyness (e.g. do they make frequent bogus claims?, known biases and employer(s). It's not really realistic to comprehensively log all of this, of course. But it sure would help me assessing the relative quality of some sources, especially now that sources for news are so varied and fragmented. I suppose in the past the reputation of a publication served some of this purpose (i.e. WSJ or NYTimes having stellar reputations). I knew of a couple of databases for equity analyst performance like Starmine and Bulldog Research. So those do exist. Not sure if either are still around since I'm no longer in the finance space.
tomjen3 16y ago
The problem is that you have to take account how unlikely a given prediction would be in comming true - a person who predicts one in a million things but only gets it right one out of a thousand is still way better than somebody who predicts the results of coin-flips with 51% accuracy but in your system the latter would be ranked far higher.
- joe_the_user 16y ago
  Indeed, You would have to give each prediction an "audacity factor", how unlikely it seems. The system would work in fashion where a 50% accuracy rate and a 30% one would actually be considered the same accuracy. Either you get higher than a 50-60% rate OR you get some of your predictions rated as having a high audacity factor.
  - Goosey 16y ago
    There is a very good system for handling this type of thing: Prediction Markets (http://en.wikipedia.org/wiki/Prediction_market http://en.wikipedia.org/wiki/Prediction_market) The basic idea of a prediction market is like a stock market for outcomes with an end point. If you make good predictions you will make 'money' and if your predictions are correct but go against the common idea your rewarded even more handsomely. Of course a true prediction market has a shifting price point where you can choose to sell your 'shares' at any point, yada yada, but I think the concept applies. The trick is you have to have the pundits participate in order to make it work. I suppose there could be some third-party that pulls their articles and 'ghost trades' for them? However I think the truth is that pundits DON'T WANT to be held accountable. Even if they are shown to be 'the most correct' they are still going to be wrong some of the time and those will always be used as examples to attack them and their views. Just look at another comment in this thread that tears into this pundit for his poor job predicting the Sony vs MSFT vs Nintendo war over this console generation. This is a SINGLE data point being used to try and dismiss his views. edit: For anyone interested in prediction markets check out Inkling (http://inklingmarkets.com/ http://inklingmarkets.com/). I don't work for them, but I just think they have done an awesome job of creating some useful software. I am trying to push for this to be adopted internally for creating 'confidence metrics' of how stable a build is going to be, or if a deadline is going to be hit.
amirmc 16y ago
I've seen such a website somewhere but can't find it. It was in relation to an apple prediction but there were plenty of others there too (inc political).
- Teese 16y ago
  your probably thinking of http://wrongtomorrow.com/ http://wrongtomorrow.com/ which no longer seems to be up - and it never seemed to get much traction anyways.
  - amirmc 16y ago
    Yes, that's the one. Thanks from Google cache: Site: http://bit.ly/9JiRrz http://bit.ly/9JiRrz & FAQ: http://bit.ly/9cQeTb http://bit.ly/9cQeTb

A database that list all pundits' specific prediction and give them a rating based on if they succeeded or not?

by kiba 16y ago

A database that list all pundits' specific prediction and give them a rating based on if they succeeded or not?

papa 16y ago

That could be a component. I can't put my finger on the other ones, and it was just a slapdash thought, but I'd just love a scorecard that maybe considered: predictive success, factual error rate/truthyness (e.g. do they make frequent bogus claims?, known biases and employer(s). It's not really realistic to comprehensively log all of this, of course. But it sure would help me assessing the relative quality of some sources, especially now that sources for news are so varied and fragmented. I suppose in the past the reputation of a publication served some of this purpose (i.e. WSJ or NYTimes having stellar reputations). I knew of a couple of databases for equity analyst performance like Starmine and Bulldog Research. So those do exist. Not sure if either are still around since I'm no longer in the finance space.

tomjen3 16y ago

The problem is that you have to take account how unlikely a given prediction would be in comming true - a person who predicts one in a million things but only gets it right one out of a thousand is still way better than somebody who predicts the results of coin-flips with 51% accuracy but in your system the latter would be ranked far higher.

joe_the_user 16y ago
Indeed, You would have to give each prediction an "audacity factor", how unlikely it seems. The system would work in fashion where a 50% accuracy rate and a 30% one would actually be considered the same accuracy. Either you get higher than a 50-60% rate OR you get some of your predictions rated as having a high audacity factor.
- Goosey 16y ago
  There is a very good system for handling this type of thing: Prediction Markets (http://en.wikipedia.org/wiki/Prediction_market http://en.wikipedia.org/wiki/Prediction_market) The basic idea of a prediction market is like a stock market for outcomes with an end point. If you make good predictions you will make 'money' and if your predictions are correct but go against the common idea your rewarded even more handsomely. Of course a true prediction market has a shifting price point where you can choose to sell your 'shares' at any point, yada yada, but I think the concept applies. The trick is you have to have the pundits participate in order to make it work. I suppose there could be some third-party that pulls their articles and 'ghost trades' for them? However I think the truth is that pundits DON'T WANT to be held accountable. Even if they are shown to be 'the most correct' they are still going to be wrong some of the time and those will always be used as examples to attack them and their views. Just look at another comment in this thread that tears into this pundit for his poor job predicting the Sony vs MSFT vs Nintendo war over this console generation. This is a SINGLE data point being used to try and dismiss his views. edit: For anyone interested in prediction markets check out Inkling (http://inklingmarkets.com/ http://inklingmarkets.com/). I don't work for them, but I just think they have done an awesome job of creating some useful software. I am trying to push for this to be adopted internally for creating 'confidence metrics' of how stable a build is going to be, or if a deadline is going to be hit.

amirmc 16y ago

I've seen such a website somewhere but can't find it. It was in relation to an apple prediction but there were plenty of others there too (inc political).

Teese 16y ago
your probably thinking of http://wrongtomorrow.com/ http://wrongtomorrow.com/ which no longer seems to be up - and it never seemed to get much traction anyways.
- amirmc 16y ago
  Yes, that's the one. Thanks from Google cache: Site: http://bit.ly/9JiRrz http://bit.ly/9JiRrz & FAQ: http://bit.ly/9cQeTb http://bit.ly/9cQeTb