6 ms·

It's too bad there's no way to assign a universal "reputation" score to the various pundits out there so that readers can better assess the value of their publi

by papa 16y ago

It's too bad there's no way to assign a universal "reputation" score to the various pundits out there so that readers can better assess the value of their published predictions. Michael Pachter, for instance, has been wrong on a number of occasions (he predicted Sony would dominate the current console generation and Nintendo would languish and MSFT would be #2 -- at least he got the last part right). Like the iPad which many pundits couldn't figure out, the consumers never wrote off Netflix and that's pretty much all that matters.

kiba 16y ago
A database that list all pundits' specific prediction and give them a rating based on if they succeeded or not?
- papa 16y ago
  That could be a component. I can't put my finger on the other ones, and it was just a slapdash thought, but I'd just love a scorecard that maybe considered: predictive success, factual error rate/truthyness (e.g. do they make frequent bogus claims?, known biases and employer(s). It's not really realistic to comprehensively log all of this, of course. But it sure would help me assessing the relative quality of some sources, especially now that sources for news are so varied and fragmented. I suppose in the past the reputation of a publication served some of this purpose (i.e. WSJ or NYTimes having stellar reputations). I knew of a couple of databases for equity analyst performance like Starmine and Bulldog Research. So those do exist. Not sure if either are still around since I'm no longer in the finance space.
- tomjen3 16y ago
  The problem is that you have to take account how unlikely a given prediction would be in comming true - a person who predicts one in a million things but only gets it right one out of a thousand is still way better than somebody who predicts the results of coin-flips with 51% accuracy but in your system the latter would be ranked far higher.
  - joe_the_user 16y ago
    Indeed, You would have to give each prediction an "audacity factor", how unlikely it seems. The system would work in fashion where a 50% accuracy rate and a 30% one would actually be considered the same accuracy. Either you get higher than a 50-60% rate OR you get some of your predictions rated as having a high audacity factor.
    - Goosey 16y ago
      There is a very good system for handling this type of thing: Prediction Markets (http://en.wikipedia.org/wiki/Prediction_market http://en.wikipedia.org/wiki/Prediction_market) The basic idea of a prediction market is like a stock market for outcomes with an end point. If you make good predictions you will make 'money' and if your predictions are correct but go against the common idea your rewarded even more handsomely. Of course a true prediction market has a shifting price point where you can choose to sell your 'shares' at any point, yada yada, but I think the concept applies. The trick is you have to have the pundits participate in order to make it work. I suppose there could be some third-party that pulls their articles and 'ghost trades' for them? However I think the truth is that pundits DON'T WANT to be held accountable. Even if they are shown to be 'the most correct' they are still going to be wrong some of the time and those will always be used as examples to attack them and their views. Just look at another comment in this thread that tears into this pundit for his poor job predicting the Sony vs MSFT vs Nintendo war over this console generation. This is a SINGLE data point being used to try and dismiss his views. edit: For anyone interested in prediction markets check out Inkling (http://inklingmarkets.com/ http://inklingmarkets.com/). I don't work for them, but I just think they have done an awesome job of creating some useful software. I am trying to push for this to be adopted internally for creating 'confidence metrics' of how stable a build is going to be, or if a deadline is going to be hit.
- amirmc 16y ago
  I've seen such a website somewhere but can't find it. It was in relation to an apple prediction but there were plenty of others there too (inc political).
  - Teese 16y ago
    your probably thinking of http://wrongtomorrow.com/ http://wrongtomorrow.com/ which no longer seems to be up - and it never seemed to get much traction anyways.
    - amirmc 16y ago
      Yes, that's the one. Thanks from Google cache: Site: http://bit.ly/9JiRrz http://bit.ly/9JiRrz & FAQ: http://bit.ly/9cQeTb http://bit.ly/9cQeTb
dopeyman 16y ago
There's a project working on this very issue. Developers, check out http://nextny.jobthread.com/job/lead-developer-new-york-ny-united-7c77e4a0a5/?d=1&excluded_view=1 http://nextny.jobthread.com/job/lead-developer-new-york-ny-u...
opoloqo 16y ago
The Long Now Foundation put up longbets.org to encourage some kind of accountability for predictions.
brandnewlow 16y ago
Why couldn't it be a combo Wiki + Browser toolbar app? When you read a story on any of X sites, a popup appears in your browser informing you that the writer of this story has an entry on JournoWiki (or whatever), and that independent readers like you have deemed him/her "Full of Crap." The popup would then have a link to share something on Twitter of Facebook indicating whether you thought their latest story was better or worse than what people thought of them. That share would have a unique tag in it so it could be picked up by your app and sucked onto his/her profile page.
epe 16y ago
It appears to be down at the moment, but there's a site called Wrong Tomorrow (http://wrongtomorrow.com/ http://wrongtomorrow.com/) that seems to me like a good start in this direction. Here's the blog post announcing it: http://www.idlewords.com/2009/04/wrong_tomorrow.htm http://www.idlewords.com/2009/04/wrong_tomorrow.htm
sgoraya 16y ago
> the consumers never wrote off Netflix and that's pretty much all that matters. Not that I necessarily wrote off netflix, but I did jump ship to Blockbuster's online service for short while when it was first introduced. The ability to return the movies I got in the mail with any movie at their physical store was a pretty nice feature, in addition I got to rent one game per month as well. A couple of months later, blockbuster started changing their policies and I switched back to trusty netflix. When blockbuster first introduced their service I thought that netflix might be in trouble...but my unpublished prediction was unfounded! ;)
stinkytaco 16y ago
I think that pundits' predictions are largely colored by their own bias, much like political pundits. I have to admit that I'm a consumer and I still don't get the iPad. It's like an underpowered laptop that I have hold instead of letting it rest in my lap. What could the benefit possibly be? But... here I am, eating my proverbial hat. Netflix, on the other hand, I get. I had it early and people would say, "Why would you want to wait for your movie when you could get it at the video store?" I would explain that 1. I wanted movies I couldn't get at the store and 2. I generally had 2 or 3 going, so I wasn't without something. It's just a matter of what you value and then projecting those values onto the observed behaviors of a large population. If you see a bunch of people going to the river you might think that they are thirsty, but in reality they want to go swimming. That uncertainty makes predicting hard.

by papa 16y ago

kiba 16y ago

A database that list all pundits' specific prediction and give them a rating based on if they succeeded or not?

papa 16y ago
That could be a component. I can't put my finger on the other ones, and it was just a slapdash thought, but I'd just love a scorecard that maybe considered: predictive success, factual error rate/truthyness (e.g. do they make frequent bogus claims?, known biases and employer(s). It's not really realistic to comprehensively log all of this, of course. But it sure would help me assessing the relative quality of some sources, especially now that sources for news are so varied and fragmented. I suppose in the past the reputation of a publication served some of this purpose (i.e. WSJ or NYTimes having stellar reputations). I knew of a couple of databases for equity analyst performance like Starmine and Bulldog Research. So those do exist. Not sure if either are still around since I'm no longer in the finance space.
tomjen3 16y ago
The problem is that you have to take account how unlikely a given prediction would be in comming true - a person who predicts one in a million things but only gets it right one out of a thousand is still way better than somebody who predicts the results of coin-flips with 51% accuracy but in your system the latter would be ranked far higher.
- joe_the_user 16y ago
  Indeed, You would have to give each prediction an "audacity factor", how unlikely it seems. The system would work in fashion where a 50% accuracy rate and a 30% one would actually be considered the same accuracy. Either you get higher than a 50-60% rate OR you get some of your predictions rated as having a high audacity factor.
  - Goosey 16y ago
    There is a very good system for handling this type of thing: Prediction Markets (http://en.wikipedia.org/wiki/Prediction_market http://en.wikipedia.org/wiki/Prediction_market) The basic idea of a prediction market is like a stock market for outcomes with an end point. If you make good predictions you will make 'money' and if your predictions are correct but go against the common idea your rewarded even more handsomely. Of course a true prediction market has a shifting price point where you can choose to sell your 'shares' at any point, yada yada, but I think the concept applies. The trick is you have to have the pundits participate in order to make it work. I suppose there could be some third-party that pulls their articles and 'ghost trades' for them? However I think the truth is that pundits DON'T WANT to be held accountable. Even if they are shown to be 'the most correct' they are still going to be wrong some of the time and those will always be used as examples to attack them and their views. Just look at another comment in this thread that tears into this pundit for his poor job predicting the Sony vs MSFT vs Nintendo war over this console generation. This is a SINGLE data point being used to try and dismiss his views. edit: For anyone interested in prediction markets check out Inkling (http://inklingmarkets.com/ http://inklingmarkets.com/). I don't work for them, but I just think they have done an awesome job of creating some useful software. I am trying to push for this to be adopted internally for creating 'confidence metrics' of how stable a build is going to be, or if a deadline is going to be hit.
amirmc 16y ago
I've seen such a website somewhere but can't find it. It was in relation to an apple prediction but there were plenty of others there too (inc political).
- Teese 16y ago
  your probably thinking of http://wrongtomorrow.com/ http://wrongtomorrow.com/ which no longer seems to be up - and it never seemed to get much traction anyways.
  - amirmc 16y ago
    Yes, that's the one. Thanks from Google cache: Site: http://bit.ly/9JiRrz http://bit.ly/9JiRrz & FAQ: http://bit.ly/9cQeTb http://bit.ly/9cQeTb

dopeyman 16y ago

There's a project working on this very issue. Developers, check out http://nextny.jobthread.com/job/lead-developer-new-york-ny-united-7c77e4a0a5/?d=1&excluded_view=1 http://nextny.jobthread.com/job/lead-developer-new-york-ny-u...

opoloqo 16y ago

The Long Now Foundation put up longbets.org to encourage some kind of accountability for predictions.

brandnewlow 16y ago

Why couldn't it be a combo Wiki + Browser toolbar app? When you read a story on any of X sites, a popup appears in your browser informing you that the writer of this story has an entry on JournoWiki (or whatever), and that independent readers like you have deemed him/her "Full of Crap." The popup would then have a link to share something on Twitter of Facebook indicating whether you thought their latest story was better or worse than what people thought of them. That share would have a unique tag in it so it could be picked up by your app and sucked onto his/her profile page.

epe 16y ago

It appears to be down at the moment, but there's a site called Wrong Tomorrow (http://wrongtomorrow.com/ http://wrongtomorrow.com/) that seems to me like a good start in this direction. Here's the blog post announcing it: http://www.idlewords.com/2009/04/wrong_tomorrow.htm http://www.idlewords.com/2009/04/wrong_tomorrow.htm

sgoraya 16y ago

> the consumers never wrote off Netflix and that's pretty much all that matters. Not that I necessarily wrote off netflix, but I did jump ship to Blockbuster's online service for short while when it was first introduced. The ability to return the movies I got in the mail with any movie at their physical store was a pretty nice feature, in addition I got to rent one game per month as well. A couple of months later, blockbuster started changing their policies and I switched back to trusty netflix. When blockbuster first introduced their service I thought that netflix might be in trouble...but my unpublished prediction was unfounded! ;)

stinkytaco 16y ago

I think that pundits' predictions are largely colored by their own bias, much like political pundits. I have to admit that I'm a consumer and I still don't get the iPad. It's like an underpowered laptop that I have hold instead of letting it rest in my lap. What could the benefit possibly be? But... here I am, eating my proverbial hat. Netflix, on the other hand, I get. I had it early and people would say, "Why would you want to wait for your movie when you could get it at the video store?" I would explain that 1. I wanted movies I couldn't get at the store and 2. I generally had 2 or 3 going, so I wasn't without something. It's just a matter of what you value and then projecting those values onto the observed behaviors of a large population. If you see a bunch of people going to the river you might think that they are thirsty, but in reality they want to go swimming. That uncertainty makes predicting hard.