6 ms·

Wrong Tomorrow - holding pundits accountable for predictions

93 points by gabrielroth 18y ago

msluyter 18y ago
Are there any categories besides right and wrong? I'd imagine that the success of some predictions is open to interpretation and/or ambiguous enough to defy easy categorization. A perfect example would be "Podcasting will be a factor in the 2006 congressional elections in the US." How much counts as "a factor?" Any amount > 0? How can we be sure one way or the other about this one, without some extensive analysis? At the very least, I think some explanation of how the determination is made should be required.
- Semiapies 18y ago
  The "WRONG" ones have some commentary: http://wrongtomorrow.com/predictions/26 http://wrongtomorrow.com/predictions/26
  - msluyter 18y ago
    Not all of them. For example: http://wrongtomorrow.com/predictions/47 http://wrongtomorrow.com/predictions/47
    - Semiapies 18y ago
      Good eye. (Not that I can think much of any counter-example, there's still no explanation.)
sethg 18y ago
I have this mental list of "things I would love to see and am too lazy to actually implement right now and maybe someone else will come up with the same idea and spare me the work". And now I can cross another item off of that list.
- deleted 18y ago
  [deleted]
- smanek 18y ago
  Yep, I just wish people could bet on these predictions too. Unfortunately, that is illegal in the US ...
  - ashot 18y ago
    virtual currency
    - smanek 18y ago
      That defeats the whole point, unless the virtual currency is worth something in the real world - at the very least considerable social currency or (ideally) it should be exchangeable for real goods and services. People bet differently when they have something real at stake - ever tried to play poker with no money behind the chips? It's a completely different (and completely boring) game. It's the reason the market instantly knew who was responsible for the Challenger crash (when it took Richard Feynman and the rest of genius scientists on the panel months to even have a plausible idea).
      - gamache 18y ago
        I'd love to see a reference on that Challenger remark -- seriously.
        smanek 18y ago
        http://www.scribd.com/doc/8142600/the-Stock-Market-Reaction-to-the-Challenger-Crash http://www.scribd.com/doc/8142600/the-Stock-Market-Reaction-... Sorry for the scribd link. It's the first copy I found that wasn't behind a pay wall. The gist is that within a few hours of the Challenger crash the stock for the company responsible for the component that failed went down far more than the stocks for other companies involved in building challenger. Interestingly, the equity value the stock lost (~200 Million USD) is almost exactly the same as the estimated total loss in profit (fines, lawsuits, updates to safety standards, lost contracts, etc).
snewe 18y ago
If you peruse the Bloomberg wire and keep track of stock analyst predictions, this site could grow into a "forecast analyzer" of stock "experts." Of course, someone needs to login and edit whether the prediction was correct.
- HSO 18y ago
  Actually, I remember reading sth about a hedge fund that employed just that approach. They had every analyst or broker send in their predictions or collected them from somewhere and kept track of who did well and who did not. The better someone did (the article did not say which measure the fund used), the more business the fund would send their way (in terms of order flow), betting on their recommendations to go long or short. Presumably, their portfolio would have been optimized to account for forecast accuracy, expected gains, and some measure of risk. The difficult question here would be which measure to take as "forecast accuracy", and that also touches on the site. How wrong is wrong, and how right is right? You would need some measure of dispersion to judge whether someone was right. If there is a lot of noise in some variable (say the oil price), then basically anyone will have been right sometime when the forecast horizon is long enough. On the other hand, you couldn't really say someone was "wrong" in his forecast if he said X and, by the measure of dispersion used, the actual number lies within reasonable bounds around X. Another question I already mailed the site's creator is how the variables are actually defined and tracked. Yet another question is whether and how the forecasters can adjust their forecasts. Obviously, you should be able to update your 5y forecast if tomorrow something happens that topples your assumptions. It's totally legit to do that, and I would argue such a mechanism is the real point of the otherwise pointless forecasting game. EDIT: There is one more question I find interesting. The measure by which forecast quality is judged should also somehow account for time dimension. Concretely, if the process you are tracking conforms 9 times out of 10 with your forecast path and then "explodes" or "collapses", are you good or bad? How far does it have to collapse or explode that you are "bad", how is the situation if the granularity is finer and you can observe 100 times instead of 10 (so the process is 98 or 99 times fine and then explodes)?
  - joshu 18y ago
    One hedge fund? This is a fairly standard strategy. Analysts publish their predictions. Services like First Call aggregate them, etc.
davidw 18y ago
Seems to be down at the moment... and now it's ok. Weird. I agree completely with sethg... I had this idea floating around, and it's nice to see it. Do make it easier to see some statistics, and see wrong/right predictions, as that's a bit more interesting for the casual browser. BTW, it's really slow again.
vaksel 18y ago
For some reason the #s in authors, don't match up with the actual author profile.
smanek 18y ago
Reminds me of overcoming bias' stuff. For example: http://www.overcomingbias.com/2009/02/who-are-macro-experts.html http://www.overcomingbias.com/2009/02/who-are-macro-experts.... We really need this sort of thing to take off ... Unfortunately, the public hates them. When DARPA created a prediction market (PAM) there was so much public outrage that the head of the agency was fired.
- adamhowell 18y ago
  The public hates prediction markets that try and predict terrorist attacks. Ones that make Scoble look like an ass are something else entirely.
cjlars 18y ago
I love this in concept, but somehow feel like it'd be better established as some sort of data blog. Perhaps you could find some aspiring young writer out there willing to work for equity?
banned_man 18y ago
Of all those, I tend to think T. Boone Pickens is the one who's most likely to turn out right. The question is, if someone turns out to be right, does he get removed from the list?
acgourley 18y ago
I, and I think lots of others, all have had this on a todo list. I think the reason they never seem to get built or take off is that there is such a lag between their establishment and their payoff. I assume the trick is to have some dedicated submitters for a year, and then to really push the interesting right/wrong answers into the right social news sites. That should get them over the hump. I hope this site manages to do that. Bookmarked.
sam_in_nyc 18y ago
Very, very cool. Kind of like the Obama promise-keeper website that I thought was cool, but then promptly forgot about. I'd like to suggest a couple of features, without which I probably will promptly forget this site as well: 1) E-mail or alert me when a prediction is verified correct, or expires, whichever comes first. 2) Mirror the URL at which the prediction is made. Great work launching the site, and best of luck to you.
Eliezer 18y ago
This is extraordinarily important work to see done. Whether or not anyone is bothering to keep track of successful predictions affects the whole quality of predictions that will be publicly made, which affects the sanity of the whole society. Mod parent up and link to this project on your blog.
ivankirigin 18y ago
Related: http://www.longbets.org/ http://www.longbets.org/

93 points by gabrielroth 18y ago

Are there any categories besides right and wrong? I'd imagine that the success of some predictions is open to interpretation and/or ambiguous enough to defy easy categorization. A perfect example would be "Podcasting will be a factor in the 2006 congressional elections in the US." How much counts as "a factor?" Any amount > 0? How can we be sure one way or the other about this one, without some extensive analysis? At the very least, I think some explanation of how the determination is made should be required.

Semiapies 18y ago
The "WRONG" ones have some commentary: http://wrongtomorrow.com/predictions/26 http://wrongtomorrow.com/predictions/26
- msluyter 18y ago
  Not all of them. For example: http://wrongtomorrow.com/predictions/47 http://wrongtomorrow.com/predictions/47
  - Semiapies 18y ago
    Good eye. (Not that I can think much of any counter-example, there's still no explanation.)

sethg 18y ago

I have this mental list of "things I would love to see and am too lazy to actually implement right now and maybe someone else will come up with the same idea and spare me the work". And now I can cross another item off of that list.

deleted 18y ago
[deleted]
smanek 18y ago
Yep, I just wish people could bet on these predictions too. Unfortunately, that is illegal in the US ...
- ashot 18y ago
  virtual currency
  - smanek 18y ago
    That defeats the whole point, unless the virtual currency is worth something in the real world - at the very least considerable social currency or (ideally) it should be exchangeable for real goods and services. People bet differently when they have something real at stake - ever tried to play poker with no money behind the chips? It's a completely different (and completely boring) game. It's the reason the market instantly knew who was responsible for the Challenger crash (when it took Richard Feynman and the rest of genius scientists on the panel months to even have a plausible idea).
    - gamache 18y ago
      I'd love to see a reference on that Challenger remark -- seriously.
      - smanek 18y ago
        http://www.scribd.com/doc/8142600/the-Stock-Market-Reaction-to-the-Challenger-Crash http://www.scribd.com/doc/8142600/the-Stock-Market-Reaction-... Sorry for the scribd link. It's the first copy I found that wasn't behind a pay wall. The gist is that within a few hours of the Challenger crash the stock for the company responsible for the component that failed went down far more than the stocks for other companies involved in building challenger. Interestingly, the equity value the stock lost (~200 Million USD) is almost exactly the same as the estimated total loss in profit (fines, lawsuits, updates to safety standards, lost contracts, etc).

snewe 18y ago

If you peruse the Bloomberg wire and keep track of stock analyst predictions, this site could grow into a "forecast analyzer" of stock "experts." Of course, someone needs to login and edit whether the prediction was correct.

HSO 18y ago
Actually, I remember reading sth about a hedge fund that employed just that approach. They had every analyst or broker send in their predictions or collected them from somewhere and kept track of who did well and who did not. The better someone did (the article did not say which measure the fund used), the more business the fund would send their way (in terms of order flow), betting on their recommendations to go long or short. Presumably, their portfolio would have been optimized to account for forecast accuracy, expected gains, and some measure of risk. The difficult question here would be which measure to take as "forecast accuracy", and that also touches on the site. How wrong is wrong, and how right is right? You would need some measure of dispersion to judge whether someone was right. If there is a lot of noise in some variable (say the oil price), then basically anyone will have been right sometime when the forecast horizon is long enough. On the other hand, you couldn't really say someone was "wrong" in his forecast if he said X and, by the measure of dispersion used, the actual number lies within reasonable bounds around X. Another question I already mailed the site's creator is how the variables are actually defined and tracked. Yet another question is whether and how the forecasters can adjust their forecasts. Obviously, you should be able to update your 5y forecast if tomorrow something happens that topples your assumptions. It's totally legit to do that, and I would argue such a mechanism is the real point of the otherwise pointless forecasting game. EDIT: There is one more question I find interesting. The measure by which forecast quality is judged should also somehow account for time dimension. Concretely, if the process you are tracking conforms 9 times out of 10 with your forecast path and then "explodes" or "collapses", are you good or bad? How far does it have to collapse or explode that you are "bad", how is the situation if the granularity is finer and you can observe 100 times instead of 10 (so the process is 98 or 99 times fine and then explodes)?
- joshu 18y ago
  One hedge fund? This is a fairly standard strategy. Analysts publish their predictions. Services like First Call aggregate them, etc.

davidw 18y ago

Seems to be down at the moment... and now it's ok. Weird. I agree completely with sethg... I had this idea floating around, and it's nice to see it. Do make it easier to see some statistics, and see wrong/right predictions, as that's a bit more interesting for the casual browser. BTW, it's really slow again.

vaksel 18y ago

For some reason the #s in authors, don't match up with the actual author profile.

smanek 18y ago

Reminds me of overcoming bias' stuff. For example: http://www.overcomingbias.com/2009/02/who-are-macro-experts.html http://www.overcomingbias.com/2009/02/who-are-macro-experts.... We really need this sort of thing to take off ... Unfortunately, the public hates them. When DARPA created a prediction market (PAM) there was so much public outrage that the head of the agency was fired.

adamhowell 18y ago
The public hates prediction markets that try and predict terrorist attacks. Ones that make Scoble look like an ass are something else entirely.

cjlars 18y ago

I love this in concept, but somehow feel like it'd be better established as some sort of data blog. Perhaps you could find some aspiring young writer out there willing to work for equity?

banned_man 18y ago

Of all those, I tend to think T. Boone Pickens is the one who's most likely to turn out right. The question is, if someone turns out to be right, does he get removed from the list?

acgourley 18y ago

I, and I think lots of others, all have had this on a todo list. I think the reason they never seem to get built or take off is that there is such a lag between their establishment and their payoff. I assume the trick is to have some dedicated submitters for a year, and then to really push the interesting right/wrong answers into the right social news sites. That should get them over the hump. I hope this site manages to do that. Bookmarked.

sam_in_nyc 18y ago

Very, very cool. Kind of like the Obama promise-keeper website that I thought was cool, but then promptly forgot about. I'd like to suggest a couple of features, without which I probably will promptly forget this site as well: 1) E-mail or alert me when a prediction is verified correct, or expires, whichever comes first. 2) Mirror the URL at which the prediction is made. Great work launching the site, and best of luck to you.

Eliezer 18y ago

This is extraordinarily important work to see done. Whether or not anyone is bothering to keep track of successful predictions affects the whole quality of predictions that will be publicly made, which affects the sanity of the whole society. Mod parent up and link to this project on your blog.

ivankirigin 18y ago

Related: http://www.longbets.org/ http://www.longbets.org/