3 ms·

For me the most frustrating part of predicting outcomes (elections, games, etc.) is that unless you're predicting with 100% confidence (which I've never seen),

For me the most frustrating part of predicting outcomes (elections, games, etc.) is that unless you're predicting with 100% confidence (which I've never seen), that no one can essentially call your prediction wrong. If a prediction model was consistently accurate, the creators would be breaking Vegas instead of publishing predictions on a blog.

klodolph 9y ago
It's not relevant and not interesting that you can't call an individual prediction "wrong". The article is just presenting a way to extend Elo to point spread. So it's better than win-loss Elo, not better than more complicated statistical models, which the authors explicitly state: > Our findings indicate the probabilistic predictions are highly accurate with a mean absolute error that is comparable to, albeit slightly worse than, Vegas betting lines. Meanwhile, tons of people are throwing ML models at sports bets, models that they often don't understand, and Vegas is keeping up with it by adjusting the payouts.
richard___ 9y ago
No one's breaking vegas because most games and payouts are designed probabilistically for the house to win
- joosters 9y ago
  I think they are referring to sports betting rather than games of chance. There’s no point calculating an elo rating for number 5 on the roulette wheel.
  - deleted 9y ago
    [deleted]
- deleted 9y ago
  [deleted]
JumpCrisscross 9y ago
> no one can essentially call your prediction wrong For a single match, no. Across a sample of predictions? Elementary. (If 60% of the games you give white a 90% chance of winning are, in fact, won by black, you don’t need a lot of games to determine your model is wrong.)
- trendia 9y ago
  There is an easy, quantifiable metric that uses this idea: the Brier score. Essentially, if you make 100 predictions, the Brier score will measure how well you are able to make precise predictions. If you always estimate the odds that X will win to be 50/50, you won't do nearly as well as someone who correctly predicts the outcome each time. https://en.m.wikipedia.org/wiki/Brier_score https://en.m.wikipedia.org/wiki/Brier_score
  - morelandjs 9y ago
    I didn't know about this, thanks. In the paper (long read I know), I use a modified quantile-quantile test to demonstrate that my predicted uncertainties are accurate. https://en.wikipedia.org/wiki/Q%E2%80%93Q_plot https://en.wikipedia.org/wiki/Q%E2%80%93Q_plot

klodolph 9y ago

It's not relevant and not interesting that you can't call an individual prediction "wrong". The article is just presenting a way to extend Elo to point spread. So it's better than win-loss Elo, not better than more complicated statistical models, which the authors explicitly state: > Our findings indicate the probabilistic predictions are highly accurate with a mean absolute error that is comparable to, albeit slightly worse than, Vegas betting lines. Meanwhile, tons of people are throwing ML models at sports bets, models that they often don't understand, and Vegas is keeping up with it by adjusting the payouts.

richard___ 9y ago

No one's breaking vegas because most games and payouts are designed probabilistically for the house to win

joosters 9y ago
I think they are referring to sports betting rather than games of chance. There’s no point calculating an elo rating for number 5 on the roulette wheel.
- deleted 9y ago
  [deleted]
deleted 9y ago
[deleted]

JumpCrisscross 9y ago

> no one can essentially call your prediction wrong For a single match, no. Across a sample of predictions? Elementary. (If 60% of the games you give white a 90% chance of winning are, in fact, won by black, you don’t need a lot of games to determine your model is wrong.)

trendia 9y ago
There is an easy, quantifiable metric that uses this idea: the Brier score. Essentially, if you make 100 predictions, the Brier score will measure how well you are able to make precise predictions. If you always estimate the odds that X will win to be 50/50, you won't do nearly as well as someone who correctly predicts the outcome each time. https://en.m.wikipedia.org/wiki/Brier_score https://en.m.wikipedia.org/wiki/Brier_score
- morelandjs 9y ago
  I didn't know about this, thanks. In the paper (long read I know), I use a modified quantile-quantile test to demonstrate that my predicted uncertainties are accurate. https://en.wikipedia.org/wiki/Q%E2%80%93Q_plot https://en.wikipedia.org/wiki/Q%E2%80%93Q_plot