10 ms·

This isn't just about having trouble with "edge cases", it's a fallacy to even think statistics are useful for predicting outcomes from such a small group where

This isn't just about having trouble with "edge cases", it's a fallacy to even think statistics are useful for predicting outcomes from such a small group where individuals have so much freedom to determine the outcome. Statistics may generate interesting facts after an event but for populations of people as small as two soccer teams there's no way they can predict individuals' actions. Maybe it works better for baseball because each individual has a sufficiently narrow range of potential impacts, but in soccer each person can do relatively whatever they want at any time.

yummyfajitas 12y ago
The fact that variation is high does not mean statistics doesn't work. Nate Silver's model never claimed to predict individual behavior.
- bioinformatics 12y ago
  No, it didn't predict individual behaviour, but it tried to predict a global outcome, which is based on the summation of 22 (plus subs) individual behaviours/actions. And that what made it fail in this specific case.
  - alttab 12y ago
    Statistics is used when a more accurate model cannot be found. This essentially assumes that there are a lot of unmeasured variables. So yes, statistically this was accounted for.
  - ajross 12y ago
    Can you rephrase that in a way that makes sense? You appear to be saying that because sports teams are small that no probabilities can be assigned to the results of their games. There's literally a whole gambling industry that exists precisely because such a thing is possible. Likewise you appear to be arguing against insurance as a business model. My guess is that you're misinterpreting Silver's work and assuming he was "predicting" that Brazil "would" win. He wasn't, and didn't claim to. He said Brazil was more likely to win, and gave numbers to that effect (that in this case happened to be wildly wrong).
    - peter-row 12y ago
      I think he's saying it's a souped-up poisson model, not an agent based model. It's not trying to predict individual behavior. A poisson model is probably valid, but it should perhaps be refined a bit. Maybe it should account for a systematic drift if the whole team plays badly (or a vital player, like the goal keeper, has a bad day). But it's probably really hard to calibrate that, and it's probably not too relevant unless you want to model major upsets.
      - gbhn 12y ago
        Wouldn't that refinement basically be identifying the tails as quite a bit fatter than you'd thought before? But in that case, the tails are probably substantially fatter for everyone, and so the numbers 538 is quoting may not be far from the "real" ones. My reading of this 538 World Cup analysis is that the error bars are taken as being very large, and the relative numbers are mostly interesting as a means of comparing the impacts of various players and other factors in the model. Obviously there can still be major flaws in this method, chief among them being that the method tends to atomize the team's contributions. But that's not a horrible first pass, and still can yield some interesting insights.
        peter-row 12y ago
        Yes and yes. It's still probably the biggest upset ever, but not as unlikely as their model suggests.
zecho 12y ago
I think you're conflating statistics and probability.
- gumballhead 12y ago
  Right, but I think that's what he's saying. Statistics can only describe the average outcome, which is only interesting when evaluating how extraordinary the actual outcome was.
  - yummyfajitas 12y ago
    Statistics can only describe the average outcome... Um, no. Please go read any statistics tutorial containing the word "standard deviation" to see one example of why you are wrong.
    - saganus 12y ago
      I think he meant average in the sense of generality, not in the sense of the average function.
crag 12y ago
It's hard to predict a small group. Reliably. The masses are easy. Wall Street and Madison Ave do it all the time. But in football (soccer); which is fast paced, unpredictable and where an individual can have major influence you have a cluster fuck. Anyone's prediction is just a guess. Now baseball is another matter. A much more rigid game. Easier to predict. You know, we see this all the time in politics. How many upsets over the years? The models need to be thrown out every year. But it's expensive. So the US media clings to their same old models and is "shocked" when an upset occurs. And then they usually search so fast for a reason, they miss the real reason in the process. Hollywood has been missing the mark too. They also need to throw out their models and start over. BADLY. Anyway I sometimes wonder just how far away we really are from Asimov's "Psychohistory ".
- prostoalex 12y ago
  It seems that it wasn't so much about the rigidness of the game, as much as it was Neymar missing from the field. Brazil is very superstar-oriented, with excellent offense followed by pretty good midfielders, followed by okay defense, followed by bad goalkeeper. Predictions on baseball teams probably fall through when a star player is injured.
  - crag 12y ago
    " Brazil is very superstar-oriented" Which is exactly why their current models failed. The need to be flexible. Lots of people lost money. Maybe it was done on purpose? The models I mean. Cause I bet someone made a fortune. Someone short-selling the game? PS: Or maybe Neymar's injury was "managed". Humm.. a nice juicy conspiracy theory. Stranger things have happened in the game before.
    - leorocky 12y ago
      > Lots of people lost money. Maybe it was done on purpose? The models I mean. Cause I bet someone made a fortune. Someone short-selling the game? The considerable amount of collusion required across fifa, the teams and players, multiple publishing industries, organizations, countries and cultures to have sports betting and publications like fivethirtyeight make shit up (to the point that they created a very detailed scoring system for their predictions, all for the purpose of lying about a lop sided 7-1 game) would have to be considerably large. I think the the better, and frankly incredibly blindingly obvious, way to look at this is that this was just an unusual game that nobody expected and you can maybe take a break today from being suspicious about the whole world being manipulated by a nefarious few who conspire to create every fucking notable turn in history to screw you or someone else over.
rjtavares 12y ago
Statistics could be, and I'd say will be, a great tool for soccer. We're only just starting in this massivly complex sport, and unfortunately no comprehensive data is available to the public. I'm pretty sure in four years we will be discussing metrics than don't even exist today.
hooande 12y ago
If you were correct then soccer oddsmakers would have gone bankrupt a long time ago. The fact is that the average outcome occurs the majority of the time. High profile surprises like this cause people to question statistics, but this line of thinking is based on a fundamental misunderstanding. The purpose of statistical inference isn't to predict the outcome of every event, but to be right slightly more often than not. This world cup has been remarkably predictable on the whole. The favored team won the overwhelming majority of matches in group stages and in the rounds of 16 and 8. And most of favored teams qualified for the Cup in the first place. Soccer isn't unpredictable because of the impact of individuals or size of the teams or the nature of the game. It's just as predictable as any other sport, which is why bookmakers are able to set neutral odds that attract an equal amount of betting on both sides. Of course there will be highly unpredictable results, and they will cause people to question the common sense idea that the most likely thing usually happens. Five Thirty Eight is a bit on their heels here because they have chosen to give themselves the job of explaining a predictable world to human minds that aren't built for understanding statistics. It's a difficult job and I don't envy them. This won't be the last time we see them talking about edge cases and outliers instead of just saying "The odds are right most of the time, which implies that sometimes they will be very wrong."
- Ntrails 12y ago
  My understanding is that bookies work on the risk neutral probability, not on any predictive power.
  - jarvuschris 12y ago
    ^^ this. Statistics help them average out their wins/losses across an entire season. But if the bookie's business model was to predict only the final game of each season and make odds for it they would _not_ be in business for long
- jsnell 12y ago
  538 predicted the winner in the group stage correctly in 28 of 48 matches. For teams advancing out of the group stage, they got 7 out of 16 right. Maybe the results were "predictable", but only if you assume that people actually making predictions for some reason took no advantage of that predictability. (Source: http://www.sportingintelligence.com/2014/06/24/bankers-and-bookies-oust-fifa-as-better-bets-for-world-cup-forecasts-240601/ http://www.sportingintelligence.com/2014/06/24/bankers-and-b...)

yummyfajitas 12y ago

The fact that variation is high does not mean statistics doesn't work. Nate Silver's model never claimed to predict individual behavior.

bioinformatics 12y ago
No, it didn't predict individual behaviour, but it tried to predict a global outcome, which is based on the summation of 22 (plus subs) individual behaviours/actions. And that what made it fail in this specific case.
- alttab 12y ago
  Statistics is used when a more accurate model cannot be found. This essentially assumes that there are a lot of unmeasured variables. So yes, statistically this was accounted for.
- ajross 12y ago
  Can you rephrase that in a way that makes sense? You appear to be saying that because sports teams are small that no probabilities can be assigned to the results of their games. There's literally a whole gambling industry that exists precisely because such a thing is possible. Likewise you appear to be arguing against insurance as a business model. My guess is that you're misinterpreting Silver's work and assuming he was "predicting" that Brazil "would" win. He wasn't, and didn't claim to. He said Brazil was more likely to win, and gave numbers to that effect (that in this case happened to be wildly wrong).
  - peter-row 12y ago
    I think he's saying it's a souped-up poisson model, not an agent based model. It's not trying to predict individual behavior. A poisson model is probably valid, but it should perhaps be refined a bit. Maybe it should account for a systematic drift if the whole team plays badly (or a vital player, like the goal keeper, has a bad day). But it's probably really hard to calibrate that, and it's probably not too relevant unless you want to model major upsets.
    - gbhn 12y ago
      Wouldn't that refinement basically be identifying the tails as quite a bit fatter than you'd thought before? But in that case, the tails are probably substantially fatter for everyone, and so the numbers 538 is quoting may not be far from the "real" ones. My reading of this 538 World Cup analysis is that the error bars are taken as being very large, and the relative numbers are mostly interesting as a means of comparing the impacts of various players and other factors in the model. Obviously there can still be major flaws in this method, chief among them being that the method tends to atomize the team's contributions. But that's not a horrible first pass, and still can yield some interesting insights.
      - peter-row 12y ago
        Yes and yes. It's still probably the biggest upset ever, but not as unlikely as their model suggests.

zecho 12y ago

I think you're conflating statistics and probability.

gumballhead 12y ago
Right, but I think that's what he's saying. Statistics can only describe the average outcome, which is only interesting when evaluating how extraordinary the actual outcome was.
- yummyfajitas 12y ago
  Statistics can only describe the average outcome... Um, no. Please go read any statistics tutorial containing the word "standard deviation" to see one example of why you are wrong.
  - saganus 12y ago
    I think he meant average in the sense of generality, not in the sense of the average function.

crag 12y ago

It's hard to predict a small group. Reliably. The masses are easy. Wall Street and Madison Ave do it all the time. But in football (soccer); which is fast paced, unpredictable and where an individual can have major influence you have a cluster fuck. Anyone's prediction is just a guess. Now baseball is another matter. A much more rigid game. Easier to predict. You know, we see this all the time in politics. How many upsets over the years? The models need to be thrown out every year. But it's expensive. So the US media clings to their same old models and is "shocked" when an upset occurs. And then they usually search so fast for a reason, they miss the real reason in the process. Hollywood has been missing the mark too. They also need to throw out their models and start over. BADLY. Anyway I sometimes wonder just how far away we really are from Asimov's "Psychohistory ".

prostoalex 12y ago
It seems that it wasn't so much about the rigidness of the game, as much as it was Neymar missing from the field. Brazil is very superstar-oriented, with excellent offense followed by pretty good midfielders, followed by okay defense, followed by bad goalkeeper. Predictions on baseball teams probably fall through when a star player is injured.
- crag 12y ago
  " Brazil is very superstar-oriented" Which is exactly why their current models failed. The need to be flexible. Lots of people lost money. Maybe it was done on purpose? The models I mean. Cause I bet someone made a fortune. Someone short-selling the game? PS: Or maybe Neymar's injury was "managed". Humm.. a nice juicy conspiracy theory. Stranger things have happened in the game before.
  - leorocky 12y ago
    > Lots of people lost money. Maybe it was done on purpose? The models I mean. Cause I bet someone made a fortune. Someone short-selling the game? The considerable amount of collusion required across fifa, the teams and players, multiple publishing industries, organizations, countries and cultures to have sports betting and publications like fivethirtyeight make shit up (to the point that they created a very detailed scoring system for their predictions, all for the purpose of lying about a lop sided 7-1 game) would have to be considerably large. I think the the better, and frankly incredibly blindingly obvious, way to look at this is that this was just an unusual game that nobody expected and you can maybe take a break today from being suspicious about the whole world being manipulated by a nefarious few who conspire to create every fucking notable turn in history to screw you or someone else over.

rjtavares 12y ago

Statistics could be, and I'd say will be, a great tool for soccer. We're only just starting in this massivly complex sport, and unfortunately no comprehensive data is available to the public. I'm pretty sure in four years we will be discussing metrics than don't even exist today.

hooande 12y ago

If you were correct then soccer oddsmakers would have gone bankrupt a long time ago. The fact is that the average outcome occurs the majority of the time. High profile surprises like this cause people to question statistics, but this line of thinking is based on a fundamental misunderstanding. The purpose of statistical inference isn't to predict the outcome of every event, but to be right slightly more often than not. This world cup has been remarkably predictable on the whole. The favored team won the overwhelming majority of matches in group stages and in the rounds of 16 and 8. And most of favored teams qualified for the Cup in the first place. Soccer isn't unpredictable because of the impact of individuals or size of the teams or the nature of the game. It's just as predictable as any other sport, which is why bookmakers are able to set neutral odds that attract an equal amount of betting on both sides. Of course there will be highly unpredictable results, and they will cause people to question the common sense idea that the most likely thing usually happens. Five Thirty Eight is a bit on their heels here because they have chosen to give themselves the job of explaining a predictable world to human minds that aren't built for understanding statistics. It's a difficult job and I don't envy them. This won't be the last time we see them talking about edge cases and outliers instead of just saying "The odds are right most of the time, which implies that sometimes they will be very wrong."

Ntrails 12y ago
My understanding is that bookies work on the risk neutral probability, not on any predictive power.
- jarvuschris 12y ago
  ^^ this. Statistics help them average out their wins/losses across an entire season. But if the bookie's business model was to predict only the final game of each season and make odds for it they would _not_ be in business for long
jsnell 12y ago
538 predicted the winner in the group stage correctly in 28 of 48 matches. For teams advancing out of the group stage, they got 7 out of 16 right. Maybe the results were "predictable", but only if you assume that people actually making predictions for some reason took no advantage of that predictability. (Source: http://www.sportingintelligence.com/2014/06/24/bankers-and-bookies-oust-fifa-as-better-bets-for-world-cup-forecasts-240601/ http://www.sportingintelligence.com/2014/06/24/bankers-and-b...)