4 ms·

As fun as the story of Carl Diggler is, his work is statistically insignificant. The podcast Reply All (which you should be listening to) asked a professor who

by apozem 10y ago

As fun as the story of Carl Diggler is, his work is statistically insignificant. The podcast Reply All (which you should be listening to) asked a professor who studies the science of predictions whether the Diggler guys are onto something [1]. They are not. The professor wanted some kind of statistically significant proof that they were any better than Paul, the octopus who successfully predicted an astonishingly high number of soccer matches by random chance [2]. Say what you will about 538, but at least they care about not using anecdotal or imagined evidence. At least they're rigorous with their methodology. They're also actually good at it, unlike most pundits [3]. [1]: https://gimletmedia.com/episode/58-earth-pony/ https://gimletmedia.com/episode/58-earth-pony/ [2]: https://en.wikipedia.org/wiki/Paul_the_Octopus https://en.wikipedia.org/wiki/Paul_the_Octopus [3]: https://en.wikipedia.org/wiki/FiveThirtyEight#Final_projections_of_2008_elections https://en.wikipedia.org/wiki/FiveThirtyEight#Final_projecti...

huac 10y ago
The point is that even with 'rigorous methodology' and 'evidence' 538 doesn't do as well as Diggler. Predicting elections is, for the most part, an easy task. Polls are usually good enough to get some picture of the election at hand. One major shortcoming of a primarily evidence-based approach is how to call races without much polling (small sample size). Silver has consistently refused to cover these races at all. But Silver commits two sins that you ignore: he wants to show off his 'accuracy' and he wants to improve on the polls. These two problems kind of intertwine - the 538 'polls-plus,' which incorporates predictors Silver thinks are significant e.g. endorsements or Nate's gut, performs worse than the polls forecast, which is just a weighted average of recent state polls. What kind of rigorous methodology allows for two different predictions and then picking the best one, after the fact? What kind of rigorous methodology results in a prediction method that's shown to be worse than reading the newspaper, time and time again? Where's the evidence behind endorsements working this year? As a side note, Philip Tetlock is the Wharton professor, and Dan Gardner (interviewed here) is some dude.
- neaden 10y ago
  The part you are ignoring is that there are many potential "Digglers" out there. If Diggler hadn't done well, you wouldn't have heard about it. Buy maybe you would have heard about some other guy who predicted the races correctly based on what kind of treat his horse wanted to eat that day.
  - huac 10y ago
    That's fair, and I wouldn't bet the bank on his predictions moving forward, but it's valid to consider what Diggler means for Silver's method. Diggler and Cafe.com are important as critics of the media at large. When what passes for journalism is some dudes making multiple predictions and asking you to choose the one that turns out to be closest to correct, that should be lampooned! Silver wrote a faux mea culpa earlier this year about how he was wrong about Trump and Sanders and so on. My takeaway was that he apologized for inserting his own opinions where he claimed to use data. Isn't that what Diggler does?
  - pdeuchler 10y ago
    That's exactly the point the Diggler parody is making about Nate Silver
  - smcl 10y ago
    If Diggler had bombed I most certainly would have heard about it - I was reading about him before all this 538 stuff, when he had a Hostage crisis in Russia[1]. However if that had happened we would just have a different but equally humourous narrative, perhaps blaming his family court issues, or some other nefarious plot :D Really you're overthinking it, and you shouldn't feel the need to protect Nate Silver et al, they're completely unaffected by Carl (afaik they haven't even acknowledged his "existence") so don't worry about it [1] = http://www.cafe.com/breaking-missing-journalist-carl-diggler-emerges-in-hostage-video-on-russian-state-television/ http://www.cafe.com/breaking-missing-journalist-carl-diggler...

by apozem 10y ago

huac 10y ago

The point is that even with 'rigorous methodology' and 'evidence' 538 doesn't do as well as Diggler. Predicting elections is, for the most part, an easy task. Polls are usually good enough to get some picture of the election at hand. One major shortcoming of a primarily evidence-based approach is how to call races without much polling (small sample size). Silver has consistently refused to cover these races at all. But Silver commits two sins that you ignore: he wants to show off his 'accuracy' and he wants to improve on the polls. These two problems kind of intertwine - the 538 'polls-plus,' which incorporates predictors Silver thinks are significant e.g. endorsements or Nate's gut, performs worse than the polls forecast, which is just a weighted average of recent state polls. What kind of rigorous methodology allows for two different predictions and then picking the best one, after the fact? What kind of rigorous methodology results in a prediction method that's shown to be worse than reading the newspaper, time and time again? Where's the evidence behind endorsements working this year? As a side note, Philip Tetlock is the Wharton professor, and Dan Gardner (interviewed here) is some dude.

neaden 10y ago
The part you are ignoring is that there are many potential "Digglers" out there. If Diggler hadn't done well, you wouldn't have heard about it. Buy maybe you would have heard about some other guy who predicted the races correctly based on what kind of treat his horse wanted to eat that day.
- huac 10y ago
  That's fair, and I wouldn't bet the bank on his predictions moving forward, but it's valid to consider what Diggler means for Silver's method. Diggler and Cafe.com are important as critics of the media at large. When what passes for journalism is some dudes making multiple predictions and asking you to choose the one that turns out to be closest to correct, that should be lampooned! Silver wrote a faux mea culpa earlier this year about how he was wrong about Trump and Sanders and so on. My takeaway was that he apologized for inserting his own opinions where he claimed to use data. Isn't that what Diggler does?
- pdeuchler 10y ago
  That's exactly the point the Diggler parody is making about Nate Silver
- smcl 10y ago
  If Diggler had bombed I most certainly would have heard about it - I was reading about him before all this 538 stuff, when he had a Hostage crisis in Russia[1]. However if that had happened we would just have a different but equally humourous narrative, perhaps blaming his family court issues, or some other nefarious plot :D Really you're overthinking it, and you shouldn't feel the need to protect Nate Silver et al, they're completely unaffected by Carl (afaik they haven't even acknowledged his "existence") so don't worry about it [1] = http://www.cafe.com/breaking-missing-journalist-carl-diggler-emerges-in-hostage-video-on-russian-state-television/ http://www.cafe.com/breaking-missing-journalist-carl-diggler...