2 ms·

Can what Nate Silver be doing be considered science? He's dealing with terribly sketchy data. Response rates to political polls in America are south of 10%, an

Can what Nate Silver be doing be considered science? He's dealing with terribly sketchy data. Response rates to political polls in America are south of 10%, and there's no proof that the portion of the population willing to have a conversation about their political preferences with a stranger is representative of the population as a whole. He's also dealing with incomplete data. The 'likely voter' screen pollsters use to determine who's actually going to the polls isn't always revealed, and it varies from organization to organization. Polling is a cash-strapped industry and some polls' likely-voter screens are much more porous, cutting down on staff required and voter contacts needed to generate a 'statistically-significant' sample. But details of the likely-voter screens and response rates aren't always available. His results are also completely unverifiable. Let's say his final prediction is 80-20 in favor of one candidate, but the other candidate wins. Well, his model did say that'll happen one time in five, so you can't really criticize it. The appropriate amount of precision to conclude with, given all of the fuzzy inputs, is something like 'well, we can't tell who's going to win - it's close, although this one candidate's chances look slightly better'. It's not 'Candidate A's chances improved from 74.6% yesterday to 76.5% today'. That's what makes Nate Silver so irritating - he doesn't know any more than the journalists, but he claims he does.

smackfu 14y ago
>His results are also completely unverifiable. Let's say his final prediction is 80-20 in favor of one candidate, but the other candidate wins. Well, his model did say that'll happen one time in five, so you can't really criticize it. That's a good point. He is using percentage chances everywhere but then running only one trial (election).
- defen 14y ago
  Clearly a wager is the way to solve this problem (Bayesian interpretation of probability). Is it legal to gamble on an election result? They could use the following payouts - J represents a regular journalist using 50/50 odds, N represents Nate Silver using 75/25 Obama/Romney: Romney win: J: +1000 N: -1000 Obama win: J: -600 N: +600 With those payouts each side has an expected value of +200 dollars for this wager, using the specified odds they believe to be true, so both should be willing to take it.
  - Evbn 14y ago
    Imtrade and trade sports do this. It is also trivial to simulate with play money over many elections and races.
- halostatue 14y ago
  This is, from my recollection, an incorrect assessment of Silver’s methodology. He runs (continuous?) Monte Carlo simulations over his inputs, and when he says that there's an 80-20 chance in favour of candidate O, he's saying that 80% of the simulations resulted in a win for candidate O, not candidate R.
brown9-2 14y ago
I don't think Silver claims to have data or knowledge that is better than what is available to the public. What he does different - and his isn't the only model built off of polling data, nor are 538's probabilities that different than other models - is just trying to decipher all the available polling data to reduce the noise and focus on the signal. I also don't think he sees himself as being in the business of picking a winner, as in saying "I believe X will win tomorrow". One thing you might be missing is that his predictions are verifiable at the state level - a result where he predicted the overall winner but only got 30 of 50 states right versus getting 49 out of 50 states right are pretty different. Polling aggregation and models are as much "science" as the practice of polling is in the first place.
- ryanwaggoner 14y ago
  One thing you might be missing is that his predictions are verifiable at the state level - a result where he predicted the overall winner but only got 30 of 50 states right versus getting 49 out of 50 states right are pretty different. This really isn't true, because something like 35-40 of the states aren't even remotely up for grabs and anyone intelligent could predict them after 15 mins of looking at recent polls and previous election results. So he's trying to pick winners for 10-15 states where the race is closer, but for probably half of those, it's really not that close and you'd have a pretty good chance of getting them right if you just eyeballed it. So now we're talking about 5-7 really tight swing states, and you're basically saying that predicting 44/50 states is very different than predicting 49/50. Not so sure that it is, especially given that it was a 50/50 choice for each of those states, and he did it for one election. Also, lots of pundits and bloggers made predictions in 2008; you'd expect someone to be mostly right. Fooled by Randomness and all that. For what it's worth, I think Silver probably is right and Obama will win (though I'd prefer he didn't), but his methods do lean towards being relatively unverifiable on a short timeline.
  - brown9-2 14y ago
    but his methods do lean towards being relatively unverifiable on a short timeline. Perhaps. But I think a lot of his critics fall into this trap of thinking that what Silver thinks is that he is offering some sort of revolutionary and perfect prediction, or that he thinks he is doing some sort of groundbreaking science. He's just trying to make a prediction based on all of the available information. If election predictions are inherently "relatively unverifiable", Nate Silver isn't capable of changing that - just working with what he is given (and doing a great job of explaining all of the ins and outs of this stuff to the layman).
localhost3000 14y ago
The guy made a model that he (and the nytimes) think is reasonable. The model takes inputs and spits out a probability. As the inputs change slightly so does the output. He's well within his right to publish these changes. What's the problem? That you just can't accept the validity of day-to-day basis-point changes in 'chances of something happening' when the inputs might be dirty? Can't you just say 'oh, that's nice' and take it as a directional signal without allowing it to irritate you? I'll take a thoughtful statistical analysis any day over the words of a single television journalist giving me second-hand information from a 'source within the campaign' especially when any campaign insider has an absurdly strong incentive to manufacture public perception in his candidates favor.
shardling 14y ago
>The appropriate amount of precision to conclude with, given all of the fuzzy inputs, is something like 'well, we can't tell who's going to win - it's close, although this one candidate's chances look slightly better'. It's not 'Candidate A's chances improved from 74.6% yesterday to 76.5% today'. Your post shows a horrible misunderstanding of how this all works. When you have sketchy data, you don't just disregard it -- it's still telling you something and the role of statistics here is to extract, from amidst all the noise, that information. Just because something is hard doesn't mean it's not worth the attempt. >His results are also completely unverifiable. Kind of. But how his methodology performs over all, over a wide range of elections, is verifiable. Wikipeida mentions this: >The accuracy of his November 2008 presidential election predictions—he correctly predicted the winner of 49 of the 50 states—won Silver further attention and commendation. The only state he missed was Indiana, which went for Barack Obama by 1%. He also correctly predicted the winner of all 35 Senate races that year. Whether his models perform like this consistently is a very real way that the methodology can be verified.
mikeash 14y ago
You're forgetting that he predicts the specific numbers of the outcome, not just the result. When he predicts that candidate A will win with X electoral votes, and the result is very close to X, that's a substantial verification of his prediction, especially if it repeats several elections in a row.
andrewljohnson 14y ago
A couple of points: 1) His model is easy to verify... wait for the elections to be over, and see how much his predictions correlate with reality. He predicts a great many races, and the results speak for themselves. "The accuracy of his November 2008 presidential election predictions—he correctly predicted the winner of 49 of the 50 states—won Silver further attention and commendation. The only state he missed was Indiana, which went for Barack Obama by 1%. He also correctly predicted the winner of all 35 Senate races that year." http://en.wikipedia.org/wiki/Nate_Silver http://en.wikipedia.org/wiki/Nate_Silver 2) Also, his model accounts for the response rate figures you cite - that was actually the point of his most recent blog post. The reason Nate gives Romney any chance at all to win is because his model predicts that there is less than 1/5 chance that the polls are systematically biased (based on data since 1968). He thinks that if he's wrong, it's exactly for the reason you say - that those who respond to the polls are a small group that don't represent the population. He has a large enough sample size to eliminate sampling error, and we're close enough to election day to discount error due to polls being a snapshot in time. So there is only systematic bias left to discuss... "So why, then, do we have Mr. Obama as “only” an 83.7 percent favorite to win the Electoral College, and not close to 100 percent? This is because of the other potential sources of error in polling." And that error is simply that polls don't reflect reality, and he thinks that is about a 15% chance. http://fivethirtyeight.blogs.nytimes.com/2012/11/03/nov-2-for-romney-to-win-state-polls-must-be-statistically-biased/ http://fivethirtyeight.blogs.nytimes.com/2012/11/03/nov-2-fo...
- streptomycin 14y ago
  His model is easy to verify... wait for the elections to be over, and see how much his predictions correlate with reality. FWIW, he got a bunch of congressional races wrong in 2010.
  - jfc 14y ago
    What you just posted is misleading. Silver's model gave the GOP a 2 in 3 chance of winning the House in 2010, predicting a net gain of 45-50 House seats, and the majority - http://fivethirtyeight.blogs.nytimes.com/2010/09/10/g-o-p-has-2-in-3-chance-of-taking-house-model-forecasts/ http://fivethirtyeight.blogs.nytimes.com/2010/09/10/g-o-p-ha...

smackfu 14y ago

>His results are also completely unverifiable. Let's say his final prediction is 80-20 in favor of one candidate, but the other candidate wins. Well, his model did say that'll happen one time in five, so you can't really criticize it. That's a good point. He is using percentage chances everywhere but then running only one trial (election).

defen 14y ago
Clearly a wager is the way to solve this problem (Bayesian interpretation of probability). Is it legal to gamble on an election result? They could use the following payouts - J represents a regular journalist using 50/50 odds, N represents Nate Silver using 75/25 Obama/Romney: Romney win: J: +1000 N: -1000 Obama win: J: -600 N: +600 With those payouts each side has an expected value of +200 dollars for this wager, using the specified odds they believe to be true, so both should be willing to take it.
- Evbn 14y ago
  Imtrade and trade sports do this. It is also trivial to simulate with play money over many elections and races.
halostatue 14y ago
This is, from my recollection, an incorrect assessment of Silver’s methodology. He runs (continuous?) Monte Carlo simulations over his inputs, and when he says that there's an 80-20 chance in favour of candidate O, he's saying that 80% of the simulations resulted in a win for candidate O, not candidate R.

brown9-2 14y ago

I don't think Silver claims to have data or knowledge that is better than what is available to the public. What he does different - and his isn't the only model built off of polling data, nor are 538's probabilities that different than other models - is just trying to decipher all the available polling data to reduce the noise and focus on the signal. I also don't think he sees himself as being in the business of picking a winner, as in saying "I believe X will win tomorrow". One thing you might be missing is that his predictions are verifiable at the state level - a result where he predicted the overall winner but only got 30 of 50 states right versus getting 49 out of 50 states right are pretty different. Polling aggregation and models are as much "science" as the practice of polling is in the first place.

ryanwaggoner 14y ago
One thing you might be missing is that his predictions are verifiable at the state level - a result where he predicted the overall winner but only got 30 of 50 states right versus getting 49 out of 50 states right are pretty different. This really isn't true, because something like 35-40 of the states aren't even remotely up for grabs and anyone intelligent could predict them after 15 mins of looking at recent polls and previous election results. So he's trying to pick winners for 10-15 states where the race is closer, but for probably half of those, it's really not that close and you'd have a pretty good chance of getting them right if you just eyeballed it. So now we're talking about 5-7 really tight swing states, and you're basically saying that predicting 44/50 states is very different than predicting 49/50. Not so sure that it is, especially given that it was a 50/50 choice for each of those states, and he did it for one election. Also, lots of pundits and bloggers made predictions in 2008; you'd expect someone to be mostly right. Fooled by Randomness and all that. For what it's worth, I think Silver probably is right and Obama will win (though I'd prefer he didn't), but his methods do lean towards being relatively unverifiable on a short timeline.
- brown9-2 14y ago
  but his methods do lean towards being relatively unverifiable on a short timeline. Perhaps. But I think a lot of his critics fall into this trap of thinking that what Silver thinks is that he is offering some sort of revolutionary and perfect prediction, or that he thinks he is doing some sort of groundbreaking science. He's just trying to make a prediction based on all of the available information. If election predictions are inherently "relatively unverifiable", Nate Silver isn't capable of changing that - just working with what he is given (and doing a great job of explaining all of the ins and outs of this stuff to the layman).

localhost3000 14y ago

The guy made a model that he (and the nytimes) think is reasonable. The model takes inputs and spits out a probability. As the inputs change slightly so does the output. He's well within his right to publish these changes. What's the problem? That you just can't accept the validity of day-to-day basis-point changes in 'chances of something happening' when the inputs might be dirty? Can't you just say 'oh, that's nice' and take it as a directional signal without allowing it to irritate you? I'll take a thoughtful statistical analysis any day over the words of a single television journalist giving me second-hand information from a 'source within the campaign' especially when any campaign insider has an absurdly strong incentive to manufacture public perception in his candidates favor.

shardling 14y ago

>The appropriate amount of precision to conclude with, given all of the fuzzy inputs, is something like 'well, we can't tell who's going to win - it's close, although this one candidate's chances look slightly better'. It's not 'Candidate A's chances improved from 74.6% yesterday to 76.5% today'. Your post shows a horrible misunderstanding of how this all works. When you have sketchy data, you don't just disregard it -- it's still telling you something and the role of statistics here is to extract, from amidst all the noise, that information. Just because something is hard doesn't mean it's not worth the attempt. >His results are also completely unverifiable. Kind of. But how his methodology performs over all, over a wide range of elections, is verifiable. Wikipeida mentions this: >The accuracy of his November 2008 presidential election predictions—he correctly predicted the winner of 49 of the 50 states—won Silver further attention and commendation. The only state he missed was Indiana, which went for Barack Obama by 1%. He also correctly predicted the winner of all 35 Senate races that year. Whether his models perform like this consistently is a very real way that the methodology can be verified.

mikeash 14y ago

You're forgetting that he predicts the specific numbers of the outcome, not just the result. When he predicts that candidate A will win with X electoral votes, and the result is very close to X, that's a substantial verification of his prediction, especially if it repeats several elections in a row.

andrewljohnson 14y ago

A couple of points: 1) His model is easy to verify... wait for the elections to be over, and see how much his predictions correlate with reality. He predicts a great many races, and the results speak for themselves. "The accuracy of his November 2008 presidential election predictions—he correctly predicted the winner of 49 of the 50 states—won Silver further attention and commendation. The only state he missed was Indiana, which went for Barack Obama by 1%. He also correctly predicted the winner of all 35 Senate races that year." http://en.wikipedia.org/wiki/Nate_Silver http://en.wikipedia.org/wiki/Nate_Silver 2) Also, his model accounts for the response rate figures you cite - that was actually the point of his most recent blog post. The reason Nate gives Romney any chance at all to win is because his model predicts that there is less than 1/5 chance that the polls are systematically biased (based on data since 1968). He thinks that if he's wrong, it's exactly for the reason you say - that those who respond to the polls are a small group that don't represent the population. He has a large enough sample size to eliminate sampling error, and we're close enough to election day to discount error due to polls being a snapshot in time. So there is only systematic bias left to discuss... "So why, then, do we have Mr. Obama as “only” an 83.7 percent favorite to win the Electoral College, and not close to 100 percent? This is because of the other potential sources of error in polling." And that error is simply that polls don't reflect reality, and he thinks that is about a 15% chance. http://fivethirtyeight.blogs.nytimes.com/2012/11/03/nov-2-for-romney-to-win-state-polls-must-be-statistically-biased/ http://fivethirtyeight.blogs.nytimes.com/2012/11/03/nov-2-fo...

streptomycin 14y ago
His model is easy to verify... wait for the elections to be over, and see how much his predictions correlate with reality. FWIW, he got a bunch of congressional races wrong in 2010.
- jfc 14y ago
  What you just posted is misleading. Silver's model gave the GOP a 2 in 3 chance of winning the House in 2010, predicting a net gain of 45-50 House seats, and the majority - http://fivethirtyeight.blogs.nytimes.com/2010/09/10/g-o-p-has-2-in-3-chance-of-taking-house-model-forecasts/ http://fivethirtyeight.blogs.nytimes.com/2010/09/10/g-o-p-ha...