3 ms·

I think it's interesting to talk about the failure of polls to predict the magnitude of the result. Here are the results of the final polls from six major polli

I think it's interesting to talk about the failure of polls to predict the magnitude of the result. Here are the results of the final polls from six major polling organizations, taken 1-3 days before the vote Organization Yes No Number* --------------------------------------- Opinium 0.479 0.521 1087 ICM 0.477 0.523 1010 Panelbase 0.474 0.526 954 Survation 0.473 0.527 910 YouGov 0.474 0.526 3043 Ipsos Mori 0.474 0.526 951 --------------------------------------- TOTAL 0.475 0.525 7955 * excluding undecided voters On one hand they were accurate in that they forecast a "no" result with high probability - the proportion of "no" voters is more than four standard errors away from 50%; the standard error being s.e. = sqrt(0.475 * 0.525 / 7955) = 0.006 On the other hand, the final proportion of "no" voters was 55.3%, which is more than four standard errors away from the forecast value of 52.5%. So the polls were significantly underestimating the proportion of "no" voters out there. A similar effect was seen in the Quebec independence referendum, where the polls actually predicted a slight advantage for "yes" but the final result was a decisive victory for "no".

mike_hearn 12y ago
http://en.wikipedia.org/wiki/Shy_Tory_Factor http://en.wikipedia.org/wiki/Shy_Tory_Factor
- nknighthb 12y ago
  A related question I'd be curious about is the likely voter issue. In the US, left-wingers are less likely to actually go vote than right-wingers. Pollsters try to correct for this with a "likely voter" model (which varies by pollster, but tries to discover how likely a respondent actually is to vote). If you poll everyone eligible to vote, you get left-wing results. If you poll registered voters, it pulls a bit right. If you poll voters that match a reasonable "likely voter" model, you'll get results that are even further right and tend to be closer to the reality of votes actually cast. Does this dichotomy exist in the UK and/or Scotland specifically, and if so, do pollsters there try to correct for it?
  - adaml_623 12y ago
    I read that a problem the pollsters were having was simply that they didn't have any precedents. There was a referendum last century asking a different question but that wasn't that helpful. Essentially you can't make up a model without observations and opinion polls don't count
    - soneil 12y ago
      This is my understanding of the issue too. Polls don't just ask the question. Each cycle they take their input, the actual results, and refine their model. Over iterations the model should get more and more accurate. This event is a one-off. No cycles, no iterations, leaving a model which is essentially a best-guess.
  - M2Ys4U 12y ago
    Some pollsters ask how likely one is to turn out to vote. Amongst those that do, they weight responses differently. Some will only report those who are 10/10, 9/10 and 8/10 likely to vote in their topline figures. Some will multiply the likeliness with the voting intention, so that a 10/10 likely voter would count as a whole vote, a 9/10 likely voter as 9/10ths of a vote and so on. The UK Polling Report[0] - run by the guy in charge of YouGov, if I recall correctly - is a very useful site for explaining UK polling oddities! [0] http://ukpollingreport.co.uk/ http://ukpollingreport.co.uk/
adaml_623 12y ago
I'm not sure because I don't know where you've sourced your data but where are the figures for undecided voters? That was always a large number and explains why your s.e. is too small.
- throwaway283719 12y ago
  That's a good point, and I've now corrected for that, but it makes very little difference to the top line result (that the polls underestimated the support for "no" by a large margin). The proportion of undecided voters in these polls was about 7.1%. Source is http://whatscotlandthinks.org/opinion-polls http://whatscotlandthinks.org/opinion-polls
jiggy2011 12y ago
My guess is that a lot of people like the idea of an independent Scotland but are frightened of the reality and bottled it on the way to the polling station.
- philjohn 12y ago
  http://www.thedailymash.co.uk/news/international/salmond-votes-no-2014091890806 http://www.thedailymash.co.uk/news/international/salmond-vot... ;)
UVB-76 12y ago
Independence was the high-risk option. I don't think it's surprising that people are more risk-averse on polling day, when their vote actually matters.
oliwarner 12y ago
And there were notes that went along with these figures. Certainly YouGov's. They said that if correct, their figures were likely going to be multiple-points out in favour of No because of various social factors: - "No" is fundamentally negative, people don't like to be negative - Supporting the Conservative party is seen as violently offensive anywhere north Sheffield. People that do are less likely to give their opinion in public. Given the lack of precedence, these are factors they couldn't adjust for.

mike_hearn 12y ago

http://en.wikipedia.org/wiki/Shy_Tory_Factor http://en.wikipedia.org/wiki/Shy_Tory_Factor

nknighthb 12y ago
A related question I'd be curious about is the likely voter issue. In the US, left-wingers are less likely to actually go vote than right-wingers. Pollsters try to correct for this with a "likely voter" model (which varies by pollster, but tries to discover how likely a respondent actually is to vote). If you poll everyone eligible to vote, you get left-wing results. If you poll registered voters, it pulls a bit right. If you poll voters that match a reasonable "likely voter" model, you'll get results that are even further right and tend to be closer to the reality of votes actually cast. Does this dichotomy exist in the UK and/or Scotland specifically, and if so, do pollsters there try to correct for it?
- adaml_623 12y ago
  I read that a problem the pollsters were having was simply that they didn't have any precedents. There was a referendum last century asking a different question but that wasn't that helpful. Essentially you can't make up a model without observations and opinion polls don't count
  - soneil 12y ago
    This is my understanding of the issue too. Polls don't just ask the question. Each cycle they take their input, the actual results, and refine their model. Over iterations the model should get more and more accurate. This event is a one-off. No cycles, no iterations, leaving a model which is essentially a best-guess.
- M2Ys4U 12y ago
  Some pollsters ask how likely one is to turn out to vote. Amongst those that do, they weight responses differently. Some will only report those who are 10/10, 9/10 and 8/10 likely to vote in their topline figures. Some will multiply the likeliness with the voting intention, so that a 10/10 likely voter would count as a whole vote, a 9/10 likely voter as 9/10ths of a vote and so on. The UK Polling Report[0] - run by the guy in charge of YouGov, if I recall correctly - is a very useful site for explaining UK polling oddities! [0] http://ukpollingreport.co.uk/ http://ukpollingreport.co.uk/

adaml_623 12y ago

I'm not sure because I don't know where you've sourced your data but where are the figures for undecided voters? That was always a large number and explains why your s.e. is too small.

throwaway283719 12y ago
That's a good point, and I've now corrected for that, but it makes very little difference to the top line result (that the polls underestimated the support for "no" by a large margin). The proportion of undecided voters in these polls was about 7.1%. Source is http://whatscotlandthinks.org/opinion-polls http://whatscotlandthinks.org/opinion-polls

jiggy2011 12y ago

My guess is that a lot of people like the idea of an independent Scotland but are frightened of the reality and bottled it on the way to the polling station.

philjohn 12y ago
http://www.thedailymash.co.uk/news/international/salmond-votes-no-2014091890806 http://www.thedailymash.co.uk/news/international/salmond-vot... ;)

UVB-76 12y ago

Independence was the high-risk option. I don't think it's surprising that people are more risk-averse on polling day, when their vote actually matters.

oliwarner 12y ago

And there were notes that went along with these figures. Certainly YouGov's. They said that if correct, their figures were likely going to be multiple-points out in favour of No because of various social factors: - "No" is fundamentally negative, people don't like to be negative - Supporting the Conservative party is seen as violently offensive anywhere north Sheffield. People that do are less likely to give their opinion in public. Given the lack of precedence, these are factors they couldn't adjust for.