3 ms·

On page 11 there is a mention of taking the result of self reporting (surveys) at their word. I’ve wondered about this issue not just in science but other situa

On page 11 there is a mention of taking the result of self reporting (surveys) at their word. I’ve wondered about this issue not just in science but other situations. For example political polling, data point in time surveys, census, etc. Without verification, what good is the data? And yet you often see such self reported data quoted by articles or papers as if it were factual.

fithisux 2y ago
It is a proxy.
tsimionescu 2y ago
For many types of data you'd like, there is no alternative. For example for political polls, especially early ones where you're asking people who they would vote for if so and so were running, what other tool could you use? Or how would you verify the results? Of course, given the weakness of the data, you have to remember to place very little trust in the results. Even so, "X% of people say they would vote for Y" can be an interesting information in itself, even if those people were lying.
- deleted 2y ago
  [deleted]
- graemep 2y ago
  Polls may or may not be carried out in a neutral way. A good many are deliberately biased, even more have known flaws that are glossed over. A humorous take on this I love (from the 80s BBC comedy Yes Minister, which is often scarily realistic): https://www.youtube.com/watch?v=G0ZZJXw4MTA https://www.youtube.com/watch?v=G0ZZJXw4MTA
  - tsimionescu 2y ago
    Oh, absolutely, even in more subtle, unintentional ways (order of responses, whether you offer a neutral option or not, etc). But that doesn't mean polling is entirely useless, or that you need to verify the responses in the way that the original seemed to suggest.
culebron21 2y ago
Pollsters know about this, and the rule of thumb is that you can reliably ask people of what they currently do routinely, almost daily. E.g. if you ask them who they voted 10 years ago, they might remember it wrong. If you ask what they're gonna do, this is completely unreliable. In political polls, this is the only thing you can do. I guess, they have corrections for the real attendance of social/age strata. In deep interviews, the sociologists are more interested in how people explain their world picture and what they point at, and how react to questions, than to what they tell about themselves.
michaelt 2y ago
> For example political polling, data point in time surveys, census, etc. Without verification, what good is the data? Political polls are regularly wrong https://www.bbc.co.uk/news/articles/cj4ve004llxo https://www.bbc.co.uk/news/articles/cj4ve004llxo In my country, supporters of one political party seem a lot more likely to tell polling companies they're "undecided" - so the pollsters try to adjust their predictions, based on what they've seen in previous elections. But the problem is getting the adjustment right is very difficult. The political landscape can change a great deal between elections. If John McCain and Donald Trump appeal to very different sets of voters, why should voter behaviour observed with McCain hold true with Trump?
HPsquared 2y ago
It's like the uncertainty principle: even the act of asking someone a question will likely change their views.
jjk166 2y ago
There are multiple levels to data. If 57% of respondents to a self reported survey say they are going to do X, it doesn't mean that 57% of people are actually going to do X. But if you do the same survey again and now only 43% of respondents say they will do X, then that is clear evidence something has clearly changed, even if you don't know exactly what effect that something will have on X. That is very useful. The problem is when people only look at self reporting. For example if in the previous scenario 57% responded X but only 54% actually did X, then someone might naively assume that after 43% respond X that 40% will actually do X. Or just as naively they could say 54% will still do X. Or they could apply any of an infinite number of other models. There exists a model that will spit out any given answer for any given input, so without that followup work to actually verify and understand the underlying mechanism, models are worthless.

fithisux 2y ago

It is a proxy.

tsimionescu 2y ago

For many types of data you'd like, there is no alternative. For example for political polls, especially early ones where you're asking people who they would vote for if so and so were running, what other tool could you use? Or how would you verify the results? Of course, given the weakness of the data, you have to remember to place very little trust in the results. Even so, "X% of people say they would vote for Y" can be an interesting information in itself, even if those people were lying.

deleted 2y ago
[deleted]
graemep 2y ago
Polls may or may not be carried out in a neutral way. A good many are deliberately biased, even more have known flaws that are glossed over. A humorous take on this I love (from the 80s BBC comedy Yes Minister, which is often scarily realistic): https://www.youtube.com/watch?v=G0ZZJXw4MTA https://www.youtube.com/watch?v=G0ZZJXw4MTA
- tsimionescu 2y ago
  Oh, absolutely, even in more subtle, unintentional ways (order of responses, whether you offer a neutral option or not, etc). But that doesn't mean polling is entirely useless, or that you need to verify the responses in the way that the original seemed to suggest.

culebron21 2y ago

Pollsters know about this, and the rule of thumb is that you can reliably ask people of what they currently do routinely, almost daily. E.g. if you ask them who they voted 10 years ago, they might remember it wrong. If you ask what they're gonna do, this is completely unreliable. In political polls, this is the only thing you can do. I guess, they have corrections for the real attendance of social/age strata. In deep interviews, the sociologists are more interested in how people explain their world picture and what they point at, and how react to questions, than to what they tell about themselves.

michaelt 2y ago

> For example political polling, data point in time surveys, census, etc. Without verification, what good is the data? Political polls are regularly wrong https://www.bbc.co.uk/news/articles/cj4ve004llxo https://www.bbc.co.uk/news/articles/cj4ve004llxo In my country, supporters of one political party seem a lot more likely to tell polling companies they're "undecided" - so the pollsters try to adjust their predictions, based on what they've seen in previous elections. But the problem is getting the adjustment right is very difficult. The political landscape can change a great deal between elections. If John McCain and Donald Trump appeal to very different sets of voters, why should voter behaviour observed with McCain hold true with Trump?

HPsquared 2y ago

It's like the uncertainty principle: even the act of asking someone a question will likely change their views.

jjk166 2y ago

There are multiple levels to data. If 57% of respondents to a self reported survey say they are going to do X, it doesn't mean that 57% of people are actually going to do X. But if you do the same survey again and now only 43% of respondents say they will do X, then that is clear evidence something has clearly changed, even if you don't know exactly what effect that something will have on X. That is very useful. The problem is when people only look at self reporting. For example if in the previous scenario 57% responded X but only 54% actually did X, then someone might naively assume that after 43% respond X that 40% will actually do X. Or just as naively they could say 54% will still do X. Or they could apply any of an infinite number of other models. There exists a model that will spit out any given answer for any given input, so without that followup work to actually verify and understand the underlying mechanism, models are worthless.