4 ms·

> But 538 isn't necessarily claiming that the per-state error will be normally distributed around their predictions. If per-state error isn't normally distribu

> But 538 isn't necessarily claiming that the per-state error will be normally distributed around their predictions. If per-state error isn't normally distributed, that's evidence of bias, or bad polling.

roenxi 6y ago
Yep. But 538 does not run polls.
- listenallyall 6y ago
  The inputs it uses to spit out probabilities is known to be bad. Any scientist or researcher who claimed to get valid results from known bad inputs would be ridiculed.
  - roenxi 6y ago
    Science does not require an unbiased sample from a normal distribution to work. Bias is a technical term that the field of statistics is very comfortable working with. Scientists can also often get good results out of biased inputs. 538 has corrections for bias already. They seem to have worked in this instance - I repeat myself but: massive surprise, Biden still president.
    - alentist 6y ago
      > 538 has corrections for bias already. They seem to have worked in this instance See https://news.ycombinator.com/item?id=25017810 https://news.ycombinator.com/item?id=25017810
      - roenxi 6y ago
        You are pointing at evidence that 538 correctly called 11/12 races using statistics, and their confident call on a Biden president withstood a 4-7% swing (!!). The existence of bias doesn't invalidate their predictions. Everyone knows that polls can be badly off target in a biased way - that isn't a new phenomenon. When they talk about X% chance of Y being president they should be optimising to the outcome, not the margins.
    - deleted 6y ago
      [deleted]
  - Karrot_Kream 6y ago
    To offer a concrete example here, survery respondents are often biased based on who actually sees and fills out a survey. A common technique used to overcome this non-representative sample is called post-stratification. There are, of course, limits to post stratification especially in instances of low sample sizes, but techniques to overcome issues with data are well known.
Karrot_Kream 6y ago
Not necessarily. Errors, like outcomes, are not independently distributed in US elections. Politics are intertwined and expecting errors and votes to be independent on a state (or even county) basis is overly simplistic. This is also what makes modelling US elections so difficult.
- listenallyall 6y ago
  Sampling errors are random, and expected. Other types of misses are not simple "errors" but polling flaws, like sampling a non-representative group, ignoring non-responders or assuming they break the same as the responders, asking poorly-worded questions, etc. Occasional flaws in polling is understandable and tolerated. But when those misses repeatedly line up the same way, and are rather sizeable, that's evidence of either systematic flaws, or outright bias.
  - Karrot_Kream 6y ago
    I'm not sure what a "sampling error" is. To echo the sibling poster, per-state sentiment is not normally distributed. For example, we know Trump is more popular among white men than other demographics. This means that if we were to create a random variable that reflected the sentiment of white men throughout the US, we would (probably though I'd have to dig deeper into the data) presume to see a higher median vote count in this demographic. However, we cannot say that Trump's popularity in Massachusetts is independent from his popularity in New York, because his popularity in the white male demographic is the dependent variable between both random variables.
    - listenallyall 6y ago
      > I'm not sure what a "sampling error" is. Perhaps you shouldn't be commenting on polling then.
      - Karrot_Kream 6y ago
        I was discussing in good faith, so I'm not sure why you chose to be snarky. Let's clarify here, I'm not sure what "sampling error" in this case would be, such that it is distinct from electoral trends at large. The random variables in question _are_ demographic groups. How is it meaningful to discuss sampling error if your assumption is that state and county data is independently distributed? The poll data that Gelman et al used is public data, I urge you to take a look and work with it.
JamisonM 6y ago
> If per-state error isn't normally distributed, that's evidence of bias, or bad polling. No! Assuming the per-state error would be normally distributed in some neutral world is making huge assumptions about the nature of the electorate, polling, and the correlations of errors between states, you can't do that! You would specifically /not/ expect per-state error to be evenly distributed because the nature of the error would have similar impacts on similar populations and there are similar populations of people that live in different states in differing numbers. You should review the literature about the nature of the (fairly small) polling misses that impacted the swing states and thus disproportionately the outcome in the 2016 election. You will probably find it interesting.
- listenallyall 6y ago
  Yes! There are unavoidable, expected, sampling errors which are, by definition, random. That's why valid, trusted polls calculate a confidence interval instead of a single discrete result. Other types of "errors" -- election results that repeatedly fall outside the confidence interval, or are consistently on only one side of the mean -- only arise when the poll is flawed for some reason. Maybe you relied on landlines only, maybe you spoke with too many men, or too many young people, asked bad questions, miscalculated "likely voter," whatever. Accurate, valid, trusted polls don't have these flaws, the ONLY errors are small, random, expected sampling errors.
  - Karrot_Kream 6y ago
    > That's why valid, trusted polls calculate a confidence interval instead of a single discrete result. That is what each of these statistical models did, yes. And the actual outcomes fell into these confidence intervals. > Other types of "errors" -- election results that repeatedly fall outside the confidence interval, or are consistently on only one side of the mean -- only arise when the poll is flawed for some reason. Or the model was inaccurate. Perhaps the priors were too specific. Perhaps the data was missing, misrecorded, not tabulated properly, who knows. Again, the results fell within the CI of most models, the problem was simply that the result fell too close to the mean for most statisticians' comfort.

roenxi 6y ago

Yep. But 538 does not run polls.

listenallyall 6y ago
The inputs it uses to spit out probabilities is known to be bad. Any scientist or researcher who claimed to get valid results from known bad inputs would be ridiculed.
- roenxi 6y ago
  Science does not require an unbiased sample from a normal distribution to work. Bias is a technical term that the field of statistics is very comfortable working with. Scientists can also often get good results out of biased inputs. 538 has corrections for bias already. They seem to have worked in this instance - I repeat myself but: massive surprise, Biden still president.
  - alentist 6y ago
    > 538 has corrections for bias already. They seem to have worked in this instance See https://news.ycombinator.com/item?id=25017810 https://news.ycombinator.com/item?id=25017810
    - roenxi 6y ago
      You are pointing at evidence that 538 correctly called 11/12 races using statistics, and their confident call on a Biden president withstood a 4-7% swing (!!). The existence of bias doesn't invalidate their predictions. Everyone knows that polls can be badly off target in a biased way - that isn't a new phenomenon. When they talk about X% chance of Y being president they should be optimising to the outcome, not the margins.
  - deleted 6y ago
    [deleted]
- Karrot_Kream 6y ago
  To offer a concrete example here, survery respondents are often biased based on who actually sees and fills out a survey. A common technique used to overcome this non-representative sample is called post-stratification. There are, of course, limits to post stratification especially in instances of low sample sizes, but techniques to overcome issues with data are well known.

Karrot_Kream 6y ago

Not necessarily. Errors, like outcomes, are not independently distributed in US elections. Politics are intertwined and expecting errors and votes to be independent on a state (or even county) basis is overly simplistic. This is also what makes modelling US elections so difficult.

listenallyall 6y ago
Sampling errors are random, and expected. Other types of misses are not simple "errors" but polling flaws, like sampling a non-representative group, ignoring non-responders or assuming they break the same as the responders, asking poorly-worded questions, etc. Occasional flaws in polling is understandable and tolerated. But when those misses repeatedly line up the same way, and are rather sizeable, that's evidence of either systematic flaws, or outright bias.
- Karrot_Kream 6y ago
  I'm not sure what a "sampling error" is. To echo the sibling poster, per-state sentiment is not normally distributed. For example, we know Trump is more popular among white men than other demographics. This means that if we were to create a random variable that reflected the sentiment of white men throughout the US, we would (probably though I'd have to dig deeper into the data) presume to see a higher median vote count in this demographic. However, we cannot say that Trump's popularity in Massachusetts is independent from his popularity in New York, because his popularity in the white male demographic is the dependent variable between both random variables.
  - listenallyall 6y ago
    > I'm not sure what a "sampling error" is. Perhaps you shouldn't be commenting on polling then.
    - Karrot_Kream 6y ago
      I was discussing in good faith, so I'm not sure why you chose to be snarky. Let's clarify here, I'm not sure what "sampling error" in this case would be, such that it is distinct from electoral trends at large. The random variables in question _are_ demographic groups. How is it meaningful to discuss sampling error if your assumption is that state and county data is independently distributed? The poll data that Gelman et al used is public data, I urge you to take a look and work with it.

JamisonM 6y ago

> If per-state error isn't normally distributed, that's evidence of bias, or bad polling. No! Assuming the per-state error would be normally distributed in some neutral world is making huge assumptions about the nature of the electorate, polling, and the correlations of errors between states, you can't do that! You would specifically /not/ expect per-state error to be evenly distributed because the nature of the error would have similar impacts on similar populations and there are similar populations of people that live in different states in differing numbers. You should review the literature about the nature of the (fairly small) polling misses that impacted the swing states and thus disproportionately the outcome in the 2016 election. You will probably find it interesting.

listenallyall 6y ago
Yes! There are unavoidable, expected, sampling errors which are, by definition, random. That's why valid, trusted polls calculate a confidence interval instead of a single discrete result. Other types of "errors" -- election results that repeatedly fall outside the confidence interval, or are consistently on only one side of the mean -- only arise when the poll is flawed for some reason. Maybe you relied on landlines only, maybe you spoke with too many men, or too many young people, asked bad questions, miscalculated "likely voter," whatever. Accurate, valid, trusted polls don't have these flaws, the ONLY errors are small, random, expected sampling errors.
- Karrot_Kream 6y ago
  > That's why valid, trusted polls calculate a confidence interval instead of a single discrete result. That is what each of these statistical models did, yes. And the actual outcomes fell into these confidence intervals. > Other types of "errors" -- election results that repeatedly fall outside the confidence interval, or are consistently on only one side of the mean -- only arise when the poll is flawed for some reason. Or the model was inaccurate. Perhaps the priors were too specific. Perhaps the data was missing, misrecorded, not tabulated properly, who knows. Again, the results fell within the CI of most models, the problem was simply that the result fell too close to the mean for most statisticians' comfort.