3 ms·

It's effectively worthless because it needs to be qualified so heavily. Dropping the qualification (as will happen sometimes) makes it more like misinformation,

by jbert 18y ago

It's effectively worthless because it needs to be qualified so heavily. Dropping the qualification (as will happen sometimes) makes it more like misinformation, which is worse than worthless, so it averages out to zero... "The median poll selection of a HN visitor who clicked on a poll which was on the front page on the following weekdays (and national holidays in the following countries:...) was..." etc. If the information content is sufficiently hard to extract usefully that it would be easier to redo the sample in a sensible way, then you could call the sample worthless. (A bit like uneconomic oil reserves).

IsaacSchlueter 18y ago
So, doesn't that just mean that a statistically significant portion of the regular HN users would have to respond? Of course, you'd have to figure out how to define "regular HN users" and what would be a significant portion of them. If this poll were to get that much response, it would also tell you something interesting: that HN users respond to polls in statistically significant numbers :) So, voluntary response polls are not worthless if enough people voluntarily respond to them, but this is a tricky problem.
- jbert 18y ago
  > So, doesn't that just mean that a statistically significant portion of the regular HN users would have to respond? No. If 50% of HN visitors voted but 99% of those over 40 didn't (due to whatever confounding factor you like - say embarrasment), you'd still wouldn't be able to extrapolate from the sample to the category of "HN visitors".
  - tokenadult 18y ago
    Correct. It doesn't matter how large a sample size is if the sample is biased. This should be something that every high school student knows after any high school statistics course, but it is a crucial consideration that is widely ignored.
    - whatusername 18y ago
      although we can all think of an edge case.. (especially if we know how the sample is biased) eg: a gender survey in the general population that transgender respondees do not answer -- ends up with a 99% sample size (all but the transgenders) -- and thus gives a pretty significant bit of data on male:female ratios, etc.. Agreed in principle though.

by jbert 18y ago

IsaacSchlueter 18y ago

So, doesn't that just mean that a statistically significant portion of the regular HN users would have to respond? Of course, you'd have to figure out how to define "regular HN users" and what would be a significant portion of them. If this poll were to get that much response, it would also tell you something interesting: that HN users respond to polls in statistically significant numbers :) So, voluntary response polls are not worthless if enough people voluntarily respond to them, but this is a tricky problem.

jbert 18y ago
> So, doesn't that just mean that a statistically significant portion of the regular HN users would have to respond? No. If 50% of HN visitors voted but 99% of those over 40 didn't (due to whatever confounding factor you like - say embarrasment), you'd still wouldn't be able to extrapolate from the sample to the category of "HN visitors".
- tokenadult 18y ago
  Correct. It doesn't matter how large a sample size is if the sample is biased. This should be something that every high school student knows after any high school statistics course, but it is a crucial consideration that is widely ignored.
  - whatusername 18y ago
    although we can all think of an edge case.. (especially if we know how the sample is biased) eg: a gender survey in the general population that transgender respondees do not answer -- ends up with a 99% sample size (all but the transgenders) -- and thus gives a pretty significant bit of data on male:female ratios, etc.. Agreed in principle though.