3 ms·

Reminds me of the small “sample size misconception” https://www.dailykos.com/story/2008/11/9/656465/- https://www.dailykos.com/story/2008/11/9/656465/-

by baxtr 7y ago

dredmorbius 7y ago
Very true, though this can cut several different ways. I've done several measurements of various aspects of use and engagement on Google+, as an independent outsider. From a randomised sample of fewer than 100 randomly selected profiles (of 2.2 billion total), it was clear that the active fraction was about 10%, and the highly active fraction a minuscule portion of that. I ended up checking 50k profiles, and another (fully independent) analysis by Stone Temple Consulting of 500k profiles largely re-demonstrated the initial 9% finding. But, with more profiles, it was possible to dial in on the very few (about 0.16%) highly active users. Which is another sampling problem -- you're looking for the 1 in 1,000 users who are active, and need to get a sufficient sample (typically 30-300) of those. Let's call it 100, for round numbers. Which means you're looking for a sample of a one-in-a-thousand subpopulation, meaning that 100 rare high-use users requires sampling 100 * 1000 = 100k of the total population. (This presumes no other way of subsetting the population.) I ran into this looking at G+ Communities -- 8 million in total, with again a very small fraction of highly-active ones. Initial samples of 12k and 36k subsets were useful (and tractable on a residential DSL connection), but it was being gifted with a full 8-million record population summary of community activity that allowed full and detailed statistics to be calculated. https://social.antefriguserat.de/index.php/Migrating_Google%2B_Communities#Google.2B_Community_Characteristics_and_Membership https://social.antefriguserat.de/index.php/Migrating_Google%...
feanaro 7y ago
This article presents standard error as the quantity representing "by how much our prediction could be off" without mentioning that this is only so for a single standard deviation. For a normally distributed value, one standard deviation only covers around 68% of the cases so you'd have to at least double that error value if you want to be pretty sure of your conclusion (which would then cover around 95% of the cases).

Reminds me of the small “sample size misconception” https://www.dailykos.com/story/2008/11/9/656465/- https://www.dailykos.com/story/2008/11/9/656465/-

by baxtr 7y ago

Reminds me of the small “sample size misconception” https://www.dailykos.com/story/2008/11/9/656465/- https://www.dailykos.com/story/2008/11/9/656465/-

dredmorbius 7y ago

Very true, though this can cut several different ways. I've done several measurements of various aspects of use and engagement on Google+, as an independent outsider. From a randomised sample of fewer than 100 randomly selected profiles (of 2.2 billion total), it was clear that the active fraction was about 10%, and the highly active fraction a minuscule portion of that. I ended up checking 50k profiles, and another (fully independent) analysis by Stone Temple Consulting of 500k profiles largely re-demonstrated the initial 9% finding. But, with more profiles, it was possible to dial in on the very few (about 0.16%) highly active users. Which is another sampling problem -- you're looking for the 1 in 1,000 users who are active, and need to get a sufficient sample (typically 30-300) of those. Let's call it 100, for round numbers. Which means you're looking for a sample of a one-in-a-thousand subpopulation, meaning that 100 rare high-use users requires sampling 100 * 1000 = 100k of the total population. (This presumes no other way of subsetting the population.) I ran into this looking at G+ Communities -- 8 million in total, with again a very small fraction of highly-active ones. Initial samples of 12k and 36k subsets were useful (and tractable on a residential DSL connection), but it was being gifted with a full 8-million record population summary of community activity that allowed full and detailed statistics to be calculated. https://social.antefriguserat.de/index.php/Migrating_Google%2B_Communities#Google.2B_Community_Characteristics_and_Membership https://social.antefriguserat.de/index.php/Migrating_Google%...

feanaro 7y ago

This article presents standard error as the quantity representing "by how much our prediction could be off" without mentioning that this is only so for a single standard deviation. For a normally distributed value, one standard deviation only covers around 68% of the cases so you'd have to at least double that error value if you want to be pretty sure of your conclusion (which would then cover around 95% of the cases).