4 ms·

> One, small data can be extremely illuminating. I love this. I think we dismiss "small data" too quickly. I see this all the time here on HN and in other pla

by cle 6y ago

> One, small data can be extremely illuminating. I love this. I think we dismiss "small data" too quickly. I see this all the time here on HN and in other places, where someone's experience will be dismissed as "just anecdotes", as if an anecdote contains no useful information at all. Statistical summaries by definition lose information--it can be incredibly valuable to continuously ask ourselves over time "What information is important to retain in our models/statistics, and what information is okay to discard?" To me, it seems obvious when I say it--well duh that's the scientific method--but it's so easy to become attached to our models and statistics and use those to dismiss important information as noise.

wenc 6y ago
Well, it seems to me that it's not so much the size of the data that really matters -- it's whether the amount of data and the method of collection is appropriate to the question at hand. Certain questions only require small samples to answer. If you frame your null hypothesis as "the virus is aerosol therefore everyone in an indoor space has equal probability of being infected after a period of time", then a few counterexamples is all that is necessary to reject the hypothesis. In the classic black swan example "all swans are white", it only took 1 counterexample (small data) to disprove the premise, while mountains of examples of white swans (big data) would be insufficient evidence to actually prove it. There's an asymmetry here. Coming back to the article, the author recognized this exact asymmetry in the question at hand: "While some of this may look discouraging, I find these “small data” studies greatly empowering. They don’t just tell us what happened, they tell us what didn’t happen." "These small studies cannot tell us the proportion of transmission that occurs indoors, but they highlight how it occurs: droplets and aerosols being carried through the air. Clearly, the closer you are to the person, the more likely you are to get hit But equally clearly, air flow and the positioning of people are huge variables, too. Just talking doesn’t seem to generate aerosols to the point that everyone within the room is infected (though this is possible), but singing does—at least in these well-studied cases. Sometimes, though, the most important information is that the dog did not bark." However, not all questions have this property. Let's say your hypothesis was something like "this vaccine will work in at least 95% of the population", small data approaches might not work. Bob and Jill said it worked for them -- how comfortable would we be extrapolating this anecdotal evidence to the general population? At most we can say there's a statistical signal to the affirmative, but nothing more than that. And that is the nature of self-reported anecdotal evidence (unless they were rigorously structured to remove confounding factors -- usually they are not). p.s. Furthermore, not all questions require rigor, therefore anecdotal evidence can sometimes be useful without being rigorous. For instance I've always found Myers Briggs to be a "useful" heuristic despite not having any rigorous basis.
latchkey 6y ago
John Snow was able to disprove the miasma theory with "small data" (via analysis of people using a public water pump). This is how the germ theory of disease came about and helped save countless lives from cholera. This is also how the science of (modern) epidemiology began. It is a fascinating story. https://en.wikipedia.org/wiki/John_Snow#Cholera https://en.wikipedia.org/wiki/John_Snow#Cholera

by cle 6y ago

wenc 6y ago

Well, it seems to me that it's not so much the size of the data that really matters -- it's whether the amount of data and the method of collection is appropriate to the question at hand. Certain questions only require small samples to answer. If you frame your null hypothesis as "the virus is aerosol therefore everyone in an indoor space has equal probability of being infected after a period of time", then a few counterexamples is all that is necessary to reject the hypothesis. In the classic black swan example "all swans are white", it only took 1 counterexample (small data) to disprove the premise, while mountains of examples of white swans (big data) would be insufficient evidence to actually prove it. There's an asymmetry here. Coming back to the article, the author recognized this exact asymmetry in the question at hand: "While some of this may look discouraging, I find these “small data” studies greatly empowering. They don’t just tell us what happened, they tell us what didn’t happen." "These small studies cannot tell us the proportion of transmission that occurs indoors, but they highlight how it occurs: droplets and aerosols being carried through the air. Clearly, the closer you are to the person, the more likely you are to get hit But equally clearly, air flow and the positioning of people are huge variables, too. Just talking doesn’t seem to generate aerosols to the point that everyone within the room is infected (though this is possible), but singing does—at least in these well-studied cases. Sometimes, though, the most important information is that the dog did not bark." However, not all questions have this property. Let's say your hypothesis was something like "this vaccine will work in at least 95% of the population", small data approaches might not work. Bob and Jill said it worked for them -- how comfortable would we be extrapolating this anecdotal evidence to the general population? At most we can say there's a statistical signal to the affirmative, but nothing more than that. And that is the nature of self-reported anecdotal evidence (unless they were rigorously structured to remove confounding factors -- usually they are not). p.s. Furthermore, not all questions require rigor, therefore anecdotal evidence can sometimes be useful without being rigorous. For instance I've always found Myers Briggs to be a "useful" heuristic despite not having any rigorous basis.

latchkey 6y ago

John Snow was able to disprove the miasma theory with "small data" (via analysis of people using a public water pump). This is how the germ theory of disease came about and helped save countless lives from cholera. This is also how the science of (modern) epidemiology began. It is a fascinating story. https://en.wikipedia.org/wiki/John_Snow#Cholera https://en.wikipedia.org/wiki/John_Snow#Cholera