3 ms·
Well, it seems to me that it's not so much the size of the data that really matters -- it's whether the amount of data and the method of collection is appropria
by wenc 6y ago
Well, it seems to me that it's not so much the size of the data that really matters -- it's whether the amount of data and the method of collection is appropriate to the question at hand. Certain questions only require small samples to answer. If you frame your null hypothesis as "the virus is aerosol therefore everyone in an indoor space has equal probability of being infected after a period of time", then a few counterexamples is all that is necessary to reject the hypothesis.
In the classic black swan example "all swans are white", it only took 1 counterexample (small data) to disprove the premise, while mountains of examples of white swans (big data) would be insufficient evidence to actually prove it. There's an asymmetry here.
Coming back to the article, the author recognized this exact asymmetry in the question at hand: "While some of this may look discouraging, I find these “small data” studies greatly empowering. They don’t just tell us what happened, they tell us what didn’t happen." "These small studies cannot tell us the proportion of transmission that occurs indoors, but they highlight how it occurs: droplets and aerosols being carried through the air. Clearly, the closer you are to the person, the more likely you are to get hit But equally clearly, air flow and the positioning of people are huge variables, too. Just talking doesn’t seem to generate aerosols to the point that everyone within the room is infected (though this is possible), but singing does—at least in these well-studied cases. Sometimes, though, the most important information is that the dog did not bark."
However, not all questions have this property. Let's say your hypothesis was something like "this vaccine will work in at least 95% of the population", small data approaches might not work. Bob and Jill said it worked for them -- how comfortable would we be extrapolating this anecdotal evidence to the general population? At most we can say there's a statistical signal to the affirmative, but nothing more than that. And that is the nature of self-reported anecdotal evidence (unless they were rigorously structured to remove confounding factors -- usually they are not).
p.s. Furthermore, not all questions require rigor, therefore anecdotal evidence can sometimes be useful without being rigorous. For instance I've always found Myers Briggs to be a "useful" heuristic despite not having any rigorous basis.