4 ms·

Imagine a treatment that would being people back to life. Surely if you saw that 11 times you would believe it worked. In fact, after n = 1, you would be so ama

by kilbuz 7y ago

Imagine a treatment that would being people back to life. Surely if you saw that 11 times you would believe it worked. In fact, after n = 1, you would be so amazed you'd be thinking there is a trick. The point is that, while N is important, the N needed to uncover a signal differs wildly with the outcome you're trying to affect, specifically the outcome's variance. If one of these drug cocktails 'works' (which has many different definitions in clinical research), the signal may be obvious enough that n = 11 is enough to hone in on a promising treatment for further study. It's complicated.

onlyrealcuzzo 7y ago
I can only think in the context of A/B testing, but I really don't get it. Can anyone explain how you can have any confidence with these small sample sizes? Imagine you have a purchase form that converts at 0.1%. You test a new change, and with the first 22 users (11 in each group), if the control group randomly gets a conversion, it will have AT LEAST a ~9% CVR now. Okay, maybe you ignore that data, since you can just compare to historical data. Whatever. Even if the new form is dramatically better and has a 30% CVR -- There's a 3% chance it would show a ~9% CVR also, and about the same that it would show a 0% CVR. Similarly, there's about a 6% chance it would show a 91%+ CVR -- skewing your results just as badly in the opposite direction. There's only like a 30% chance you get close to the "true" CVR. Unless these hydroxycloroquine is close to 100% effective (and that doesn't seem to be what any of these studies say) I just don't understand how there's any confidence with small sample sizes. Does anyone have a link to how this works? I've asked a few friends in medicine, and they haven't given a good answer. Haven't found anything on Wikipedia. Probably don't know what to search for.
- kilbuz 7y ago
  You're thinking about things correctly. The statistical and scientific methodology of clinical trials are by and large the same thing as A/B tests. I don't think anyone is drawing much from this sample of n = 11 except to say it's definitely not a miracle cure. No approval would normally be given to a new treatment for any study of this size. If an A/B test took a form from almost 0% conversion to almost 100% conversion, you might only need 11 users to show that. But even in that case, the devil is in the details. Were your users really a completely random sample from your population, etc? The other side of the coin is, can we say these are non-interesting treatments with only 11 subjects? Or are the studies too underpowered to say much of all?
dooglius 7y ago
That wouldn't really be n=11 because there is a huge implicit control group of people who haven't come back to life, and if you do the math it would very strongly reject the null hypothesis.
simondw 7y ago
I don't think your scenario is as different as you think. Depending on what we mean by "back to life", a trick is in fact much more likely than something thought to be biologically impossible. If we eliminated the possibility of a trick -- that is, proved beyond a doubt that 11 people were dead, and then alive -- I would absolutely not be ready to attribute it to some new treatment. Natural law is being violated here! I would instead start checking for other possibilities, like: - Are other people spontaneously reanimating around the world, or just in this study? - Did these 11 people have anything else in common? Extraterrestrial origin, perhaps? Unusual religious beliefs? - Hang on, let's double (triple, quadruple, etc.) check that they were really dead. And then go back and check again. - And by all means, start a proper clinical trial of the new treatment while we're trying to understand how resurrection is even possible.

by kilbuz 7y ago

onlyrealcuzzo 7y ago

I can only think in the context of A/B testing, but I really don't get it. Can anyone explain how you can have any confidence with these small sample sizes? Imagine you have a purchase form that converts at 0.1%. You test a new change, and with the first 22 users (11 in each group), if the control group randomly gets a conversion, it will have AT LEAST a ~9% CVR now. Okay, maybe you ignore that data, since you can just compare to historical data. Whatever. Even if the new form is dramatically better and has a 30% CVR -- There's a 3% chance it would show a ~9% CVR also, and about the same that it would show a 0% CVR. Similarly, there's about a 6% chance it would show a 91%+ CVR -- skewing your results just as badly in the opposite direction. There's only like a 30% chance you get close to the "true" CVR. Unless these hydroxycloroquine is close to 100% effective (and that doesn't seem to be what any of these studies say) I just don't understand how there's any confidence with small sample sizes. Does anyone have a link to how this works? I've asked a few friends in medicine, and they haven't given a good answer. Haven't found anything on Wikipedia. Probably don't know what to search for.

kilbuz 7y ago
You're thinking about things correctly. The statistical and scientific methodology of clinical trials are by and large the same thing as A/B tests. I don't think anyone is drawing much from this sample of n = 11 except to say it's definitely not a miracle cure. No approval would normally be given to a new treatment for any study of this size. If an A/B test took a form from almost 0% conversion to almost 100% conversion, you might only need 11 users to show that. But even in that case, the devil is in the details. Were your users really a completely random sample from your population, etc? The other side of the coin is, can we say these are non-interesting treatments with only 11 subjects? Or are the studies too underpowered to say much of all?

dooglius 7y ago

That wouldn't really be n=11 because there is a huge implicit control group of people who haven't come back to life, and if you do the math it would very strongly reject the null hypothesis.

simondw 7y ago

I don't think your scenario is as different as you think. Depending on what we mean by "back to life", a trick is in fact much more likely than something thought to be biologically impossible. If we eliminated the possibility of a trick -- that is, proved beyond a doubt that 11 people were dead, and then alive -- I would absolutely not be ready to attribute it to some new treatment. Natural law is being violated here! I would instead start checking for other possibilities, like: - Are other people spontaneously reanimating around the world, or just in this study? - Did these 11 people have anything else in common? Extraterrestrial origin, perhaps? Unusual religious beliefs? - Hang on, let's double (triple, quadruple, etc.) check that they were really dead. And then go back and check again. - And by all means, start a proper clinical trial of the new treatment while we're trying to understand how resurrection is even possible.