5 ms·

n = 11?

by sna1l 7y ago

n = 11?

michael_storm 7y ago
Believe it or not, it's pretty hard to conduct a large-scale study in the middle of the largest pandemic in modern history.
- rattray 7y ago
  Why is that?
- monadic2 7y ago
  I'd estimate they were commenting on the quality of the results, not critiquing the personal efforts of the researchers.
- SpicyLemonZest 7y ago
  I don’t believe it. Studying how to effectively treat coronavirus patients is literally the most important problem in the world right now, and I’m sure a lot more than 11 severe cases are being treated with hydrochloroquine.
  - kilbuz 7y ago
    While your point is a good one, meaningful clinical research is hard enough under normal conditions. Forms, enrollment, protocols, data collection, statistical analysis plans, reporting, etc. It's an entire industry used to operating in scales that often span years for a single meaningful study.
    - SpicyLemonZest 7y ago
      And I agree with that as far as it goes, but I think it reflects the problem - clinical research contains a bunch of hurdles that have nothing to do with uncovering the truth. So in a sudden emergency, where clinical research starts to diverge from the treatment decisions of doctors and the drug supply actions of governments, I'm not going to put much weight on clinical research.
- lostmsu 7y ago
  Is it really in the age of ubiquitous Internet access? A bit of self-promotion: https://news.ycombinator.com/item?id=22787162 https://news.ycombinator.com/item?id=22787162 In fact, I am impressed, this is not a thing yet.
- MikeTheGreat 7y ago
  Is it too morbid to joke about it actually being a great time to get a lot of samples for a study? I mean, that plus Zoom and you're like halfway to a world-wide sample size, more-or-less On a serious note: I'm extremely grateful for everyone who's working to solve this big and complicated problem, and (like the other poster said) it's entirely reasonable that it's really difficult to get larger studies going.
kilbuz 7y ago
Imagine a treatment that would being people back to life. Surely if you saw that 11 times you would believe it worked. In fact, after n = 1, you would be so amazed you'd be thinking there is a trick. The point is that, while N is important, the N needed to uncover a signal differs wildly with the outcome you're trying to affect, specifically the outcome's variance. If one of these drug cocktails 'works' (which has many different definitions in clinical research), the signal may be obvious enough that n = 11 is enough to hone in on a promising treatment for further study. It's complicated.
- onlyrealcuzzo 7y ago
  I can only think in the context of A/B testing, but I really don't get it. Can anyone explain how you can have any confidence with these small sample sizes? Imagine you have a purchase form that converts at 0.1%. You test a new change, and with the first 22 users (11 in each group), if the control group randomly gets a conversion, it will have AT LEAST a ~9% CVR now. Okay, maybe you ignore that data, since you can just compare to historical data. Whatever. Even if the new form is dramatically better and has a 30% CVR -- There's a 3% chance it would show a ~9% CVR also, and about the same that it would show a 0% CVR. Similarly, there's about a 6% chance it would show a 91%+ CVR -- skewing your results just as badly in the opposite direction. There's only like a 30% chance you get close to the "true" CVR. Unless these hydroxycloroquine is close to 100% effective (and that doesn't seem to be what any of these studies say) I just don't understand how there's any confidence with small sample sizes. Does anyone have a link to how this works? I've asked a few friends in medicine, and they haven't given a good answer. Haven't found anything on Wikipedia. Probably don't know what to search for.
  - kilbuz 7y ago
    You're thinking about things correctly. The statistical and scientific methodology of clinical trials are by and large the same thing as A/B tests. I don't think anyone is drawing much from this sample of n = 11 except to say it's definitely not a miracle cure. No approval would normally be given to a new treatment for any study of this size. If an A/B test took a form from almost 0% conversion to almost 100% conversion, you might only need 11 users to show that. But even in that case, the devil is in the details. Were your users really a completely random sample from your population, etc? The other side of the coin is, can we say these are non-interesting treatments with only 11 subjects? Or are the studies too underpowered to say much of all?
crygin 7y ago
Generally speaking, a criticism of a study that focuses only on something like this is uninteresting -- while it _may_ be a valid criticism, it is not always (many valid studies have very small samples!), and so stating it unsupported is actually more like evidence that you don't know how to critically evaluate a study. As Zeynep Tufekci said: 'Anybody immediately responds to a correlation with “but correlation does not imply causation” probably doesn’t know what they’re talking about. Don’t have much to say, throw around smart sounding cocktail phrase.'
dang 7y ago
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- hpcjoe 7y ago
  That's not a shallow dismissal. Fundamentally, what I read was a cohort size of 11. 1 died. No control group. No double blind (as no control group). There's no aspect of this that would be statistically distinguishable from noise. Then again, the original study suffered from this as well. n=11 may be succinct, but it is not shallow. There is a world of statistical DoE, analytics, etc. behind this. My concern is the rush to headlines about studies without any real resolving power. The bad part of this is that we don't have time to do proper studies (this takes months and many people to make work). The n=11 comment actually infers this issue. Sort of like -1 = exp(sqrt(-1)*pi), you don't need many letters to convey a tremendous amount of meaning.
  - elcomet 7y ago
    It is shallow because it doesn't teach anything to people that are not already familiar with this kind of work. Something not shallow would be like "only eleven people participated in this study, this is too little to get a clear result that's not due to luck."
    - dang 7y ago
      That would be just as shallow. The issue isn't brevity, it's genericness. Are we really making the assumption that every study with a small sample is worthless, and that our reflexes know better than the people who worked on it? Obviously we need to do better than that. Generic discussion is shallow discussion (https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=by%3Adang%20generic%20discussion&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...). Here's a comparable comment that at least begins to engage with the specifics: https://news.ycombinator.com/item?id=22798031 https://news.ycombinator.com/item?id=22798031. I don't know if it's a good point, but it's at least actually about this particular study. We wouldn't moderate such a comment.

n = 11?

by sna1l 7y ago

n = 11?

Believe it or not, it's pretty hard to conduct a large-scale study in the middle of the largest pandemic in modern history.

rattray 7y ago
Why is that?
monadic2 7y ago
I'd estimate they were commenting on the quality of the results, not critiquing the personal efforts of the researchers.
SpicyLemonZest 7y ago
I don’t believe it. Studying how to effectively treat coronavirus patients is literally the most important problem in the world right now, and I’m sure a lot more than 11 severe cases are being treated with hydrochloroquine.
- kilbuz 7y ago
  While your point is a good one, meaningful clinical research is hard enough under normal conditions. Forms, enrollment, protocols, data collection, statistical analysis plans, reporting, etc. It's an entire industry used to operating in scales that often span years for a single meaningful study.
  - SpicyLemonZest 7y ago
    And I agree with that as far as it goes, but I think it reflects the problem - clinical research contains a bunch of hurdles that have nothing to do with uncovering the truth. So in a sudden emergency, where clinical research starts to diverge from the treatment decisions of doctors and the drug supply actions of governments, I'm not going to put much weight on clinical research.
lostmsu 7y ago
Is it really in the age of ubiquitous Internet access? A bit of self-promotion: https://news.ycombinator.com/item?id=22787162 https://news.ycombinator.com/item?id=22787162 In fact, I am impressed, this is not a thing yet.
MikeTheGreat 7y ago
Is it too morbid to joke about it actually being a great time to get a lot of samples for a study? I mean, that plus Zoom and you're like halfway to a world-wide sample size, more-or-less On a serious note: I'm extremely grateful for everyone who's working to solve this big and complicated problem, and (like the other poster said) it's entirely reasonable that it's really difficult to get larger studies going.

kilbuz 7y ago

Imagine a treatment that would being people back to life. Surely if you saw that 11 times you would believe it worked. In fact, after n = 1, you would be so amazed you'd be thinking there is a trick. The point is that, while N is important, the N needed to uncover a signal differs wildly with the outcome you're trying to affect, specifically the outcome's variance. If one of these drug cocktails 'works' (which has many different definitions in clinical research), the signal may be obvious enough that n = 11 is enough to hone in on a promising treatment for further study. It's complicated.

onlyrealcuzzo 7y ago
I can only think in the context of A/B testing, but I really don't get it. Can anyone explain how you can have any confidence with these small sample sizes? Imagine you have a purchase form that converts at 0.1%. You test a new change, and with the first 22 users (11 in each group), if the control group randomly gets a conversion, it will have AT LEAST a ~9% CVR now. Okay, maybe you ignore that data, since you can just compare to historical data. Whatever. Even if the new form is dramatically better and has a 30% CVR -- There's a 3% chance it would show a ~9% CVR also, and about the same that it would show a 0% CVR. Similarly, there's about a 6% chance it would show a 91%+ CVR -- skewing your results just as badly in the opposite direction. There's only like a 30% chance you get close to the "true" CVR. Unless these hydroxycloroquine is close to 100% effective (and that doesn't seem to be what any of these studies say) I just don't understand how there's any confidence with small sample sizes. Does anyone have a link to how this works? I've asked a few friends in medicine, and they haven't given a good answer. Haven't found anything on Wikipedia. Probably don't know what to search for.
- kilbuz 7y ago
  You're thinking about things correctly. The statistical and scientific methodology of clinical trials are by and large the same thing as A/B tests. I don't think anyone is drawing much from this sample of n = 11 except to say it's definitely not a miracle cure. No approval would normally be given to a new treatment for any study of this size. If an A/B test took a form from almost 0% conversion to almost 100% conversion, you might only need 11 users to show that. But even in that case, the devil is in the details. Were your users really a completely random sample from your population, etc? The other side of the coin is, can we say these are non-interesting treatments with only 11 subjects? Or are the studies too underpowered to say much of all?

crygin 7y ago

Generally speaking, a criticism of a study that focuses only on something like this is uninteresting -- while it _may_ be a valid criticism, it is not always (many valid studies have very small samples!), and so stating it unsupported is actually more like evidence that you don't know how to critically evaluate a study. As Zeynep Tufekci said: 'Anybody immediately responds to a correlation with “but correlation does not imply causation” probably doesn’t know what they’re talking about. Don’t have much to say, throw around smart sounding cocktail phrase.'

dang 7y ago

"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

hpcjoe 7y ago
That's not a shallow dismissal. Fundamentally, what I read was a cohort size of 11. 1 died. No control group. No double blind (as no control group). There's no aspect of this that would be statistically distinguishable from noise. Then again, the original study suffered from this as well. n=11 may be succinct, but it is not shallow. There is a world of statistical DoE, analytics, etc. behind this. My concern is the rush to headlines about studies without any real resolving power. The bad part of this is that we don't have time to do proper studies (this takes months and many people to make work). The n=11 comment actually infers this issue. Sort of like -1 = exp(sqrt(-1)*pi), you don't need many letters to convey a tremendous amount of meaning.
- elcomet 7y ago
  It is shallow because it doesn't teach anything to people that are not already familiar with this kind of work. Something not shallow would be like "only eleven people participated in this study, this is too little to get a clear result that's not due to luck."
  - dang 7y ago
    That would be just as shallow. The issue isn't brevity, it's genericness. Are we really making the assumption that every study with a small sample is worthless, and that our reflexes know better than the people who worked on it? Obviously we need to do better than that. Generic discussion is shallow discussion (https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=by%3Adang%20generic%20discussion&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...). Here's a comparable comment that at least begins to engage with the specifics: https://news.ycombinator.com/item?id=22798031 https://news.ycombinator.com/item?id=22798031. I don't know if it's a good point, but it's at least actually about this particular study. We wouldn't moderate such a comment.