5 ms·

Sample size doesn't tell you everything about a study. You don't need to throw 40,000 people out of an airplane to determine it's safer with a parachute.

yibg 1y ago
This gets brought up often and while true isn't very useful. We know jumping out of plane is dangerous because we have a good understanding of the physics and we have many many comparable examples (jumping out of buildings etc). We understand the mechanism of action of parachutes, so we also know for example a tiny little parachute made of paper towel won't work, and we don't need to test it to know that. Do we understand the cause of autism or how this supposed cure works?
- OkayPhysicist 1y ago
  The key is effect size. We didn't know the exact method of action when we started injecting comatose diabetic children with insulin, but when the coma patient sits up and starts chatting with you, you can establish that a drug is effective with very small sample sizes.
  - yibg 1y ago
    Sure, but even there you need some reasonable sample size. i.e. a sample of 1 won't be sufficient and more importantly it needs to be reproducible. Inject 1 patient and they wake up, but if you inject 10 others and nothing happens and it may not be effective (not that it's definitely not effective). This would require a then a larger sample size.
  - thaw13579 1y ago
    Autism isn't as cut and dry as this. Falling out of planes and diabetic comas are truly black-and-white outcomes, but ASD assessment involves behavioral testing of social and verbal ability, which has high variability between and within individuals. The linked studies about folinic acid are not finding a binary conversion of of ASD kids from non-verbal to verbal, rather they are detecting X% increase in Y behavioral test, which is subtle and something that is found with behavioral treatment. I would love to live in a world where we discover a treatment for ASD that is as clearly successful as insulin for diabetes, but we're not there yet.
bunderbunder 1y ago
It does tell you something. But that kind of sample size is arguably underpowered for anything but a preliminary study. It's the statistical equivalent of Hubble before it had its corrective optics installed: still fine for seeing big unsubtle things like whether parachutes are warranted when jumping from airplanes, but unable to resolve all the details you want to know before concluding that a medication is safe and effective.
- dsr_ 1y ago
  It's worse than that, because: https://www.bmj.com/content/363/bmj.k5094 https://www.bmj.com/content/363/bmj.k5094 Participants 92 aircraft passengers aged 18 and over were screened for participation. 23 agreed to be enrolled and were randomized. Intervention Jumping from an aircraft (airplane or helicopter) with a parachute versus an empty backpack (unblinded). Main outcome measures Composite of death or major traumatic injury (defined by an Injury Severity Score over 15) upon impact with the ground measured immediately after landing. ... go read it to find out what happened. No, really.
  - buttercraft 1y ago
    What the... Is this some sort of joke I'm not getting? "The participants who did ultimately enroll, agreed with the knowledge that the aircraft were stationary and on the ground."
    - bunderbunder 1y ago
      It's gently reminding medical researchers not to forget about participation bias (https://en.wikipedia.org/wiki/Participation_bias https://en.wikipedia.org/wiki/Participation_bias). Another favorite of mine along these lines is "Cigarette smoking: an underused tool in high-performance endurance training". (https://pmc.ncbi.nlm.nih.gov/articles/PMC3001541/ https://pmc.ncbi.nlm.nih.gov/articles/PMC3001541/) This one might actually be quite pertinent in this case, because the FDA's decision appears to rely heavily on exactly the kind of reasoning that this article satirizes.
    - 1y ago
thaw13579 1y ago
I reviewed the linked studies of folinic acid treatment in ASD, and they uniformly say that larger studies are needed before considering this as a widely available treatment. The main issues that need to be worked out are: * who this applies to (some studies suggest genotype and autism subtype matter for getting positive outcomes) * what the side effects are (12 week studies of <100 people are not enough to safely deploy this as a long-term treatment at scale) * how this compares to behavioral treatment (ABA and sensory interventions have reliable positive outcomes as well) I think there's a useful signal there, but we need to be cautious rolling things out at a national scale without bigger studies.

yibg 1y ago

This gets brought up often and while true isn't very useful. We know jumping out of plane is dangerous because we have a good understanding of the physics and we have many many comparable examples (jumping out of buildings etc). We understand the mechanism of action of parachutes, so we also know for example a tiny little parachute made of paper towel won't work, and we don't need to test it to know that. Do we understand the cause of autism or how this supposed cure works?

OkayPhysicist 1y ago
The key is effect size. We didn't know the exact method of action when we started injecting comatose diabetic children with insulin, but when the coma patient sits up and starts chatting with you, you can establish that a drug is effective with very small sample sizes.
- yibg 1y ago
  Sure, but even there you need some reasonable sample size. i.e. a sample of 1 won't be sufficient and more importantly it needs to be reproducible. Inject 1 patient and they wake up, but if you inject 10 others and nothing happens and it may not be effective (not that it's definitely not effective). This would require a then a larger sample size.
- thaw13579 1y ago
  Autism isn't as cut and dry as this. Falling out of planes and diabetic comas are truly black-and-white outcomes, but ASD assessment involves behavioral testing of social and verbal ability, which has high variability between and within individuals. The linked studies about folinic acid are not finding a binary conversion of of ASD kids from non-verbal to verbal, rather they are detecting X% increase in Y behavioral test, which is subtle and something that is found with behavioral treatment. I would love to live in a world where we discover a treatment for ASD that is as clearly successful as insulin for diabetes, but we're not there yet.

bunderbunder 1y ago

It does tell you something. But that kind of sample size is arguably underpowered for anything but a preliminary study. It's the statistical equivalent of Hubble before it had its corrective optics installed: still fine for seeing big unsubtle things like whether parachutes are warranted when jumping from airplanes, but unable to resolve all the details you want to know before concluding that a medication is safe and effective.

dsr_ 1y ago
It's worse than that, because: https://www.bmj.com/content/363/bmj.k5094 https://www.bmj.com/content/363/bmj.k5094 Participants 92 aircraft passengers aged 18 and over were screened for participation. 23 agreed to be enrolled and were randomized. Intervention Jumping from an aircraft (airplane or helicopter) with a parachute versus an empty backpack (unblinded). Main outcome measures Composite of death or major traumatic injury (defined by an Injury Severity Score over 15) upon impact with the ground measured immediately after landing. ... go read it to find out what happened. No, really.
- buttercraft 1y ago
  What the... Is this some sort of joke I'm not getting? "The participants who did ultimately enroll, agreed with the knowledge that the aircraft were stationary and on the ground."
  - bunderbunder 1y ago
    It's gently reminding medical researchers not to forget about participation bias (https://en.wikipedia.org/wiki/Participation_bias https://en.wikipedia.org/wiki/Participation_bias). Another favorite of mine along these lines is "Cigarette smoking: an underused tool in high-performance endurance training". (https://pmc.ncbi.nlm.nih.gov/articles/PMC3001541/ https://pmc.ncbi.nlm.nih.gov/articles/PMC3001541/) This one might actually be quite pertinent in this case, because the FDA's decision appears to rely heavily on exactly the kind of reasoning that this article satirizes.
  - 1y ago

thaw13579 1y ago

I reviewed the linked studies of folinic acid treatment in ASD, and they uniformly say that larger studies are needed before considering this as a widely available treatment. The main issues that need to be worked out are: * who this applies to (some studies suggest genotype and autism subtype matter for getting positive outcomes) * what the side effects are (12 week studies of <100 people are not enough to safely deploy this as a long-term treatment at scale) * how this compares to behavioral treatment (ABA and sensory interventions have reliable positive outcomes as well) I think there's a useful signal there, but we need to be cautious rolling things out at a national scale without bigger studies.