4 ms·

This is obviously a hard and serious problem to address. That said, I little part of me smiled a bit at 5 withdrawals out of 93 drugs. It's bizarrely close to w

This is obviously a hard and serious problem to address. That said, I little part of me smiled a bit at 5 withdrawals out of 93 drugs. It's bizarrely close to what blindly applying 95% confidence intervals get you. I'm both terrified and impressed that it's that close enough.

jdashg 7y ago
Expected value of 95% with 93 trials is 88.3, so right on target.
SamReidHughes 7y ago
Passing a 95% significance test doesn't have any connection to the notion that there is a 5% chance the drug is no good. This is what you would expect from a blind application of a 95% confidence interval only if 50% of tested drugs were good.
- rifung 7y ago
  Care to clarify? My understanding was that a 95% confidence interval implies the chance of a false positive of 5%. From http://www.mit.edu/~6.s085/notes/lecture2.pdf http://www.mit.edu/~6.s085/notes/lecture2.pdf "In particular, we can ask for an interval around ˆp for any sample so that in 95% of samples, the true mean p will lie inside this interval. Such an interval is called a confidence interval."
  - Retric 7y ago
    Suppose you test for 1 true positive out of 1001 cases. You are likely to find the actual positive and 50 false ones. However, that’s with a single test. If you run 5+ different tests you are likely able to distinguish the true positive.
    - 6gvONxR4sf7o 7y ago
      To add to this, you can't just run 5+ tests. You have to run wholly independent ones.
  - 6gvONxR4sf7o 7y ago
    The false positive rate has nothing to do with positive examples (hugely counterintuitively). A false positive happens when there's nothing there but you think there is. The rate of these is compared to all incidents where there's nothing there. So it's the fraction of negative instances where you think there's a positive. No actual positives have to come into the picture.
    - rifung 7y ago
      Ah you're right that makes a lot of sense thanks for the explanation
refurb 7y ago
It's not so straightforward as the way a positive result is measured is also changing over tim, e.g. biomarker -> overall survival

jdashg 7y ago

Expected value of 95% with 93 trials is 88.3, so right on target.

SamReidHughes 7y ago

Passing a 95% significance test doesn't have any connection to the notion that there is a 5% chance the drug is no good. This is what you would expect from a blind application of a 95% confidence interval only if 50% of tested drugs were good.

rifung 7y ago
Care to clarify? My understanding was that a 95% confidence interval implies the chance of a false positive of 5%. From http://www.mit.edu/~6.s085/notes/lecture2.pdf http://www.mit.edu/~6.s085/notes/lecture2.pdf "In particular, we can ask for an interval around ˆp for any sample so that in 95% of samples, the true mean p will lie inside this interval. Such an interval is called a confidence interval."
- Retric 7y ago
  Suppose you test for 1 true positive out of 1001 cases. You are likely to find the actual positive and 50 false ones. However, that’s with a single test. If you run 5+ different tests you are likely able to distinguish the true positive.
  - 6gvONxR4sf7o 7y ago
    To add to this, you can't just run 5+ tests. You have to run wholly independent ones.
- 6gvONxR4sf7o 7y ago
  The false positive rate has nothing to do with positive examples (hugely counterintuitively). A false positive happens when there's nothing there but you think there is. The rate of these is compared to all incidents where there's nothing there. So it's the fraction of negative instances where you think there's a positive. No actual positives have to come into the picture.
  - rifung 7y ago
    Ah you're right that makes a lot of sense thanks for the explanation

refurb 7y ago

It's not so straightforward as the way a positive result is measured is also changing over tim, e.g. biomarker -> overall survival