5 ms·

I recently listened to an advanced statistics lecture. As an illustration the Professor said, medical science (still) uses 2 sigma statistics, jet engine builde

I recently listened to an advanced statistics lecture. As an illustration the Professor said, medical science (still) uses 2 sigma statistics, jet engine builders use 6 sigma. That's why jet engines never (sic) fail. Yes, and then there is the influence of the drug makers too.

mikecsh 12y ago
Do you have a link/reference to this lecture?
tomp 12y ago
The biggest difference is that jet engine builders can conduct an unlimited number proper controlled scientific experiments, while the fields that study humans (psychology, medicine, sociology, economy) can usually only conduct a limited number of field trials (which, even though they are "controlled" and possibly "double blind" are nowhere as rigorous as in proper sciences).
- paulojreis 12y ago
  This is a very important point that seems to be forgotten frequently. Particularly when doing methodological comparison between disciplines. The other point is complexity. The number of variables needed to create a useful model of a jet engine is a lot smaller than the number needed to model an organic system. This means that today we have useful models of jet engines and we can reliably test something with simulations - we may generate loads of valid data and easily apply "top-notch" statistics. The same, unfortunately, is not true for medicine.
- icegreentea 12y ago
  Yes. There was the other article sitting around recently asking why the rate of blockbuster drugs have slowed down. Getting even the level of confidence required now is hugely expensive. Requiring higher confidence is an expensive choice that must be weighed closely. I think it's becoming clearer each day that many drug side effects, and general drug efficacy is effected by the genetics of the drug taker. Without being able to tease out these genetic effects, blindly aiming for broad population safety or efficacy levels is always going to be troublesome. This is separate from the data publication issue that the article focuses on.
  - tommorris 12y ago
    > There was the other article sitting around recently asking why the rate of blockbuster drugs have slowed down. Getting even the level of confidence required now is hugely expensive. Requiring higher confidence is an expensive choice that must be weighed closely. Not just expensive but also potentially unethical. Consider this: post-exposure prophylaxis for HIV. It's used both for sexual and non-sexual exposure to the HIV virus. It's a month-long series of antiretrovirals (the exact protocols used differ). It's effective if you start taking it within 72 hours of exposure. The drugs themselves have been tested for use by people who are HIV+, but nobody has done clinical trials on using them as post-exposure prophylaxis. We know it works, but we don't know how well because testing would be unethical. You'd have to take half the people who are exposed and give them placebo and put them at risk of getting HIV. At a certain point, you have to throw the dice and say "given the evidence we have, this is what we are going to do". (The fact that such judgement calls are being distorted by evidence being intentionally withheld by drug companies is disgraceful though.)
  - bainsfather 12y ago
    "Requiring higher confidence is an expensive choice that must be weighed closely." Confidence of effectiveness is very easy to get - you just need to find an effective drug. e.g. when antibiotics were first tried - they are massively effective compared to previous treatments - you get high confidence of effectiveness from just a small sample size. A reason it is hard for new drugs to get high confidence is because they are only marginally better than existing drugs.
  - fluidcruft 12y ago
    What is a "blockbuster" drug? One that makes a lot of money for the various stakeholders or one that has a large impact on human health? I'm not sure I care too much about the former and it's not at all obvious how "level of confidence" would affect the latter. Yes, as a result of past medical progress it is more difficult to improve upon the current state of the art. But new therapies have to compete with existing therapies in the market anyway. If a new therapy can't outperform existing ones, what actual value does it have? Perhaps drug companies should think about solving problems that aren't already solved rather than bemoaning their inability to reap massive profits while reinventing the wheel or pushing minor formulation tweaks to renew their patents.
- BenoitEssiambre 12y ago
  The main difficulty is that it is unethical and illegal to test on humans to any degree of significance for treatments that have important ramifications. The moment results start leaning one way, researchers are obligated to move all the patients to the beneficial or non dangerous side. You are not allowed to let humans get sick or die in order to keep a clean control group.
sqrt28 12y ago
Oh come on! How is this sigma counting relevant? What would you prefer, a drug that improved the condition of 1% tested patients (p = 10^{-6}) or a drug that improved the condition of 95% tested patients (p = 0.05)? Take any drug, test in on millions of patients, here is your six sigma. What problem did you solve?
- mironathetin 12y ago
  You name a problem. Many years ago, I used to teach physics and with it also error treatment and some basic statistics to medical students. My overall impression was, they never got it, nor did the majority even understand its importance, when results are interpreted. Most medical student appeared to be exceptionally smart, even smarter as physics students. But somehow statistics never appealed to them. Your comment shows me, that not much has changed.

mikecsh 12y ago

Do you have a link/reference to this lecture?

tomp 12y ago

The biggest difference is that jet engine builders can conduct an unlimited number proper controlled scientific experiments, while the fields that study humans (psychology, medicine, sociology, economy) can usually only conduct a limited number of field trials (which, even though they are "controlled" and possibly "double blind" are nowhere as rigorous as in proper sciences).

paulojreis 12y ago
This is a very important point that seems to be forgotten frequently. Particularly when doing methodological comparison between disciplines. The other point is complexity. The number of variables needed to create a useful model of a jet engine is a lot smaller than the number needed to model an organic system. This means that today we have useful models of jet engines and we can reliably test something with simulations - we may generate loads of valid data and easily apply "top-notch" statistics. The same, unfortunately, is not true for medicine.
icegreentea 12y ago
Yes. There was the other article sitting around recently asking why the rate of blockbuster drugs have slowed down. Getting even the level of confidence required now is hugely expensive. Requiring higher confidence is an expensive choice that must be weighed closely. I think it's becoming clearer each day that many drug side effects, and general drug efficacy is effected by the genetics of the drug taker. Without being able to tease out these genetic effects, blindly aiming for broad population safety or efficacy levels is always going to be troublesome. This is separate from the data publication issue that the article focuses on.
- tommorris 12y ago
  > There was the other article sitting around recently asking why the rate of blockbuster drugs have slowed down. Getting even the level of confidence required now is hugely expensive. Requiring higher confidence is an expensive choice that must be weighed closely. Not just expensive but also potentially unethical. Consider this: post-exposure prophylaxis for HIV. It's used both for sexual and non-sexual exposure to the HIV virus. It's a month-long series of antiretrovirals (the exact protocols used differ). It's effective if you start taking it within 72 hours of exposure. The drugs themselves have been tested for use by people who are HIV+, but nobody has done clinical trials on using them as post-exposure prophylaxis. We know it works, but we don't know how well because testing would be unethical. You'd have to take half the people who are exposed and give them placebo and put them at risk of getting HIV. At a certain point, you have to throw the dice and say "given the evidence we have, this is what we are going to do". (The fact that such judgement calls are being distorted by evidence being intentionally withheld by drug companies is disgraceful though.)
- bainsfather 12y ago
  "Requiring higher confidence is an expensive choice that must be weighed closely." Confidence of effectiveness is very easy to get - you just need to find an effective drug. e.g. when antibiotics were first tried - they are massively effective compared to previous treatments - you get high confidence of effectiveness from just a small sample size. A reason it is hard for new drugs to get high confidence is because they are only marginally better than existing drugs.
- fluidcruft 12y ago
  What is a "blockbuster" drug? One that makes a lot of money for the various stakeholders or one that has a large impact on human health? I'm not sure I care too much about the former and it's not at all obvious how "level of confidence" would affect the latter. Yes, as a result of past medical progress it is more difficult to improve upon the current state of the art. But new therapies have to compete with existing therapies in the market anyway. If a new therapy can't outperform existing ones, what actual value does it have? Perhaps drug companies should think about solving problems that aren't already solved rather than bemoaning their inability to reap massive profits while reinventing the wheel or pushing minor formulation tweaks to renew their patents.
BenoitEssiambre 12y ago
The main difficulty is that it is unethical and illegal to test on humans to any degree of significance for treatments that have important ramifications. The moment results start leaning one way, researchers are obligated to move all the patients to the beneficial or non dangerous side. You are not allowed to let humans get sick or die in order to keep a clean control group.

sqrt28 12y ago

Oh come on! How is this sigma counting relevant? What would you prefer, a drug that improved the condition of 1% tested patients (p = 10^{-6}) or a drug that improved the condition of 95% tested patients (p = 0.05)? Take any drug, test in on millions of patients, here is your six sigma. What problem did you solve?

mironathetin 12y ago
You name a problem. Many years ago, I used to teach physics and with it also error treatment and some basic statistics to medical students. My overall impression was, they never got it, nor did the majority even understand its importance, when results are interpreted. Most medical student appeared to be exceptionally smart, even smarter as physics students. But somehow statistics never appealed to them. Your comment shows me, that not much has changed.