5 ms·

How much of this could be solved by replication? Some sort of rule where we say, formally, not to take any study seriously unless it's been replicated? On the

by graeme 12y ago

How much of this could be solved by replication? Some sort of rule where we say, formally, not to take any study seriously unless it's been replicated? On the one hand, it's very bad that the FDA has covered this up. But on the other hand, there was science before the FDA. I may be interpreting the article wrong, but it seems as though the FDA is merely failing to prevent bad science. Presumably, journals that cited these experiments didn't replicate that and knew they weren't replicated. Is science too complex and expensive to replicate these days? If so, is it even still science? (Those are actual questions, not rhetorical ones. I don't know the answers here). Edit: For some reason, at present my comment seems to be the top comment. I don't want to detract from the story itself. The actual issues at the FDA raised by the article deserve discussion.

Scaevolus 12y ago
Clinical trials are very expensive to run. The only people financially interested in running them are the pharmaceutical companies trying to make new products available. Replication is hard enough to achieve in hard sciences where replicating a study might cost tens of thousands of dollars. Finding the money to replicate or refute a clinical trial for millions of dollars is very difficult.
- graeme 12y ago
  So can we still call it science, or not? And if not, what do we call it?
  - chris_va 12y ago
    Well, technically they are replicating the study over 50+ different hospitals until they reach some confidence interval on the safety data. If you get fraud in a small fraction of them, it should not (and if you read the actual testimony, does not) impact the net result.
    - static_noise 12y ago
      If you need 50 experiments to actually show that the effect exists, then even a small fraction of fraudulent experiments can heavily skew the result when looking at confidence intervals. Think of selfhealing as a coin toss with 50/50 chance and a medication increasing this chance: a) Medication increases healing chances to 80% -> Only few experiments needed. b) Medication increases healing chances to 55% -> Many experiments needed to prove that the effect exists. c) Medication is not effective but 5% fraud is introduced so that data shows 55% healing chances -> The same as in b) but with only "a little" fraud involoved.
      - kirsebaer 12y ago
        Yes. Many medications are not that effective. The supposed effect seen in clinical trials is very small and could be easily manipulated.
      - logfromblammo 12y ago
        Don't forget the placebo effect. A major flaw I see in most medications studies is that the market prices for the experimental drug or placebo are not included in the study. My hypothesis is that a placebo described as costing $1000 per dose will produce a greater effect than a placebo described as costing $10 per dose, even if the patients are not asked to pay that cost. Could it be that those drugs that barely outperform a gratis placebo are only producing that effect because the act of charging more for them makes them better placebos? If that's the case, the fraud would only be needed to eliminate evidence of harm to get approval for a bogus remedy. All the effectiveness would come from magnified placebo effect.
        static_noise 12y ago
        Current stdies should account and compensate for the placebo effect (double blind study). I think there was a study which compared different placebos and the more expensive ones showed better effects. So your hypothesis may be right.
  - gerry_shaw 12y ago
    If all the steps are made available in the paper to reproduce the experiment I would consider it science. Just because people don't replicate the study because of other factors like time or money shouldn't take away from the first experiment. Somebody has to be first. That said, is an interesting question.
  - Intermernet 12y ago
    You can call it research. If it's done according to the scientific method, then it can be called scientific research. If not, it's just research.
- tjradcliffe 12y ago
  This is an area where some kind of crowd-sourcing seems like it ought to work. The facts on side effects and morality exist once a drug is out in the wild, but we don't have any way of collecting it, curating it, or analyzing it. Yet odds are that most of the people taking it (for any value of "it") have smartphones and could be prospectively matched with people not taking it (I'm an enemy of case-control studies for lots of reasons, but this may be an area where a well-designed one would work.)
- threatofrain 12y ago
  Also, I don't think people do pure replications. People generally do replication with a twist, which is essentially a new branched experiment. Rather than confirm one specific part of a theory more than once, it's better to confirm various wings or implications of a theory.
- emcrazyone 12y ago
  @Scaevolus so very true. I use to work in this field. I was the IT director for a small pharmaceutical company that produced prescription creams and ointments and worked with a nearby university to work on new drugs. My role was to develop the custom software that ran on laptops we gave to hospitals involved in the clinical trial. In the drug manufacturing world, anytime computer software is used in a clinical trial or in the tracking of drug production, you have to have the software validated. Software validation is basically a process by which you hire an outside company to go over what your software does. Similar to an ISO certification. The folks & companies you hire are very clueless... They know enough buzz words to ask good questions but they often over look things. Anyway, long story short, the whole process of just collecting data is riddled with security problems. None of this, as far as I know, is exposed to the public either. So if buggy software is, somehow, corrupting test results, there is not feedback loop to correct or discard the data. Companies like the one I worked for at the time, would throw out the data as a policy and start over. Hence, very expensive situation and very crucial to make sure your systems do the right thing the first time. This was in the late 90s and in one situation I was in charge of some software that we were developing that was expected run past Y2K. Back then there was a lot of hoopla about the Y2K century roll over and what it would do to computer systems. The company we hired validated our software for Y2K use cases but they often over looked basic things like data encryption, plain text storage of user passwords, etc... I imagine the validation companies have gotten better over the years but in the end I learned that software validation is nothing more than a big CYA tactic. It's a paper trail to try and claim we did our due diligence if the shit were to hit the fan.
  - EpicEng 12y ago
    I'm also in the medical device industry, though we don't farm our V&V work out. The bugs are supposed to be caught in the verification phase; validation is all about making sure that the software is functioning correctly from a user perspective? Are you just conflating the terms here for the sake of simplicity?
refurb 12y ago
How much of this could be solved by replication? The FDA won't approve a drug without at least two positive phase III trials (some exceptions for oncology and other fatal diseases).
- kirsebaer 12y ago
  Two positive trials, but if there are additional negative trials those can be ignored. There is no requirement to combine all available trials in a meta-analysis. Very easy for the pharm company to come up with some excuse to throw away "bad" data.
  - refurb 12y ago
    I don't disagree, but I also don't think it's a bad thing. Most of the time the multiple failed trials occur in diseases where our current tools for measuring effect suck (CNS drugs). The FDA acknowledges this. Keep in mind that drug companies must submit all clinical trial data to the FDA (including all of the failures). It's not like the drug companies are pulling a fast one on the FDA. The FDA is looking at the totality of the evidence and making a call.
snowwrestler 12y ago
The validity of scientific knowledge does not depend on replication, it depends on confirmation and agreement. Merely repeating the original experiment might simply repeat the design errors of the original experiment. For example what if I drop a feather a bunch of times and find a value for g (gravitational acceleration) of 3.4 m/s^2? If you drop a similar feather you'll probably get a similar value, but that doesn't make it right. The real question is not replication, it's whether I've found an accurate value for g. In that respect, science is better advanced if you attempt to confirm this value by designing your own experiment to test it, by dropping something else (a rock, maybe) instead of a feather. We're well beside the original point of this thread now, but the focus on replication is a bit of a pet peeve of mine. edit: clarity

by graeme 12y ago

Scaevolus 12y ago

Clinical trials are very expensive to run. The only people financially interested in running them are the pharmaceutical companies trying to make new products available. Replication is hard enough to achieve in hard sciences where replicating a study might cost tens of thousands of dollars. Finding the money to replicate or refute a clinical trial for millions of dollars is very difficult.

graeme 12y ago
So can we still call it science, or not? And if not, what do we call it?
- chris_va 12y ago
  Well, technically they are replicating the study over 50+ different hospitals until they reach some confidence interval on the safety data. If you get fraud in a small fraction of them, it should not (and if you read the actual testimony, does not) impact the net result.
  - static_noise 12y ago
    If you need 50 experiments to actually show that the effect exists, then even a small fraction of fraudulent experiments can heavily skew the result when looking at confidence intervals. Think of selfhealing as a coin toss with 50/50 chance and a medication increasing this chance: a) Medication increases healing chances to 80% -> Only few experiments needed. b) Medication increases healing chances to 55% -> Many experiments needed to prove that the effect exists. c) Medication is not effective but 5% fraud is introduced so that data shows 55% healing chances -> The same as in b) but with only "a little" fraud involoved.
    - kirsebaer 12y ago
      Yes. Many medications are not that effective. The supposed effect seen in clinical trials is very small and could be easily manipulated.
    - logfromblammo 12y ago
      Don't forget the placebo effect. A major flaw I see in most medications studies is that the market prices for the experimental drug or placebo are not included in the study. My hypothesis is that a placebo described as costing $1000 per dose will produce a greater effect than a placebo described as costing $10 per dose, even if the patients are not asked to pay that cost. Could it be that those drugs that barely outperform a gratis placebo are only producing that effect because the act of charging more for them makes them better placebos? If that's the case, the fraud would only be needed to eliminate evidence of harm to get approval for a bogus remedy. All the effectiveness would come from magnified placebo effect.
      - static_noise 12y ago
        Current stdies should account and compensate for the placebo effect (double blind study). I think there was a study which compared different placebos and the more expensive ones showed better effects. So your hypothesis may be right.
- gerry_shaw 12y ago
  If all the steps are made available in the paper to reproduce the experiment I would consider it science. Just because people don't replicate the study because of other factors like time or money shouldn't take away from the first experiment. Somebody has to be first. That said, is an interesting question.
- Intermernet 12y ago
  You can call it research. If it's done according to the scientific method, then it can be called scientific research. If not, it's just research.
tjradcliffe 12y ago
This is an area where some kind of crowd-sourcing seems like it ought to work. The facts on side effects and morality exist once a drug is out in the wild, but we don't have any way of collecting it, curating it, or analyzing it. Yet odds are that most of the people taking it (for any value of "it") have smartphones and could be prospectively matched with people not taking it (I'm an enemy of case-control studies for lots of reasons, but this may be an area where a well-designed one would work.)
threatofrain 12y ago
Also, I don't think people do pure replications. People generally do replication with a twist, which is essentially a new branched experiment. Rather than confirm one specific part of a theory more than once, it's better to confirm various wings or implications of a theory.
emcrazyone 12y ago
@Scaevolus so very true. I use to work in this field. I was the IT director for a small pharmaceutical company that produced prescription creams and ointments and worked with a nearby university to work on new drugs. My role was to develop the custom software that ran on laptops we gave to hospitals involved in the clinical trial. In the drug manufacturing world, anytime computer software is used in a clinical trial or in the tracking of drug production, you have to have the software validated. Software validation is basically a process by which you hire an outside company to go over what your software does. Similar to an ISO certification. The folks & companies you hire are very clueless... They know enough buzz words to ask good questions but they often over look things. Anyway, long story short, the whole process of just collecting data is riddled with security problems. None of this, as far as I know, is exposed to the public either. So if buggy software is, somehow, corrupting test results, there is not feedback loop to correct or discard the data. Companies like the one I worked for at the time, would throw out the data as a policy and start over. Hence, very expensive situation and very crucial to make sure your systems do the right thing the first time. This was in the late 90s and in one situation I was in charge of some software that we were developing that was expected run past Y2K. Back then there was a lot of hoopla about the Y2K century roll over and what it would do to computer systems. The company we hired validated our software for Y2K use cases but they often over looked basic things like data encryption, plain text storage of user passwords, etc... I imagine the validation companies have gotten better over the years but in the end I learned that software validation is nothing more than a big CYA tactic. It's a paper trail to try and claim we did our due diligence if the shit were to hit the fan.
- EpicEng 12y ago
  I'm also in the medical device industry, though we don't farm our V&V work out. The bugs are supposed to be caught in the verification phase; validation is all about making sure that the software is functioning correctly from a user perspective? Are you just conflating the terms here for the sake of simplicity?

refurb 12y ago

How much of this could be solved by replication? The FDA won't approve a drug without at least two positive phase III trials (some exceptions for oncology and other fatal diseases).

kirsebaer 12y ago
Two positive trials, but if there are additional negative trials those can be ignored. There is no requirement to combine all available trials in a meta-analysis. Very easy for the pharm company to come up with some excuse to throw away "bad" data.
- refurb 12y ago
  I don't disagree, but I also don't think it's a bad thing. Most of the time the multiple failed trials occur in diseases where our current tools for measuring effect suck (CNS drugs). The FDA acknowledges this. Keep in mind that drug companies must submit all clinical trial data to the FDA (including all of the failures). It's not like the drug companies are pulling a fast one on the FDA. The FDA is looking at the totality of the evidence and making a call.

snowwrestler 12y ago

The validity of scientific knowledge does not depend on replication, it depends on confirmation and agreement. Merely repeating the original experiment might simply repeat the design errors of the original experiment. For example what if I drop a feather a bunch of times and find a value for g (gravitational acceleration) of 3.4 m/s^2? If you drop a similar feather you'll probably get a similar value, but that doesn't make it right. The real question is not replication, it's whether I've found an accurate value for g. In that respect, science is better advanced if you attempt to confirm this value by designing your own experiment to test it, by dropping something else (a rock, maybe) instead of a feather. We're well beside the original point of this thread now, but the focus on replication is a bit of a pet peeve of mine. edit: clarity