4 ms·

Medical Science Has a Data Problem

88 points by juanplusjuan 12y ago

tomohawk 12y ago
It appears that the FDA also has a data problem, as it refuses to respond to FOIA requests for data that is obviously in the public interest to be disclosed, and taxpayers end up paying $1.3 billion to stockpile a medicine that is probably not as effective as advertised. How does this practice keep us safer?
- Alex3917 12y ago
  They also actually put a gag order on at least one pharma company preventing from disclosing the fact that antidepressants were not effective in children and increased the risk of suicide. The thing about medical studies though is that only the ones the pharma companies want to get published make it into journals, and there is no requirement that what they are saying in the journal articles matches the clinical trial data at all. And the FDA doesn't publish any of the raw clinical trial data, so you'll never know. This is especially problematic since pharma companies only need two studies showing a drug plausibly works to get approval, regardless of how many trials fail to show any benefit. So usually what you see is one or two journal articles showing a drug works, which may or may not correspond to the findings of one or two clinical trials, and the rest of the trial data is buried forever. And then other authors publish metareviews based on this completely inaccurate data. Essentially journal articles should only be ever be viewed as advertisements for the pharma industry, and should no way ever be taken as science. The one exception is the largescale trials funded by the NIH.
  - ihnorton 12y ago
    There have been some changes [1] in recent years attempting to address this. Since 2007, all drug, biologic, and device trials in the US must be registered with ClinicalTrials.gov after phase 1 (which, at least in principle, addresses safety only - not efficacy), and "results" must be reported. There could certainly be some loopholes or limitations on the data release that I am unaware of. However, it is true that most of the controversial, blockbuster drugs that are discussed in the original article were approved long before these requirements came into effect. [1] http://clinicaltrials.gov/ct2/manage-recs/fdaaa http://clinicaltrials.gov/ct2/manage-recs/fdaaa
    - Alex3917 12y ago
      > Since 2007, all drug, biologic, and device trials in the US must be registered with ClinicalTrials.gov after phase 1 (which, at least in principle, addresses safety only - not efficacy), and "results" must be reported. - A substantial percentage of trials haven't reported their results within the required timeframe. - The number of fines the FDA has levied for non-compliance is zero. - The data isn't actually published on clinicaltrials.gov, only a brief summary of the results which doesn't really tell you anything. - As you say, the requirements aren't retroactive. - There are several large loopholes, e.g. phase IV studies aren't required to be registered.
mironathetin 12y ago
I recently listened to an advanced statistics lecture. As an illustration the Professor said, medical science (still) uses 2 sigma statistics, jet engine builders use 6 sigma. That's why jet engines never (sic) fail. Yes, and then there is the influence of the drug makers too.
- mikecsh 12y ago
  Do you have a link/reference to this lecture?
- tomp 12y ago
  The biggest difference is that jet engine builders can conduct an unlimited number proper controlled scientific experiments, while the fields that study humans (psychology, medicine, sociology, economy) can usually only conduct a limited number of field trials (which, even though they are "controlled" and possibly "double blind" are nowhere as rigorous as in proper sciences).
  - paulojreis 12y ago
    This is a very important point that seems to be forgotten frequently. Particularly when doing methodological comparison between disciplines. The other point is complexity. The number of variables needed to create a useful model of a jet engine is a lot smaller than the number needed to model an organic system. This means that today we have useful models of jet engines and we can reliably test something with simulations - we may generate loads of valid data and easily apply "top-notch" statistics. The same, unfortunately, is not true for medicine.
  - icegreentea 12y ago
    Yes. There was the other article sitting around recently asking why the rate of blockbuster drugs have slowed down. Getting even the level of confidence required now is hugely expensive. Requiring higher confidence is an expensive choice that must be weighed closely. I think it's becoming clearer each day that many drug side effects, and general drug efficacy is effected by the genetics of the drug taker. Without being able to tease out these genetic effects, blindly aiming for broad population safety or efficacy levels is always going to be troublesome. This is separate from the data publication issue that the article focuses on.
Houshalter 12y ago
Silly question, why do so many drugs have rare side effects? Why does it work on 999 people and then the 1,000th person takes it and becomes suicidal? If there is something different biologically different about the 1,000th person, could it be tested for?
- rojost 12y ago
  That's part of the aim of personalised medicine. http://en.wikipedia.org/wiki/Personalized_medicine http://en.wikipedia.org/wiki/Personalized_medicine
- Alex3917 12y ago
  It doesn't work on any of the 999 people, that's the point of the article.
- davecap1 12y ago
  That's not really a silly question :) Drugs act on proteins which are little machines in the body, the instructions of which are encoded in our genome. The slight variations in our genetic code leads to some proteins having slight variations in their structure (and their function). Most of the time this is harmful, but sometimes it causes them to react differently when they interact with a drug or with each other. The manifestation of this can sometimes be harmful. Understanding "pharmacogenomic" effects will help us mitigate these side-effects going forward.
- cryoshon 12y ago
  A person's genome, epigenetic profile, physical environment, social environment, microbiome, parasite-ome, neural connectome, metabolism, enteric connectome, and diet are all unique. Some of these factors regulate the others; some have two-way regulation. There are far more combinations and variations of the above variables and sub-variables than can be tested for in clinical trials. This fact means that some obscure combination of these variables may result in unintended negative (or positive, less frequently) consequences upon consumption. Theoretically, you could test for differences in the 1000th person-- with today's technology, it would be a test which described the person's genome. Once scientists had that genomic data in hand, they'd certainly find a profile of alleles which could feasibly describe why a negative reaction occurred. Importantly, everyone's genome is unique, meaning that a correlation between genomes and negative drug reactions must be exceptionally rigorous in order to be believable. Currently, this correlation poses a problem, as our ability to quantify and understand genomes is far ahead of the other dimensions which I mentioned at the start of my post. As our understanding of the microbiome, enteric connectome, and epigenetics increases, it'll be more possible to predict adverse drug reactions beforehand, assuming the patients in question have been profiled.
pron 12y ago
Medical science does indeed have a data (or, rather, rigorous statistics problem), but it has a much more serious, intrinsic, problem: dimensionality. The number of variables is so high that proving something (clinically, when the biological process is unclear, which is most of the time) to a high degree of certainty is almost impossible.
- robrenaud 12y ago
  Why doesn't randomization defeat dimensionality?
  - lmkg 12y ago
    Randomization lets you ignore dimensions that you're not interested in. It doesn't do much in the way of letting you attribute an effect to one or more dimensions from a large set of them. There's a fundamental lower bound to the number of data-points you need to make in order to achieve statistical significance. This lower bound grows exponentially in the number of dimensions, particularly if you cannot make the simplifying assumption that the dimensions are statistically independent (and biological systems are often coupled in strange ways). Randomization is a good way of achieving that lower bound, but it cannot defeat it.
tokenadult 12y ago
I read the article last evening before I turned in in this time zone, and have just read all the comments posted till now overnight. Readers of the writings of Dr. John P.A. Ioannidis[1] will already be familiar with many of these issues, which dog most research on human health and medical treatments. Several of the previous comments here are very helpful in pointing out that human physiology is based on much more complicated system interactions than most man-made machines, and subtle individual variations in genome or personal environment that are now poorly understood can make a different in how drugs work for different patients. That said, an important point that hasn't been made here yet is that "alternative" treatments to the treatments developed by medical science often have a much worse data problem, because no one is gathering data about the alternative treatments for safety or for effectiveness in the first place. In the United States, "natural supplements" are regulated in a way that exempts billions of dollars of annual revenue from any review about whether the supplements do any good at all.[2] Chiropractic manipulation can kill, but chiropractors don't look into that issue carefully.[3] What you as a person who desires to stay healthy for many years have to do is turn in the first instance to medicine that works[4] and to continually ask your doctors for explanations of what they are doing and what the rationale is for what they are doing. Patients thinking more will encourage physicians to think more. Voters telling governments to regulate treatments more rationally will also help, as will voters supporting more funding of basic medical research. [1] https://med.stanford.edu/profiles/john-ioannidis?tab=publications https://med.stanford.edu/profiles/john-ioannidis?tab=publica... [2] "Dietary supplement industry says “no” to more information for consumers (again)" http://www.sciencebasedmedicine.org/big-supp-resists-giving-consumers-safety-and-effectiveness-information/ http://www.sciencebasedmedicine.org/big-supp-resists-giving-... [3] "Stroke Death from Chiropractic Neck Manipulation" http://www.sciencebasedmedicine.org/stroke-death-from-chiropractic-neck-manipulation/ http://www.sciencebasedmedicine.org/stroke-death-from-chirop... [4] "There is no such thing as alternative medicine. There is only medicine that works and medicine that doesn't." http://books.google.com/books?id=zOh0t3QFXdoC&dq=There+is+no+such+thing+as+alternative+medicine.+There+is+only+medicine+that+works+and+medicine+that+doesn%27t http://books.google.com/books?id=zOh0t3QFXdoC&dq=There+is+no...
leoc 12y ago
Ben Goldacre has a book on the subject http://www.amazon.com/gp/offer-listing/0007350740/ http://www.amazon.com/gp/offer-listing/0007350740/ in addition to that TED talk.
conorh 12y ago
I don't think that anyone outside the field can understand just how bad and pervasive this problem is, it is everywhere and harming patients every day. One more example - my wife is a surgeon working in a specialized field and in this field there are panels of 'experts' that get together and set the guidelines for how to treat the disease: - The panel is stacked with visiting researchers working for the most senior researcher on the panel. - The most senior experts on this panel recommend treating with a particular drug. It turns out that these experts also happen to receive money from these drug companies. - These researchers then use the results of this panel when applying for NIH grants as proof that this is where the treatment is heading AND THEN, next time the panel happens, also use this research as proof that this is where the treatment is heading. Talk about self interest and circular logic. What is incredibly sad about this is that the results of consensus panels are then quoted by doctors every day and used to lead their treatment of patients.
radnam 12y ago
Clinical trials data is one piece of the puzzle and definitely a important one. One thing which in my opinion would dramatically change patient outcomes if care provider starts listening to the patient. In an anecdotal experience, my wife was wrongly diagnosed for almost a year impacting every day of her life to be filled with pain. Issue as it turned out was a simple infection and was fixed by an exceptional surgeon with tweezers in few minutes.

88 points by juanplusjuan 12y ago

It appears that the FDA also has a data problem, as it refuses to respond to FOIA requests for data that is obviously in the public interest to be disclosed, and taxpayers end up paying $1.3 billion to stockpile a medicine that is probably not as effective as advertised. How does this practice keep us safer?

Alex3917 12y ago
They also actually put a gag order on at least one pharma company preventing from disclosing the fact that antidepressants were not effective in children and increased the risk of suicide. The thing about medical studies though is that only the ones the pharma companies want to get published make it into journals, and there is no requirement that what they are saying in the journal articles matches the clinical trial data at all. And the FDA doesn't publish any of the raw clinical trial data, so you'll never know. This is especially problematic since pharma companies only need two studies showing a drug plausibly works to get approval, regardless of how many trials fail to show any benefit. So usually what you see is one or two journal articles showing a drug works, which may or may not correspond to the findings of one or two clinical trials, and the rest of the trial data is buried forever. And then other authors publish metareviews based on this completely inaccurate data. Essentially journal articles should only be ever be viewed as advertisements for the pharma industry, and should no way ever be taken as science. The one exception is the largescale trials funded by the NIH.
- ihnorton 12y ago
  There have been some changes [1] in recent years attempting to address this. Since 2007, all drug, biologic, and device trials in the US must be registered with ClinicalTrials.gov after phase 1 (which, at least in principle, addresses safety only - not efficacy), and "results" must be reported. There could certainly be some loopholes or limitations on the data release that I am unaware of. However, it is true that most of the controversial, blockbuster drugs that are discussed in the original article were approved long before these requirements came into effect. [1] http://clinicaltrials.gov/ct2/manage-recs/fdaaa http://clinicaltrials.gov/ct2/manage-recs/fdaaa
  - Alex3917 12y ago
    > Since 2007, all drug, biologic, and device trials in the US must be registered with ClinicalTrials.gov after phase 1 (which, at least in principle, addresses safety only - not efficacy), and "results" must be reported. - A substantial percentage of trials haven't reported their results within the required timeframe. - The number of fines the FDA has levied for non-compliance is zero. - The data isn't actually published on clinicaltrials.gov, only a brief summary of the results which doesn't really tell you anything. - As you say, the requirements aren't retroactive. - There are several large loopholes, e.g. phase IV studies aren't required to be registered.

mironathetin 12y ago

I recently listened to an advanced statistics lecture. As an illustration the Professor said, medical science (still) uses 2 sigma statistics, jet engine builders use 6 sigma. That's why jet engines never (sic) fail. Yes, and then there is the influence of the drug makers too.

mikecsh 12y ago
Do you have a link/reference to this lecture?
tomp 12y ago
The biggest difference is that jet engine builders can conduct an unlimited number proper controlled scientific experiments, while the fields that study humans (psychology, medicine, sociology, economy) can usually only conduct a limited number of field trials (which, even though they are "controlled" and possibly "double blind" are nowhere as rigorous as in proper sciences).
- paulojreis 12y ago
  This is a very important point that seems to be forgotten frequently. Particularly when doing methodological comparison between disciplines. The other point is complexity. The number of variables needed to create a useful model of a jet engine is a lot smaller than the number needed to model an organic system. This means that today we have useful models of jet engines and we can reliably test something with simulations - we may generate loads of valid data and easily apply "top-notch" statistics. The same, unfortunately, is not true for medicine.
- icegreentea 12y ago
  Yes. There was the other article sitting around recently asking why the rate of blockbuster drugs have slowed down. Getting even the level of confidence required now is hugely expensive. Requiring higher confidence is an expensive choice that must be weighed closely. I think it's becoming clearer each day that many drug side effects, and general drug efficacy is effected by the genetics of the drug taker. Without being able to tease out these genetic effects, blindly aiming for broad population safety or efficacy levels is always going to be troublesome. This is separate from the data publication issue that the article focuses on.

Houshalter 12y ago

Silly question, why do so many drugs have rare side effects? Why does it work on 999 people and then the 1,000th person takes it and becomes suicidal? If there is something different biologically different about the 1,000th person, could it be tested for?

rojost 12y ago
That's part of the aim of personalised medicine. http://en.wikipedia.org/wiki/Personalized_medicine http://en.wikipedia.org/wiki/Personalized_medicine
Alex3917 12y ago
It doesn't work on any of the 999 people, that's the point of the article.
davecap1 12y ago
That's not really a silly question :) Drugs act on proteins which are little machines in the body, the instructions of which are encoded in our genome. The slight variations in our genetic code leads to some proteins having slight variations in their structure (and their function). Most of the time this is harmful, but sometimes it causes them to react differently when they interact with a drug or with each other. The manifestation of this can sometimes be harmful. Understanding "pharmacogenomic" effects will help us mitigate these side-effects going forward.
cryoshon 12y ago
A person's genome, epigenetic profile, physical environment, social environment, microbiome, parasite-ome, neural connectome, metabolism, enteric connectome, and diet are all unique. Some of these factors regulate the others; some have two-way regulation. There are far more combinations and variations of the above variables and sub-variables than can be tested for in clinical trials. This fact means that some obscure combination of these variables may result in unintended negative (or positive, less frequently) consequences upon consumption. Theoretically, you could test for differences in the 1000th person-- with today's technology, it would be a test which described the person's genome. Once scientists had that genomic data in hand, they'd certainly find a profile of alleles which could feasibly describe why a negative reaction occurred. Importantly, everyone's genome is unique, meaning that a correlation between genomes and negative drug reactions must be exceptionally rigorous in order to be believable. Currently, this correlation poses a problem, as our ability to quantify and understand genomes is far ahead of the other dimensions which I mentioned at the start of my post. As our understanding of the microbiome, enteric connectome, and epigenetics increases, it'll be more possible to predict adverse drug reactions beforehand, assuming the patients in question have been profiled.

pron 12y ago

Medical science does indeed have a data (or, rather, rigorous statistics problem), but it has a much more serious, intrinsic, problem: dimensionality. The number of variables is so high that proving something (clinically, when the biological process is unclear, which is most of the time) to a high degree of certainty is almost impossible.

robrenaud 12y ago
Why doesn't randomization defeat dimensionality?
- lmkg 12y ago
  Randomization lets you ignore dimensions that you're not interested in. It doesn't do much in the way of letting you attribute an effect to one or more dimensions from a large set of them. There's a fundamental lower bound to the number of data-points you need to make in order to achieve statistical significance. This lower bound grows exponentially in the number of dimensions, particularly if you cannot make the simplifying assumption that the dimensions are statistically independent (and biological systems are often coupled in strange ways). Randomization is a good way of achieving that lower bound, but it cannot defeat it.

tokenadult 12y ago

I read the article last evening before I turned in in this time zone, and have just read all the comments posted till now overnight. Readers of the writings of Dr. John P.A. Ioannidis[1] will already be familiar with many of these issues, which dog most research on human health and medical treatments. Several of the previous comments here are very helpful in pointing out that human physiology is based on much more complicated system interactions than most man-made machines, and subtle individual variations in genome or personal environment that are now poorly understood can make a different in how drugs work for different patients. That said, an important point that hasn't been made here yet is that "alternative" treatments to the treatments developed by medical science often have a much worse data problem, because no one is gathering data about the alternative treatments for safety or for effectiveness in the first place. In the United States, "natural supplements" are regulated in a way that exempts billions of dollars of annual revenue from any review about whether the supplements do any good at all.[2] Chiropractic manipulation can kill, but chiropractors don't look into that issue carefully.[3] What you as a person who desires to stay healthy for many years have to do is turn in the first instance to medicine that works[4] and to continually ask your doctors for explanations of what they are doing and what the rationale is for what they are doing. Patients thinking more will encourage physicians to think more. Voters telling governments to regulate treatments more rationally will also help, as will voters supporting more funding of basic medical research. [1] https://med.stanford.edu/profiles/john-ioannidis?tab=publications https://med.stanford.edu/profiles/john-ioannidis?tab=publica... [2] "Dietary supplement industry says “no” to more information for consumers (again)" http://www.sciencebasedmedicine.org/big-supp-resists-giving-consumers-safety-and-effectiveness-information/ http://www.sciencebasedmedicine.org/big-supp-resists-giving-... [3] "Stroke Death from Chiropractic Neck Manipulation" http://www.sciencebasedmedicine.org/stroke-death-from-chiropractic-neck-manipulation/ http://www.sciencebasedmedicine.org/stroke-death-from-chirop... [4] "There is no such thing as alternative medicine. There is only medicine that works and medicine that doesn't." http://books.google.com/books?id=zOh0t3QFXdoC&dq=There+is+no+such+thing+as+alternative+medicine.+There+is+only+medicine+that+works+and+medicine+that+doesn%27t http://books.google.com/books?id=zOh0t3QFXdoC&dq=There+is+no...

leoc 12y ago

Ben Goldacre has a book on the subject http://www.amazon.com/gp/offer-listing/0007350740/ http://www.amazon.com/gp/offer-listing/0007350740/ in addition to that TED talk.

conorh 12y ago

I don't think that anyone outside the field can understand just how bad and pervasive this problem is, it is everywhere and harming patients every day. One more example - my wife is a surgeon working in a specialized field and in this field there are panels of 'experts' that get together and set the guidelines for how to treat the disease: - The panel is stacked with visiting researchers working for the most senior researcher on the panel. - The most senior experts on this panel recommend treating with a particular drug. It turns out that these experts also happen to receive money from these drug companies. - These researchers then use the results of this panel when applying for NIH grants as proof that this is where the treatment is heading AND THEN, next time the panel happens, also use this research as proof that this is where the treatment is heading. Talk about self interest and circular logic. What is incredibly sad about this is that the results of consensus panels are then quoted by doctors every day and used to lead their treatment of patients.

radnam 12y ago

Clinical trials data is one piece of the puzzle and definitely a important one. One thing which in my opinion would dramatically change patient outcomes if care provider starts listening to the patient. In an anecdotal experience, my wife was wrongly diagnosed for almost a year impacting every day of her life to be filled with pain. Issue as it turned out was a simple infection and was fixed by an exceptional surgeon with tweezers in few minutes.