10 ms·

Make journals report clinical trials properly

109 points by bootload 11y ago

timrpeterson 11y ago
Irony for Nature to be talking ill of other journals' research reporting "standards". Pot. Kettle. Black.
- chaz72 11y ago
  If they could make progress on the problem, I might forgive many kettles, I mean ills.
- capnrefsmmat 11y ago
  I was reasonably impressed with Nature's new(ish) reporting checklist, at least, which exceeds what most other journals are doing: http://www.nature.com/authors/policies/checklist.pdf http://www.nature.com/authors/policies/checklist.pdf
  - ekianjo 11y ago
    Yet Nature regularly publishes studies which are not properly statistically powered or with wrong p-values conclusions.
    - tokenadult 11y ago
      For example?
- awakeasleep 11y ago
  "The pot calling the kettle black" is only meant to argue against a defensive counter-accusation. It is same idea as "two wrongs don't make a right" The pot can legitimately call the kettle black and vice versa. They both have the problem.
  - harryjo 11y ago
    There are two common interpretations of the "The pot calling the kettle black", neither of which is as you describe. https://en.wikipedia.org/wiki/The_pot_calling_the_kettle_black https://en.wikipedia.org/wiki/The_pot_calling_the_kettle_bla...
- dekhn 11y ago
  This is an editorial by an external author. It does not necessarily represent Nature's leadership's position.
Pyxl101 11y ago
It's always interesting to look at past ages through the lens of the modern age. I wonder if, decades or centuries from now, the misuse of statistics and inadequate up-front documentation of research methods in science will be seen as a great scientific scandal, or scientific failure, of the 20th and 21st centuries. We should have known better, but many of us failed in these basic ways. An NPR episode of Planet Money covered the "Experiment Experiment": http://www.npr.org/sections/money/2016/01/15/463237871/episode-677-the-experiment-experiment http://www.npr.org/sections/money/2016/01/15/463237871/episo... and described how the Reproducibility Project was unable to successfully produce a significant percent of studies that they undertook to reproduce. There is legitimate concern that data-dredging leads to misunderstanding or misrepresenting of results; and even if some of the studies are valid science, it's clear that the documentation of the studies do not suffice to allow others to replicate it, which is the bedrock of science. The episode above also discussed a number of psychological fallacies and errors that experimenters make. Imagine that you begin a study of 200 people, measuring some variable. You see promising results, so you decide to extend the study and add more people; you test another 100 people. By adding 100 more people to your study, have you increased or decreased the likelihood that your results were due to statistical chance? Counter-intuitively, the answer is that you have increased the odds that the result was due to chance. You cannot add or remove people to a study while it is in progress, based on preliminary results in the study, without impacting its statistical validity in important ways. These and other changes that experimenters make during the course of an experiment create a great risk of conducting bad science. I for one support any effort to encourage all research studies and methods to be declared ahead of time.
- nonbel 11y ago
  >"Imagine that you begin a study of 200 people, measuring some variable. You see promising results, so you decide to extend the study and add more people; you test another 100 people. By adding 100 more people to your study, have you increased or decreased the likelihood that your results were due to statistical chance? Counter-intuitively, the answer is that you have increased the odds that the result was due to chance." I'll assume you are talking about using a t-test to see if two groups are samples from the same population. The problem you point out has nothing to do with chance. It is that you started with a null hypothesis that your groups were independent samples from the same distribution, but by having the second 100 people sampled conditional on the results of the first 200, you have ensured this is not true. Such research designs are just a way of making sure the null hypothesis is false. If you get rid of the incorrect attribution to "chance", I don't see what is counter-intuitive about it.
gdewilde 11y ago
John Ioannidis: "Reproducible Research: True or False?" | Talks at Google https://www.youtube.com/watch?v=GPYzY9I78CI https://www.youtube.com/watch?v=GPYzY9I78CI
Pyxl101 11y ago
I would like to eventually see a "science results build system", where experimenters can register (preferably ahead of time) the analytical methods they will use to analyze their data, expressed as runnable code. Any data sets already available, on which they plan to rely, will be similarly registered and made available to the "build system". When the study is complete, the researchers upload their data, in as raw a form as technology permits. All analysis performed on the data to achieve the results should be expressed through runnable code evaluated by the build system. Ideally 100% of this code would be registered in advance, though to be realistic, in practice we'd find some of it to be written along the way. Whether the code is registered in advance or written and registered along the way, in both cases the build system will have the complete code and data set, and will be able to reproduce the results. Ideally the experimenters simply take the output of the build system as their actual results. The scientific community will have complete visibility into the analytical methods of the study, since all data and analysis will have been captured by the build system. The build system would ideally be agnostic to tooling and data format. Perhaps it is something like a data store alongside a runtime platform, capable of taking a virtual machine image and a data set, and executing the VM with the data set as input. Tagline: "GitHub for scientific data and runnable experimental analysis"
- pjschlic 11y ago
  This reminds me of the recent Planet Money "The Experiment Experiment" and they pointed out some journals requesting pre-registered experiments (pre-commited experiment methods and analysis submitted to somewhere online before the experiment is run), it's at the 18-ish minute mark.
- kriro 11y ago
  This sounds like a pretty great idea if I understand it correctly. My cynical view of the business of science says it'll never get traction but the base idea sounds like a great way to get rid of a lot of cheating methods. It would also result in research that I'd immediately trust. Alas it's probably possible to game this theoretical system because you can still upload faulty data that leads to the desired results since the registered code can probably be reversed somewhat easily/would be open. I think there's two issues (which can also be combined). 1) Fake/false data. 2) Fiddling with methods until you get the results you want given a fixed data set. Your solution would be excellent for getting rid of problem #2. I think #1 is quite a bit harder and I haven't come up with anything better than "as open as possible"

109 points by bootload 11y ago

timrpeterson 11y ago

Irony for Nature to be talking ill of other journals' research reporting "standards". Pot. Kettle. Black.

chaz72 11y ago
If they could make progress on the problem, I might forgive many kettles, I mean ills.
capnrefsmmat 11y ago
I was reasonably impressed with Nature's new(ish) reporting checklist, at least, which exceeds what most other journals are doing: http://www.nature.com/authors/policies/checklist.pdf http://www.nature.com/authors/policies/checklist.pdf
- ekianjo 11y ago
  Yet Nature regularly publishes studies which are not properly statistically powered or with wrong p-values conclusions.
  - tokenadult 11y ago
    For example?
awakeasleep 11y ago
"The pot calling the kettle black" is only meant to argue against a defensive counter-accusation. It is same idea as "two wrongs don't make a right" The pot can legitimately call the kettle black and vice versa. They both have the problem.
- harryjo 11y ago
  There are two common interpretations of the "The pot calling the kettle black", neither of which is as you describe. https://en.wikipedia.org/wiki/The_pot_calling_the_kettle_black https://en.wikipedia.org/wiki/The_pot_calling_the_kettle_bla...
dekhn 11y ago
This is an editorial by an external author. It does not necessarily represent Nature's leadership's position.

Pyxl101 11y ago

It's always interesting to look at past ages through the lens of the modern age. I wonder if, decades or centuries from now, the misuse of statistics and inadequate up-front documentation of research methods in science will be seen as a great scientific scandal, or scientific failure, of the 20th and 21st centuries. We should have known better, but many of us failed in these basic ways. An NPR episode of Planet Money covered the "Experiment Experiment": http://www.npr.org/sections/money/2016/01/15/463237871/episode-677-the-experiment-experiment http://www.npr.org/sections/money/2016/01/15/463237871/episo... and described how the Reproducibility Project was unable to successfully produce a significant percent of studies that they undertook to reproduce. There is legitimate concern that data-dredging leads to misunderstanding or misrepresenting of results; and even if some of the studies are valid science, it's clear that the documentation of the studies do not suffice to allow others to replicate it, which is the bedrock of science. The episode above also discussed a number of psychological fallacies and errors that experimenters make. Imagine that you begin a study of 200 people, measuring some variable. You see promising results, so you decide to extend the study and add more people; you test another 100 people. By adding 100 more people to your study, have you increased or decreased the likelihood that your results were due to statistical chance? Counter-intuitively, the answer is that you have increased the odds that the result was due to chance. You cannot add or remove people to a study while it is in progress, based on preliminary results in the study, without impacting its statistical validity in important ways. These and other changes that experimenters make during the course of an experiment create a great risk of conducting bad science. I for one support any effort to encourage all research studies and methods to be declared ahead of time.

nonbel 11y ago
>"Imagine that you begin a study of 200 people, measuring some variable. You see promising results, so you decide to extend the study and add more people; you test another 100 people. By adding 100 more people to your study, have you increased or decreased the likelihood that your results were due to statistical chance? Counter-intuitively, the answer is that you have increased the odds that the result was due to chance." I'll assume you are talking about using a t-test to see if two groups are samples from the same population. The problem you point out has nothing to do with chance. It is that you started with a null hypothesis that your groups were independent samples from the same distribution, but by having the second 100 people sampled conditional on the results of the first 200, you have ensured this is not true. Such research designs are just a way of making sure the null hypothesis is false. If you get rid of the incorrect attribution to "chance", I don't see what is counter-intuitive about it.

gdewilde 11y ago

John Ioannidis: "Reproducible Research: True or False?" | Talks at Google https://www.youtube.com/watch?v=GPYzY9I78CI https://www.youtube.com/watch?v=GPYzY9I78CI

Pyxl101 11y ago

I would like to eventually see a "science results build system", where experimenters can register (preferably ahead of time) the analytical methods they will use to analyze their data, expressed as runnable code. Any data sets already available, on which they plan to rely, will be similarly registered and made available to the "build system". When the study is complete, the researchers upload their data, in as raw a form as technology permits. All analysis performed on the data to achieve the results should be expressed through runnable code evaluated by the build system. Ideally 100% of this code would be registered in advance, though to be realistic, in practice we'd find some of it to be written along the way. Whether the code is registered in advance or written and registered along the way, in both cases the build system will have the complete code and data set, and will be able to reproduce the results. Ideally the experimenters simply take the output of the build system as their actual results. The scientific community will have complete visibility into the analytical methods of the study, since all data and analysis will have been captured by the build system. The build system would ideally be agnostic to tooling and data format. Perhaps it is something like a data store alongside a runtime platform, capable of taking a virtual machine image and a data set, and executing the VM with the data set as input. Tagline: "GitHub for scientific data and runnable experimental analysis"

pjschlic 11y ago
This reminds me of the recent Planet Money "The Experiment Experiment" and they pointed out some journals requesting pre-registered experiments (pre-commited experiment methods and analysis submitted to somewhere online before the experiment is run), it's at the 18-ish minute mark.
kriro 11y ago
This sounds like a pretty great idea if I understand it correctly. My cynical view of the business of science says it'll never get traction but the base idea sounds like a great way to get rid of a lot of cheating methods. It would also result in research that I'd immediately trust. Alas it's probably possible to game this theoretical system because you can still upload faulty data that leads to the desired results since the registered code can probably be reversed somewhat easily/would be open. I think there's two issues (which can also be combined). 1) Fake/false data. 2) Fiddling with methods until you get the results you want given a fixed data set. Your solution would be excellent for getting rid of problem #2. I think #1 is quite a bit harder and I haven't come up with anything better than "as open as possible"