5 ms·

Is this even true? Most of the progress is in "hard sciences" - physics, cellular biology, computational biology, genetics ... where experiments (and presumably

by tomp 5y ago

Is this even true? Most of the progress is in "hard sciences" - physics, cellular biology, computational biology, genetics ... where experiments (and presumably reproducibility) is just fine.

epistasis 5y ago
As somebody in computational biology, I think it's important to note that a ton of cellular biology and even computational biology is "not reproducible." This could be anything from bad documentation of how something was run (computational scientists are often terrible at lab notebook culture), to having a model system that lived for a while and no longer exists in the form that initial discoveries were made with. Or it could be that there was just that one batch of reagents that reliably reproduced a phenomenon, and all future orders of the reagents no longer get the same phenomenon, and whatever the difference was is lost to time. But I would claim that especially in biology, though this is less than ideal for writing up near little explanations of reality, it is still extremely useful for understanding what's going on. Scientific papers aren't meant to be ever-lasting truth, like a textbook. They are communications amongst specialists about "look here something cool happened that may be useful to you too." It is only through lots of work that a phenomenon can be established as widespread in biology, and sharing information before doing 10 years of work helps accelerate everything. This is why I roll my eyes at complaints like that one about "foundational cancer research papers not being reproducible." It was written by scientists in industry who wanted to take a new paper and develop an entire drug program around something particularly novel and surprising. Sure, that would be ideal if it worked, but the scientific literature is a lot more than a catalog of ideas ready for commercialization. Scientists in practice understand the limitations of taking a journal paper as gospel. They always try to get something working in their own hands before basing a lot of research off another paper.
- SubiculumCode 5y ago
  As a fellow scientist, thank you for your dissemination of how science is actually conducted, and what any single paper or series of papers mean. Progress almost always occurs due to the assessment of converging methods and analytic approaches, almost like individual papers are weak classifiers in a big beautiful bag.
  - epistasis 5y ago
    Exactly, the phenomena are often way too complex for even a series of papers to come up with the proper abstractions to plan the right experiments. The history of chromatin marks is really illustrative of this. We had lots of early experiments to establish the antibodies to even measure the chromatin marks, and then then two decades of gathering enough data to lead to useful abstractions. And now it's all converging with the 3D organizational data from Hi-C experiments, and DNA methylation from methyl-Seq, and over the past decades the measurement technology (driven by DNA sequencing) has advanced so much that somebody in 2001 trying to anticipate this trajectory would be amazed. What was a single paper back then? Reproducible? Maybe, but that's much less important than pointing to the correct direction in this n-dimensional search space of what to examine next. I've never looked at the historical literature from, say, 100 years ago, but I suspect that it's quite the same as today in all regards. There's a ton of great ideas that never get the attention they deserve, a bunch of ideas that get way too much attention due to fashion or due to influencers that chose the wrong path. But who could we trust to chart a better path? Just like startups, we must accept a high failure rate when exploring the unknown.
querez 5y ago
Counterpoint: I have a friend who works in a biomed lab where results would regularly just be made up. If a lab culture takes 5 years to grow, reproducibility is more of a theoretical than a practical matter. Also, not every reviewer has a particle accelerator in their back yard. Or the money to reproduce large computational models. A lot of published results only pass peer review because you essentially trust the authors not to have made up their numbers.
derbOac 5y ago
A lot of the biomedical sciences has similar problems with reproducibility. A survey found similar problems in pharmacology (I think there was one article quoting someone saying the pharmaceutical companies they were familiar with internally budgeted for about 2/3 of published academic articles to be false) and oncology, and others. Whatever you might have to say about research in psychology, it's also the field primarily turning the microscope on itself. This is part of a tradition in the field -- modern meta-analysis has its origins there. I'm less familiar with physics but there's a lot of problems with reproducibility in many fields.
- tomp 5y ago
  > A lot of the biomedical sciences has similar problems with reproducibility. Indeed, which is why I specifically singled out cell biology. It's the part of medicine that's closest to physics. Oncology, pharmacology etc. study humans/whole bodies, so not only are the experiments more expensive to run and much more noisy, there's also all kinds of ethical issues. Most of these aren't there, or are at least reduced, when dealing with just cells (I didn't expect "cell cultures take a long time to grow" and "the chemicals used are non-reproducible" issues that sibling comments pointed out, so I guess not quite physics)
  - derbOac 5y ago
    To be honest, it would be nice to me to figure out what's different about biomedicine, if that's actually the case. It's easy to say "money" but money is everywhere.
microtherion 5y ago
As for biology, I recommend following e.g. the work of Elisabeth Bik to see the hair raising amount of fraud going on, and getting published for years (or, if you like extra spice in your diet, you can follow Leonid Schneider). For physics, consider how difficult it is to settle the controversies around something like the EmDrive, even among experimenters with solid reputations and impeccable professionalism.
cafebeen 5y ago
A good example from physics is this observation related by Feynman in 1974: "[...] It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher. Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of—this history—because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong—and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that..."

by tomp 5y ago

Is this even true? Most of the progress is in "hard sciences" - physics, cellular biology, computational biology, genetics ... where experiments (and presumably reproducibility) is just fine.

epistasis 5y ago

As somebody in computational biology, I think it's important to note that a ton of cellular biology and even computational biology is "not reproducible." This could be anything from bad documentation of how something was run (computational scientists are often terrible at lab notebook culture), to having a model system that lived for a while and no longer exists in the form that initial discoveries were made with. Or it could be that there was just that one batch of reagents that reliably reproduced a phenomenon, and all future orders of the reagents no longer get the same phenomenon, and whatever the difference was is lost to time. But I would claim that especially in biology, though this is less than ideal for writing up near little explanations of reality, it is still extremely useful for understanding what's going on. Scientific papers aren't meant to be ever-lasting truth, like a textbook. They are communications amongst specialists about "look here something cool happened that may be useful to you too." It is only through lots of work that a phenomenon can be established as widespread in biology, and sharing information before doing 10 years of work helps accelerate everything. This is why I roll my eyes at complaints like that one about "foundational cancer research papers not being reproducible." It was written by scientists in industry who wanted to take a new paper and develop an entire drug program around something particularly novel and surprising. Sure, that would be ideal if it worked, but the scientific literature is a lot more than a catalog of ideas ready for commercialization. Scientists in practice understand the limitations of taking a journal paper as gospel. They always try to get something working in their own hands before basing a lot of research off another paper.

SubiculumCode 5y ago
As a fellow scientist, thank you for your dissemination of how science is actually conducted, and what any single paper or series of papers mean. Progress almost always occurs due to the assessment of converging methods and analytic approaches, almost like individual papers are weak classifiers in a big beautiful bag.
- epistasis 5y ago
  Exactly, the phenomena are often way too complex for even a series of papers to come up with the proper abstractions to plan the right experiments. The history of chromatin marks is really illustrative of this. We had lots of early experiments to establish the antibodies to even measure the chromatin marks, and then then two decades of gathering enough data to lead to useful abstractions. And now it's all converging with the 3D organizational data from Hi-C experiments, and DNA methylation from methyl-Seq, and over the past decades the measurement technology (driven by DNA sequencing) has advanced so much that somebody in 2001 trying to anticipate this trajectory would be amazed. What was a single paper back then? Reproducible? Maybe, but that's much less important than pointing to the correct direction in this n-dimensional search space of what to examine next. I've never looked at the historical literature from, say, 100 years ago, but I suspect that it's quite the same as today in all regards. There's a ton of great ideas that never get the attention they deserve, a bunch of ideas that get way too much attention due to fashion or due to influencers that chose the wrong path. But who could we trust to chart a better path? Just like startups, we must accept a high failure rate when exploring the unknown.

querez 5y ago

Counterpoint: I have a friend who works in a biomed lab where results would regularly just be made up. If a lab culture takes 5 years to grow, reproducibility is more of a theoretical than a practical matter. Also, not every reviewer has a particle accelerator in their back yard. Or the money to reproduce large computational models. A lot of published results only pass peer review because you essentially trust the authors not to have made up their numbers.

derbOac 5y ago

A lot of the biomedical sciences has similar problems with reproducibility. A survey found similar problems in pharmacology (I think there was one article quoting someone saying the pharmaceutical companies they were familiar with internally budgeted for about 2/3 of published academic articles to be false) and oncology, and others. Whatever you might have to say about research in psychology, it's also the field primarily turning the microscope on itself. This is part of a tradition in the field -- modern meta-analysis has its origins there. I'm less familiar with physics but there's a lot of problems with reproducibility in many fields.

tomp 5y ago
> A lot of the biomedical sciences has similar problems with reproducibility. Indeed, which is why I specifically singled out cell biology. It's the part of medicine that's closest to physics. Oncology, pharmacology etc. study humans/whole bodies, so not only are the experiments more expensive to run and much more noisy, there's also all kinds of ethical issues. Most of these aren't there, or are at least reduced, when dealing with just cells (I didn't expect "cell cultures take a long time to grow" and "the chemicals used are non-reproducible" issues that sibling comments pointed out, so I guess not quite physics)
- derbOac 5y ago
  To be honest, it would be nice to me to figure out what's different about biomedicine, if that's actually the case. It's easy to say "money" but money is everywhere.

microtherion 5y ago

As for biology, I recommend following e.g. the work of Elisabeth Bik to see the hair raising amount of fraud going on, and getting published for years (or, if you like extra spice in your diet, you can follow Leonid Schneider). For physics, consider how difficult it is to settle the controversies around something like the EmDrive, even among experimenters with solid reputations and impeccable professionalism.

cafebeen 5y ago

A good example from physics is this observation related by Feynman in 1974: "[...] It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher. Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of—this history—because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong—and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that..."