7 ms·

In my view, reproducibility is neither strictly necessary nor sufficient for scientific progress. I'm sure if you went back through the literature on something

In my view, reproducibility is neither strictly necessary nor sufficient for scientific progress. I'm sure if you went back through the literature on something like electromagnetism, you would find results that fail to replicate, yet the theory of electromagnetism if applied properly is remarkably robust. Scientific results can be strengthened by replication, but that's just one thing, and if that's all you do, then you end up with a science that does nothing but generate independent factoids. On the other hand, robust science tends to look at a particular phenomenon from many different angles, and manages to connect multiple results together into a framework that can survive retracting individual studies without collapsing. This is how electromagnetism developed. Sciences that are in the factoid phase are not necessarily junk. Discovery of a psychological "effect" is perfectly scientific and interesting. But some sciences have barely progressed beyond the factoid phase. And if those sciences are also plagued by irreproducibility, then they may embrace scientific methodology without producing a useful scientific knowledge base.

atty 5y ago
As a physicist, I think how we normally describe what you’re talking about is that there must be a robust and explanatory theory that goes along with experimental measurements. The most fruitful areas of research are the ones where theory and experiment exist on approximately the same level of what they can predict or measure. These strong explanatory theories are the things that generate truly robust, reproducible experiments. In cases where there is significant differences in the maturity between theory and experiment, you either end up with theorists playing games with math, making up tons of unconstrained theories (string theory, for instance), or in the other direction you end up with experimentalists measuring anything and everything they can imagine, half of which is probably not theoretically enlightening, waiting for theorists to constrain their space of possible experiments to potentially fruitful paths. Obviously this is harder for fields that don’t have models as robust as we do in physics, but I’d guess the same phenomenon happens everywhere.
- analog31 5y ago
  I'm a physicist too. A bad analogy is that theory and experiment are like plywood. The individual layers are weak, but criss crossing them against one another and gluing them together makes something that's quite strong.
  - atty 5y ago
    I actually quite like that analogy - I may use that the next time I have a chance to teach.
    - wombatmobile 5y ago
      Yeah, why is it a "bad analogy"? I think there's something useful in it, possibly even essential.
- wombatmobile 5y ago
  Clearly, for there to be chronic scientific progress, there must be some value in experimental practice. And yet widespread irreproducibility of results negates that value, by definition. So, where is the progress coming from? Your argument that "there must be a robust and explanatory theory that goes along with experimental results" helps make sense of TFA, which otherwise presents as a paradox, or a ruse to excuse shoddy practices. The presence of a viable theory is a useful heuristic for sorting useful experimental practice from less useful. At least, I imagine it is. How can we know for sure? Has anyone published a meta analysis that checks for this? And has that meta analysis been independently reproduced?
LatteLazy 5y ago
Wait, what electromagnetic results are not reproducable!?
- enkid 5y ago
  I think the parent poster is saying some EM experiments weren't initially replicated because of the peculiarities of the set up, etc. Another example is that some people couldn't see the Galilean moons initially. This made initial report of them somewhat questionable, as some observed them and some didn't.
AlotOfReading 5y ago
Of course reproducibility isn't necessary. There's a whole subset of sciences (often called the historical sciences) where reproduction is generally impossible because of the arrow of time. You can't replicate evolving humans again, as an example. Nonetheless, reproducibility is a really nice property. You have to be incredibly careful with your theory if it can't be justified.
- euanc 5y ago
  That subset of "sciences" aren't sciences by definition if they are not reproducible. They may be useful but they are not sciences.
  - AlotOfReading 5y ago
    If your personal definition of sciences doesn't include evolutionary biology, geology, or large parts of astronomy, it's a very unusual definition. All I can tell you is that most people would disagree.
    - euanc 5y ago
      That's ok with me. I think it's very important to distinguish between the two endeavors and ideally have different names for them so we don't get led into these kinds of otherwise unnecessary discussions. The domains you mention seem (to me) generally worthwhile. But it's very hard to articulate exactly why they seem that way given we can't reproduce them which makes their results hard to justify believing in.
TeeMassive 5y ago
> Scientific results can be strengthened by replication, but that's just one thing, and if that's all you do, then you end up with a science that does nothing but generate independent factoids. Scientific results are what makes science universal; since they can be reproduced everywhere by anyone.
- analog31 5y ago
  Indeed, at least until they can't be. The whole "reproducibility" business is because of an emerging "replication crisis," where it has been found that a large swath of published studies fail attempts at replication. Articles about this situation usually say something to the effect of "even in physics," though usually without evidence. A variety of causes have been proposed, such as abuse / ignorance of statistics, bad experimental design, and even outright chicanery.
dotcommand 5y ago
> In my view, reproducibility is neither strictly necessary nor sufficient for scientific progress. In your view? Reproducibility is essential to scientific progress because it defines what science is. It's the backbone of science. Hypothesis,testing/experiments. It's what separates science from math, arts and pseudosciences like social "sciences". > I'm sure if you went back through the literature on something like electromagnetism, you would find results that fail to replicate, yet the theory of electromagnetism if applied properly is remarkably robust. How would it be "robust" if it couldn't be replicated? How would the theory of electromagnetism be robust if it failed to produce replicable results? > Scientific results can be strengthened by replication No. It can only be a "scientific result" if it can be replicated/tested. It's definitional. > Discovery of a psychological "effect" is perfectly scientific and interesting No it isn't if it can't be test/replicated/reproduced. It's not science. It's something else altogether. > And if those sciences are also plagued by irreproducibility, then they may embrace scientific methodology without producing a useful scientific knowledge base. There is no "scientific methodology" without testability/replication/reproducibility. All science is based on hypothesis-testing.
- analog31 5y ago
  I get what you're saying, but even within the more reputable sciences, the body of known or published results is peppered with results that will fail to replicate. They have not all been identified. There will be more. The good results are mixed with junk results yet somehow science progresses. The robustness of science has to come from somewhere other than the complete absence of junk results. This is what I meant by my first sentence. I read that in the century after Newton, the French Academy offered a prize for evidence of the failure of Newton's laws. They gave out the prize dozens of times, yet Newtonian physics kept getting stronger and stronger. Eventually they stopped giving out the prize. Many of the contradictions apparently had to do with the lunar orbit, which was poorly understood.
- NeuNeurosis 5y ago
  Check this article out https://aeon.co/essays/how-popperian-falsification-enabled-the-rise-of-neoliberalism https://aeon.co/essays/how-popperian-falsification-enabled-t... Deals with some of your points.

atty 5y ago

As a physicist, I think how we normally describe what you’re talking about is that there must be a robust and explanatory theory that goes along with experimental measurements. The most fruitful areas of research are the ones where theory and experiment exist on approximately the same level of what they can predict or measure. These strong explanatory theories are the things that generate truly robust, reproducible experiments. In cases where there is significant differences in the maturity between theory and experiment, you either end up with theorists playing games with math, making up tons of unconstrained theories (string theory, for instance), or in the other direction you end up with experimentalists measuring anything and everything they can imagine, half of which is probably not theoretically enlightening, waiting for theorists to constrain their space of possible experiments to potentially fruitful paths. Obviously this is harder for fields that don’t have models as robust as we do in physics, but I’d guess the same phenomenon happens everywhere.

analog31 5y ago
I'm a physicist too. A bad analogy is that theory and experiment are like plywood. The individual layers are weak, but criss crossing them against one another and gluing them together makes something that's quite strong.
- atty 5y ago
  I actually quite like that analogy - I may use that the next time I have a chance to teach.
  - wombatmobile 5y ago
    Yeah, why is it a "bad analogy"? I think there's something useful in it, possibly even essential.
wombatmobile 5y ago
Clearly, for there to be chronic scientific progress, there must be some value in experimental practice. And yet widespread irreproducibility of results negates that value, by definition. So, where is the progress coming from? Your argument that "there must be a robust and explanatory theory that goes along with experimental results" helps make sense of TFA, which otherwise presents as a paradox, or a ruse to excuse shoddy practices. The presence of a viable theory is a useful heuristic for sorting useful experimental practice from less useful. At least, I imagine it is. How can we know for sure? Has anyone published a meta analysis that checks for this? And has that meta analysis been independently reproduced?

LatteLazy 5y ago

Wait, what electromagnetic results are not reproducable!?

enkid 5y ago
I think the parent poster is saying some EM experiments weren't initially replicated because of the peculiarities of the set up, etc. Another example is that some people couldn't see the Galilean moons initially. This made initial report of them somewhat questionable, as some observed them and some didn't.

AlotOfReading 5y ago

Of course reproducibility isn't necessary. There's a whole subset of sciences (often called the historical sciences) where reproduction is generally impossible because of the arrow of time. You can't replicate evolving humans again, as an example. Nonetheless, reproducibility is a really nice property. You have to be incredibly careful with your theory if it can't be justified.

euanc 5y ago
That subset of "sciences" aren't sciences by definition if they are not reproducible. They may be useful but they are not sciences.
- AlotOfReading 5y ago
  If your personal definition of sciences doesn't include evolutionary biology, geology, or large parts of astronomy, it's a very unusual definition. All I can tell you is that most people would disagree.
  - euanc 5y ago
    That's ok with me. I think it's very important to distinguish between the two endeavors and ideally have different names for them so we don't get led into these kinds of otherwise unnecessary discussions. The domains you mention seem (to me) generally worthwhile. But it's very hard to articulate exactly why they seem that way given we can't reproduce them which makes their results hard to justify believing in.

TeeMassive 5y ago

> Scientific results can be strengthened by replication, but that's just one thing, and if that's all you do, then you end up with a science that does nothing but generate independent factoids. Scientific results are what makes science universal; since they can be reproduced everywhere by anyone.

analog31 5y ago
Indeed, at least until they can't be. The whole "reproducibility" business is because of an emerging "replication crisis," where it has been found that a large swath of published studies fail attempts at replication. Articles about this situation usually say something to the effect of "even in physics," though usually without evidence. A variety of causes have been proposed, such as abuse / ignorance of statistics, bad experimental design, and even outright chicanery.

dotcommand 5y ago

> In my view, reproducibility is neither strictly necessary nor sufficient for scientific progress. In your view? Reproducibility is essential to scientific progress because it defines what science is. It's the backbone of science. Hypothesis,testing/experiments. It's what separates science from math, arts and pseudosciences like social "sciences". > I'm sure if you went back through the literature on something like electromagnetism, you would find results that fail to replicate, yet the theory of electromagnetism if applied properly is remarkably robust. How would it be "robust" if it couldn't be replicated? How would the theory of electromagnetism be robust if it failed to produce replicable results? > Scientific results can be strengthened by replication No. It can only be a "scientific result" if it can be replicated/tested. It's definitional. > Discovery of a psychological "effect" is perfectly scientific and interesting No it isn't if it can't be test/replicated/reproduced. It's not science. It's something else altogether. > And if those sciences are also plagued by irreproducibility, then they may embrace scientific methodology without producing a useful scientific knowledge base. There is no "scientific methodology" without testability/replication/reproducibility. All science is based on hypothesis-testing.

analog31 5y ago
I get what you're saying, but even within the more reputable sciences, the body of known or published results is peppered with results that will fail to replicate. They have not all been identified. There will be more. The good results are mixed with junk results yet somehow science progresses. The robustness of science has to come from somewhere other than the complete absence of junk results. This is what I meant by my first sentence. I read that in the century after Newton, the French Academy offered a prize for evidence of the failure of Newton's laws. They gave out the prize dozens of times, yet Newtonian physics kept getting stronger and stronger. Eventually they stopped giving out the prize. Many of the contradictions apparently had to do with the lunar orbit, which was poorly understood.
NeuNeurosis 5y ago
Check this article out https://aeon.co/essays/how-popperian-falsification-enabled-the-rise-of-neoliberalism https://aeon.co/essays/how-popperian-falsification-enabled-t... Deals with some of your points.