27 ms·

Reproducibility trial: Biologists get different results from same data sets

90 points by EvgeniyZh 3y ago

RandomLensman 3y ago
Not surprised, given the rather vague question asked in relation to the data (e.g., "how does x influence y") - many choices from there.
- mcphage 3y ago
  Phrased as "how does X influence Y" makes it sound vague, but I think that the question they asked would be better phrased "to what degree does X influence Y"—they got results which varied along a continuum, but not in random different ways. So the actual questions themselves weren't vague, and the researchers all treated it as meaning the same thing, they just came up with different degrees in the results.
  - RandomLensman 3y ago
    But how does that really relate to replication? What they found out is more like: with interpretability come different results.
    - mcphage 3y ago
      I agree (and said as much in another comment). The different groups didn't come up with different effects, they came up with different magnitudes of the same effect. This doesn't seem like a replication crisis.
- light_hue_1 3y ago
  That's what research is. It's not homework. No one defines the question for you. You just become curious about some relationships. If they had asked very specific questions the whole study would be meaningless. The questions they asked are spon on: "To what extent is the growth of nestling blue tits (Cyanistes caeruleus) influenced by competition with siblings?" and "How does grass cover influence Eucalyptus spp. seedling recruitment?"
  - RandomLensman 3y ago
    I disagree for a study on replication, as people having different interpretations is totally fine then but not really related to replication as such. Many valid answers here to the questions. They studied comprehensiveness or something like that (edit: and to some extent craftsmanship of sorts) . Imagine in chemistry they had asked to synthesize a compound given a range of inputs and then see a variety of yields based on different synthesis approaches.
mcphage 3y ago
This doesn't seem like quite the same issue as the reproducibility problems in psychology, for instance. Researchers were presented with some data, and they came up with different analyses for it. That doesn't seem especially surprising—for a complex phenomenon, where you don't have a ton of data, and maybe the thing you're studying isn't very strong—yeah, you're going to get different results. It's like people arguing over which sports team is better—the facts of the game are the facts of the game, but there's plenty of room for disagreement over what stats are most important, who has an easier season—and even if the two teams were to play each other directly, it's not always that the "better" team wins. But the reproducibility crisis in social sciences seemed much deeper—one team would say that some effect exists, and other teams would try to reproduce the results and come up with nothing.
- tgv 3y ago
  There is a relation between them, but not a strong one, I agree. The more conclusions you can draw from a data set, the more will be wrong. A possible reason, hinted at in the article, is underestimating the noise. It's one of the things that went bad in psych studies, but also in genetics (in a lecture some 10 years ago, a statistician explained how results in genetics would be reproducible only at 5 sigma or higher).
mjburgess 3y ago
The issue here is the same across all 'science' which is unable to effectively control causes, and which studies non-linear phenomena (or, more simply, cases where the distribution of means isnt defined, or is far away from normal). The issue is: you can't do stats. The reality is it is highly likely that all data here is consistent with any effect size you care to choose. What this study does, nevertheless, is quantify the 'model risk' ie., the "p-value of the method itself" -- how often does research provide reliable effect sizes? Rarely. So the research methods themselves have a huge 'epistemic discounting' associated with them. But we can go much further than this -- recurse: what is the risk of using this methodology to assess methodology? Actually, it's fairly high. This isn't a very reliable method for judging research methods. And so on. What we find here is that in the vast majority of cases, where we'd care to research anything, recursing risks lead to an essential insurmountable error: measurement error, model error, methodology error, research-assumptions-error, etc. There isnt enough data to resolve between competing possibilities at the outer-most stage of error (often you'd need much more data than practically possible to collect). What do we do with this? I'd say, mostly, throw away stats here. Do case analyses, risk-based recommendations, even broadly philosophical analysis. The alterantive is just staring at tea-leaves.
- DanielBMarkham 3y ago
  You're not wrong, but I think you may have headed out the wrong way from your conclusion. Reproducibility trumps causal models. If you call the psychic hotline and get the same results time-after-time, then keep using the psychic hotline. Leave the modeling to others. When developing new science, stats lag behind reproducibility. First you see something that correlates, then somebody is able to make A always infer B, then you start doing some statistical surveying in an effort to build up a larger model. Observation first, then Abduction (in the Peircian sense), then measurement, then modeling, and finally Deductive/Inductive reasoning. We keep trying to do it backwards where we have the model for everything, then bang actual data into various parts of the model. The saddest thing in this process is that if we're only doing incremental, normal science (Kuhn), none's the wiser. This is why reproducibility tests are so important. We lie to ourselves. We're quite good at it. BTW, I agree that with you general conclusion about case analysis and such, but only in the sense that it drives Abduction, not as a goal in itself.
thenerdhead 3y ago
> “It’s about sitting back and watching what the natural world throws at you — which is a lot of variation.” Great quote.
User23 3y ago
Is this more clearly stated something like “Biological models are not uniquely determined by data sets?”
sinuhe69 3y ago
So, essentially all statistics in (live-science) research are just unreliable? And in general, you can not draw a single conclusion from any research? That would be disastrous, isn't it?
- bannedbybros 3y ago
  [dead]
rando_dfad 3y ago
So as I understand Fischer's stance on using stats, you are supposed to start with a hypothesis. As in, you have a mechanism which could cause the effect. The stats are there to provide supporting evidence, to limit BS causal chains. Data mining is a bit of the reverse of that.
- jewayne 3y ago
  Exactly. I feel like this is asking the biologists a trick question. The correct answer is, "I don't fucking know. Are you offering me a grant to study this?"
- icegreentea2 3y ago
  Starting with a hypothesis doesn't require a detailed mechanism. Both questions posed in the article have a proposed mechanism - they clearly have a cause and proposed effect listed. This isn't the same thing as mining a giant generic database looking for arbitrary correlation. The replication issue highlighted here is how choices in analytical approach can yield differing answers. This would be true even if you had a detailed proposed mechanism.
  - jewayne 3y ago
    But neither one specified a falsifiable hypothesis. Which, to this layman's mind, means that any general statement made based upon the data is itself a hypothesis.
jewayne 3y ago
Non-scientist here. Isn't this just stating that "experiments", even those involving non-laboratory measurements, need to begin with a falsifiable hypothesis? From my understanding, you can't draw any real conclusions from the data unless you predicted that it would look that way beforehand.
- Frost1x 3y ago
  >From my understanding, you can't draw any real conclusions from the data unless you predicted that it would look that way beforehand. Oh my sweet summer child, a lot of lab work and data collection is expensive and in the game of research, you spend a lot of time gaming the system and meeting expectations relative to doing actual fundamental research. So much work wants to take the hard low return work like doing tests and collecting data and releveraging it with increasingly complex statistical approaches. I've worked in so many research environments that you find more often the case is that there's a selection towards research that could be pursued and falsified with existing data vs the other way around. Here's this set of data and how it was collected, what arbitrarily new novel thing can we say about it? It may not be something interesting but it may be statistically or theoretically valid. The result is you get a paper/publication out of it without doing the footwork. This is part of the reason researchers often hold their data tightly. You'd think scientists would want to share data but it's a highly competitive environment and if you took the risk to invest time and money in some costly data collection process, you want to do everything you can to say everything you can about it before someone else does it without any of the underlying cost. Sure, you may get a reference or footnote for your data but that's not going to help that much in the big scheme of things, not as much as a fresh publication. Also, if you're only being referenced for the data collection portion of your work... it doesn't speak alot about the work you did around that data collection.
  - jewayne 3y ago
    But doesn't that explain the reproducibility crisis, in a nutshell? If you work backwards from a dataset and look for correlations, doesn't that effectively set the p-value to 1?
amelius 3y ago
What they should test next is if the scientific method would upvote the biologists that produced the more correct results.
karmakaze 3y ago
The 'results' here isn't about confirming or rejecting a hypothesis, it's about analyzing data for a strength of effect, which seems fair enough. > The researchers gave scientist-participants one of two data sets and an accompanying research question: either “To what extent is the growth of nestling blue tits (Cyanistes caeruleus) influenced by competition with siblings?” or “How does grass cover influence Eucalyptus spp. seedling recruitment?” What follows is that the variations in choices of how the data is analyzed gives a wide range of 'strengths' that when averaged is not significant, or extreme and opposing individual strength results. What this is highlighting is that the data analysis (even for identically reproduced data) is a source of irreproducibility meaning that it itself needs to be explicit and validated.
- rrsp 3y ago
  Does anyone know if there’s a reason why we’ve not agreed on a standard way of analysing data for any given type of study or type of outcome? Maybe it’s naive but my intuition would be that if someone were conducting a randomised control trial with a hypothesis that A > B then there should be a known best practice for analysing that data to check the hypothesis. A way that is reproducible, otherwise surely it’ll become a shitfest of people trying multiple methods of analysis and then publishing whichever produced the ‘best’ result.
  - negimpactcalc 3y ago
    [flagged]
fwungy 3y ago
Replication is a real problem in modern science, especially with more complex subjects like medicine[0]. https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis

90 points by EvgeniyZh 3y ago

Not surprised, given the rather vague question asked in relation to the data (e.g., "how does x influence y") - many choices from there.

mcphage 3y ago
Phrased as "how does X influence Y" makes it sound vague, but I think that the question they asked would be better phrased "to what degree does X influence Y"—they got results which varied along a continuum, but not in random different ways. So the actual questions themselves weren't vague, and the researchers all treated it as meaning the same thing, they just came up with different degrees in the results.
- RandomLensman 3y ago
  But how does that really relate to replication? What they found out is more like: with interpretability come different results.
  - mcphage 3y ago
    I agree (and said as much in another comment). The different groups didn't come up with different effects, they came up with different magnitudes of the same effect. This doesn't seem like a replication crisis.
light_hue_1 3y ago
That's what research is. It's not homework. No one defines the question for you. You just become curious about some relationships. If they had asked very specific questions the whole study would be meaningless. The questions they asked are spon on: "To what extent is the growth of nestling blue tits (Cyanistes caeruleus) influenced by competition with siblings?" and "How does grass cover influence Eucalyptus spp. seedling recruitment?"
- RandomLensman 3y ago
  I disagree for a study on replication, as people having different interpretations is totally fine then but not really related to replication as such. Many valid answers here to the questions. They studied comprehensiveness or something like that (edit: and to some extent craftsmanship of sorts) . Imagine in chemistry they had asked to synthesize a compound given a range of inputs and then see a variety of yields based on different synthesis approaches.

mcphage 3y ago

This doesn't seem like quite the same issue as the reproducibility problems in psychology, for instance. Researchers were presented with some data, and they came up with different analyses for it. That doesn't seem especially surprising—for a complex phenomenon, where you don't have a ton of data, and maybe the thing you're studying isn't very strong—yeah, you're going to get different results. It's like people arguing over which sports team is better—the facts of the game are the facts of the game, but there's plenty of room for disagreement over what stats are most important, who has an easier season—and even if the two teams were to play each other directly, it's not always that the "better" team wins. But the reproducibility crisis in social sciences seemed much deeper—one team would say that some effect exists, and other teams would try to reproduce the results and come up with nothing.

tgv 3y ago
There is a relation between them, but not a strong one, I agree. The more conclusions you can draw from a data set, the more will be wrong. A possible reason, hinted at in the article, is underestimating the noise. It's one of the things that went bad in psych studies, but also in genetics (in a lecture some 10 years ago, a statistician explained how results in genetics would be reproducible only at 5 sigma or higher).

mjburgess 3y ago

The issue here is the same across all 'science' which is unable to effectively control causes, and which studies non-linear phenomena (or, more simply, cases where the distribution of means isnt defined, or is far away from normal). The issue is: you can't do stats. The reality is it is highly likely that all data here is consistent with any effect size you care to choose. What this study does, nevertheless, is quantify the 'model risk' ie., the "p-value of the method itself" -- how often does research provide reliable effect sizes? Rarely. So the research methods themselves have a huge 'epistemic discounting' associated with them. But we can go much further than this -- recurse: what is the risk of using this methodology to assess methodology? Actually, it's fairly high. This isn't a very reliable method for judging research methods. And so on. What we find here is that in the vast majority of cases, where we'd care to research anything, recursing risks lead to an essential insurmountable error: measurement error, model error, methodology error, research-assumptions-error, etc. There isnt enough data to resolve between competing possibilities at the outer-most stage of error (often you'd need much more data than practically possible to collect). What do we do with this? I'd say, mostly, throw away stats here. Do case analyses, risk-based recommendations, even broadly philosophical analysis. The alterantive is just staring at tea-leaves.

DanielBMarkham 3y ago
You're not wrong, but I think you may have headed out the wrong way from your conclusion. Reproducibility trumps causal models. If you call the psychic hotline and get the same results time-after-time, then keep using the psychic hotline. Leave the modeling to others. When developing new science, stats lag behind reproducibility. First you see something that correlates, then somebody is able to make A always infer B, then you start doing some statistical surveying in an effort to build up a larger model. Observation first, then Abduction (in the Peircian sense), then measurement, then modeling, and finally Deductive/Inductive reasoning. We keep trying to do it backwards where we have the model for everything, then bang actual data into various parts of the model. The saddest thing in this process is that if we're only doing incremental, normal science (Kuhn), none's the wiser. This is why reproducibility tests are so important. We lie to ourselves. We're quite good at it. BTW, I agree that with you general conclusion about case analysis and such, but only in the sense that it drives Abduction, not as a goal in itself.

thenerdhead 3y ago

> “It’s about sitting back and watching what the natural world throws at you — which is a lot of variation.” Great quote.

User23 3y ago

Is this more clearly stated something like “Biological models are not uniquely determined by data sets?”

sinuhe69 3y ago

So, essentially all statistics in (live-science) research are just unreliable? And in general, you can not draw a single conclusion from any research? That would be disastrous, isn't it?

bannedbybros 3y ago
[dead]

rando_dfad 3y ago

So as I understand Fischer's stance on using stats, you are supposed to start with a hypothesis. As in, you have a mechanism which could cause the effect. The stats are there to provide supporting evidence, to limit BS causal chains. Data mining is a bit of the reverse of that.

jewayne 3y ago
Exactly. I feel like this is asking the biologists a trick question. The correct answer is, "I don't fucking know. Are you offering me a grant to study this?"
icegreentea2 3y ago
Starting with a hypothesis doesn't require a detailed mechanism. Both questions posed in the article have a proposed mechanism - they clearly have a cause and proposed effect listed. This isn't the same thing as mining a giant generic database looking for arbitrary correlation. The replication issue highlighted here is how choices in analytical approach can yield differing answers. This would be true even if you had a detailed proposed mechanism.
- jewayne 3y ago
  But neither one specified a falsifiable hypothesis. Which, to this layman's mind, means that any general statement made based upon the data is itself a hypothesis.

jewayne 3y ago

Non-scientist here. Isn't this just stating that "experiments", even those involving non-laboratory measurements, need to begin with a falsifiable hypothesis? From my understanding, you can't draw any real conclusions from the data unless you predicted that it would look that way beforehand.

Frost1x 3y ago
>From my understanding, you can't draw any real conclusions from the data unless you predicted that it would look that way beforehand. Oh my sweet summer child, a lot of lab work and data collection is expensive and in the game of research, you spend a lot of time gaming the system and meeting expectations relative to doing actual fundamental research. So much work wants to take the hard low return work like doing tests and collecting data and releveraging it with increasingly complex statistical approaches. I've worked in so many research environments that you find more often the case is that there's a selection towards research that could be pursued and falsified with existing data vs the other way around. Here's this set of data and how it was collected, what arbitrarily new novel thing can we say about it? It may not be something interesting but it may be statistically or theoretically valid. The result is you get a paper/publication out of it without doing the footwork. This is part of the reason researchers often hold their data tightly. You'd think scientists would want to share data but it's a highly competitive environment and if you took the risk to invest time and money in some costly data collection process, you want to do everything you can to say everything you can about it before someone else does it without any of the underlying cost. Sure, you may get a reference or footnote for your data but that's not going to help that much in the big scheme of things, not as much as a fresh publication. Also, if you're only being referenced for the data collection portion of your work... it doesn't speak alot about the work you did around that data collection.
- jewayne 3y ago
  But doesn't that explain the reproducibility crisis, in a nutshell? If you work backwards from a dataset and look for correlations, doesn't that effectively set the p-value to 1?

amelius 3y ago

What they should test next is if the scientific method would upvote the biologists that produced the more correct results.

karmakaze 3y ago

The 'results' here isn't about confirming or rejecting a hypothesis, it's about analyzing data for a strength of effect, which seems fair enough. > The researchers gave scientist-participants one of two data sets and an accompanying research question: either “To what extent is the growth of nestling blue tits (Cyanistes caeruleus) influenced by competition with siblings?” or “How does grass cover influence Eucalyptus spp. seedling recruitment?” What follows is that the variations in choices of how the data is analyzed gives a wide range of 'strengths' that when averaged is not significant, or extreme and opposing individual strength results. What this is highlighting is that the data analysis (even for identically reproduced data) is a source of irreproducibility meaning that it itself needs to be explicit and validated.

rrsp 3y ago
Does anyone know if there’s a reason why we’ve not agreed on a standard way of analysing data for any given type of study or type of outcome? Maybe it’s naive but my intuition would be that if someone were conducting a randomised control trial with a hypothesis that A > B then there should be a known best practice for analysing that data to check the hypothesis. A way that is reproducible, otherwise surely it’ll become a shitfest of people trying multiple methods of analysis and then publishing whichever produced the ‘best’ result.
- negimpactcalc 3y ago
  [flagged]

fwungy 3y ago

Replication is a real problem in modern science, especially with more complex subjects like medicine[0]. https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis