8 ms·

"very difficult to handle, and it's almost impossible to make many of them nicely reproducible" Those same qualities that make a study hard to reproduce should

"very difficult to handle, and it's almost impossible to make many of them nicely reproducible" Those same qualities that make a study hard to reproduce should also create skepticism that the first study was done properly.

fabian2k 4y ago
It's more insidious than that. In a field where the core subject is difficult to handle it is nothing unusual if a different lab cannot recreate known experiments on their first try. This doesn't mean anything is wrong with the original research, it might just mean that the second lab didn't control all the variables. The part that is difficult and annoying is that the variables are not necessarily known, it takes a lot of time and experiments to actually nail down those. And even then for some sensitive stuff every detail can matter, like the exact type of tube you did the experiment in or the vendor, batch and age of every chemical you used. Very often you can control this enough to have consistent results for a set of experiments, but that kind of stuff is really hard to control across different labs.
- LudwigNagasena 4y ago
  But is there a reason to privilege the first study results over others? Sounds like a case of the baby duck syndrome.
  - mxwsn 4y ago
    The social default is in some fields where work is hard to reproduce is they still get into good journals and are believed up front. Usually this is fine. A lot of work that is hard to reproduce, but are relatively important or open up many research directions, do eventually get reproduced, though it might take a year or more. And usually the first party does their best to try to help reproduce the results.
    - vlovich123 4y ago
      I wonder what it would look like if you’re not allowed to cite papers in the field unless there’s been 5 independent labs that have managed to reproduce the result and all the labs that reproduce get to share in the credit (eg if there’s a Nobel prize). Would that change the incentive structure and create better outcomes? Science isn’t science until you have a repeatable process in place where observations match outcomes and the underlying theory. Someone proposing a bold new theory without the ability to test it usually gets significant credit even if posthumously for being a bold visionary which seems to have been sufficient throughout time as a motivator for the really foundational scientific victories.
      - toxik 4y ago
        The effort required to reproduce a result, especially an exotic one, is often prohibitive. The reward for successful reproduction is near zero, and if you fail, it’s often absolutely zero - you are not going to be able to contest a result by a failure. The original authors can simply say that your reproduction was flawed, if they need to answer at all. A journal that requires five confirmations to cite a work is a journal that will fast have zero submissions. The main reason people attempt reproduction is to continue that line of research, and that often starts with reproduction plus a slight tweak. This is why you find a lot of papers like this. This is also how irreproducible results become sort of known about but not challenged in many fields.
      - ebiester 4y ago
        So what does the incentive structure look like? Do you get tenure for running a lab that replicates? Are your postdocs in line for tenure track positions? How is it funded: do you put the secondary replications into the original grant? What happens if you can't find X number of labs to run the test? It's an interesting idea, but it all comes down to how the incentive structures happen.
        vlovich123 4y ago
        Yes. You get tenure track debunking things that have mind share in the community and adding support for theories that become foundational.
- shusaku 4y ago
  To me this is why the emphasis on “reproducibility” is misguided. You might end up failing because you lack the technical capability. Conversely, you might “succeed” by making a mistake. We should be focusing on coming up with new ways to test the same hypothesis.
  - vlovich123 4y ago
    The null hypothesis tells us that if we can reproduce repeatedly and reliably then we can actually use even relatively small number of reproductions to raise the net significance of the result (which also diminishes the ability for fraud to work as it requires a larger conspiracy instead of people just accepting people at their word). If you think reproductions have failed then it’s on you to try to have a cogent explanation for why if you’re personally motivated by a belief that the experiment is actually valid. The particle physics folks also have a different approach these days where they blind the ability to see results and do multiple cross checks of each other’s results to validate the data (because the underlying experiment is so expensive to run). That can also work in theory although hard to say what the success rate of that approach is just yet.
  - ImPostingOnHN 4y ago
    that is what multiple reproductions are for indeed, the initial experiment is not immune to such concerns, and might itself have succeeded by making a mistake or failed because it lacked the technical capability
- elcritch 4y ago
  Presuming the original lab didn't also intentionally leave out important details to make it harder for others to replicate their research quickly. Why list all the details needed to replicate your research and possibly let other labs catch up and take your next grant? Much better to add a veneer of listing details but leave out a few of the really important ones.
  - WalterGR 4y ago
    > Presuming the original lab didn't also intentionally leave out important details to make it harder for others to replicate their research quickly. Is this common?
    - vkazanov 4y ago
      Well, half of DS and CS papers are like this: shallow descriptions, no public data, no code. And things are MUCH easier with computers! Extrapolation to a more complex field tells me that it's a norm there.
      - ImPostingOnHN 4y ago
        how can extrapolation tell you anything if you don't know whether the extrapolation is appropriate, much less true?
        vkazanov 4y ago
        Yup. Never read a non-CS paper in my life. This is why I used this word - extrapolation. I am pointing out certain shady incentives in our field that might apply in another scientific field as well.
        ImPostingOnHN 4y ago
        yeah, I was just asking why you thought it was appropriate to extrapolate (if you had any reason to believe that the shady incentives you mention, translate to a different field, versus them being completely different in this regard)
- wrinklytidbits 4y ago
  Would it be more convincing if labs corroborated other lab's findings? Like Lab A found that XYZ impacts LMQ and Lab B, following Lab A's findings, was able to replicate the finding within some margin. Is there a way to reward this type of research?
buscoquadnary 4y ago
It turns out applying those might not even be enough. https://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ https://slatestarcodex.com/2014/04/28/the-control-group-is-o...
szundi 4y ago
If they wrote this in the article then bad interpretations are coming from readers. It makes sense to publish this, maybe someone else can make it stable and reproduce.
spoonjim 4y ago
I worked for a pharmaceutical company that tried to open a second factory producing more of the same thing that it was already successfully making and it never worked. They built the entire factory, it didn't work, they spent years and millions of dollars trying to debug it, it simply never worked out. They ended up just essentially running the US factory at three shifts to meet the increased demand. To this day I don't think anyone knows why the European production line never worked. Things can fail to reproduce without fraud.
- vaylian 4y ago
  Very fascinating. Were you at least able to pinpoint a part of the production process that produced different results, or was the difference only visible once the final product was "assembled"?
  - spoonjim 4y ago
    I was not closely involved with the project. They essentially built the first one to "spec" like "use this kind of steel in the tanks" and the remediation process was essentially tearing those out one by one and "use the steel that we used in the first tank."
    - Aqwis 4y ago
      Sounds like a process similar to how you would debug a difficult bug in a big and/or complicated codebase – through trying to isolate the code causing the bug, comparing the buggy code with code that's known to work, replacing other parts of the code with dummy code/mocks etc. And sometimes a bug seems impossible to solve regardless of how much developer time you throw at it, and you just have to ship the product with the bug (seems to be especially common in the game dev business).

fabian2k 4y ago

It's more insidious than that. In a field where the core subject is difficult to handle it is nothing unusual if a different lab cannot recreate known experiments on their first try. This doesn't mean anything is wrong with the original research, it might just mean that the second lab didn't control all the variables. The part that is difficult and annoying is that the variables are not necessarily known, it takes a lot of time and experiments to actually nail down those. And even then for some sensitive stuff every detail can matter, like the exact type of tube you did the experiment in or the vendor, batch and age of every chemical you used. Very often you can control this enough to have consistent results for a set of experiments, but that kind of stuff is really hard to control across different labs.

LudwigNagasena 4y ago
But is there a reason to privilege the first study results over others? Sounds like a case of the baby duck syndrome.
- mxwsn 4y ago
  The social default is in some fields where work is hard to reproduce is they still get into good journals and are believed up front. Usually this is fine. A lot of work that is hard to reproduce, but are relatively important or open up many research directions, do eventually get reproduced, though it might take a year or more. And usually the first party does their best to try to help reproduce the results.
  - vlovich123 4y ago
    I wonder what it would look like if you’re not allowed to cite papers in the field unless there’s been 5 independent labs that have managed to reproduce the result and all the labs that reproduce get to share in the credit (eg if there’s a Nobel prize). Would that change the incentive structure and create better outcomes? Science isn’t science until you have a repeatable process in place where observations match outcomes and the underlying theory. Someone proposing a bold new theory without the ability to test it usually gets significant credit even if posthumously for being a bold visionary which seems to have been sufficient throughout time as a motivator for the really foundational scientific victories.
    - toxik 4y ago
      The effort required to reproduce a result, especially an exotic one, is often prohibitive. The reward for successful reproduction is near zero, and if you fail, it’s often absolutely zero - you are not going to be able to contest a result by a failure. The original authors can simply say that your reproduction was flawed, if they need to answer at all. A journal that requires five confirmations to cite a work is a journal that will fast have zero submissions. The main reason people attempt reproduction is to continue that line of research, and that often starts with reproduction plus a slight tweak. This is why you find a lot of papers like this. This is also how irreproducible results become sort of known about but not challenged in many fields.
    - ebiester 4y ago
      So what does the incentive structure look like? Do you get tenure for running a lab that replicates? Are your postdocs in line for tenure track positions? How is it funded: do you put the secondary replications into the original grant? What happens if you can't find X number of labs to run the test? It's an interesting idea, but it all comes down to how the incentive structures happen.
      - vlovich123 4y ago
        Yes. You get tenure track debunking things that have mind share in the community and adding support for theories that become foundational.
shusaku 4y ago
To me this is why the emphasis on “reproducibility” is misguided. You might end up failing because you lack the technical capability. Conversely, you might “succeed” by making a mistake. We should be focusing on coming up with new ways to test the same hypothesis.
- vlovich123 4y ago
  The null hypothesis tells us that if we can reproduce repeatedly and reliably then we can actually use even relatively small number of reproductions to raise the net significance of the result (which also diminishes the ability for fraud to work as it requires a larger conspiracy instead of people just accepting people at their word). If you think reproductions have failed then it’s on you to try to have a cogent explanation for why if you’re personally motivated by a belief that the experiment is actually valid. The particle physics folks also have a different approach these days where they blind the ability to see results and do multiple cross checks of each other’s results to validate the data (because the underlying experiment is so expensive to run). That can also work in theory although hard to say what the success rate of that approach is just yet.
- ImPostingOnHN 4y ago
  that is what multiple reproductions are for indeed, the initial experiment is not immune to such concerns, and might itself have succeeded by making a mistake or failed because it lacked the technical capability
elcritch 4y ago
Presuming the original lab didn't also intentionally leave out important details to make it harder for others to replicate their research quickly. Why list all the details needed to replicate your research and possibly let other labs catch up and take your next grant? Much better to add a veneer of listing details but leave out a few of the really important ones.
- WalterGR 4y ago
  > Presuming the original lab didn't also intentionally leave out important details to make it harder for others to replicate their research quickly. Is this common?
  - vkazanov 4y ago
    Well, half of DS and CS papers are like this: shallow descriptions, no public data, no code. And things are MUCH easier with computers! Extrapolation to a more complex field tells me that it's a norm there.
    - ImPostingOnHN 4y ago
      how can extrapolation tell you anything if you don't know whether the extrapolation is appropriate, much less true?
      - vkazanov 4y ago
        Yup. Never read a non-CS paper in my life. This is why I used this word - extrapolation. I am pointing out certain shady incentives in our field that might apply in another scientific field as well.
        ImPostingOnHN 4y ago
        yeah, I was just asking why you thought it was appropriate to extrapolate (if you had any reason to believe that the shady incentives you mention, translate to a different field, versus them being completely different in this regard)
wrinklytidbits 4y ago
Would it be more convincing if labs corroborated other lab's findings? Like Lab A found that XYZ impacts LMQ and Lab B, following Lab A's findings, was able to replicate the finding within some margin. Is there a way to reward this type of research?

buscoquadnary 4y ago

It turns out applying those might not even be enough. https://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ https://slatestarcodex.com/2014/04/28/the-control-group-is-o...

szundi 4y ago

If they wrote this in the article then bad interpretations are coming from readers. It makes sense to publish this, maybe someone else can make it stable and reproduce.

spoonjim 4y ago

I worked for a pharmaceutical company that tried to open a second factory producing more of the same thing that it was already successfully making and it never worked. They built the entire factory, it didn't work, they spent years and millions of dollars trying to debug it, it simply never worked out. They ended up just essentially running the US factory at three shifts to meet the increased demand. To this day I don't think anyone knows why the European production line never worked. Things can fail to reproduce without fraud.

vaylian 4y ago
Very fascinating. Were you at least able to pinpoint a part of the production process that produced different results, or was the difference only visible once the final product was "assembled"?
- spoonjim 4y ago
  I was not closely involved with the project. They essentially built the first one to "spec" like "use this kind of steel in the tanks" and the remediation process was essentially tearing those out one by one and "use the steel that we used in the first tank."
  - Aqwis 4y ago
    Sounds like a process similar to how you would debug a difficult bug in a big and/or complicated codebase – through trying to isolate the code causing the bug, comparing the buggy code with code that's known to work, replacing other parts of the code with dummy code/mocks etc. And sometimes a bug seems impossible to solve regardless of how much developer time you throw at it, and you just have to ship the product with the bug (seems to be especially common in the game dev business).