11 ms·

I wrote my concerns about replicating these studies a couple of years ago [1]. In short, most papers don't have enough details to allow for a third party to rep

by kayhi 8y ago

I wrote my concerns about replicating these studies a couple of years ago [1]. In short, most papers don't have enough details to allow for a third party to replicate them. Problem areas include detailed protocols and reagents used. If you don't what someone did exactly and what they used then replicating it is going to be very difficult. [1] https://www.linkedin.com/pulse/scientific-reproducibility-reagent-variability-sean-seaver/ https://www.linkedin.com/pulse/scientific-reproducibility-re...

nonbel 8y ago
That is exactly the problem. All this stuff is supposed to be in the paper, the paper is not a press release. And everyone understands that its hard to know exactly what info someone else will be missing, so it wont be perfect at first. That is why there is supposed to be a back and forth the first time a new method is used. Figuring out what needs to go into the methods is going to be an iterative process. As it is right now its a disaster though. Like they say in this article no one even knows the most basic info like cell density. The entire system is not set up to deal with replications at all because they haven't been doing them.
- rjkennedy98 8y ago
  > All this stuff is supposed to be in the paper, the paper is not a press release. Welcome to the real world. Science doesn't work like it was described in your 7th grade science class.
  - pas 8y ago
    But it can be approximated. Repeatable experiments. Done by machines, properly accounted ingredients and so on. Physics does it. It should be perfectly doable with cell cultures too. Mice are of course a lot harder. But if we wouldn't wait decades to replicate, but could simply get in touch with the originators - and so do a collaborative effort in establishing the parameters.
  - nonbel 8y ago
    These sayings like "welcome to the real world", "there's a difference between theory and practice", thought-killers are just excuses for being lazy. Science is hard, really hard. People who think doing a good job is too hard should get out of science instead of polluting the literature with their statistically significant results or whatever. Unfortunately its been the opposite happening and the people who cant handle it have been training the new people how to get away with half-assing it. This has been going on for a long time now, at least since the 1940s in some areas...
    - timr 8y ago
      There's a huge difference between "bad science", and "it doesn't live up to the average HN reader's belief of how science works". Maybe the parent's comment is harsh, but the sentiment is, IMO, correct. Science is a system, not a series of published recipes. Even the best, most-cited papers in scientific history, when taken in isolation, are often riddled with problems. No paper is perfect, but the process of science, over time, validates the papers that are worth validating. Experiments are rarely reproduced exactly, but rather, reproduced under a thousand different conditions, thus proving that the result is robust over time. I've often felt that these efforts to mass-reproduce scientific literature are misguided. Anyone who has worked in a lab knows that most papers are difficult to reproduce for any number of reasons, and that even if you have a perfect description of materials and methods, plenty of correct and valid results are difficult to reproduce, just because lab work is hard and messy. It's not fair to fling accusations of impropriety simply because an experiment doesn't work in your hands.
      - JumpCrisscross 8y ago
        > Science is a system, not a series of published recipe Peer review and reproduction are essential components of a scientific system. If a paper does not provide—it its main text or supplements—the necessary detail to enable reproduction, it is not a scientific paper but a press release. > plenty of correct and valid results are difficult to reproduce There is lots of good science in figuring out why certain results reproduce in certain circumstances and not in others. With insufficient methodological detail, this exploration is rendered impossible.
  - p1necone 8y ago
    This isn't helpful. People questioning this is a good thing. We can't just throw our hands in the air and go "welp, shit's fucked, no point in trying to fix it".
- chrisseaton 8y ago
  > All this stuff is supposed to be in the paper I don't agree. I think this stuff should be in supplemental notes (we call it the 'artefact' in my field, not sure if that's universal). A paper should be reasonably readable, not a huge list of processes in detail.
  - nonbel 8y ago
    I'm used to biomed where the "supplements" do exist and are containing more and more of the actual information while the "paper" is becoming more of an "executive summary". Since I find all that info in the same place, I consider it part of "the paper".
drb91 8y ago
> If you don't what someone did exactly and what they used then replicating it is going to be very difficult. Why do journals accept these papers? If you can just make shit up, it kind of makes the whole value of a paid journal moot. What value do they add if not reviewing the damn paper?
- freetime2 8y ago
  I don't write research papers, so I don't really know what I'm talking about here, but I assume it's not as easy to just "make shit up" as you are implying here. There's a peer review process, there are lots of people working in the lab that make it difficult to keep a large conspiracy secret, there are serious penalties if you are caught fabricating results, etc.
  - drb91 8y ago
    There is a difference between wholesale making things up and conveniently leaving out variables that alter analysis of your data. I do agree overall that this is not trivial and there are tremendous disincentives; that is evidentally not a good tactic to reproducing experiments in the general case.
  - stonemetal 8y ago
    What do the reviewers do? They don't try to reproduce it. So yeah make shit up, as long as it is plausible no one will notice. For example Yoshihiro Sato was recently (2016-2017 time frame) caught fabricating results, turns out he had been doing it for 30 years. He had more than 20 papers retracted. Also note he wasn't caught by a fellow scientist in the field he was caught by a statistician bulk processing papers looking for anomalies. So yeah big penalties if you are caught, but the odds of getting caught are so low you have to be stunningly stupid to get caught.
    - southern_cross 8y ago
      Sadly, once the statistical methods being used here and the types of anomalies being found become well-known, it's probably not too hard for fakers to adjust their "results" so that such anomalies are far more difficult (if not impossible) to detect. For example, if an analysis would show that your experimental data is maybe a bit too clean, in a way that wouldn't normally turn up in the real world and would therefore cast doubt on its validity, then you just need to make sure that you dirty it up a little before publication.
ekianjo 8y ago
This is nothing recent though. Back in early 2000s I was trying to reproduce published reactions regarding catalysts leading to certain yields, and virtually all published stuff was absolute garbage and could not be reproduced no matter how many times you tried. That makes you wonder what journals actually do: accept articles on good faith only? That is unforgivable in a scientific society.
- comicjk 8y ago
  The peer review process is halfway between "accept on good faith" and "reproduce the results". And it's not a flawless compromise. As a peer reviewer, I check that an incoming paper is sane, says something new, and doesn't have obvious holes. But I don't reproduce the original experiment unless it's very quick. Peer review is entirely unpaid and uncredited, so I can't justify taking time from my own research. Ideally, every research team would have a remote collaborator that checks their results and gets credit for doing so (even if the answer is no!). But right now there is nothing to incentivize that kind of rigor.
- southern_cross 8y ago
  Back in the day I had a physics professor who sometimes complained about his inability to reproduce certain published physics experiments - the kind you could do in a small lab and on a limited budget. (We're not talking "Big Physics" here.) Well, the guy wasn't particularly well-liked for a variety of reasons, so we all tended to just chalk this up to probable incompetence on his part. Knowing what I know now, though, I might be more inclined to give him the benefit of the doubt here - that in fact these experiments were just not reproducible, at least not as described in the literature.
avip 8y ago
There's a startup named codeocean (I'm not affiliated) trying to fix this (though as you've mentioned, the main issue here is broken incentives).

by kayhi 8y ago

nonbel 8y ago

That is exactly the problem. All this stuff is supposed to be in the paper, the paper is not a press release. And everyone understands that its hard to know exactly what info someone else will be missing, so it wont be perfect at first. That is why there is supposed to be a back and forth the first time a new method is used. Figuring out what needs to go into the methods is going to be an iterative process. As it is right now its a disaster though. Like they say in this article no one even knows the most basic info like cell density. The entire system is not set up to deal with replications at all because they haven't been doing them.

rjkennedy98 8y ago
> All this stuff is supposed to be in the paper, the paper is not a press release. Welcome to the real world. Science doesn't work like it was described in your 7th grade science class.
- pas 8y ago
  But it can be approximated. Repeatable experiments. Done by machines, properly accounted ingredients and so on. Physics does it. It should be perfectly doable with cell cultures too. Mice are of course a lot harder. But if we wouldn't wait decades to replicate, but could simply get in touch with the originators - and so do a collaborative effort in establishing the parameters.
- nonbel 8y ago
  These sayings like "welcome to the real world", "there's a difference between theory and practice", thought-killers are just excuses for being lazy. Science is hard, really hard. People who think doing a good job is too hard should get out of science instead of polluting the literature with their statistically significant results or whatever. Unfortunately its been the opposite happening and the people who cant handle it have been training the new people how to get away with half-assing it. This has been going on for a long time now, at least since the 1940s in some areas...
  - timr 8y ago
    There's a huge difference between "bad science", and "it doesn't live up to the average HN reader's belief of how science works". Maybe the parent's comment is harsh, but the sentiment is, IMO, correct. Science is a system, not a series of published recipes. Even the best, most-cited papers in scientific history, when taken in isolation, are often riddled with problems. No paper is perfect, but the process of science, over time, validates the papers that are worth validating. Experiments are rarely reproduced exactly, but rather, reproduced under a thousand different conditions, thus proving that the result is robust over time. I've often felt that these efforts to mass-reproduce scientific literature are misguided. Anyone who has worked in a lab knows that most papers are difficult to reproduce for any number of reasons, and that even if you have a perfect description of materials and methods, plenty of correct and valid results are difficult to reproduce, just because lab work is hard and messy. It's not fair to fling accusations of impropriety simply because an experiment doesn't work in your hands.
    - JumpCrisscross 8y ago
      > Science is a system, not a series of published recipe Peer review and reproduction are essential components of a scientific system. If a paper does not provide—it its main text or supplements—the necessary detail to enable reproduction, it is not a scientific paper but a press release. > plenty of correct and valid results are difficult to reproduce There is lots of good science in figuring out why certain results reproduce in certain circumstances and not in others. With insufficient methodological detail, this exploration is rendered impossible.
- p1necone 8y ago
  This isn't helpful. People questioning this is a good thing. We can't just throw our hands in the air and go "welp, shit's fucked, no point in trying to fix it".
chrisseaton 8y ago
> All this stuff is supposed to be in the paper I don't agree. I think this stuff should be in supplemental notes (we call it the 'artefact' in my field, not sure if that's universal). A paper should be reasonably readable, not a huge list of processes in detail.
- nonbel 8y ago
  I'm used to biomed where the "supplements" do exist and are containing more and more of the actual information while the "paper" is becoming more of an "executive summary". Since I find all that info in the same place, I consider it part of "the paper".

drb91 8y ago

> If you don't what someone did exactly and what they used then replicating it is going to be very difficult. Why do journals accept these papers? If you can just make shit up, it kind of makes the whole value of a paid journal moot. What value do they add if not reviewing the damn paper?

freetime2 8y ago
I don't write research papers, so I don't really know what I'm talking about here, but I assume it's not as easy to just "make shit up" as you are implying here. There's a peer review process, there are lots of people working in the lab that make it difficult to keep a large conspiracy secret, there are serious penalties if you are caught fabricating results, etc.
- drb91 8y ago
  There is a difference between wholesale making things up and conveniently leaving out variables that alter analysis of your data. I do agree overall that this is not trivial and there are tremendous disincentives; that is evidentally not a good tactic to reproducing experiments in the general case.
- stonemetal 8y ago
  What do the reviewers do? They don't try to reproduce it. So yeah make shit up, as long as it is plausible no one will notice. For example Yoshihiro Sato was recently (2016-2017 time frame) caught fabricating results, turns out he had been doing it for 30 years. He had more than 20 papers retracted. Also note he wasn't caught by a fellow scientist in the field he was caught by a statistician bulk processing papers looking for anomalies. So yeah big penalties if you are caught, but the odds of getting caught are so low you have to be stunningly stupid to get caught.
  - southern_cross 8y ago
    Sadly, once the statistical methods being used here and the types of anomalies being found become well-known, it's probably not too hard for fakers to adjust their "results" so that such anomalies are far more difficult (if not impossible) to detect. For example, if an analysis would show that your experimental data is maybe a bit too clean, in a way that wouldn't normally turn up in the real world and would therefore cast doubt on its validity, then you just need to make sure that you dirty it up a little before publication.

ekianjo 8y ago

This is nothing recent though. Back in early 2000s I was trying to reproduce published reactions regarding catalysts leading to certain yields, and virtually all published stuff was absolute garbage and could not be reproduced no matter how many times you tried. That makes you wonder what journals actually do: accept articles on good faith only? That is unforgivable in a scientific society.

comicjk 8y ago
The peer review process is halfway between "accept on good faith" and "reproduce the results". And it's not a flawless compromise. As a peer reviewer, I check that an incoming paper is sane, says something new, and doesn't have obvious holes. But I don't reproduce the original experiment unless it's very quick. Peer review is entirely unpaid and uncredited, so I can't justify taking time from my own research. Ideally, every research team would have a remote collaborator that checks their results and gets credit for doing so (even if the answer is no!). But right now there is nothing to incentivize that kind of rigor.
southern_cross 8y ago
Back in the day I had a physics professor who sometimes complained about his inability to reproduce certain published physics experiments - the kind you could do in a small lab and on a limited budget. (We're not talking "Big Physics" here.) Well, the guy wasn't particularly well-liked for a variety of reasons, so we all tended to just chalk this up to probable incompetence on his part. Knowing what I know now, though, I might be more inclined to give him the benefit of the doubt here - that in fact these experiments were just not reproducible, at least not as described in the literature.

avip 8y ago

There's a startup named codeocean (I'm not affiliated) trying to fix this (though as you've mentioned, the main issue here is broken incentives).