3 ms·

> a postdoc in my lab told me not to continue wasting time trying (and failing) to reproduce [Top Institution]’s “Best Paper Award” results from the year prior.

by jhgb 5y ago

> a postdoc in my lab told me not to continue wasting time trying (and failing) to reproduce [Top Institution]’s “Best Paper Award” results from the year prior. He had been there when the work was done and said they manipulated the dataset until they got the numbers they wanted. Isn't that the moment where you try even harder to falsify the claims in that paper? You already know that you'll succeed so it wouldn't be a waste of time in your effort.

xjlin0 5y ago
It's harder to publish negative results.
- jhgb 5y ago
  Even good negative results? If it's a problem to publish a negative result debunking an award-winning paper, then that is a problem.
  - carlmr 5y ago
    Hence the reproducibility crisis.
  - gibba999 5y ago
    Yes, and in most cases, no one will cite negative results. The positive results continue to be cited even long after debunked. This is an example which did get cites: https://journals.sagepub.com/doi/full/10.1111/j.1539-6053.2009.01038.x https://journals.sagepub.com/doi/full/10.1111/j.1539-6053.20... But despite the high visibility, you can see the large number of papers published based on the original myth. And this refutation doesn't have great methodology (but other ones do). It's mostly cited due to strong language used.
gus_massa 5y ago
The problem with experimental results is that they are difficult to replicate. In software you can "git clone x.git & cd x & make" and replicate the correct or incorrect results. In hardware, it's more difficult. The main problem is that even if you reproduce their experiment, they can claim that you did some step wrong, perhaps you are mixing it too fast or too slow, or the temperature is not correctly controlled, or that one of your reactive have a contamination that destroy the effect, or magically realize that their reactive that is important. It's very difficult to publish papers with negative results. So there is a high chance it will not count in your total number of publications. Also, expect a low number of citation, so it's not useful for other metrics like citation count or h. For the same reason, you will not see publications of exact replications. A good paper X will be followed by almost-replications by another teams, like "we changed this and got X with a 10% improvement" or "we mixed the methods of X and Y and unsurprisingly^W got X+Y". This is somewhat good because it shows that the initial result is robust enough to survive small modifications.

by jhgb 5y ago

xjlin0 5y ago

It's harder to publish negative results.

jhgb 5y ago
Even good negative results? If it's a problem to publish a negative result debunking an award-winning paper, then that is a problem.
- carlmr 5y ago
  Hence the reproducibility crisis.
- gibba999 5y ago
  Yes, and in most cases, no one will cite negative results. The positive results continue to be cited even long after debunked. This is an example which did get cites: https://journals.sagepub.com/doi/full/10.1111/j.1539-6053.2009.01038.x https://journals.sagepub.com/doi/full/10.1111/j.1539-6053.20... But despite the high visibility, you can see the large number of papers published based on the original myth. And this refutation doesn't have great methodology (but other ones do). It's mostly cited due to strong language used.

gus_massa 5y ago

The problem with experimental results is that they are difficult to replicate. In software you can "git clone x.git & cd x & make" and replicate the correct or incorrect results. In hardware, it's more difficult. The main problem is that even if you reproduce their experiment, they can claim that you did some step wrong, perhaps you are mixing it too fast or too slow, or the temperature is not correctly controlled, or that one of your reactive have a contamination that destroy the effect, or magically realize that their reactive that is important. It's very difficult to publish papers with negative results. So there is a high chance it will not count in your total number of publications. Also, expect a low number of citation, so it's not useful for other metrics like citation count or h. For the same reason, you will not see publications of exact replications. A good paper X will be followed by almost-replications by another teams, like "we changed this and got X with a 10% improvement" or "we mixed the methods of X and Y and unsurprisingly^W got X+Y". This is somewhat good because it shows that the initial result is robust enough to survive small modifications.