3 ms·

There's a big qualitative difference between inaccurate calculations from flipped signs and software bugs to fundamentally wrong analysis. Also why aren't the

• PaulHoule 4y ago
  Peer review is not effective. A peer reviewer is not going to go through complex calculations line-by-line or find bugs in the software. The person who does find that is a person following in the footsteps of the researcher who wants to build on that result. Software bugs are common in commercial software too but if they are put in front of customers most get fixed. What boggles my mind are the poor standards in biomedical work. Math-heavy papers are often written by one or two people and a computer but clinical studies involve a lot of people, paper and you'd think formalized process. I remember one paper where they tested COVID-19 patients and could not detect Vitamin C in their blood but didn't test anyone who didn't have COVID-19. Vitamin C levels vary greatly in different populations and the test is tricky and requires careful sample handling... It could be they were unable to detect Vitamin C at all, but they only way they could disprove that would be to use proper controls. Most clinical trials are rejected when it comes time for a meta-analysis, I think that journals shouldn't be accepting studies that wouldn't be accepted for meta-analysis and work that isn't to that standard shouldn't be funded.
  • manigandham 4y ago
    Why not have every (bio/medical) paper then include the cost of a separate red team to recreate the results? Is there a fundamental issue why this wouldn't solve the reproducibility by handling it internally as part of the process itself?