3 ms·

I think this is a very important question. This is something that I struggle with. I have read a lot of papers. I generally think science can be a force for go

I think this is a very important question. This is something that I struggle with. I have read a lot of papers. I generally think science can be a force for good. I understand analytic methods developed by or used in papers from my field of interest. I generally believe that those methods are capable of answering important and interesting questions. In my view, the problem is that you can't know if an article is good or bullshit until you sit with it for, say, at least 2 or 3 hours (some papers even more). And that is for someone with my background. I tried to do this same thing when I had an undergraduate level of education and it (a) took me a lot longer (at least 10x), and (b) I missed a lot of the mistakes/scams/lies that I would not miss now. (I'm sure I am not able to detect some bullshit even still.) We should follow the good science. We should not follow the bullshit science. This sounds hard because science, being more technical, is harder to vet. But upon further reflection, it seems that society hasn't figured out how to deal with much simpler lies, either.

bluGill 5y ago
> But upon further reflection, it seems that society hasn't figured out how to deal with much simpler lies, either. outside of your field how much of the BS papers can you catch? I know enough about computers that I could probably figure out at least some of it in that field (after spending 10x longer than someone who actually reads papers regularly), but give me a paper in something else and I'm not so sure.
caddemon 5y ago
It's also the norm in some fields to provide only high level info in the methods section, often without supplemental method details accompanying. This makes it even harder to tell if they did the work correctly, because usually half the methods are intermediate steps which don't have any results directly reported in the paper. In a perfect world those methods would be uniform across labs, but in practice they definitely are not, and it makes tracing down the source of honest replication differences a nightmare. There's also no way to really know if the researcher entirely left out 10 other tests they tried that failed. Sometimes you can guess that it's a stretch because of the stupid categories they use (I'm reminded of those ESPN graphics that say things like "most home runs on a rainy Tuesday in June"). But it's harder to detect if someone straight up removes data points, repeats tests and reports only the nicest, etc. So at some point you basically need to be an insider in the field so you hear the gossip about what doesn't replicate. Or if you have access to thousands of dollars to blow you could try a dozen different variations to try getting it to replicate yourself. I think for something like COVID that is actively affecting many people, there should be funding explicitly for replicating studies, and some slots reserved in a prestigious journal for the findings of the replications. I get that it is not feasible to be replicating everything in science, but I don't see why we can't have ~one lab per relevant university department that specializes in replicating important studies. If you make that a path towards becoming a tenured prof I think that could change the culture surrounding replication studies in general.

bluGill 5y ago

> But upon further reflection, it seems that society hasn't figured out how to deal with much simpler lies, either. outside of your field how much of the BS papers can you catch? I know enough about computers that I could probably figure out at least some of it in that field (after spending 10x longer than someone who actually reads papers regularly), but give me a paper in something else and I'm not so sure.

caddemon 5y ago

It's also the norm in some fields to provide only high level info in the methods section, often without supplemental method details accompanying. This makes it even harder to tell if they did the work correctly, because usually half the methods are intermediate steps which don't have any results directly reported in the paper. In a perfect world those methods would be uniform across labs, but in practice they definitely are not, and it makes tracing down the source of honest replication differences a nightmare. There's also no way to really know if the researcher entirely left out 10 other tests they tried that failed. Sometimes you can guess that it's a stretch because of the stupid categories they use (I'm reminded of those ESPN graphics that say things like "most home runs on a rainy Tuesday in June"). But it's harder to detect if someone straight up removes data points, repeats tests and reports only the nicest, etc. So at some point you basically need to be an insider in the field so you hear the gossip about what doesn't replicate. Or if you have access to thousands of dollars to blow you could try a dozen different variations to try getting it to replicate yourself. I think for something like COVID that is actively affecting many people, there should be funding explicitly for replicating studies, and some slots reserved in a prestigious journal for the findings of the replications. I get that it is not feasible to be replicating everything in science, but I don't see why we can't have ~one lab per relevant university department that specializes in replicating important studies. If you make that a path towards becoming a tenured prof I think that could change the culture surrounding replication studies in general.