5 ms·

I think once you've seen a few papers in high-tier journals that turn out to be bullshit once you start to dig a bit deeper, there is not other choice than to a

I think once you've seen a few papers in high-tier journals that turn out to be bullshit once you start to dig a bit deeper, there is not other choice than to adopt this harsh stance on random scientific papers. Especially if you want to do work with that expands on findings on other papers that roughly look good "trust but verify" seems to be the way to go. I've only recently dipped by toes into academic life in a lab, but it very much seems that PIs generally know which are the bad apples. E.g. when discussing whether some data is good enough to be publishable the PIs reaction was something along the lines of "If we were FAMOUS_LAB_NAME it would be, but we want to do it in a way that holds up". So it seems like there are at least some barriers to how incompetence would hurt the whole field. I'm also surprised that there is no mention of the PI in GP's story. As it's a paper published by the lab, it's not just on the grad student "to do the right thing", but even more on the more senior scientist, whose reputation is also at stake.

dnautics 6y ago
Funny story, PI noticed an error in one of my papers and I (happily) issued a very minor retraction. Also in one of the threads I talked about how he did retract several year's worth of work done on a different project by the intern when she joined later. So he was alright. Plus, as a junior (2nd year grad student) you really don't want to tattle on the NIH grad student of the year. Who do you think wrote the recommendation?
TeMPOraL 6y ago
> I think once you've seen a few papers in high-tier journals that turn out to be bullshit once you start to dig a bit deeper, there is not other choice than to adopt this harsh stance on random scientific papers. Especially if you want to do work with that expands on findings on other papers that roughly look good "trust but verify" seems to be the way to go. Yeah, but I meant that in general case, you no longer "trust but verify", but "assume bullshit and hope there's a nugget of truth in the paper". This has interesting implications for consuming science for non-academic use, too. I've been accused of being "anti-science" when I said this before, but I no longer trust arguments backed by citations around soft-ish fields like social sciences, dietetics or medicine. Even if the person posting a claim does good work of selecting citations (so they're not all saying something tangentially related, or much more specific, or "in mice!"), if the claim is counterintuitive and papers seem complex enough, I mentally code this as very weak evidence - i.e. most likely bullshit, but there were some papers claiming it, so if that comes up again, many times in different contexts, I may be willing to entertain the claim being true. And stories like this make me extend this principle to biology and chemistry in general as well. I've burned myself enough times, getting excited about some result, only to later learn it was bunk. The same pattern of course repeats outside academia, but more overtly - you can hardly trust any commercial communication either. At this point, I'm wondering how are we even managing to keep a society running? It's very hard work to make progress and contribute, if you have to assume everyone is either bullshitting, or repeating bullshit they've heard elsewhere.

dnautics 6y ago

Funny story, PI noticed an error in one of my papers and I (happily) issued a very minor retraction. Also in one of the threads I talked about how he did retract several year's worth of work done on a different project by the intern when she joined later. So he was alright. Plus, as a junior (2nd year grad student) you really don't want to tattle on the NIH grad student of the year. Who do you think wrote the recommendation?

TeMPOraL 6y ago

> I think once you've seen a few papers in high-tier journals that turn out to be bullshit once you start to dig a bit deeper, there is not other choice than to adopt this harsh stance on random scientific papers. Especially if you want to do work with that expands on findings on other papers that roughly look good "trust but verify" seems to be the way to go. Yeah, but I meant that in general case, you no longer "trust but verify", but "assume bullshit and hope there's a nugget of truth in the paper". This has interesting implications for consuming science for non-academic use, too. I've been accused of being "anti-science" when I said this before, but I no longer trust arguments backed by citations around soft-ish fields like social sciences, dietetics or medicine. Even if the person posting a claim does good work of selecting citations (so they're not all saying something tangentially related, or much more specific, or "in mice!"), if the claim is counterintuitive and papers seem complex enough, I mentally code this as very weak evidence - i.e. most likely bullshit, but there were some papers claiming it, so if that comes up again, many times in different contexts, I may be willing to entertain the claim being true. And stories like this make me extend this principle to biology and chemistry in general as well. I've burned myself enough times, getting excited about some result, only to later learn it was bunk. The same pattern of course repeats outside academia, but more overtly - you can hardly trust any commercial communication either. At this point, I'm wondering how are we even managing to keep a society running? It's very hard work to make progress and contribute, if you have to assume everyone is either bullshitting, or repeating bullshit they've heard elsewhere.