5 ms·

Funny, I'm a postdoc (in biomedical research), and I and my boss, a PI, had this exact debate last night. I took the side of 781, and he was justin66. My persp

by cgiles 7y ago

Funny, I'm a postdoc (in biomedical research), and I and my boss, a PI, had this exact debate last night. I took the side of 781, and he was justin66. My perspective is that you cannot take data, or a concept, that is fundamentally broken, add a little effort, and fix it. For example, if a collaborator sends data for analysis where the sample size was too small for the effect, or worse, asked to find statistical differences between 1-2 samples per group, you can't fix that with any amount of algorithmic mumbo-jumbo. You can only fix it by redoing the experiment properly. Although we are asked to do this sort of thing almost daily (not always this blatantly). His perspective is, and maybe I'm being a bit uncharitable, you take what you're given, do your best, clearly describe what you did and honestly note the limitations, and move on. Even if the result is something highly dubious, that is for the reviewers to decide. But I personally think a little loud whining is called for in the sciences to discourage totally half-baked ideas and data from ever seeing the light of day. But the trouble is the pressures are so intense and funding is so tight, people feel compelled to halfass many things, which is the real problem. EDIT: I should add I am not talking about data falsification or major ethical breaches like that. I'm talking about things that more amount to a form of pressure-induced willful stupidity, no dishonesty involved. I would speculate this is much more of a cause of the replication crisis than outright dishonesty. People don't like to outright lie because it could cost them their careers. But they certainly can conveniently forget some basic scientific principles at times if it suits them and they can't get a decent story while being rigorous. And their excuse will be -- well, we clearly documented everything we did in the paper, and the reviewers thought it was OK (because the reviewers do the same sorts of thing). And this, folks, is how you get a lot of shitty papers.

781 7y ago
> I'm talking about things that more amount to a form of pressure-induced willful stupidity, no dishonesty involved. Isn't that still dishonesty though? If you know that you are misapplying statistical techniques to get what you want, how is that different from a pseudo-scientist talking about quantum mumbo-jumbo to get what he wants. https://rationalwiki.org/wiki/Lying_by_omission https://rationalwiki.org/wiki/Lying_by_omission On a bigger scale, this just fuels public anti-science, who is fed up of reading in the news "coffee is good for your heart" and the next day "coffee causes heart attacks". And it's not only because of journalists writing sensational titles, you can actually find meta-analysis pro/against any major food group.
- cgiles 7y ago
  It is honestly hard to tell in these situations when they are lying by omission and when they are plain stupid. I think it is usually a form of apathy whereby what is produced is bullshit (in the technical meaning of the essay, "On Bullshit", whereby people don't care whether a thing is true or false so long as it's published). > On a bigger scale, this just fuels public anti-science Yep... But people have careers to maintain and grants to get. I totally agree with you and take your side on this issue but the systemic issues are huge and I don't know what exactly it would take to fix them.
justin66 7y ago
> You can only fix it by redoing the experiment properly. I don't disagree.
astazangasta 7y ago
>For example, if a collaborator sends data for analysis where the sample size was too small for the effect, or worse, asked to find statistical differences between 1-2 samples per group, you can't fix that with any amount of algorithmic mumbo-jumbo. Au contraire: a few years back my colleagues came to me with an shRNA screen with no replication. I said you can't do this, without an estimate of variability this is meaningless (NOT the first, or last, time i gave this lecture). The reply was, analyze it anyway. I made up some bullshit. The reviewers did nothing. We published in Nature Genetics.
- cgiles 7y ago
  Ha. I have a lot of those papers too, but none that are BS in such a high tier, congrats! (or "congrats") At this point I am cynical enough to believe people include analysts on their papers as an insurance policy to have the "licensed" person to give the seal of approval, and take the blame if necessary, for their shitty data. Which they will never listen to concerns about before publication. Just today I am working on a qPCR array (yes people still use those) with one "housekeeping" gene replicated 4 times for ddCt normalization. 80 miRs go up, none go down, out of 400. No one sees any problem with this, the data is the data. Them: Oh, and can you, cgiles, also tell us about the genes regulated by these miRs? Me: Not really in any principled way, especially when the underlying data is incredibly fishy, and even if I did, we would be saying all the target genes go down. Don't you think people will find that odd? Do you think perhaps there could be a problem with your "housekeeping" gene? Them: Who cares? Just run the algorithm. Thank you for this tale. I'm glad I'm not the only one in this situation.
  - astazangasta 7y ago
    This sort of stuff, the general lack of interest in method in the drive to publish, has really killed a lot of my faith in the idea of a scientific method. It can't work, we can't produce good, useful knowledge if even the best scientists at top flight institutions just regard it as a shell game where the facsimile of a result is as good as a real result, as long as you can sneak it past publishers, who are playing the same game with the public.
- 7y ago

by cgiles 7y ago

781 7y ago

> I'm talking about things that more amount to a form of pressure-induced willful stupidity, no dishonesty involved. Isn't that still dishonesty though? If you know that you are misapplying statistical techniques to get what you want, how is that different from a pseudo-scientist talking about quantum mumbo-jumbo to get what he wants. https://rationalwiki.org/wiki/Lying_by_omission https://rationalwiki.org/wiki/Lying_by_omission On a bigger scale, this just fuels public anti-science, who is fed up of reading in the news "coffee is good for your heart" and the next day "coffee causes heart attacks". And it's not only because of journalists writing sensational titles, you can actually find meta-analysis pro/against any major food group.

cgiles 7y ago
It is honestly hard to tell in these situations when they are lying by omission and when they are plain stupid. I think it is usually a form of apathy whereby what is produced is bullshit (in the technical meaning of the essay, "On Bullshit", whereby people don't care whether a thing is true or false so long as it's published). > On a bigger scale, this just fuels public anti-science Yep... But people have careers to maintain and grants to get. I totally agree with you and take your side on this issue but the systemic issues are huge and I don't know what exactly it would take to fix them.

justin66 7y ago

> You can only fix it by redoing the experiment properly. I don't disagree.

astazangasta 7y ago

>For example, if a collaborator sends data for analysis where the sample size was too small for the effect, or worse, asked to find statistical differences between 1-2 samples per group, you can't fix that with any amount of algorithmic mumbo-jumbo. Au contraire: a few years back my colleagues came to me with an shRNA screen with no replication. I said you can't do this, without an estimate of variability this is meaningless (NOT the first, or last, time i gave this lecture). The reply was, analyze it anyway. I made up some bullshit. The reviewers did nothing. We published in Nature Genetics.

cgiles 7y ago
Ha. I have a lot of those papers too, but none that are BS in such a high tier, congrats! (or "congrats") At this point I am cynical enough to believe people include analysts on their papers as an insurance policy to have the "licensed" person to give the seal of approval, and take the blame if necessary, for their shitty data. Which they will never listen to concerns about before publication. Just today I am working on a qPCR array (yes people still use those) with one "housekeeping" gene replicated 4 times for ddCt normalization. 80 miRs go up, none go down, out of 400. No one sees any problem with this, the data is the data. Them: Oh, and can you, cgiles, also tell us about the genes regulated by these miRs? Me: Not really in any principled way, especially when the underlying data is incredibly fishy, and even if I did, we would be saying all the target genes go down. Don't you think people will find that odd? Do you think perhaps there could be a problem with your "housekeeping" gene? Them: Who cares? Just run the algorithm. Thank you for this tale. I'm glad I'm not the only one in this situation.
- astazangasta 7y ago
  This sort of stuff, the general lack of interest in method in the drive to publish, has really killed a lot of my faith in the idea of a scientific method. It can't work, we can't produce good, useful knowledge if even the best scientists at top flight institutions just regard it as a shell game where the facsimile of a result is as good as a real result, as long as you can sneak it past publishers, who are playing the same game with the public.
7y ago