2 ms·

I strongly disagree with the other comments. This PI seemed fully involved in, did the appropriate amount of oversight over, their lab's work. They studied the

I strongly disagree with the other comments. This PI seemed fully involved in, did the appropriate amount of oversight over, their lab's work. They studied the data their postdoc gave them, discussed its implications, and understood it. That's adequate. It's unreasonable to expect any team to be robust against a malicious insider attack—to be able to audit things with sufficient aggressiveness, that they could catch sophisticated liars who manipulate raw instrument data. I don't have a *clue* what the solution is; but, asking ordinary people to turn into FBI detectives is not a viable one. It won't work in the real world, with real humans. It's a fantasy. I sympathize with the outrage in this thread. But, go past your initial hot-takes: would you apply the same standards to yourself? Suppose you're in software, and one of your coworkers is sneaking subtle, obfuscated security bugs into your production code. Is it your fault if you fail to catch that? Would you consider yourself culpable if you did a code review and failed to notice some "Underhanded C Contest"-tier sophisticated backdoor? Because IMHO that is utterly impossible; and I think it's similarly impossible to catch data manipulation, reliably, if it's sophisticated enough. Solutions need to be grounded in what's achievable.

flobosg 3y ago
> that they could catch sophisticated liars who manipulate raw instrument data There was no raw data manipulation, but fabrication of derived data. These kind of too-good-to-be-true results are easily auditable, if anything by replicating the experiment. The PI’s blind enthusiasm prevented her to do her due diligence. > would you apply the same standards to yourself? While I was in academia I checked basically every kind of raw biological data I was getting. Routine QC allowed me, for instance, to find contaminants in some of the reagents the lab was using.
- achileas 3y ago
  For many labs, there are no resources, time, or ability to rerun every bit of data collection that happens. When I was in academia, our lab had a hard enough time recruiting participants (we were a cog neuro lab) and our experiments were several hours long per participant with enough individual variation to be useless without several months of data collection and analysis to find any sort of signal in the noisy data. And that’s a single lab, with PIs overseeing several and collaborating with even more, an element of trust is going to be needed if anything is to ever get done.
  - flobosg 3y ago
    In the case of this article, the experiment entailed putting a sample in an instrument, do the measurement and check the values. And the checks I propose are nothing out of the ordinary. Many academic fraud postmortems tell that the fabricated data had repeated or perfectly rounded values, strange timestamps, numeric data in a different format than expected, etc. Take a look at the (raw or processed) data, there’s unfortunately no way around it.

flobosg 3y ago

> that they could catch sophisticated liars who manipulate raw instrument data There was no raw data manipulation, but fabrication of derived data. These kind of too-good-to-be-true results are easily auditable, if anything by replicating the experiment. The PI’s blind enthusiasm prevented her to do her due diligence. > would you apply the same standards to yourself? While I was in academia I checked basically every kind of raw biological data I was getting. Routine QC allowed me, for instance, to find contaminants in some of the reagents the lab was using.

achileas 3y ago
For many labs, there are no resources, time, or ability to rerun every bit of data collection that happens. When I was in academia, our lab had a hard enough time recruiting participants (we were a cog neuro lab) and our experiments were several hours long per participant with enough individual variation to be useless without several months of data collection and analysis to find any sort of signal in the noisy data. And that’s a single lab, with PIs overseeing several and collaborating with even more, an element of trust is going to be needed if anything is to ever get done.
- flobosg 3y ago
  In the case of this article, the experiment entailed putting a sample in an instrument, do the measurement and check the values. And the checks I propose are nothing out of the ordinary. Many academic fraud postmortems tell that the fabricated data had repeated or perfectly rounded values, strange timestamps, numeric data in a different format than expected, etc. Take a look at the (raw or processed) data, there’s unfortunately no way around it.