3 ms·

My own rather short academic career involved doing lab work with three different PI-led groups. One PI was actually excellent, and I really had no idea how goo

My own rather short academic career involved doing lab work with three different PI-led groups. One PI was actually excellent, and I really had no idea how good I had it. I caught the other two engaging in deliberately fraudulent practices. For example, data they'd collect from experiments would be thrown out selectively so that they could publish better curve-fits. Another trick was fabricating data with highly obscure methods that other groups would be unlikely to replicate. They'd also apply pressure to graduate students to falsify data in order to get results that agreed with their previously published work. The main difference between the excellent PI and the two fraudsters was that the former insisted on everyone in her lab keeping highly accurate and detailed daily lab notebooks, while the other two had incredibly poor lab notebook discipline (and often didn't even keep records!). She actually caught one of her grad students fudging data via this method, before it went to publication. Another requirement was that samples had to be blindly randomized before we analyzed them, so that nobody could manipulate the analytical process to get their desired result. If you're thinking about going into academia, that's the kind of thing to look out for when visiting prospective PIs. Shoddy record keeping is a huge red flag. Inability to replicate results, and in particular no desire to replicate results, is another warning sign. And yes, a fair number of PIs have made careers out of publishing fraudulent results and never get caught, and they infest the academic system.

georgecmu 5y ago
I would say that this applies even more so outside of academia. At early stages of development, a research group's or company's product is by necessity a report or a presentation rather than a physical plant's or process's real, quantifiable performance. No malicious intent is required; it's just all too easy to fool yourself or cherry-pick data to support desired conclusions when the recordkeeping is poor. In my hard-tech experiment-heavy start-up there's no way we could have made any actual technical progress without setting up a solid data preservation and analysis framework first. For every experimental run, all the original sensor data are collected and immediately uploaded along with any photos, videos, and operator comments to a uniquely-tagged confluence page. Results and data from any further data or product analysis are linked to this original page. As an anecdotal example, we recently caught swapped dataset labels in results from analysis performed on our physical samples by a third-party lab. We were able to do this easily just because we could refer back to every other piece of information regarding these samples, including the conditions in which they were generated months prior to this analysis. As soon all the data were on display at once, the discrepancies were obvious.
ketanmaheshwari 5y ago
[PLUG] Some of what you mention are "negative results" that are quite prevelant and a necessary part of any research. However, the expected mold at publishing venues is such that they are not considered worthwhile. My colleagues and I are trying to address this by creating a platform to discuss and publish such "bad" or "negative" results. More info here: https://error-workshop.org/ https://error-workshop.org/

georgecmu 5y ago

I would say that this applies even more so outside of academia. At early stages of development, a research group's or company's product is by necessity a report or a presentation rather than a physical plant's or process's real, quantifiable performance. No malicious intent is required; it's just all too easy to fool yourself or cherry-pick data to support desired conclusions when the recordkeeping is poor. In my hard-tech experiment-heavy start-up there's no way we could have made any actual technical progress without setting up a solid data preservation and analysis framework first. For every experimental run, all the original sensor data are collected and immediately uploaded along with any photos, videos, and operator comments to a uniquely-tagged confluence page. Results and data from any further data or product analysis are linked to this original page. As an anecdotal example, we recently caught swapped dataset labels in results from analysis performed on our physical samples by a third-party lab. We were able to do this easily just because we could refer back to every other piece of information regarding these samples, including the conditions in which they were generated months prior to this analysis. As soon all the data were on display at once, the discrepancies were obvious.

ketanmaheshwari 5y ago

[PLUG] Some of what you mention are "negative results" that are quite prevelant and a necessary part of any research. However, the expected mold at publishing venues is such that they are not considered worthwhile. My colleagues and I are trying to address this by creating a platform to discuss and publish such "bad" or "negative" results. More info here: https://error-workshop.org/ https://error-workshop.org/