4 ms·

The best "researchers" are poor scientists and vice versa. I thought I had hitched my wagon to a star in grad school when I joined a lab with a high profile and

The best "researchers" are poor scientists and vice versa. I thought I had hitched my wagon to a star in grad school when I joined a lab with a high profile and big money-raising PI. After awhile I realized they did no research themselves and focused solely on hype and raising money. When I found out a software bug had invalidated a year's worth of simulations they pressured me to present the false data as true at a large talk. That's when I realized that being honest and truthful with data was not the way to succeed in academia. I also unknowingly ended my career that day by not going along with my professor's fraud. That being said, nothing surprises me anymore about academic dishonesty and research grant funding.

KittenInABox 3y ago
Unfortunately the systemic incentives support PIs that are basically hype salespeople. PIs gain prominence on metrics: number of dollars raised, number of PhDs mentored, number of post-docs on staff, number of papers landed, number of awards granted... They are not supported on number of hours of research done-- they arguably put in that time when they were PhD candidates or post-docs themselves. When I was a PhD candidate (couldn't handle the papers-or-die academic environment so I left) I really, really loved my PI. He had immense academic and intellectual humility. But even he had 10+ papers in-flight at the same time and rarely had time to do any work himself!
strikelaserclaw 3y ago
These days science is done as a "collaborative" effort, there are tons of people gaming that, like putting their names on papers where they didn't do that much, churning out papers which doesn't really add much to science etc...
- dguest 3y ago
  In my collaboration you get to be an author on every paper (around 100 a year) just by doing any work in the experiment. I'm an author on around 1000 papers that I haven't even downloaded, much less read.
  - jampekka 3y ago
    Are you in HEP? I think that's actually a good thing. Makes publication metrics unusable (which is a good thing) and sidesteps many gaming strategies.
    - dguest 3y ago
      It's a thing for sure. It would be great in a post-scarcity society where everyone can be funded regardless of what they do, but people are still competitive. For example: - There are still internal databases (which funding agencies can access), people fight over who gets on which paper there. There are people trying to game that system as much as they would try to game a public metric. - There's a whole cottage industry of people in these collaborations who simulate the experiment, show that it could do something, and then publish a paper (with a shorter author list) about it. This is so popular that many of these people never get around to doing the real experiment. Unfortunately I don't think we've broken free of Goodhart's Law so easily.
FrustratedMonky 3y ago
Also, between the lines there. Why was there a software bug that existed for a year, where was the checking. Researchers also do not care about any kind of software best practices, or the value of software. So it is all, fast and dirty hacks.
- fock 3y ago
  also fun along that: - oss-project implements paper in another framework. makes a small mistake to the paper. - original lab takes the OSS code because they switch frameworks - they find and fix the mistake - sit on that for 2 years without doing a PR.
- icedchai 3y ago
  I've worked with "research" code and the lack of best practices is very true. Few to no tests. Little to no modularization or reuse. The same code copy-and-pasted all over the place. Reinventing the wheel poorly instead of using a 3rd party package/library. Dev environments that are difficult to reproduce since various pieces are either not checked in or in some external drive share. I won't even talk about the actual code itself.
- jboggan 3y ago
  In a related note of worst practices, the main research assistant (who was doing most of the lab's work) died unexpectedly and since no one was using any form of version control we couldn't find his code and data after his death. Which was doubly tragic because we were in the middle of a promising cancer drug trial and it sputtered out after the cell line testing because the original predictions and methodology were gone. The solution from the PI was to have everyone buddy up and tell your partner where you kept code on your laptop.
Kevin09210 3y ago
Daniel Promislow ?
- jboggan 3y ago
  I have not a single bad thing to say about Dr. Promislow, I never studied under him in graduate school and only worked loosely with him in undergraduate research. I also played a lot of music with him back in the day. If you are trying to sleuth my CV you are looking at the wrong school entirely.
dheera 3y ago
The incentives are wrong in the first place. People are judged based on number of papers published to prestigious journals, and demoted when they don't have quarterly positive results, and given promotions and financial boosts accordingly. People attempting to reproduce past results for verification get little to no reward. This incentive structure is in direct contradiction to honesty and real science.
deleted 3y ago
[deleted]
isthatafact 3y ago
Knowingly publishing wrong simulation data is stupid -- (imho, no blame should have gone to you, a student, even if you had gone along with it, but rather to your supervisors in that situation) -- because published simulations would always be reproducible, by definition, if you think about it. Anyone trying to replicate the results will discover that the previous simulations are wrong, though will probably assume mistakes and not fraud by the original researchers. Of course it could take years for someone else to notice the problem (if ever), depending on how exciting were the claimed results and how easy to run are the simulations.

KittenInABox 3y ago

Unfortunately the systemic incentives support PIs that are basically hype salespeople. PIs gain prominence on metrics: number of dollars raised, number of PhDs mentored, number of post-docs on staff, number of papers landed, number of awards granted... They are not supported on number of hours of research done-- they arguably put in that time when they were PhD candidates or post-docs themselves. When I was a PhD candidate (couldn't handle the papers-or-die academic environment so I left) I really, really loved my PI. He had immense academic and intellectual humility. But even he had 10+ papers in-flight at the same time and rarely had time to do any work himself!

strikelaserclaw 3y ago

These days science is done as a "collaborative" effort, there are tons of people gaming that, like putting their names on papers where they didn't do that much, churning out papers which doesn't really add much to science etc...

dguest 3y ago
In my collaboration you get to be an author on every paper (around 100 a year) just by doing any work in the experiment. I'm an author on around 1000 papers that I haven't even downloaded, much less read.
- jampekka 3y ago
  Are you in HEP? I think that's actually a good thing. Makes publication metrics unusable (which is a good thing) and sidesteps many gaming strategies.
  - dguest 3y ago
    It's a thing for sure. It would be great in a post-scarcity society where everyone can be funded regardless of what they do, but people are still competitive. For example: - There are still internal databases (which funding agencies can access), people fight over who gets on which paper there. There are people trying to game that system as much as they would try to game a public metric. - There's a whole cottage industry of people in these collaborations who simulate the experiment, show that it could do something, and then publish a paper (with a shorter author list) about it. This is so popular that many of these people never get around to doing the real experiment. Unfortunately I don't think we've broken free of Goodhart's Law so easily.

FrustratedMonky 3y ago

Also, between the lines there. Why was there a software bug that existed for a year, where was the checking. Researchers also do not care about any kind of software best practices, or the value of software. So it is all, fast and dirty hacks.

fock 3y ago
also fun along that: - oss-project implements paper in another framework. makes a small mistake to the paper. - original lab takes the OSS code because they switch frameworks - they find and fix the mistake - sit on that for 2 years without doing a PR.
icedchai 3y ago
I've worked with "research" code and the lack of best practices is very true. Few to no tests. Little to no modularization or reuse. The same code copy-and-pasted all over the place. Reinventing the wheel poorly instead of using a 3rd party package/library. Dev environments that are difficult to reproduce since various pieces are either not checked in or in some external drive share. I won't even talk about the actual code itself.
jboggan 3y ago
In a related note of worst practices, the main research assistant (who was doing most of the lab's work) died unexpectedly and since no one was using any form of version control we couldn't find his code and data after his death. Which was doubly tragic because we were in the middle of a promising cancer drug trial and it sputtered out after the cell line testing because the original predictions and methodology were gone. The solution from the PI was to have everyone buddy up and tell your partner where you kept code on your laptop.

Kevin09210 3y ago

Daniel Promislow ?

jboggan 3y ago
I have not a single bad thing to say about Dr. Promislow, I never studied under him in graduate school and only worked loosely with him in undergraduate research. I also played a lot of music with him back in the day. If you are trying to sleuth my CV you are looking at the wrong school entirely.

dheera 3y ago

The incentives are wrong in the first place. People are judged based on number of papers published to prestigious journals, and demoted when they don't have quarterly positive results, and given promotions and financial boosts accordingly. People attempting to reproduce past results for verification get little to no reward. This incentive structure is in direct contradiction to honesty and real science.

deleted 3y ago

[deleted]

isthatafact 3y ago

Knowingly publishing wrong simulation data is stupid -- (imho, no blame should have gone to you, a student, even if you had gone along with it, but rather to your supervisors in that situation) -- because published simulations would always be reproducible, by definition, if you think about it. Anyone trying to replicate the results will discover that the previous simulations are wrong, though will probably assume mistakes and not fraud by the original researchers. Of course it could take years for someone else to notice the problem (if ever), depending on how exciting were the claimed results and how easy to run are the simulations.