4 ms·

In private I have heard of science done that is suspected to be fraudulent. The results are so intriguing and promising as to send multiple grad students acros

In private I have heard of science done that is suspected to be fraudulent. The results are so intriguing and promising as to send multiple grad students across multiple groups barreling into replicating and expanding the results. What happened? Years of time wasted with no publications. PIs and journals aren't interested in publishing negative results. The person who lied in their research is reaping career benefits of such a valuable contribution to research. Academia takes something as beautiful as the closed loop of science and opens it up only to benefit charlatans and punish the best and brightest.

gus_massa 5y ago
> The results are so intriguing and promising as to send multiple grad students across multiple groups barreling into replicating and expanding the results. One important task of the advisor is to filter bad projects. Does the paper pass the smell test? Are the authors reliable or they had published "dubious" results before? [I was recently discussing a paper in a virtual meeting. My conclusion is that Zoom does not have enough negative reactions.] Also, before assigning an experiment to a graduate student, try to have something like a plan: 1) This will surely work and is interesting enough for the dissertation and to be publish. 2) This part is dubious, because it's related to a too resent research paper or there may be some hidden technical problems that are difficult to see. 2') An underspecified alternative to 2, just in case. 3) Extra task for bonus points and perhaps a Nobel price.
newsclues 5y ago
Publishing negative results is important to solving this issue.
- atoav 5y ago
  How about instead of applying to journals with a finished study, people would apply with a proposal of a study and then publish the results independent of how spectacular/boring they turn out? Journals would then have to select studies based on their design and based on the question and scientists could just do science.
  - jononor 5y ago
    This is a good idea, and is known as "preregistration". https://en.wikipedia.org/wiki/Preregistration_(science) https://en.wikipedia.org/wiki/Preregistration_(science) Hopefully it will become more common practice.
    - FabHK 5y ago
      Well, it is preregistration plus a commitment by a journal to accept and publish the resulting paper (no matter the outcome of the study). That apparently is "Registered reports" (see same Wikipedia article), an idea I hadn't heard before. Edited to add (from Wikipedia): > Over 200 journals offer a registered reports option (Centre for Open Science, 2019),[33] and the number of journals that are adopting registered reports is approximately doubling each year (Chambers et al., 2019).[34] > Nature Human Behaviour has adopted the registered report format, as it “shift[s] the emphasis from the results of research to the questions that guide the research and the methods used to answer them”.[36] > European Journal of Personality defines this format: “In a registered report, authors create a study proposal that includes theoretical and empirical background, research questions/hypotheses, and pilot data (if available). Upon submission, this proposal will then be reviewed prior to data collection, and if accepted, the paper resulting from this peer-reviewed procedure will be published, regardless of the study outcomes.”[37]
- smitty1e 5y ago
  Incentivizing publishing negative results is a challenge.
  - tux3 5y ago
    This has potential consequences I don't like. Once negative results are incentivized the same way as positive results, cheaters could have a perverse incentive to post fraudulent failures to replicate, too. I think I agree that we need much more transparency and scientific rigor, first. Post the data whenever possible, incentivize people for creating good study protocols that make it harder for any single person to fake the data without it being obvious. Once we have enough confidence in that, let's do preregistration, sure. And then negative results should be as incentivized as positive ones, absolutely. I worry about the order in which we want to do these things. If people are faking data, we need to make this specific step harder to hide from reviewers. If you incentivize people to fail to replicate each other without solving the lag between fraudulent data and retractation, you will get fraudulent negative results. That could cause a lot of confusion. If the replication crisis itself starts to be fraudulent, scientific credibility is further damaged, negative results will be less credible, and fraudsters will profit from the confusion.
    - regardsop 5y ago
      "If negative results are published like other information, people could have a incentive to replicate post fraudulent failures"? Um sounds worse, (OT:) to add something a simpified (strict stripped) example -hope practicable: You work at a company. Orders are comming in, wares are placed as 'reserved' till paid, than shipped. With a worker perspective, you check the emails with orders ('the customer whishes'), sending 'bills' and the wares if, or if not been paid - 'you've been paid'. Viewed by the company-manager your position may a) be valid oder come b) 'overvalued' (...) Stucked in this example, you talked to other 'employeees' during break and 'there was a time meeting on weekends', so you talked about work and there was a (special) kind of 'more transparency' than without peer review. ^^ Regards, hope this helps... (-:
      - tux3 5y ago
- willis936 5y ago
  I agree. The grad students agree. Now how do you convince a group obsessed with career advancement and holding all of the cards (PIs) to agree?
  - detaro 5y ago
    By convincing the people deciding about their career advancement that negative results are worth something too. (no, that isn't easy, since it's the same people to a good degree, which is a problem with solving many problems in academia)
  - unityByFreedom 5y ago
    Enough of these "falsified science" stories come out and people will become interested in stories debunking bad science. I subscribe to a podcast called "everything hertz" for this reason. https://everythinghertz.com/ https://everythinghertz.com/ https://twitter.com/jamesheathers https://twitter.com/jamesheathers https://twitter.com/dsquintana https://twitter.com/dsquintana
  - MichaelZuo 5y ago
    Give groups larger grants for publishing negative results too? More money seems to be workable enough in the rest of the economy. Obviously it would be best if those at the top make the change and let it cascade down. But any private source of funds could also experiment with changing the grant model.
Consultant32452 5y ago
Peer review, as we know it today, is a fairly new phenomenon and most of our best science was done before it entered the academy. It is a pox on innovation.
nerdponx 5y ago
The fact that the "negative" results don't get published is the worst part. Failing to replicate a strong effect should be a result in and of itself! Failing to find support for a plausible theory is also a result. The fact that this kind of publication bias still exists doesn't make any sense to me. And it seems to be one of the primary ills of scientific publishing and academia.
mountainb 5y ago
In other fields where there is a chance for fraud, the honor system is not used. In academia, the attitude is just 'the honor system is good enough, WINK' as it relates to data. In other fields where there are opportunities for embezzlement or other abuses of trust everything is monitored, only some employees are authorized to do certain transactions, and many of those employees are bound by law and oath to not do things like embezzle or falsify records. This doesn't always work but having the capacity to disbar lawyers for, say, embezzling trust accounts or remove a CPA's license and put them in prison for falsifying records, it certainly has a deterrent effect on similar crimes of trust. How much worse would white collar crime be if everything was on the 'honor system' and there was no prison time for stealing from customers/clients/shareholders? That's what we have in academia: crime won, so now syndicates of crooks run the system. With other abuses of trust the material consequences are more readily apparent: someone's bank account is empty that should not be. With this abuse of trust, the damage is more to the integrity of the purportedly rational knowledge base. It's more of a 'reign of error' than a reign of terror.
- JumpCrisscross 5y ago
  > With this abuse of trust, the damage is more to the integrity of the purportedly rational knowledge base It corrodes trust in our scientific institutions. When that pop something talk by an acclaimed psychologist turns out to be not only bullshit but outright fraud, it’s not a huge leap to project that mistrust onto e.g. the medical establishment or climate scientists.
- matthewdgreen 5y ago
  Everything is a cost/benefit tradeoff. We devote more resources to securing your bank account against fraud because we (as a society) think that it’s worth spending the money — on the theory that recovering your money after it’s stolen is very hard and because we think it’s acceptable to cause legitimate transactions to undergo some inconvenience. We devote fewer resources to verifying every social sciences experiment because society doesn’t want to spend the money to reduce a presumably already-low fraud rate and because the worst case outcome is usually articles like this one. And yet despite these decisions, white collar financial fraud occurs all the time and plenty of lawyers stretch their profession to (or beyond) the ethical breaking point — with devastating impacts on real people.

gus_massa 5y ago

> The results are so intriguing and promising as to send multiple grad students across multiple groups barreling into replicating and expanding the results. One important task of the advisor is to filter bad projects. Does the paper pass the smell test? Are the authors reliable or they had published "dubious" results before? [I was recently discussing a paper in a virtual meeting. My conclusion is that Zoom does not have enough negative reactions.] Also, before assigning an experiment to a graduate student, try to have something like a plan: 1) This will surely work and is interesting enough for the dissertation and to be publish. 2) This part is dubious, because it's related to a too resent research paper or there may be some hidden technical problems that are difficult to see. 2') An underspecified alternative to 2, just in case. 3) Extra task for bonus points and perhaps a Nobel price.

newsclues 5y ago

Publishing negative results is important to solving this issue.

atoav 5y ago
How about instead of applying to journals with a finished study, people would apply with a proposal of a study and then publish the results independent of how spectacular/boring they turn out? Journals would then have to select studies based on their design and based on the question and scientists could just do science.
- jononor 5y ago
  This is a good idea, and is known as "preregistration". https://en.wikipedia.org/wiki/Preregistration_(science) https://en.wikipedia.org/wiki/Preregistration_(science) Hopefully it will become more common practice.
  - FabHK 5y ago
    Well, it is preregistration plus a commitment by a journal to accept and publish the resulting paper (no matter the outcome of the study). That apparently is "Registered reports" (see same Wikipedia article), an idea I hadn't heard before. Edited to add (from Wikipedia): > Over 200 journals offer a registered reports option (Centre for Open Science, 2019),[33] and the number of journals that are adopting registered reports is approximately doubling each year (Chambers et al., 2019).[34] > Nature Human Behaviour has adopted the registered report format, as it “shift[s] the emphasis from the results of research to the questions that guide the research and the methods used to answer them”.[36] > European Journal of Personality defines this format: “In a registered report, authors create a study proposal that includes theoretical and empirical background, research questions/hypotheses, and pilot data (if available). Upon submission, this proposal will then be reviewed prior to data collection, and if accepted, the paper resulting from this peer-reviewed procedure will be published, regardless of the study outcomes.”[37]
smitty1e 5y ago
Incentivizing publishing negative results is a challenge.
- tux3 5y ago
  This has potential consequences I don't like. Once negative results are incentivized the same way as positive results, cheaters could have a perverse incentive to post fraudulent failures to replicate, too. I think I agree that we need much more transparency and scientific rigor, first. Post the data whenever possible, incentivize people for creating good study protocols that make it harder for any single person to fake the data without it being obvious. Once we have enough confidence in that, let's do preregistration, sure. And then negative results should be as incentivized as positive ones, absolutely. I worry about the order in which we want to do these things. If people are faking data, we need to make this specific step harder to hide from reviewers. If you incentivize people to fail to replicate each other without solving the lag between fraudulent data and retractation, you will get fraudulent negative results. That could cause a lot of confusion. If the replication crisis itself starts to be fraudulent, scientific credibility is further damaged, negative results will be less credible, and fraudsters will profit from the confusion.
  - regardsop 5y ago
    "If negative results are published like other information, people could have a incentive to replicate post fraudulent failures"? Um sounds worse, (OT:) to add something a simpified (strict stripped) example -hope practicable: You work at a company. Orders are comming in, wares are placed as 'reserved' till paid, than shipped. With a worker perspective, you check the emails with orders ('the customer whishes'), sending 'bills' and the wares if, or if not been paid - 'you've been paid'. Viewed by the company-manager your position may a) be valid oder come b) 'overvalued' (...) Stucked in this example, you talked to other 'employeees' during break and 'there was a time meeting on weekends', so you talked about work and there was a (special) kind of 'more transparency' than without peer review. ^^ Regards, hope this helps... (-:
    - tux3 5y ago
willis936 5y ago
I agree. The grad students agree. Now how do you convince a group obsessed with career advancement and holding all of the cards (PIs) to agree?
- detaro 5y ago
  By convincing the people deciding about their career advancement that negative results are worth something too. (no, that isn't easy, since it's the same people to a good degree, which is a problem with solving many problems in academia)
- unityByFreedom 5y ago
  Enough of these "falsified science" stories come out and people will become interested in stories debunking bad science. I subscribe to a podcast called "everything hertz" for this reason. https://everythinghertz.com/ https://everythinghertz.com/ https://twitter.com/jamesheathers https://twitter.com/jamesheathers https://twitter.com/dsquintana https://twitter.com/dsquintana
- MichaelZuo 5y ago
  Give groups larger grants for publishing negative results too? More money seems to be workable enough in the rest of the economy. Obviously it would be best if those at the top make the change and let it cascade down. But any private source of funds could also experiment with changing the grant model.

Consultant32452 5y ago

Peer review, as we know it today, is a fairly new phenomenon and most of our best science was done before it entered the academy. It is a pox on innovation.

nerdponx 5y ago

The fact that the "negative" results don't get published is the worst part. Failing to replicate a strong effect should be a result in and of itself! Failing to find support for a plausible theory is also a result. The fact that this kind of publication bias still exists doesn't make any sense to me. And it seems to be one of the primary ills of scientific publishing and academia.

mountainb 5y ago

In other fields where there is a chance for fraud, the honor system is not used. In academia, the attitude is just 'the honor system is good enough, WINK' as it relates to data. In other fields where there are opportunities for embezzlement or other abuses of trust everything is monitored, only some employees are authorized to do certain transactions, and many of those employees are bound by law and oath to not do things like embezzle or falsify records. This doesn't always work but having the capacity to disbar lawyers for, say, embezzling trust accounts or remove a CPA's license and put them in prison for falsifying records, it certainly has a deterrent effect on similar crimes of trust. How much worse would white collar crime be if everything was on the 'honor system' and there was no prison time for stealing from customers/clients/shareholders? That's what we have in academia: crime won, so now syndicates of crooks run the system. With other abuses of trust the material consequences are more readily apparent: someone's bank account is empty that should not be. With this abuse of trust, the damage is more to the integrity of the purportedly rational knowledge base. It's more of a 'reign of error' than a reign of terror.

JumpCrisscross 5y ago
> With this abuse of trust, the damage is more to the integrity of the purportedly rational knowledge base It corrodes trust in our scientific institutions. When that pop something talk by an acclaimed psychologist turns out to be not only bullshit but outright fraud, it’s not a huge leap to project that mistrust onto e.g. the medical establishment or climate scientists.
matthewdgreen 5y ago
Everything is a cost/benefit tradeoff. We devote more resources to securing your bank account against fraud because we (as a society) think that it’s worth spending the money — on the theory that recovering your money after it’s stolen is very hard and because we think it’s acceptable to cause legitimate transactions to undergo some inconvenience. We devote fewer resources to verifying every social sciences experiment because society doesn’t want to spend the money to reduce a presumably already-low fraud rate and because the worst case outcome is usually articles like this one. And yet despite these decisions, white collar financial fraud occurs all the time and plenty of lawyers stretch their profession to (or beyond) the ethical breaking point — with devastating impacts on real people.