8 ms·

Defence against scientific fraud: a proposal for a new MSc course

59 points by vo2maxer 3y ago

gunshai 3y ago
> To date, the response of the scientific establishment has been wholly inadequate. There is little attempt to proactively check for fraud: science is still regarded as a gentlemanly pursuit There is no incentive mechanism, it has been the problem all along. The peer review process is clearly inadequate or not up to the task in its current form.
rdedev 3y ago
Speaking as a former MSc student in CS, a lot of ML related roles are asking for paper publications especially in top journals. This is just more bad incentives for the students to hype up their paper even if it is good but not good enough for a top journal
captainkrtek 3y ago
Great defcon talk on uncovering fake science https://youtu.be/ras_VYgA77Q?feature=shared https://youtu.be/ras_VYgA77Q?feature=shared
TrackerFF 3y ago
It is a decent idea, but it can be really hard to spot scientific fraud. As in REALLY, REALLY hard. Some fields, like machine learning, gets deluged with papers coming from all kinds of researchers - there's no unified scientific notation, so to speak. In one paper you could have a pure math authors, in some other you could have econ authors, and then maybe a theoretical physics author. All these could use wildly different notation to describe related ideas, or discuss ideas to solve problems in their respective fields. And in that case, how do you detect bogus science? You could be a leading ML scientist, but not have the faintest idea about the other domain. Or the inverse - be a leading domain expert, but have little ML knowledge. Not to mention, the people that are publishing fraudulent scientific papers aren't morons - they're likely publishing that kind of stuff because of bad incentives. Maybe you'll catch the laziest, least capable fraudsters - but trying to catch savvy researchers sounds like a daunting task. In my mind, this type of task/job is more fit for full-time scientific "auditors".
zaptheimpaler 3y ago
I think a lot of knowledge about what is fraud is already distributed among researchers in their own fields but can't be accessed because no one wants to say those kinds of things in public. The whole system of academia is shrouded in a fog because saying "X field is dead" or "Y research is fraud" is just not done. But this is the same system that is unable to move off paid journals when the web has allowed ~0 cost publishing for decades. They can't effectively retract papers when the knowledge has failed to hold up. Like we have so many ways to organize, publish and review information on the web but they haven't adopted any of them. Literally a clone of HN or Reddit with upvotes and comments could solve half the problems with publishing and peer review. Actually publishing your data and your scripts no matter how hacky they are can help replication. This is a problem that is technologically easy to at least attack and improve if not solve. But none of that has happened so i don't have much hope, for some reason they aren't able to change.
logicchains 3y ago
Such a course will just give the people who want to commit fraud the tools they need to get away with it more successfully.
kukkeliskuu 3y ago
In the HN discussions about posts like this, there tend to be several comments from people saying they left the academic world because they were disgusted with what they saw. This is a worrying signal, because it tells me that fraud is so pervasive in some fields that just exposing one paper as fraud is not going to fix it. Also, like the article suggests, there appears to be pressure to dismiss the fraud suspicions by the institutions and the journals. These tell me that the problems are systematic, because the system does not automatically correct the faults -- but tries to prevent them from being corrected. In a culture that is based on trust, like science, we should see a very different attitude towards fixing failures.
throwawaymaths 3y ago
It's worse: some fraudsters get caught and aren't denounced. See: Homme Hellinga.
Throwfi44 3y ago
[flagged]
- dang 3y ago
  Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
yashap 3y ago
I have a slightly inside opinion on this - before I became a Software Engineer, I got an MSc in a mostly unrelated field, and am a co-author on work published in a few (very middling) peer reviewed journals. My gut feel from what I saw is that outright fraud is rare, but bias seeping into the study is common. Researchers invest a huge amount of effort into a study, they really want the hypothesis to be true. A lot of studies have steps that are pretty susceptible to bias, like maybe you’re classifying results and there’s a bit of a judgement call there. It’s human nature to make biased decisions in these situations, and make enough of these biased decisions, and statistically insignificant results become statistically significant. Some people might not see a difference between bias and fraud, but I personally do. I think of fraud as a very intentional deception, straight up falsifying numbers in a conscious attempt to deceive. While I see bias as more, you’ve got a borderline case, and you view it in the light you want to see it in, even somewhat unconsciously. Like the difference between unconscious racial bias, and overt hateful racism. I think the best approach to combatting this is to spend a lot less of the overall $$ in science on novel research, and a lot more on attempting to independently reproduce results. Papers should be seen as meaningless until their results can be independently reproduced, and universities/colleges should reward reproduction studies as much as novel research. It’s kind of crazy that the system almost completely lacks these checks and balances right now - peer review is more like an editor, it’s just a very different thing than reproduction. Bishop here is suggesting a data sleuthing approach to root out fraudsters, but I dunno if that’d be effective, as I think the main issue is subtle but pervasive bias seeping into studies by most researchers, vs. a smaller number of heavy fraudsters. Independent reproduction of results, while expensive, is the only effective approach I can think of to combat this.
- hyperthesis 3y ago
  Feynman has a bit where empirixal measurements of some universal constant (electron charge?) slowly crept up to the now-accepted true value. He said scientists are a little bit ashamed of this, because it showed that higher estimates were discarded and not published - bias. BTW scientists get no upvotes for merely reproducing "known" results. Is there a way to see natural merit/discovery in reproducing (not an artificial incentive, like grants for repeating studies). Perhaps analogous to how teaching helps your own understanding?
userinanother 3y ago
Anyone dismissing fraud hasn’t tried to replicate academic results to build products. Those people know exactly how bad the fraud truly is
everybodyknows 3y ago
> those who have committed fraud can rise to positions of influence and eminence ... > ... sideline any honest young scientists who want to do things properly. I fear in some institutions this has already happened. She has some names in mind!
- greenyoda 3y ago
  The recent resignation of Stanford's president comes to mind: https://www.npr.org/2023/07/19/1188828810/stanford-university-president-resigns https://www.npr.org/2023/07/19/1188828810/stanford-universit... > The president of Stanford University has resigned after an investigation opened by the board of trustees found several academic reports he authored contained manipulated data. > Marc Tessier-Lavigne, who has spent seven years as president, authored 12 reports that contained falsified information, including lab panels that had been stitched together, panel backgrounds that were digitally altered and blot results taken from other research papers.
yawnxyz 3y ago
This article puts the onus on publishers / the scientific publication process to catch and prevent fraud. This is like asking merchants to catch and stop fraud and crime (e.g. selling alcohol to kids); it's in their best interest to not catch fraud and maximize their income. The reason they do clamp down on underage drinking is because they'll get fined/arrested / their license will be revoked. The system cares enough to catch and punish the behavior. If the funding bodies like NIH don't catch, punish and stop this behavior, it creates a system where fraudsters win more. This makes more groups, even if reluctant, participate in fraud because that's the only way to compete. It's a race to the bottom. Money drives incentives, and clawing it back while blacklisting and publicly humiliating a lab, will change behavior. That would make the lab a toxic collaborator, especially if collaborators ALSO get blacklisted from funding.
- atrettel 3y ago
  What changes to the incentive structure do you think would help here? Blacklisting seems appropriate for repeat offenders but I'd be more interested in changing the incentives for funding to prevent fraud in the first place. The problem with blacklisting in my view is that many fraudsters who get away with it will just continue anyway since they do not "learn their lesson" until long after the damage is already done.
- ta988 3y ago
  NIH is using volunteers to review grants and some programs. So same problem applies. I've even seen reviewers stealing the work from grants they reviewed (and noted really badly so it wouldn't get funded) and that same reviewer did it multiple times. Nobody cared at NIH, the program officers said they would look into it and many years later still nothing happened. There is nothing really in place to solve blatant abuse.
atrettel 3y ago
I can sympathize with the idea of a "scientific police force" looking for fraud, but the fact is that we already attempt the appearance of that through peer review. The problem is that peer review right now only provides the appearance of oversight. Referees are not given the time, money, or resources to really dig into papers and look for issues. I have peer reviewed 4 papers this year, averaging around 3 hours per review. The onus is largely on the referees/editors/publishers to immediately find something wrong in the paper to reject it rather than on the authors to really prove their case. And I mean "immediately". One journal that I review for requires peer reviews within 2 weeks of accepting the assignment, and they will nag you if you do not have it done within a week. If the onus must be on the referees, they need to be given much more time and resources, and that might even include money, and also might include waiting for reproduction of the results before publishing. "Publish or perish" also plays a big role here, because scientists need to get papers published to prove they are productive and worthy of funding and employment. Authors will just re-submit their manuscripts to different journals until they are published rather than re-evaluating the research in any fundamental manner (is my methodology flawed? etc.). So putting the onus on referees isn't going to work in the first place, since peer reviews are not always shared.
- vlovich123 3y ago
  Peer review’s fundamental flaw is that science only works through replication and peer review tries to sidestep that as a cost saving measure. It doesn’t mean there’s no value - having peers review your work can be valuable. But as any professional coder can tell you, a code review is an extremely low quality signal that can only catch obvious bugs, typos, and stylistic/basic rule conformance issues. It can’t explain why you made various design decisions and whether those were any good to begin with. I think its current role as gatekeeper is probably overall more harmful than helpful as it lends legitimacy it doesn’t have a right to (ie this paper is right) but I don’t have a better suggestion other than independent replication. Maybe more separation between who designs, conducts and analyses experiments with the analysis people being rewarded if there’s no result while the design and experiment people being paid if there is to set up a feedback loop?
dang 3y ago
Related: https://statmodeling.stat.columbia.edu/2023/11/19/dorothy-bishop-on-the-prevalence-of-scientific-fraud/ https://statmodeling.stat.columbia.edu/2023/11/19/dorothy-bi... (via https://news.ycombinator.com/item?id=38336432 https://news.ycombinator.com/item?id=38336432, but we merged that thread hither)
- perihelions 3y ago
  Also, one of the papers OP mentions had its own thread, https://news.ycombinator.com/item?id=28107614 https://news.ycombinator.com/item?id=28107614 ("Tortured phrases: A dubious writing style emerging in science"—256 comments)

59 points by vo2maxer 3y ago

> To date, the response of the scientific establishment has been wholly inadequate. There is little attempt to proactively check for fraud: science is still regarded as a gentlemanly pursuit There is no incentive mechanism, it has been the problem all along. The peer review process is clearly inadequate or not up to the task in its current form.

rdedev 3y ago

Speaking as a former MSc student in CS, a lot of ML related roles are asking for paper publications especially in top journals. This is just more bad incentives for the students to hype up their paper even if it is good but not good enough for a top journal

captainkrtek 3y ago

Great defcon talk on uncovering fake science https://youtu.be/ras_VYgA77Q?feature=shared https://youtu.be/ras_VYgA77Q?feature=shared

TrackerFF 3y ago

It is a decent idea, but it can be really hard to spot scientific fraud. As in REALLY, REALLY hard. Some fields, like machine learning, gets deluged with papers coming from all kinds of researchers - there's no unified scientific notation, so to speak. In one paper you could have a pure math authors, in some other you could have econ authors, and then maybe a theoretical physics author. All these could use wildly different notation to describe related ideas, or discuss ideas to solve problems in their respective fields. And in that case, how do you detect bogus science? You could be a leading ML scientist, but not have the faintest idea about the other domain. Or the inverse - be a leading domain expert, but have little ML knowledge. Not to mention, the people that are publishing fraudulent scientific papers aren't morons - they're likely publishing that kind of stuff because of bad incentives. Maybe you'll catch the laziest, least capable fraudsters - but trying to catch savvy researchers sounds like a daunting task. In my mind, this type of task/job is more fit for full-time scientific "auditors".

zaptheimpaler 3y ago

I think a lot of knowledge about what is fraud is already distributed among researchers in their own fields but can't be accessed because no one wants to say those kinds of things in public. The whole system of academia is shrouded in a fog because saying "X field is dead" or "Y research is fraud" is just not done. But this is the same system that is unable to move off paid journals when the web has allowed ~0 cost publishing for decades. They can't effectively retract papers when the knowledge has failed to hold up. Like we have so many ways to organize, publish and review information on the web but they haven't adopted any of them. Literally a clone of HN or Reddit with upvotes and comments could solve half the problems with publishing and peer review. Actually publishing your data and your scripts no matter how hacky they are can help replication. This is a problem that is technologically easy to at least attack and improve if not solve. But none of that has happened so i don't have much hope, for some reason they aren't able to change.

logicchains 3y ago

Such a course will just give the people who want to commit fraud the tools they need to get away with it more successfully.

kukkeliskuu 3y ago

In the HN discussions about posts like this, there tend to be several comments from people saying they left the academic world because they were disgusted with what they saw. This is a worrying signal, because it tells me that fraud is so pervasive in some fields that just exposing one paper as fraud is not going to fix it. Also, like the article suggests, there appears to be pressure to dismiss the fraud suspicions by the institutions and the journals. These tell me that the problems are systematic, because the system does not automatically correct the faults -- but tries to prevent them from being corrected. In a culture that is based on trust, like science, we should see a very different attitude towards fixing failures.

throwawaymaths 3y ago

It's worse: some fraudsters get caught and aren't denounced. See: Homme Hellinga.

Throwfi44 3y ago

[flagged]

dang 3y ago
Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

yashap 3y ago

I have a slightly inside opinion on this - before I became a Software Engineer, I got an MSc in a mostly unrelated field, and am a co-author on work published in a few (very middling) peer reviewed journals. My gut feel from what I saw is that outright fraud is rare, but bias seeping into the study is common. Researchers invest a huge amount of effort into a study, they really want the hypothesis to be true. A lot of studies have steps that are pretty susceptible to bias, like maybe you’re classifying results and there’s a bit of a judgement call there. It’s human nature to make biased decisions in these situations, and make enough of these biased decisions, and statistically insignificant results become statistically significant. Some people might not see a difference between bias and fraud, but I personally do. I think of fraud as a very intentional deception, straight up falsifying numbers in a conscious attempt to deceive. While I see bias as more, you’ve got a borderline case, and you view it in the light you want to see it in, even somewhat unconsciously. Like the difference between unconscious racial bias, and overt hateful racism. I think the best approach to combatting this is to spend a lot less of the overall $$ in science on novel research, and a lot more on attempting to independently reproduce results. Papers should be seen as meaningless until their results can be independently reproduced, and universities/colleges should reward reproduction studies as much as novel research. It’s kind of crazy that the system almost completely lacks these checks and balances right now - peer review is more like an editor, it’s just a very different thing than reproduction. Bishop here is suggesting a data sleuthing approach to root out fraudsters, but I dunno if that’d be effective, as I think the main issue is subtle but pervasive bias seeping into studies by most researchers, vs. a smaller number of heavy fraudsters. Independent reproduction of results, while expensive, is the only effective approach I can think of to combat this.

hyperthesis 3y ago
Feynman has a bit where empirixal measurements of some universal constant (electron charge?) slowly crept up to the now-accepted true value. He said scientists are a little bit ashamed of this, because it showed that higher estimates were discarded and not published - bias. BTW scientists get no upvotes for merely reproducing "known" results. Is there a way to see natural merit/discovery in reproducing (not an artificial incentive, like grants for repeating studies). Perhaps analogous to how teaching helps your own understanding?

userinanother 3y ago

Anyone dismissing fraud hasn’t tried to replicate academic results to build products. Those people know exactly how bad the fraud truly is

everybodyknows 3y ago

> those who have committed fraud can rise to positions of influence and eminence ... > ... sideline any honest young scientists who want to do things properly. I fear in some institutions this has already happened. She has some names in mind!

greenyoda 3y ago
The recent resignation of Stanford's president comes to mind: https://www.npr.org/2023/07/19/1188828810/stanford-university-president-resigns https://www.npr.org/2023/07/19/1188828810/stanford-universit... > The president of Stanford University has resigned after an investigation opened by the board of trustees found several academic reports he authored contained manipulated data. > Marc Tessier-Lavigne, who has spent seven years as president, authored 12 reports that contained falsified information, including lab panels that had been stitched together, panel backgrounds that were digitally altered and blot results taken from other research papers.

yawnxyz 3y ago

This article puts the onus on publishers / the scientific publication process to catch and prevent fraud. This is like asking merchants to catch and stop fraud and crime (e.g. selling alcohol to kids); it's in their best interest to not catch fraud and maximize their income. The reason they do clamp down on underage drinking is because they'll get fined/arrested / their license will be revoked. The system cares enough to catch and punish the behavior. If the funding bodies like NIH don't catch, punish and stop this behavior, it creates a system where fraudsters win more. This makes more groups, even if reluctant, participate in fraud because that's the only way to compete. It's a race to the bottom. Money drives incentives, and clawing it back while blacklisting and publicly humiliating a lab, will change behavior. That would make the lab a toxic collaborator, especially if collaborators ALSO get blacklisted from funding.

atrettel 3y ago
What changes to the incentive structure do you think would help here? Blacklisting seems appropriate for repeat offenders but I'd be more interested in changing the incentives for funding to prevent fraud in the first place. The problem with blacklisting in my view is that many fraudsters who get away with it will just continue anyway since they do not "learn their lesson" until long after the damage is already done.
ta988 3y ago
NIH is using volunteers to review grants and some programs. So same problem applies. I've even seen reviewers stealing the work from grants they reviewed (and noted really badly so it wouldn't get funded) and that same reviewer did it multiple times. Nobody cared at NIH, the program officers said they would look into it and many years later still nothing happened. There is nothing really in place to solve blatant abuse.

atrettel 3y ago

I can sympathize with the idea of a "scientific police force" looking for fraud, but the fact is that we already attempt the appearance of that through peer review. The problem is that peer review right now only provides the appearance of oversight. Referees are not given the time, money, or resources to really dig into papers and look for issues. I have peer reviewed 4 papers this year, averaging around 3 hours per review. The onus is largely on the referees/editors/publishers to immediately find something wrong in the paper to reject it rather than on the authors to really prove their case. And I mean "immediately". One journal that I review for requires peer reviews within 2 weeks of accepting the assignment, and they will nag you if you do not have it done within a week. If the onus must be on the referees, they need to be given much more time and resources, and that might even include money, and also might include waiting for reproduction of the results before publishing. "Publish or perish" also plays a big role here, because scientists need to get papers published to prove they are productive and worthy of funding and employment. Authors will just re-submit their manuscripts to different journals until they are published rather than re-evaluating the research in any fundamental manner (is my methodology flawed? etc.). So putting the onus on referees isn't going to work in the first place, since peer reviews are not always shared.

vlovich123 3y ago
Peer review’s fundamental flaw is that science only works through replication and peer review tries to sidestep that as a cost saving measure. It doesn’t mean there’s no value - having peers review your work can be valuable. But as any professional coder can tell you, a code review is an extremely low quality signal that can only catch obvious bugs, typos, and stylistic/basic rule conformance issues. It can’t explain why you made various design decisions and whether those were any good to begin with. I think its current role as gatekeeper is probably overall more harmful than helpful as it lends legitimacy it doesn’t have a right to (ie this paper is right) but I don’t have a better suggestion other than independent replication. Maybe more separation between who designs, conducts and analyses experiments with the analysis people being rewarded if there’s no result while the design and experiment people being paid if there is to set up a feedback loop?

dang 3y ago

Related: https://statmodeling.stat.columbia.edu/2023/11/19/dorothy-bishop-on-the-prevalence-of-scientific-fraud/ https://statmodeling.stat.columbia.edu/2023/11/19/dorothy-bi... (via https://news.ycombinator.com/item?id=38336432 https://news.ycombinator.com/item?id=38336432, but we merged that thread hither)

perihelions 3y ago
Also, one of the papers OP mentions had its own thread, https://news.ycombinator.com/item?id=28107614 https://news.ycombinator.com/item?id=28107614 ("Tortured phrases: A dubious writing style emerging in science"—256 comments)