7 ms·

If one wants less fraud change incentives. 1. Grants and faculty positions should be less dependent on number of publications, and more on the quality of publi

If one wants less fraud change incentives. 1. Grants and faculty positions should be less dependent on number of publications, and more on the quality of publications. 2. Place more emphasis on basic research and less on translational research. The latter encourages hyperbole and p-hacking. In the late 90s, the NIH initiated a strong emphasis that all research should have an immediate practical value. This is not reasonable and as a result, researchers grossly exaggerate the impact of their work, and some even engage in an outright fraud.

nradov 2y ago
How do we measure quality of publications? I know there have been attempts to do so based on journal impact factor and quantity of citations but those metrics are also manipulated.
- desert_rue 2y ago
  I know we would have to change a lot from how it is set up today, but here are some ideas: - How many times has the results been reproduced by others? - Add another layer of blindness: the person doing the lab work is not the person crunching the results. It could even be where two different groups crunch the numbers, and all groups are unknown to the others. - Avoid p hacking with pre determined (and registered!) p values - Register the experiment with hypothesis before the start of the experiment - Register authors as they are added and removed, and have that history be publicly available - All results have to be uploaded to a website before publication - The method of calculation has to be public on a website before publication. So a high quality paper is one where - the experiment was logged in advance - the history of authors is known - there is a distinct separation of experimenter and cruncher - the public can get the results themselves and run the same analysis to confirm (or not!) the results - the experiment is repeated and confirmed by others. Even if the first experiment is bad or a fraud, and the second one doesn’t confirm it, a third experiment could be the tie breaker. It would be more traceable to understand if it was the lab or the cruncher that made a mistake or was committing fraud. Overall, we have to incentivize good science.
  - mike_hearn 2y ago
    Many of these ideas have been tried already. Unfortunately they don't work. 1. Pre-registration. Great idea, it's often being done. Doesn't work because universities don't enforce it. They'd have to proactively check that papers match their pre-registrations and then fire people when they don't. 2. Reproducibility. Nobody gets funded to do this but even if they did, there are lots of bogus studies that can easily be reproduced. The problems are in the methods rather than the data. If you specify a logically invalid or unscientific method, then people following your instructions will calculate the same results and still be wrong. 3. Blindness/splitting work up. This is already how many papers are made and academics turn it around as a defense. Notice how in every single case where this stuff happens, the people whose names are on the paper claim the fraud was done by someone else and they had no idea. Universities invariably accept this claim without question. 4. All results have to be uploaded before publication. Did you mean raw data? Results are the primary content of the paper, so it's unclear what this would mean. Researchers in some fields heavily resist publishing raw data for various (bad) reasons, like academic culture rewarding papers but not data collection efforts, so they're incentivized to squeeze as many papers out of a dataset as possible before revealing it to competitors. In a few fields (like climatology) they often refuse to publish raw data because they're afraid of empowering people who might check their papers for errors, who they see as a sort of shadowy enemy. 5. Authorship history. Which frauds are you thinking of that this would fix? I spent a lot of lockdown time looking at this question. You've listed five ideas, I churned through maybe 15-20. None of them work. On investigation they've usually been tried before and academics immediately work around them. Science is littered with integrity theatre in which systems are put in place in response to some fraud, and they appear to be operating on the surface, but nothing is actually being checked or enforced. > Overall, we have to incentivize good science. I'm by now 95% convinced the only way to do this is for scientific research to be done commercially. No government or charitable funding of science at all. As long as science is being firehosed with money by people who don't actually use or consume the resulting papers, the incentive will always be to provide the appearance of doing science without actually doing it. It's just so much more effective for fundraising. Commercial science that's purchased by someone has a customer who will actually try to check they got what they paid for at some point, and can vote with their wallet if they didn't. Also legal protections and systems to stop fraud in markets are well developed. Notice that Elizabeth Holmes went to prison for defrauding investors. If she'd done the same thing at a university she'd have gotten away with it scot free.
otherme123 2y ago
How do you determine quality? Right now, publishing below Q2 or even below Q1 in some cases is the same as not publishing at all. I've seen grants that only accept D1 papers. As a curiosity, Gregor Mendel original work was published in a small and newly created local Brno journal. It was cited three times in the following 35 years. By all metrics, it was a low quality work. Only 40 years after being published it was rediscovered as a fundamental work. That on the clean part. I've also seen papers published well above its merit just because the authors know the editors, or the paper comes from a reputable lab so it must be good. Also, the opposite is true: your work is from a small or unknown lab, or goes against the grain, and you'll be lucky if you get published at all.
- kergonath 2y ago
  > How do you determine quality? Certainly not by citations counts. Citations have more to do with the authors social or scientific network than the worth of the papers. > It was cited three times in the following 35 years. By all metrics, it was a low quality work. It was not low quality work (although it did spark quite a bit of controversy due to its perceived or actual issues). It was just an article written by an unknown author in an unknown journal.
- czl 2y ago
  FYI: Some have suggested that Mendel's data might be too perfect, indicating possible manipulation or fraud. (1) Mendel's results are unusually close to the expected ratios of 3:1 for dominant and recessive traits. Some argue that real-world data often show more variation. (2) In 1936, statistician Ronald A. Fisher analyzed Mendel's data and suggested it was "too good to be true." Fisher believed the results might have been altered or selectively reported to better match Mendel's hypothesis. (3) Despite these concerns, many of Mendel's experiments have been replicated, and his fundamental findings remain valid. Most scientists believe any discrepancies in Mendel's data were not due to intentional fraud but possibly unconscious bias or error.
- admissionsguy 2y ago
  > Also, the opposite is true: your work is from a small or unknown lab, or goes against the grain, and you'll be lucky if you get published at all. Unfortunately, many papers that come from obscure labs and go against the grain are both bad and wrong. It's a hard problem.
vlovich123 2y ago
Why not criminal charges for fraud? Besides the fact that this kind of stuff is used to obtain funding in the first place (defrauding the funding agency), it’s theft (stealing funds that would otherwise be allocated to competitors), and breaking the public trust since usually these kinds of funds are allocated by government agencies.
mike_hearn 2y ago
Are you arguing that Alzheimer's research isn't basic research? That it's got so much fraud because people demand immediate practical value from it? How does that square with the fact that they've been researching it for decades, haven't found a cure or even a solid understanding of the cause, and still get funded? Surely this is an exemplar of basic research?
- AceyMan 2y ago
  'Basic research' has a specific meaning in the natural sciences, which, I suspect, is how the OP was using the phrase.
  - mike_hearn 2y ago
    Interesting. What's the definition they use?
- BenFranklin100 2y ago
  If the authors are pushing a novel or shaky hypothesis that purports to provide a potential cure (as they often do), yes this might not fall under basic research. For instance, arguing a particular pathway is the cause of AD without first characterizing the pathway and how it affects humans can waste tens of million dollars and cost years. In the balance between trying to quickly develop a therapeutic and understanding the basic mechanisms driving the disease etiology, we see a lot of research groups with under powered studies playing fast and loose and hyping their research. Is it outright fraud? Rarely. But it’s not good science either.
hi-v-rocknroll 2y ago
E.g., return to the rigor and quality-aspects of research standards c. 1940 before it declined into the Sears, Roebuck and Co. model we find today. The commercialization of academia rewards churn and volume rather than $upporting (a semi-orthogonal constraint) and promoting landmark hard-work that takes longer to gather data or develop. Perhaps data-oriented research needs to consider extraordinary rigor as the new normal, perhaps with internal and external audits to assure standards and practices are sound.
kylebenzle 2y ago
As it is now, bad researches get rewarded and good researches who speak up get pushed out. One of the most despicable people I've ever met is a long term "successful" academic researcher at a major university.

nradov 2y ago

How do we measure quality of publications? I know there have been attempts to do so based on journal impact factor and quantity of citations but those metrics are also manipulated.

desert_rue 2y ago
I know we would have to change a lot from how it is set up today, but here are some ideas: - How many times has the results been reproduced by others? - Add another layer of blindness: the person doing the lab work is not the person crunching the results. It could even be where two different groups crunch the numbers, and all groups are unknown to the others. - Avoid p hacking with pre determined (and registered!) p values - Register the experiment with hypothesis before the start of the experiment - Register authors as they are added and removed, and have that history be publicly available - All results have to be uploaded to a website before publication - The method of calculation has to be public on a website before publication. So a high quality paper is one where - the experiment was logged in advance - the history of authors is known - there is a distinct separation of experimenter and cruncher - the public can get the results themselves and run the same analysis to confirm (or not!) the results - the experiment is repeated and confirmed by others. Even if the first experiment is bad or a fraud, and the second one doesn’t confirm it, a third experiment could be the tie breaker. It would be more traceable to understand if it was the lab or the cruncher that made a mistake or was committing fraud. Overall, we have to incentivize good science.
- mike_hearn 2y ago
  Many of these ideas have been tried already. Unfortunately they don't work. 1. Pre-registration. Great idea, it's often being done. Doesn't work because universities don't enforce it. They'd have to proactively check that papers match their pre-registrations and then fire people when they don't. 2. Reproducibility. Nobody gets funded to do this but even if they did, there are lots of bogus studies that can easily be reproduced. The problems are in the methods rather than the data. If you specify a logically invalid or unscientific method, then people following your instructions will calculate the same results and still be wrong. 3. Blindness/splitting work up. This is already how many papers are made and academics turn it around as a defense. Notice how in every single case where this stuff happens, the people whose names are on the paper claim the fraud was done by someone else and they had no idea. Universities invariably accept this claim without question. 4. All results have to be uploaded before publication. Did you mean raw data? Results are the primary content of the paper, so it's unclear what this would mean. Researchers in some fields heavily resist publishing raw data for various (bad) reasons, like academic culture rewarding papers but not data collection efforts, so they're incentivized to squeeze as many papers out of a dataset as possible before revealing it to competitors. In a few fields (like climatology) they often refuse to publish raw data because they're afraid of empowering people who might check their papers for errors, who they see as a sort of shadowy enemy. 5. Authorship history. Which frauds are you thinking of that this would fix? I spent a lot of lockdown time looking at this question. You've listed five ideas, I churned through maybe 15-20. None of them work. On investigation they've usually been tried before and academics immediately work around them. Science is littered with integrity theatre in which systems are put in place in response to some fraud, and they appear to be operating on the surface, but nothing is actually being checked or enforced. > Overall, we have to incentivize good science. I'm by now 95% convinced the only way to do this is for scientific research to be done commercially. No government or charitable funding of science at all. As long as science is being firehosed with money by people who don't actually use or consume the resulting papers, the incentive will always be to provide the appearance of doing science without actually doing it. It's just so much more effective for fundraising. Commercial science that's purchased by someone has a customer who will actually try to check they got what they paid for at some point, and can vote with their wallet if they didn't. Also legal protections and systems to stop fraud in markets are well developed. Notice that Elizabeth Holmes went to prison for defrauding investors. If she'd done the same thing at a university she'd have gotten away with it scot free.

otherme123 2y ago

How do you determine quality? Right now, publishing below Q2 or even below Q1 in some cases is the same as not publishing at all. I've seen grants that only accept D1 papers. As a curiosity, Gregor Mendel original work was published in a small and newly created local Brno journal. It was cited three times in the following 35 years. By all metrics, it was a low quality work. Only 40 years after being published it was rediscovered as a fundamental work. That on the clean part. I've also seen papers published well above its merit just because the authors know the editors, or the paper comes from a reputable lab so it must be good. Also, the opposite is true: your work is from a small or unknown lab, or goes against the grain, and you'll be lucky if you get published at all.

kergonath 2y ago
> How do you determine quality? Certainly not by citations counts. Citations have more to do with the authors social or scientific network than the worth of the papers. > It was cited three times in the following 35 years. By all metrics, it was a low quality work. It was not low quality work (although it did spark quite a bit of controversy due to its perceived or actual issues). It was just an article written by an unknown author in an unknown journal.
czl 2y ago
FYI: Some have suggested that Mendel's data might be too perfect, indicating possible manipulation or fraud. (1) Mendel's results are unusually close to the expected ratios of 3:1 for dominant and recessive traits. Some argue that real-world data often show more variation. (2) In 1936, statistician Ronald A. Fisher analyzed Mendel's data and suggested it was "too good to be true." Fisher believed the results might have been altered or selectively reported to better match Mendel's hypothesis. (3) Despite these concerns, many of Mendel's experiments have been replicated, and his fundamental findings remain valid. Most scientists believe any discrepancies in Mendel's data were not due to intentional fraud but possibly unconscious bias or error.
admissionsguy 2y ago
> Also, the opposite is true: your work is from a small or unknown lab, or goes against the grain, and you'll be lucky if you get published at all. Unfortunately, many papers that come from obscure labs and go against the grain are both bad and wrong. It's a hard problem.

vlovich123 2y ago

Why not criminal charges for fraud? Besides the fact that this kind of stuff is used to obtain funding in the first place (defrauding the funding agency), it’s theft (stealing funds that would otherwise be allocated to competitors), and breaking the public trust since usually these kinds of funds are allocated by government agencies.

mike_hearn 2y ago

Are you arguing that Alzheimer's research isn't basic research? That it's got so much fraud because people demand immediate practical value from it? How does that square with the fact that they've been researching it for decades, haven't found a cure or even a solid understanding of the cause, and still get funded? Surely this is an exemplar of basic research?

AceyMan 2y ago
'Basic research' has a specific meaning in the natural sciences, which, I suspect, is how the OP was using the phrase.
- mike_hearn 2y ago
  Interesting. What's the definition they use?
BenFranklin100 2y ago
If the authors are pushing a novel or shaky hypothesis that purports to provide a potential cure (as they often do), yes this might not fall under basic research. For instance, arguing a particular pathway is the cause of AD without first characterizing the pathway and how it affects humans can waste tens of million dollars and cost years. In the balance between trying to quickly develop a therapeutic and understanding the basic mechanisms driving the disease etiology, we see a lot of research groups with under powered studies playing fast and loose and hyping their research. Is it outright fraud? Rarely. But it’s not good science either.

hi-v-rocknroll 2y ago

E.g., return to the rigor and quality-aspects of research standards c. 1940 before it declined into the Sears, Roebuck and Co. model we find today. The commercialization of academia rewards churn and volume rather than $upporting (a semi-orthogonal constraint) and promoting landmark hard-work that takes longer to gather data or develop. Perhaps data-oriented research needs to consider extraordinary rigor as the new normal, perhaps with internal and external audits to assure standards and practices are sound.

kylebenzle 2y ago

As it is now, bad researches get rewarded and good researches who speak up get pushed out. One of the most despicable people I've ever met is a long term "successful" academic researcher at a major university.