4 ms·

I don't have much to add, only that it's been my personal experience as well. Fraud and non-reproducibility used to be such ethereal concepts to me, issues that

I don't have much to add, only that it's been my personal experience as well. Fraud and non-reproducibility used to be such ethereal concepts to me, issues that are discussed at a systemic level as an intellectual exercise. But after becoming a scientist, coming across so many fraudulent papers, so much non-reproducible and poorly done research. And seeing how it's affected my own, the thing that's dearest to me, having to build on top of those results and work in that global environment. It's been heart breaking, I feel no love for science anymore.

epistasis 4y ago
What field are you in where you are coming across lots of fraudulent papers?! I spend most of my career reanalyzing others' data and combining it together into larger datasets, in molecular biology and genomics. The only time I encountered mistakes it was from my own labeling errors or bugs, pre-publication. And in pre-publications datasets I would sometimes detect accidental swaps on the labels of samples as part of QC checks.
- ad404b8a372f2b9 4y ago
  Machine learning
  - riffraff 4y ago
    But surely ML did advance a lot in recent years? As the saying goes, science advances one funeral at a time, but it does advance. It seems overall you can still have an optimistic point of view of science while accepting it does not work as well as it could.
    - ad404b8a372f2b9 4y ago
      That's fair but that's the systemic view of the issue. When you are in the trenches you can't look at the good and throw away the bad. You have to compare your work to the fraudulent work, spend a lot of time reading and trying to reproduce the fraudulent work, cite the fraudulent work, compete with the fraudulent work, publish as fast they do, with results better than they produce. And even the work that's not fraudulent but just bad or rushed. It's a race to the bottom, and how can we ask that people compromise the quality of their work to compete when the only motivation for doing this work is the love of the craft. It's certainly not the money, or the environment, or the work-life balance, or the prestige.
      - epistasis 4y ago
        > When you are in the trenches you can't look at the good and throw away the bad. When in the trenches of science this is literally your job. What seems like the likely hypothesis, what model that explains the data is good or bad. Papers are not textbooks. They are the boiling cauldron from with some bits eventually emerged as fully cooked facts. The only way that I can square others' experience of science with my own is if they take the Discussion sections, which are free-form extrapolation about future directions, and treat them as if they were asserted as truth.
        ad404b8a372f2b9 4y ago
        None of what you said relates to the problem I described. It relates superficially to the sentence you quoted, but only if you strip it of its context. I'm talking about malicious, fraudulent results. Made up numbers, code obfuscated and manipulated to do things different from what's being claimed. I resent the implication that my issues stem from a naive, idealistic reading of the discussion section rather than a thorough examination of the methods. Sorting the good from the bad is one thing, creating and evaluating models, good models or bad models is one thing. Having to compete AND CITE fraudulent work, lies, imaginary models to be able to participate in the system is not "literally" science. And it's not the job any self-respecting scientist should want to do.
        epistasis 4y ago
        I'm sorry. The idea that there is outright fraud running rampant is so foreign to me that I clearly can not interpret your experience very well.
- nemo44x 4y ago
  Social sciences have a reproducibility problem and is littered with fraudulent data. Of course most of it isn’t really science but they (social scientists, politicians, activists) that rely on it to justify applications of power insist it is. Because otherwise it’s just anecdotal, which it mainly is. “Sokal Squared” is a great example of the problems in the field.
  - themitigating 4y ago
    Do you have evidence showing this?
    - nemo44x 4y ago
      I literally gave you a place to look in the post, Sokal Squared so here's a link to help you along. https://en.wikipedia.org/wiki/Grievance_studies_affair https://en.wikipedia.org/wiki/Grievance_studies_affair Regarding the reproducibility crisis: https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
ncmncm 4y ago
Fraud is probably not evenly distributed. The more money is involved, the more fraud to expect. This should not surprise: fraud goes where the money is. And where there is money, the stakes are higher. But I have seen reprehensible behavior even in opponents of the Younger Dryas Impact Hypothesis, where practically no money seemed to be at stake.
- lupire 4y ago
  For one person, a small amount of money is life changing motivation.
- themitigating 4y ago
  This generalization would occur in any occupation and probably doesn't. There should be consequences except many in thread now cast doubt on the whole of "science"
jostmey 4y ago
I still love science. I just don’t think the academic model and peer review publications makes sense anymore
atrettel 4y ago
I would not give up hope in science. In the short term, fraud is a problem, but science is a process and in the long term it will (eventually) self-correct, though the time scales for that process may be quite long indeed. To that end, does anyone know of any list of techniques for spotting or identifying fraud in scientific works? This is a subject that I wish somebody would write a book on, so I am curious as to what techniques people use for this purpose.
- origin_path 4y ago
  A list of resources for you to read up on this is below. But: science isn't really self correcting. Someone has to step up and fight to correct it, every single time. Spend some time learning about this problem - as I have - and you will find the overriding attitude is one of despair. Science isn't self correcting, it's arguably self corrupting: 1. There's no incentive to pick fights with colleagues so it almost never happens 2. Even when someone does pick that fight the institutions go to ground and defend their people, so there's no outcome. 3. The first response of journals and universities on being informed of fraud is to turn around and tell the fraudster everything, so there's no way to keep fraud detection techniques secret. 4. And usually after that they ask the fraudster to submit a "correction" (i.e. higher quality fake). The idea that you should maybe NOT let a fraudster have a second try once caught, does not seem to occur to the brightest of sparks that run our scientific institutions. There are actually cases where this has happened and then the original complainants spotted that the newly submitted correction was also fraudulent. How do these papers get spotted? Spend time on PubPeer. It's a site where people compare notes on dodgy papers and how they're being detected. Also read the blogs of people like Elizabeth Bik, Smut Clyde, read the old blog posts by Joe Hilgard when he was still in academia, follow https://twitter.com/steamtraen https://twitter.com/steamtraen tl;dr There's a range of techniques used for different fields, often by looking for re-used images or data across papers that shouldn't be re-using them, or by spotting internal inconsistencies in reported data tables.
  - themitigating 4y ago
    "3. The first response of journals and universities on being informed of fraud is to turn around and tell the fraudster everything, so there's no way to keep fraud detection techniques secret." Do all of them do this?

epistasis 4y ago

What field are you in where you are coming across lots of fraudulent papers?! I spend most of my career reanalyzing others' data and combining it together into larger datasets, in molecular biology and genomics. The only time I encountered mistakes it was from my own labeling errors or bugs, pre-publication. And in pre-publications datasets I would sometimes detect accidental swaps on the labels of samples as part of QC checks.

ad404b8a372f2b9 4y ago
Machine learning
- riffraff 4y ago
  But surely ML did advance a lot in recent years? As the saying goes, science advances one funeral at a time, but it does advance. It seems overall you can still have an optimistic point of view of science while accepting it does not work as well as it could.
  - ad404b8a372f2b9 4y ago
    That's fair but that's the systemic view of the issue. When you are in the trenches you can't look at the good and throw away the bad. You have to compare your work to the fraudulent work, spend a lot of time reading and trying to reproduce the fraudulent work, cite the fraudulent work, compete with the fraudulent work, publish as fast they do, with results better than they produce. And even the work that's not fraudulent but just bad or rushed. It's a race to the bottom, and how can we ask that people compromise the quality of their work to compete when the only motivation for doing this work is the love of the craft. It's certainly not the money, or the environment, or the work-life balance, or the prestige.
    - epistasis 4y ago
      > When you are in the trenches you can't look at the good and throw away the bad. When in the trenches of science this is literally your job. What seems like the likely hypothesis, what model that explains the data is good or bad. Papers are not textbooks. They are the boiling cauldron from with some bits eventually emerged as fully cooked facts. The only way that I can square others' experience of science with my own is if they take the Discussion sections, which are free-form extrapolation about future directions, and treat them as if they were asserted as truth.
      - ad404b8a372f2b9 4y ago
        None of what you said relates to the problem I described. It relates superficially to the sentence you quoted, but only if you strip it of its context. I'm talking about malicious, fraudulent results. Made up numbers, code obfuscated and manipulated to do things different from what's being claimed. I resent the implication that my issues stem from a naive, idealistic reading of the discussion section rather than a thorough examination of the methods. Sorting the good from the bad is one thing, creating and evaluating models, good models or bad models is one thing. Having to compete AND CITE fraudulent work, lies, imaginary models to be able to participate in the system is not "literally" science. And it's not the job any self-respecting scientist should want to do.
        epistasis 4y ago
        I'm sorry. The idea that there is outright fraud running rampant is so foreign to me that I clearly can not interpret your experience very well.
nemo44x 4y ago
Social sciences have a reproducibility problem and is littered with fraudulent data. Of course most of it isn’t really science but they (social scientists, politicians, activists) that rely on it to justify applications of power insist it is. Because otherwise it’s just anecdotal, which it mainly is. “Sokal Squared” is a great example of the problems in the field.
- themitigating 4y ago
  Do you have evidence showing this?
  - nemo44x 4y ago
    I literally gave you a place to look in the post, Sokal Squared so here's a link to help you along. https://en.wikipedia.org/wiki/Grievance_studies_affair https://en.wikipedia.org/wiki/Grievance_studies_affair Regarding the reproducibility crisis: https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis

ncmncm 4y ago

Fraud is probably not evenly distributed. The more money is involved, the more fraud to expect. This should not surprise: fraud goes where the money is. And where there is money, the stakes are higher. But I have seen reprehensible behavior even in opponents of the Younger Dryas Impact Hypothesis, where practically no money seemed to be at stake.

lupire 4y ago
For one person, a small amount of money is life changing motivation.
themitigating 4y ago
This generalization would occur in any occupation and probably doesn't. There should be consequences except many in thread now cast doubt on the whole of "science"

jostmey 4y ago

I still love science. I just don’t think the academic model and peer review publications makes sense anymore

atrettel 4y ago

I would not give up hope in science. In the short term, fraud is a problem, but science is a process and in the long term it will (eventually) self-correct, though the time scales for that process may be quite long indeed. To that end, does anyone know of any list of techniques for spotting or identifying fraud in scientific works? This is a subject that I wish somebody would write a book on, so I am curious as to what techniques people use for this purpose.

origin_path 4y ago
A list of resources for you to read up on this is below. But: science isn't really self correcting. Someone has to step up and fight to correct it, every single time. Spend some time learning about this problem - as I have - and you will find the overriding attitude is one of despair. Science isn't self correcting, it's arguably self corrupting: 1. There's no incentive to pick fights with colleagues so it almost never happens 2. Even when someone does pick that fight the institutions go to ground and defend their people, so there's no outcome. 3. The first response of journals and universities on being informed of fraud is to turn around and tell the fraudster everything, so there's no way to keep fraud detection techniques secret. 4. And usually after that they ask the fraudster to submit a "correction" (i.e. higher quality fake). The idea that you should maybe NOT let a fraudster have a second try once caught, does not seem to occur to the brightest of sparks that run our scientific institutions. There are actually cases where this has happened and then the original complainants spotted that the newly submitted correction was also fraudulent. How do these papers get spotted? Spend time on PubPeer. It's a site where people compare notes on dodgy papers and how they're being detected. Also read the blogs of people like Elizabeth Bik, Smut Clyde, read the old blog posts by Joe Hilgard when he was still in academia, follow https://twitter.com/steamtraen https://twitter.com/steamtraen tl;dr There's a range of techniques used for different fields, often by looking for re-used images or data across papers that shouldn't be re-using them, or by spotting internal inconsistencies in reported data tables.
- themitigating 4y ago
  "3. The first response of journals and universities on being informed of fraud is to turn around and tell the fraudster everything, so there's no way to keep fraud detection techniques secret." Do all of them do this?