3 ms·

DARPA is doing a relatively large scale effort to see if social sciences papers replicate, and how easy it is to look at a paper and guess if it will replicate

DARPA is doing a relatively large scale effort to see if social sciences papers replicate, and how easy it is to look at a paper and guess if it will replicate the preliminary estimation phase predicts like 50% rates of replication let's be real, the average social science paper is not good edit: there are lots of garbage papers in every field, the culture of academia is deeply flawed, the low standards are just even lower in some fields

p_l 3y ago
The problem is that replication studies aren't done hard enough. And by replication studies I mean "take this paper, try to replicate, write a paper on it". Every singular paper can be wrong, that's why you're supposed to replicate them - that's at least what was taught in psychology, which many consider "social science of less rigour" - well, it was probably the course most hardcore about experimental design in my education and ability to replicate a study...
- maximus-decimus 3y ago
  Are they not done or are they done and nobody wants to publish them?
  - neuronerdgirl 3y ago
    Nobody CAN publish them. An actual criterion for most top journals is literally "novelty". Reviewers will immediately levee "this has been done before" as a damning critique in peer review. There is no replication because you can't currently share replications, let alone sustain a career on them.
- zilti 3y ago
  My experience of "representative" studies in psychology (university of Berne) was that they'd come up with something, then gather a group of "volunteer" psychology students - 80% women - who need to participate at x studies as requirement to pass, publish it as representative serious research, and call it a day.
- the_snooze 3y ago
  >The problem is that replication studies aren't done hard enough. >And by replication studies I mean "take this paper, try to replicate, write a paper on it". You don't necessarily need to perform a straight-up "replication study" in order to do some useful sanity checking on published results. It can be done as part of another novel study that builds upon prior work. That's basically what happened with the fraudulent UCLA political persuasion study several years ago [1]. Some Berkeley researchers wanted to extend the ideas in that earlier published paper, reused similar methods under the assumption that the UCLA folks were correct, and couldn't get anything reasonable within the ballpark of what was published. [1] https://fivethirtyeight.com/features/how-two-grad-students-uncovered-michael-lacour-fraud-and-a-way-to-change-opinions-on-transgender-rights/ https://fivethirtyeight.com/features/how-two-grad-students-u...
WhitneyLand 3y ago
Yes there’s a serious issue, but thrust of the previous comment was that it’s not productive to criticize poor research using poor research. Conservatively 90%? Bullshit. The real numbers are already terrible so why use unintentional hyperbole to make the argument it just weakens it.
- Palomides 3y ago
  I don't think there's any objective way to say that a majority of papers are bullshit, even if they are. Failure to replicate is "easy" to show, but doesn't say anything about the utility or quality of the science done, beyond the absolute basics of a paper describing what a scientist did in detail and not lying about the results. A "sky is blue" paper will replicate, but I'm not sure how you'd find a "fair" way to call it useless. Anyway, jumping from "50% of papers will probably fail an absolute bare minimum test" to "90% of papers are garbage" is subjectively reasonable to me. If 90% seems shocking to someone, I suggest they skim a bunch of papers and develop their own feel for how bad the situation is.
  - orwin 3y ago
    I read a lot of history and some linguistics, it seems really fine to me. Linguistic especially, they work on _a lot_ of data, you don't have the "tested on 15 subjects" from biology or medecine effect. Even if i agreed with your "subjectively reasonable" idea, the issue is that non-STEM science encapsulate a lot of different fields.
    - Palomides 3y ago
      what's non-STEM science? the incentives of publishing and funding systems are similar across most fields (publish papers in the cool journals, get as many citations as possible, leverage that for more funding), I would be astonished if there were fields substantially exempt from bad science. I feel like my position is fair for any field publishing papers with falsifiable claims based on evidence?
  - WhitneyLand 3y ago

p_l 3y ago

The problem is that replication studies aren't done hard enough. And by replication studies I mean "take this paper, try to replicate, write a paper on it". Every singular paper can be wrong, that's why you're supposed to replicate them - that's at least what was taught in psychology, which many consider "social science of less rigour" - well, it was probably the course most hardcore about experimental design in my education and ability to replicate a study...

maximus-decimus 3y ago
Are they not done or are they done and nobody wants to publish them?
- neuronerdgirl 3y ago
  Nobody CAN publish them. An actual criterion for most top journals is literally "novelty". Reviewers will immediately levee "this has been done before" as a damning critique in peer review. There is no replication because you can't currently share replications, let alone sustain a career on them.
zilti 3y ago
My experience of "representative" studies in psychology (university of Berne) was that they'd come up with something, then gather a group of "volunteer" psychology students - 80% women - who need to participate at x studies as requirement to pass, publish it as representative serious research, and call it a day.
the_snooze 3y ago
>The problem is that replication studies aren't done hard enough. >And by replication studies I mean "take this paper, try to replicate, write a paper on it". You don't necessarily need to perform a straight-up "replication study" in order to do some useful sanity checking on published results. It can be done as part of another novel study that builds upon prior work. That's basically what happened with the fraudulent UCLA political persuasion study several years ago [1]. Some Berkeley researchers wanted to extend the ideas in that earlier published paper, reused similar methods under the assumption that the UCLA folks were correct, and couldn't get anything reasonable within the ballpark of what was published. [1] https://fivethirtyeight.com/features/how-two-grad-students-uncovered-michael-lacour-fraud-and-a-way-to-change-opinions-on-transgender-rights/ https://fivethirtyeight.com/features/how-two-grad-students-u...

WhitneyLand 3y ago

Yes there’s a serious issue, but thrust of the previous comment was that it’s not productive to criticize poor research using poor research. Conservatively 90%? Bullshit. The real numbers are already terrible so why use unintentional hyperbole to make the argument it just weakens it.

Palomides 3y ago
I don't think there's any objective way to say that a majority of papers are bullshit, even if they are. Failure to replicate is "easy" to show, but doesn't say anything about the utility or quality of the science done, beyond the absolute basics of a paper describing what a scientist did in detail and not lying about the results. A "sky is blue" paper will replicate, but I'm not sure how you'd find a "fair" way to call it useless. Anyway, jumping from "50% of papers will probably fail an absolute bare minimum test" to "90% of papers are garbage" is subjectively reasonable to me. If 90% seems shocking to someone, I suggest they skim a bunch of papers and develop their own feel for how bad the situation is.
- orwin 3y ago
  I read a lot of history and some linguistics, it seems really fine to me. Linguistic especially, they work on _a lot_ of data, you don't have the "tested on 15 subjects" from biology or medecine effect. Even if i agreed with your "subjectively reasonable" idea, the issue is that non-STEM science encapsulate a lot of different fields.
  - Palomides 3y ago
    what's non-STEM science? the incentives of publishing and funding systems are similar across most fields (publish papers in the cool journals, get as many citations as possible, leverage that for more funding), I would be astonished if there were fields substantially exempt from bad science. I feel like my position is fair for any field publishing papers with falsifiable claims based on evidence?
- WhitneyLand 3y ago