6 ms·

As the replication crisis grows, or we're simply becoming more aware of it, I do wonder how much of this is due to poorly aligned incentives causing researchers

• cratermoon 4y ago
  In this case I'm pretty sure that the research in question was about the same quality as the work produced by Tobacco Industry Research Committee or the Sugar Research Foundation.
• pvaldes 4y ago
  > what needs to change in the education process so studies are better designed Remove politics, free kidnapped and gagged science.
  • seventytwo 4y ago
    Politics is an emergent property of human interaction. It cannot be removed.
  • dapf 4y ago
    > Remove politics That needs the remotion of taxation. I agree.
  • karaterobot 4y ago
    I think that's a separate issue that would not have affected this particular case.
  • xhkkffbf 4y ago
    That's just hard to do with a topic like this. There are committed people on both sides of the issue who know what they want to hear. That's not a good environment for science to find a neutral answer.
  • mumblemumble 4y ago
    Politics and education are certainly factors, but I would argue that they are minor. The bigger issue is the natural incentive structures that academia has created. If you want to have a career, you get both attention and funding in return for headline-grabbing results. This creates two huge second-order effects: first, everyone's incentivized to look for big impressive new effects. Second, focusing on confirmatory work is a great way to tank your career. And those create a third order effect: people who cut corners thrive, and there's not much to keep that in check because people who cautiously vet their work and others' get stuck in adjunct professor hell until they finally give up and exit the field. Politics just adds some spin onto that. Education, I just don't think that's even a possible solution. The math involved in some of these fancier statistical techniques can be devilishly hard to get right, and it's often simply asking too much to expect someone to be an expert at both that and molecular biology. You could argue that that means people should look for competent co-authors to cover that side of things. But, see above - current incentive structures don't really favor the researcher who says, "I know! Let's bring on some nerd whose only job is to find reasons why we shouldn't publish!"
    • pvaldes 4y ago
      If we assign grant points to people that flood the precious journal space with fake articles claiming over-hyped promises, maybe we should give grant points also to the people able to spot the lies on those kind of articles. After all they are serving to the advance of science also (and saving thousands of expensive hours of research to other). If you want to promote it, just create incentives. Otherwise, If the cheater wins in a 95% of the times and the verifier does not have anything to gain (and will damage their own research by the time invested on this) after a while only cheaters remain [1]. At middle term, all research tend towards a sort of expensive shamanism in well illuminated white rooms. [1] (And of course the non cheaters will be actively blocked from this point).
      • MPSimmons 4y ago
        Like a bug bounty for academic publications?
• colechristensen 4y ago
  I think it is not enough of a sense of quality and too much pressure or gamification of getting things published. And fundamentally lots of researchers need to be more interested in finding the truth and considerably better at statistics.
• dale_glass 4y ago
  Individual studies shouldn't be the primary focus. It shouldn't fall to every individual researcher to be fully accurate and honest. This is just not possible because people are imperfect, and have all the wrong incentives in many cases. What we need is a system designed to deal with this better. My suggestion would be greater incentives to the publication of negative results, and incentives to reproduce research. Create a system that encourages reproduction as a second step to peer review. This should also increase quality. If your research can't be easily reproduced it's probably not doing a whole lot of good even if ultimately you got the right answers. Test driven research, if you will.
  • jonhohle 4y ago
    Or even requirements for replication amongst groups with separate funding sources. If everyone needs replication for publishing, the incentive to replicate other’s work increases.
  • Tao3300 4y ago
    > It shouldn't fall to every individual researcher to be fully... honest. Really? I don't know, "Don't be a fraud" might be a good Rule No. 1
    • fennecfoxen 4y ago
      It’s more that a system should not allow a single fraud to shape its understanding and its policy. Assume fraud exists.
    • naasking 4y ago
      Fraud is only a small part of the problem. No open data, poorly described methodology so reproductions sometimes aren't even possible, too little funding for reproductions, p-hacking, no incentives to publish negative results, journals incentivized to publish sensational results to increase their impact factor even though sensational results are likely false thus perpetuating misinformation. These are the main drivers.
  • FrenchAmerican 4y ago
• dc-programmer 4y ago
  Personally this is one of the reasons I chose to go into software instead of statistics when I was a young math major. The idea that you could make a small mistake that invalidates all your work and you might not even know it until much later or even never always made me uncomfortable. With software I can write tests till my neuroticism has been placated
  • jefftk 4y ago
    If all the programmers who write tests go into industry and none of them go into academia, then the state of software in academia will be... oh, I see, the status quo.
    • pca006132 4y ago
      No, they do write tests, just 1. Include the ones that passed and discard those that failed. 2. Run the tests several times and report only the best result. 3. Keep some of the details secret (data, some weird parameters, or maybe the whole repository), tweak something, run again, get some nice figures and publish a new paper.
      • WalterBright 4y ago
        Caltech has Millikan's (oil drop experiment demonstrating quantization of charge) notebooks. He repeated the experiment until the data fit his theory. Fortunately, he turned out to be right.
• jwond 4y ago
  > I do wonder how much of this is due to poorly aligned incentives causing researchers to error on the side of what will get the most attention versus researchers in general simply being sloppy. I think we should also consider the possibility that some of it might be ideologically motivated. For example someone who thinks people shouldn’t eat red meat due to climate change, animal rights, etc. (and who possibly belongs to a social circle where such beliefs are popular) might be motivated to fabricate results that could convince some people to eat less red meat (hypothetically, not saying that is the case here).
  • deleted 4y ago
    [deleted]
• yboris 4y ago
  "Poorly aligned incentives" in the nutritional sciences is, as far as I can tell, mostly various food lobbies getting their product to look good. The reason there is "confusion" about whether many animal products are good or bad is because the corporations making billions of dollars off the product fund biased research. Consider looking at the research digests from a non-profit (that sells no products, has no ads, etc) focusing on improving your health: https://nutritionfacts.org/ https://nutritionfacts.org/
  • netmonk 4y ago
    How a non-profit pays for hosting website cost, writing report, and so on and so on… being a non-profit doesn’t mean you are tied to corporations and free of any conflict of interest. If you start looking carefully you will see that most non-profit org aren’t unbiased and honest. And since long time… For instance this whole vegan trend is based on some radical religious opinion to control fertility and lower class reproduction by promoting a hormonal dysfonctionnal diet. All those from “official scientific paper” written by those religious fanatic few decades ago.
    • yboris 4y ago
      > vegan trend is based on some radical religious opinion What a bold empirical claim. Could you share any evidence of this?
• contravariant 4y ago
  It has probably been a mistake to have so many articles that are effectively "We measured A and its value suggests the following theory" rather than "To test theory B we measured A and found it did(n't) agree" I reckon that lack of a good theory is the by far best predictor for non-replicable results.
  • pca006132 4y ago
    I think there are several issues with this. In these fields, theories (not so fundamental ones) may not be very useful, in contrast with fields like Mathematics or Physics where a hypothesis may lead to a lot of other theoretical results. For example, if material A has property X, there will be a lot of applications for it, but this hypothesis cannot be used as a foundation for other theoretical research. As a result, it is not useful to just state a hypothesis: you have to verify it as well, and negative results are not really valued because it means nothing to others if you rejected something they don't care about.
    • contravariant 4y ago
      Not sure if properties of materials is the best example, those tend to have some theory behind them that, if not explains, at least defines the property and its consequences. If one research team measures the stiffness of a material it's easy enough to make a test that replicates their result. Medical research in particular seems to suffer a lot from papers where something has been measured with no supporting theory whatsoever and consequently no real information other than "If you do what we did you probably get the same result". Rather unsurprisingly this is hard to replicate because there's no theory telling you what is and isn't relevant to replicate. I'm not saying this is easy to fix, but maybe we should focus more effort on papers that expand the theory, rather than papers that at best show that something might be the case but we don't know why.
      • watwut 4y ago
        It makes sense to me to first collect small amounts of data, only then formulate theories and only then verify those with larger experiments. Especially when experiments are expensive. It does not make sense to act like small studies are proof of anything the way people like to use those studies on the internet. It is wrong that researchers are effectively punished for reproduction, verification etc. But the existence of small data collection studies on itself is not wrong.
• derbOac 4y ago
  You're framing this as if they're mutually exclusive options, which they're not. My impression is that it's "all of the above": the incentive structure in academics is completely broken, and this incentivizes both unethical practices, as well as being sloppy "to get things done". I can think of numerous examples of both from my career, from working with various colleagues and in stories I've heard. All of this is the tip of the iceberg I think: what you hear about usually are cases of blatant fraud, with high-profile or high-importance research. The bulk of greyer malpractice, with softer squishier levels of ethical dubiousness, and more indirect, cumulative impacts on peoples' careers and literature trajectories, are more hidden. It's gotten to the point where I no longer put much stock in the reputation of individual researchers. Too many times I know in the best case there's an institution, or department, or subcommunity, or doctoral students or postdocs behind it, with who knows what sort of manipulation or luck. In the worst case there's years of fraud or other forms of corruption. Call me inappropriately cynical but I didn't start out this way.