4 ms·

> I can't blame researchers for tailoring their research to help their own careers but this should help prevent that. It regularly astonishes me how easily peo

by hannob 8y ago

> I can't blame researchers for tailoring their research to help their own careers but this should help prevent that. It regularly astonishes me how easily people are willing to accept scientific malpractice with such excuses. While I agree that mechanisms like registered reports are the way to fix these things at the core, I do think that these mentalities ("it's just the system, I can't do anything but cheat with my scientific publications, my career!") are a big part of the problem. Good read on the issue: http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-incentives-its-you/ http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-inc...

deleted 8y ago
[deleted]
Devils_Avocado 8y ago
So, do 'scientists' not have dopamine pathways and an innate biological drive to maximise their personal longevity & reproductive success by securing control of as many resources as possible for as little risk as possible? "It's not the system; we only consistently get the same negative result that's different from what we want & expect because we have the wrong people staffing it. Don't change the status quo, change human neurological architecture".
gaius 8y ago
It regularly astonishes me how easily people are willing to accept scientific malpractice with such excuses. Likewise. The BBC reports that 2/3 of scientific research cannot be reproduced. Let’s be honest about what that means: 2/3 of scientists are fraudulent or incompetent. https://www.bbc.com/news/science-environment-39054778 https://www.bbc.com/news/science-environment-39054778
- lancebeet 8y ago
  If we want to give people the benefit of the doubt, it could also mean that scientific research is difficult and even competent individuals often make mistakes.
- SubiculumCode 8y ago
  This is incorrect and unhelpful. Science in fields with lots of complexity, such as neuroscience and psychology, suffer from hard problems such as unmeasured causal variables and divergent sample characteristics. You might chalk this up to incompetence, but there are are in fact real concerns of feasibility. You cannot hope to measure or control all relevant cognitive factors, you just try to balance your sample groups on the more obvious measures, increase your sample size, then hope that other differences average out with random assignment.
  - lrem 8y ago
    To put it otherwise: for some problems, none of the 7 billions humans alive is competent enough right now. For some of them, we'll collectively eventually get there, if we don't stop trying.
- AllegedAlec 8y ago
  > Let’s be honest about what that means: 2/3 of scientists are fraudulent or incompetent. No, it means that 2/3rds of the attempts to reproduce research fails. When you work on the very edge of science, the lab you're working on it probably either the only one, of one of a very small number who are even able to perform the research that you're doing, due to a combination of highly specialised equipment, subject knowledge and just plain experience. Without that combination of factors, it's quite easy to mess up an experiment and therefore fail to reproduce the research.
rumcajz 8y ago
I think one of the problems is that it often doesn't feel like cheating. You have a hypothesis, but you don't preregister it. It lives only in your head. You do an experiment, you look at the results. They fail to confirm the hypothesis. But if you tweak the hypothesis just a little, the data suddenly confirm it. So why not publish the tweaked version? This is a malpractice that effectively invalidates the research. But it doesn't feel so. It feels more like a thought-crime.
- AllegedAlec 8y ago
  > You do an experiment, you look at the results. They fail to confirm the hypothesis. But if you tweak the hypothesis just a little, the data suddenly confirm it. So why not publish the tweaked version? If between tweaking the hypothesis and publishing it, you add the step "you perform another experiment which tests the tweaked hypothesis", you have just described the scientific method.
  - raverbashing 8y ago
    I'm sure the budget for this new experiment will be given straight away, no questions asked. Though of course, there's a difference between "drug A doesn't work for condition B but seems to work slightly for C" and "drug A doesn't work for condition B but 10 of 12 individuals with condition C have shown significant improvement"
    - AllegedAlec 8y ago
      > I'm sure the budget for this new experiment will be given straight away, no questions asked. Most of the cases: yes, although your example of clinical trials is slightly different, and in that case, I do think the data should be publicly available to other researchers even in the case of a null.
    - xamuel 8y ago
      If the original experiment was large enough [1], you could almost always find some C such that "drug A doesn't work for condition B but 90% [2] of individuals with condition C showed significant improvement". In fact, you could replace [2] by a number arbitrarily close to 100% by increasing [1] accordingly.
chiefalchemist 8y ago
> "It regularly astonishes me how easily people are willing to accept scientific malpractice with such excuses." Agreed in theory. In practice such don't even qualify - by definition - as science. Using the word science in such contexts gives such things far more credit and legitimacy than they deserve.
wpietri 8y ago
> It regularly astonishes me how easily people are willing to accept scientific malpractice with such excuses. I get that. But the other side is that we pump up a lot of kids about science, then filter for very ambitious, very dedicated people and shove them into a resource-constrained environment where some kinds of results are much more strongly rewarded than others. It's absurd to set up a system like that and expect that science will be done to the level of frankness that you (and I) want. I know some really smart, hardworking, very honest PhDs who have shifted to the tech industry because the roll of the research dice didn't come up right for them early in their careers. They are far from happy about that. I wish they could have stayed in science. And I'd note that the person you're replying to is not "accepting scientific malpractice". They're saying they can't blame people for trying to survive in the system that they're caught in. If you want to be mad at somebody, be mad at the people who have set up the system, who could change it but don't. It's unscientific and illogical to expect humans to sacrifice their lives on the altar of your belief: statistically, that's not what they're like. Especially in a system that filters out the most honest, least career-focused people early on.
grigjd3 8y ago
Whether you blame researchers or not is inconsequential. People respond to incentives with a spectrum of responses from incredibly principled to entirely unprincipled. If the unprincipled are rewarded and the principled are penalized, the principled will leave the field for something else. Moralize all you like but the incentives define the field.

by hannob 8y ago

deleted 8y ago

[deleted]

Devils_Avocado 8y ago

So, do 'scientists' not have dopamine pathways and an innate biological drive to maximise their personal longevity & reproductive success by securing control of as many resources as possible for as little risk as possible? "It's not the system; we only consistently get the same negative result that's different from what we want & expect because we have the wrong people staffing it. Don't change the status quo, change human neurological architecture".

gaius 8y ago

It regularly astonishes me how easily people are willing to accept scientific malpractice with such excuses. Likewise. The BBC reports that 2/3 of scientific research cannot be reproduced. Let’s be honest about what that means: 2/3 of scientists are fraudulent or incompetent. https://www.bbc.com/news/science-environment-39054778 https://www.bbc.com/news/science-environment-39054778

lancebeet 8y ago
If we want to give people the benefit of the doubt, it could also mean that scientific research is difficult and even competent individuals often make mistakes.
SubiculumCode 8y ago
This is incorrect and unhelpful. Science in fields with lots of complexity, such as neuroscience and psychology, suffer from hard problems such as unmeasured causal variables and divergent sample characteristics. You might chalk this up to incompetence, but there are are in fact real concerns of feasibility. You cannot hope to measure or control all relevant cognitive factors, you just try to balance your sample groups on the more obvious measures, increase your sample size, then hope that other differences average out with random assignment.
- lrem 8y ago
  To put it otherwise: for some problems, none of the 7 billions humans alive is competent enough right now. For some of them, we'll collectively eventually get there, if we don't stop trying.
AllegedAlec 8y ago
> Let’s be honest about what that means: 2/3 of scientists are fraudulent or incompetent. No, it means that 2/3rds of the attempts to reproduce research fails. When you work on the very edge of science, the lab you're working on it probably either the only one, of one of a very small number who are even able to perform the research that you're doing, due to a combination of highly specialised equipment, subject knowledge and just plain experience. Without that combination of factors, it's quite easy to mess up an experiment and therefore fail to reproduce the research.

rumcajz 8y ago

I think one of the problems is that it often doesn't feel like cheating. You have a hypothesis, but you don't preregister it. It lives only in your head. You do an experiment, you look at the results. They fail to confirm the hypothesis. But if you tweak the hypothesis just a little, the data suddenly confirm it. So why not publish the tweaked version? This is a malpractice that effectively invalidates the research. But it doesn't feel so. It feels more like a thought-crime.

AllegedAlec 8y ago
> You do an experiment, you look at the results. They fail to confirm the hypothesis. But if you tweak the hypothesis just a little, the data suddenly confirm it. So why not publish the tweaked version? If between tweaking the hypothesis and publishing it, you add the step "you perform another experiment which tests the tweaked hypothesis", you have just described the scientific method.
- raverbashing 8y ago
  I'm sure the budget for this new experiment will be given straight away, no questions asked. Though of course, there's a difference between "drug A doesn't work for condition B but seems to work slightly for C" and "drug A doesn't work for condition B but 10 of 12 individuals with condition C have shown significant improvement"
  - AllegedAlec 8y ago
    > I'm sure the budget for this new experiment will be given straight away, no questions asked. Most of the cases: yes, although your example of clinical trials is slightly different, and in that case, I do think the data should be publicly available to other researchers even in the case of a null.
  - xamuel 8y ago
    If the original experiment was large enough [1], you could almost always find some C such that "drug A doesn't work for condition B but 90% [2] of individuals with condition C showed significant improvement". In fact, you could replace [2] by a number arbitrarily close to 100% by increasing [1] accordingly.

chiefalchemist 8y ago

> "It regularly astonishes me how easily people are willing to accept scientific malpractice with such excuses." Agreed in theory. In practice such don't even qualify - by definition - as science. Using the word science in such contexts gives such things far more credit and legitimacy than they deserve.

wpietri 8y ago

> It regularly astonishes me how easily people are willing to accept scientific malpractice with such excuses. I get that. But the other side is that we pump up a lot of kids about science, then filter for very ambitious, very dedicated people and shove them into a resource-constrained environment where some kinds of results are much more strongly rewarded than others. It's absurd to set up a system like that and expect that science will be done to the level of frankness that you (and I) want. I know some really smart, hardworking, very honest PhDs who have shifted to the tech industry because the roll of the research dice didn't come up right for them early in their careers. They are far from happy about that. I wish they could have stayed in science. And I'd note that the person you're replying to is not "accepting scientific malpractice". They're saying they can't blame people for trying to survive in the system that they're caught in. If you want to be mad at somebody, be mad at the people who have set up the system, who could change it but don't. It's unscientific and illogical to expect humans to sacrifice their lives on the altar of your belief: statistically, that's not what they're like. Especially in a system that filters out the most honest, least career-focused people early on.

grigjd3 8y ago

Whether you blame researchers or not is inconsequential. People respond to incentives with a spectrum of responses from incredibly principled to entirely unprincipled. If the unprincipled are rewarded and the principled are penalized, the principled will leave the field for something else. Moralize all you like but the incentives define the field.