4 ms·

My understanding is that fabricated results are so common that less than half of papers can be replicated. Nobody wants to go after the cheaters because it woul

by rdtwo 7y ago

My understanding is that fabricated results are so common that less than half of papers can be replicated. Nobody wants to go after the cheaters because it would embarrass the institution and the university would loose grant money.

gbrown 7y ago
That's not generally about outright fabrication, but instead due to invalid statistical procedures. Many students are still taught, in 2019, that the appropriate way to conduct a multiple linear regression analysis is to: 1. Fit univariate comparisons to all your explanatory variables of interest 2. Take the significant variables and put them in a multiple regression analysis 3. Report the p-values from the regression analysis as valid and meaningful. This is incorrect for several reasons (type 1 error rate inflation, confusion about marginal and conditional effects etc.), but people still do it. That's before we even account for publication bias, in which null results are often "shelved" or not accepted for publication.
- rscho 7y ago
  Indeed, I'm finishing a MSc degree in epidemiology and I was pretty surprised that they presented that as "the only correct way" (program rated 2nd worldwide for epidemiology). However, I do not agree that this is the most common problem. Ten years as a professional have shown me that the real issue is that 95% of people using statistics don't have a single clue about how the techniques they use daily really work. It's especially alarming given the booming machine learning fad.
  - gbrown 7y ago
    Yes, I was just picking out one example I've seen where even if people do what they're taught, they'll end up doing bad statistics. As you say the problems don't end there.
mbreese 7y ago
It isn’t because of fabricated results, but rather that some of these analyses are very difficult to perform. Some papers require specific techniques or tools that aren’t widely available. Some depend on biological samples that were exhausted. Some used poor experimental design. There are lots of reasons why a particular paper might be difficult to replicate. Academic publishing tends to reward those that make big discoveries or find something novel. Unfortunately, reproducibility isn’t as high on the priority list as it should be. But all that said - the amount of outright fabricated data is pretty small. Many people can make mistakes or overstate the meaning of their data. But the amount of people who flat out fake their data is really small. And institutions are more likely to loose grant money if they turn a blind eye than if they self regulate.
- biofox 7y ago
  100% agree. Just to add, as well as being difficult to perform, some experiments are highly sensitive to noise. In my experience, it's not uncommon in biology to sometimes have difficulty replicating your own experiments. There are always more variables than can be controlled, e.g. environmental conditions, different batches of reagents, how you handle samples, how animals are housed, electrostatic effects from different glassware / plastics, small changes in the genetic background, etc.

by rdtwo 7y ago

gbrown 7y ago

That's not generally about outright fabrication, but instead due to invalid statistical procedures. Many students are still taught, in 2019, that the appropriate way to conduct a multiple linear regression analysis is to: 1. Fit univariate comparisons to all your explanatory variables of interest 2. Take the significant variables and put them in a multiple regression analysis 3. Report the p-values from the regression analysis as valid and meaningful. This is incorrect for several reasons (type 1 error rate inflation, confusion about marginal and conditional effects etc.), but people still do it. That's before we even account for publication bias, in which null results are often "shelved" or not accepted for publication.

rscho 7y ago
Indeed, I'm finishing a MSc degree in epidemiology and I was pretty surprised that they presented that as "the only correct way" (program rated 2nd worldwide for epidemiology). However, I do not agree that this is the most common problem. Ten years as a professional have shown me that the real issue is that 95% of people using statistics don't have a single clue about how the techniques they use daily really work. It's especially alarming given the booming machine learning fad.
- gbrown 7y ago
  Yes, I was just picking out one example I've seen where even if people do what they're taught, they'll end up doing bad statistics. As you say the problems don't end there.

mbreese 7y ago

It isn’t because of fabricated results, but rather that some of these analyses are very difficult to perform. Some papers require specific techniques or tools that aren’t widely available. Some depend on biological samples that were exhausted. Some used poor experimental design. There are lots of reasons why a particular paper might be difficult to replicate. Academic publishing tends to reward those that make big discoveries or find something novel. Unfortunately, reproducibility isn’t as high on the priority list as it should be. But all that said - the amount of outright fabricated data is pretty small. Many people can make mistakes or overstate the meaning of their data. But the amount of people who flat out fake their data is really small. And institutions are more likely to loose grant money if they turn a blind eye than if they self regulate.

biofox 7y ago
100% agree. Just to add, as well as being difficult to perform, some experiments are highly sensitive to noise. In my experience, it's not uncommon in biology to sometimes have difficulty replicating your own experiments. There are always more variables than can be controlled, e.g. environmental conditions, different batches of reagents, how you handle samples, how animals are housed, electrostatic effects from different glassware / plastics, small changes in the genetic background, etc.