4 ms·

After learning about the perverse incentives for publishing scientific papers, I find it extremely difficult to trust new research. Any tips for reversing this

After learning about the perverse incentives for publishing scientific papers, I find it extremely difficult to trust new research. Any tips for reversing this mindset? I know it's not too great to be feeling this way. Scientists have immense incentives for publishing new results. As someone quipped, "there is no cost in getting something wrong; the cost is not being published." Toxoplasmosis in particular seems intuitively like the perfect combination of new research plus massive prestige for whoever publishes papers on the topic. And we can't trust articles written on the topic either; they do little more than parrot whatever the scientists say, and don't really cast a critical eye on the methodology or findings. Which makes sense. I mean, how could they? None of us are experts the way a scientist is an expert. But it used to be a hobby of mine to dig through research papers and find subtle inconsistencies. Take fatal familial insomnia, for example. It's not fatal due to the insomnia. That's a misnomer. But the fact that it wasn't related to the insomnia was buried on the 7th page of the report as basically a footnote. I don't know. I just wish it were easier to take a lot of this at face value.

theprotocol 9y ago
I share your concerns, yet can't think of a solution. I think the only thing that can be done is to factor in a large amount of doubt, and amortize that doubt as more definitive studies come in over time.
autokad 9y ago
the biggest problem is the reliance on linear models and their interpretations. its easy to have something statistically significant but yet produce a shitty prediction. they then take this horrible model and use its coefficients to make ridiculous claims, like doing X makes you Y times more likely to Z. thats great for a business wanting to find ways to make money, but when this crap gets in the government level these crappy models are used to legislate us.
bogomipz 9y ago
>"After learning about the perverse incentives for publishing scientific papers ..." Can you be more specific, what was the source or your learning? Was there a particular piece or pieces you read? If so would you be able to share them?
DanBC 9y ago
> I find it extremely difficult to trust new research. Any tips for reversing this mindset? I know it's not too great to be feeling this way. Why is it not great to feel this way? You give a bunch of good reasons why it is okay to feel as you do. In general you should ignore any new research until it's had a large well run meta-analysis of it. Until then, it's all "we think" and "maybe". And that's ok, unless you're forcing people to take a treatment or depriving them of their liberty based on the research.
epistasis 9y ago
>Toxoplasmosis in particular seems intuitively like the perfect combination of new research plus massive prestige for whoever publishes papers on the topic Eh, not really much prestige, but perhaps lots of press. The big thing is to not take any single paper as "scientific truth" like what you read in a textbook. The scientific literature is not a textbook, it's the place where science's self-correction happens. Things in text books have been battle-tested, and widely accepted usually due to replication. And even then, textbooks have errors and perpetuate some falsehoods! Any individual scientific paper will not even likely be replicated, because it's working on something that did not turn out to be fundamental to the path of science. Scientific papers in the literature are how scientists share their small steps while exploring the n-dimensional world. That n is very high, so there are lots of directions to explore, and most of them will not be that interesting. If it is, somebody else may decide to take a look too. If it's a hot-topic, then many many people will. Scientists have to operate in a world where they are exploring what is and is not true, and their relationship with uncertainty is strong; it's absolutely essential to embrace uncertainty. The biggest thing is to distrust news reporting about science. It's correct about 1% of the time. The motivations of a reporter trying to somehow make an interesting article by deadline on a topic they don't understand are far more distorted than the scientists publishing a paper they expect 30-50 people to ever read. And that's if you actually get a real reporter to look at these things; most articles are just press releases that are cut and pasted from a university press lackey into the newspaper's publishing system. Those are usually even worse.
- asdfgadsfgasfdg 9y ago
  > And that's if you actually get a real reporter to look at these things; most articles are just press releases that are cut and pasted from a university press lackey into the newspaper's publishing system. Those are usually even worse. Actually if the researchers expect the paper to be high impact they will often provide the press release to the University's press department themselves - these types of press releases are merely a layman directed abstract and tend to be reasonably OK. This raises two interesting points. 1. Sometimes the lowest quality newspapers have the best quality article on a given publication. 2. Why aren't all research publications required to be accompanied by a second abstract written in plain low jargon language that can be understood at some level by someone with high school science?
prepaciously 9y ago
What would it mean to trust or believe in a theory? What finishing line in our minds does it cross beyond which we accept it as proven or regard it as confirmed? I don't think there is one. We are fallible and even our best theories are tentative and provisional. When we do rely on ideas it only because we know no better alternatives.
lima 9y ago
> Toxoplasmosis in particular seems intuitively like the perfect combination of new research plus massive prestige for whoever publishes papers on the topic While I absolutely agree with the general sentiment, some of the findings related to toxoplasmosis are well-replicated and supported by meta analysis, especially the link to schizophrenia and reaction time. Flegr was mostly ignored for a decade and his findings are only now reaching mainstream recognition, after replication by independent teams. Some of the recent "toxoplasmosis linked to X" papers are indeed of low-quality (low sample size and worse).
unixhero 9y ago
Listen, the science presented is better than no science. I think you are conflating published peer reviewed articles with the incentives of Academic Institutions, Journals and publishing houses. In my view it's unfair to let the scientists be the antagonist in this debate.
RA_Fisher 9y ago
I'm a statistician and I read medical studies and experimental studies pretty much daily. My advice on a simple rule of thumb that is likely to separate out quality research: are the data and statistical code made available? Another, are if the study hypotheses and design are pre-registered. The Cox model is VERY popular, but you have to ask two things: is the hazard reasonably proportional? If you thought learning to code meant you had to be careful about mistakes and typos, statistics is another layer of a similar size.

theprotocol 9y ago

I share your concerns, yet can't think of a solution. I think the only thing that can be done is to factor in a large amount of doubt, and amortize that doubt as more definitive studies come in over time.

autokad 9y ago

the biggest problem is the reliance on linear models and their interpretations. its easy to have something statistically significant but yet produce a shitty prediction. they then take this horrible model and use its coefficients to make ridiculous claims, like doing X makes you Y times more likely to Z. thats great for a business wanting to find ways to make money, but when this crap gets in the government level these crappy models are used to legislate us.

bogomipz 9y ago

>"After learning about the perverse incentives for publishing scientific papers ..." Can you be more specific, what was the source or your learning? Was there a particular piece or pieces you read? If so would you be able to share them?

DanBC 9y ago

> I find it extremely difficult to trust new research. Any tips for reversing this mindset? I know it's not too great to be feeling this way. Why is it not great to feel this way? You give a bunch of good reasons why it is okay to feel as you do. In general you should ignore any new research until it's had a large well run meta-analysis of it. Until then, it's all "we think" and "maybe". And that's ok, unless you're forcing people to take a treatment or depriving them of their liberty based on the research.

epistasis 9y ago

>Toxoplasmosis in particular seems intuitively like the perfect combination of new research plus massive prestige for whoever publishes papers on the topic Eh, not really much prestige, but perhaps lots of press. The big thing is to not take any single paper as "scientific truth" like what you read in a textbook. The scientific literature is not a textbook, it's the place where science's self-correction happens. Things in text books have been battle-tested, and widely accepted usually due to replication. And even then, textbooks have errors and perpetuate some falsehoods! Any individual scientific paper will not even likely be replicated, because it's working on something that did not turn out to be fundamental to the path of science. Scientific papers in the literature are how scientists share their small steps while exploring the n-dimensional world. That n is very high, so there are lots of directions to explore, and most of them will not be that interesting. If it is, somebody else may decide to take a look too. If it's a hot-topic, then many many people will. Scientists have to operate in a world where they are exploring what is and is not true, and their relationship with uncertainty is strong; it's absolutely essential to embrace uncertainty. The biggest thing is to distrust news reporting about science. It's correct about 1% of the time. The motivations of a reporter trying to somehow make an interesting article by deadline on a topic they don't understand are far more distorted than the scientists publishing a paper they expect 30-50 people to ever read. And that's if you actually get a real reporter to look at these things; most articles are just press releases that are cut and pasted from a university press lackey into the newspaper's publishing system. Those are usually even worse.

asdfgadsfgasfdg 9y ago
> And that's if you actually get a real reporter to look at these things; most articles are just press releases that are cut and pasted from a university press lackey into the newspaper's publishing system. Those are usually even worse. Actually if the researchers expect the paper to be high impact they will often provide the press release to the University's press department themselves - these types of press releases are merely a layman directed abstract and tend to be reasonably OK. This raises two interesting points. 1. Sometimes the lowest quality newspapers have the best quality article on a given publication. 2. Why aren't all research publications required to be accompanied by a second abstract written in plain low jargon language that can be understood at some level by someone with high school science?

prepaciously 9y ago

What would it mean to trust or believe in a theory? What finishing line in our minds does it cross beyond which we accept it as proven or regard it as confirmed? I don't think there is one. We are fallible and even our best theories are tentative and provisional. When we do rely on ideas it only because we know no better alternatives.

lima 9y ago

> Toxoplasmosis in particular seems intuitively like the perfect combination of new research plus massive prestige for whoever publishes papers on the topic While I absolutely agree with the general sentiment, some of the findings related to toxoplasmosis are well-replicated and supported by meta analysis, especially the link to schizophrenia and reaction time. Flegr was mostly ignored for a decade and his findings are only now reaching mainstream recognition, after replication by independent teams. Some of the recent "toxoplasmosis linked to X" papers are indeed of low-quality (low sample size and worse).

unixhero 9y ago

Listen, the science presented is better than no science. I think you are conflating published peer reviewed articles with the incentives of Academic Institutions, Journals and publishing houses. In my view it's unfair to let the scientists be the antagonist in this debate.

RA_Fisher 9y ago

I'm a statistician and I read medical studies and experimental studies pretty much daily. My advice on a simple rule of thumb that is likely to separate out quality research: are the data and statistical code made available? Another, are if the study hypotheses and design are pre-registered. The Cox model is VERY popular, but you have to ask two things: is the hazard reasonably proportional? If you thought learning to code meant you had to be careful about mistakes and typos, statistics is another layer of a similar size.