3 ms·
So let's go to the beginning: >"Because positive results are being published as fast as possible with little checking on them." Why do you think this is so? I
by nonbel 9y ago
So let's go to the beginning:
>"Because positive results are being published as fast as possible with little checking on them."
Why do you think this is so? It is because they think the stats are telling them the probability their theory is true, etc. A significant p-value means the result is "real", etc.
- IanCal 9y agoReplicating a study is not something that is likely to get you published, and putting in twice the work to run an experiment twice is unlikely to give you twice the 'return' on your time investment (assuming the funder even supports spending twice as much).
- dgfgfdagasdfgfa 9y agoWhy, though? Assumably the replication would be the interesting part, and presumably both teams work would be equally valuable showing the same result. Innovation is overrated; maybe we should just pay scientists more to do inglorious but immensely valuable work. I was certainly repulsed from lab sciences because I had no desire to do the work that's in demand—pulling results out of your ass to get more funding.
- IanCal 9y agoThey are likely to not get cited as much, which would lower a journals impact factor, reducing the incentive to the journal. This makes it worse for the scientist as there's work without a high "impact" paper, and since they get largely to choose the work they do if it's less "fun" it might be done less. The funder also has to choose between funding new work or replicating previous work. Something I'd be very interested in is paying independent labs to replicate studies. This removes the incentive issues surrounding the journals and researchers, leaving the funding issue.
- nonbel 9y agoWhy are they less likely to get cited as much? Does your typical researcher have some misunderstanding about how reliable the original results were?
- Fomite 9y agoNo one gets particularly large amounts of glory for confirmatory research because if it's successful than the first paper is still "The first paper to describe X".
- nonbel 9y agoI don't understand what studies you are referring to. Each additional paper should be improving whatever parameter estimates are being made, it makes no sense to only look at one result and ignore all the others.
- Fomite 9y agoWhile they'll all get swept up in meta-analyses, systematic reviews, etc. a huge portion of a paper's citation count can come from narrative sections, where that first paper will reap a lot of citations in "First describe by...", "As established in..." etc. The Nobel Prize goes to the people who discovered something. Not the people who confirmed that they were right.
- IanCal 9y agoWhy would you need to cite a paper that confirms a result? If several people have confirmed it, do you need to cite them all every time? If these were the case you'd have the opposite problem. People would be hugely incentivised to reproduce highly cited papers again, when what you want is to reproduce papers with low levels of confirmation.
- nonbel 9y agoYes, we want to know what observations we can "hang our hats on", so that we can come up with actual theories (not vague crap like "this drug makes that disease happen less"). The more replications published the better until it is being done in undergrad/high school classrooms. You can't rely on anything in the current half-assed, no replication environment. In many areas it is literally not worth coming up with an mathematical model to explain the data because it is all so questionable.
- vkou 9y agoNot to mention that if your study is irreproducible, because you made a mistake in experiment design, or failed to control for some variable, running your experiments twice may not necessarily catch this issue.
- IanCal 9y agoIndeed. Having someone else run the experiments is also required to get past something we have to deal with a lot in the tech world. The "it works on my machine" result. You may have run the experiment twice, but can anyone else? Does your setup depend on something not specified in your paper? In the tech world, how many times has the setup documentation you wrote turned out to be missing some detail when a new starter actually tries it? No setup will ever be perfect, but we can look at reducing common points of error.