6 ms·
From my perspective as a former academic there is no simple solution here. We have created a system where the pressure to publish first and ask questions later
by danieltillett 11y ago
From my perspective as a former academic there is no simple solution here. We have created a system where the pressure to publish first and ask questions later is totally overwhelming. This is a cancer within science that will ultimately destroy public support for funding scientific research.
If we are going to solve this problem we need to reduce the pressure on scientists to publish at all costs. This publish-or-die thinking is driven by the extreme competition for grants and tenure tracked positions. Given I think it unlikely that the amount of money available for research is going to quadruple anytime soon (this is about the amount of money we would need to put into the system to get rid of the publish-at-all-costs mentality) what can be done?
1. Break the link between publication and grant funding. We could move to a system where once you had proved that you could do research (say published a few papers) and could come up with a viable research idea that was doable assessed by a simple grant proposal, then you go into a pool and the grants allocated by lottery. We should aim to have 50% to 75% of all scientists in this pool. We then fund as many grants as we can given the money available.
2. Spend some of the current research money on long-term fellowships (say 10 years) that are awarded in the same lottery way as grants.
3. Drastically reduce PhD student training to get supply and demand back into line. e.g. make it such that a supervisor can not supervise more than one student at a time.
- patkai 11y agoBut let's not forget that a significant proportion of stakeholders have no interest in changing the system. The same "broken" system is actually a way to provide hidden subsidies for industry, decrease unemployment (in many countries), show the public that "research funding was increased by the government", and who knows what else.
- danieltillett 11y agoSure there is a strong vested interest in the status quo by those that have succeeded in it, but it has reached such an extreme level that it is killing science. My experience is that the successful scientists within the current system love science enough to change even if it puts them at a disadvantage. Having said this we could grandfather in the current “winners” and move the funding to the new system over a long time period (say 20 years). We really need to do something as the current system will destroy science if we let it continue.
- patkai 11y ago"the current system will destroy science" I'm uncertain about this. The scientific method can luckily not be destroyed, and I wonder whether the self-destruction of academia is a good or bad thing. It is certainly not painless and will certainly take innocent victims. If you consider non-equilibrium thermodynamics analogies, the quicker the system destroys itself the better it is, as it more or less _has to_ explode before it reorganises. In Finland it has started already, as universities are required to get part of their funding from industry, and it's especially hard to accomplish in smaller cities.
- danieltillett 11y agoThe scientific can’t be destroyed, but the scientific infrastructure can be. To do experiments and make scientific breakthroughs you need access to a wide range of skills, equipment, and reagents. If this goes it will not be easy to get it back.
- ars 11y ago> We have created a system where the pressure to publish first and ask questions later is totally overwhelming. Why can't people simply publish negative results? > Break the link between publication and grant funding. Same answer - in fact give extra priority to those who publish lots of negative results.
- stkni 11y ago>> Why can't people simply publish negative results? The journal's peer review process tends to reject them. Or so I'm told by a researcher friend of mine.
- adrianN 11y agoBecause negative results are a dime a dozen. If you want to publish a negative result you have to do a lot more work to convince your peers that the thing you did was actually interesting. It's hard enough to do that with papers that show a real effect, scientists tend to be rather dismissive of other's work.
- IshKebab 11y ago> Why can't people simply publish negative results? Mainly because it's not as interesting. For every positive result researchers find there are dozens of things they test that just don't work. Imagine if the first test of "romantic priming" didn't work. Would the researchers really bother trying to publish "Hypothesised romantic priming effect was just a wild hunch after all."? And would the journal accept it? No way. Of course if you get a negative effect for an established "fact", as in this case it can be published. But you have to be really really sure otherwise the reviewers will just say you did your experiment wrong.
- iamsohungry 11y ago> Mainly because it's not as interesting. For every positive result researchers find there are dozens of things they test that just don't work. That seems to be the core of the problem. Reality doesn't have a bias toward results that are interesting, humans do. > Imagine if the first test of "romantic priming" didn't work. Would the researchers really bother trying to publish "Hypothesised romantic priming effect was just a wild hunch after all."? And would the journal accept it? No way. So maybe they should.
- chris_wot 11y agoI think it would be far better if major journals demanded datasets be made available and to make it easier to publish peer reviewed pieces that find anomalies or flatly contradict the conclusions of published papers.
- danieltillett 11y agoYes this would help, but it is not the driver of all these bad behaviours. We need to fix the root cause, not try and treat the symptoms.
- chris_wot 11y agoThe issue, as you probably correctly point out, is the pressure to publish too early and take shortcuts. There is only so much money for scientific research as all areas of science are vast. Is it really surprising that there is so much pressure?
- danieltillett 11y agoThere has always been problems in science due to the demand being greater than the dollars available, the problem with the current system is we are rewarding people cutting corners and doing bad science. This needs to change.
- adrianN 11y agoGiven limited funds I think we should not distribute them via a lottery. That just doesn't seem like the most efficient use of the money we have. I fully agree with point 3 though. Currently we produce PhDs waay above replacement rate. Every tenured professor produces dozens of PhDs during his career, but his tenured position only becomes available once he retires. That's just not sustainable, no matter how much money you throw at universities and research institutes.
- danieltillett 11y agoThe reason why is to break the link between track record and grant funding. It has got so extreme that it has perverted the whole of science. It just is not possible by peer review to determine the difference between a grant that is ranked in the top 10% and one that is ranked in the top 30%. Given we only have enough money to give less than 10% of all grants money, continuing the current process is just creating massive problems.
- tikhonj 11y agoNot just a lottery, but it feels like randomness should be part of the process. The mechanisms and people we have in place to make decisions are horribly imperfect, so there's a lot of noise in the system as is. Many perfectly good grants don't get funded, often for unclear reasons. Adding randomness past a certain minimum bar just makes that noise explicit and ensures that it's not biased or spuriously correlated. This would also deemphasize the importance of narrow skills specific to writing and getting grants. Right now, understanding the grant process and granting agency is probably just as important as, you know, coming up with good proposals. Not ideal! Among other things, this could also help academics to spread out a bit—right now, research agendas are extremely fashion-driven, in large part because of how grants are awarded. Think about it like a search algorithm which, with some probability, randomly jumps out of its current state as a way to avoid getting stuck in local maxima. Originally, I thought of this idea as applied to college admissions. Harvard likes to talk about how it could assemble a few classes of qualified students each year, but only has room for one. At that point, it's distinctly non-obvious that the process they use to choose between similarly qualified candidates is any good; some actual randomness could have benefits. But this idea feels even more applicable to research grants.
- mandor 11y ago4. The grant agencies (e.g. the NSF) could use some of its money to pay some labs to reproduce the study during the end-of-grant review process, in the same way as it already pays reviewers and committees for the initial review process. This would add a direct pressure to publish things that are easy to reproduce. Of course, that would mean even less grants... 5. Each PI should have a single grant at a time (and a long one), to end the "money competition" (that mirrors the publication competition) and spread the grants better.
- danieltillett 11y agoIn regards 4 the basic problem is it is really difficult to reproduce work. If an experiment does not work is it because the original paper was wrong, or is it because you are just not doing things correctly? The person best able to ensure everything is correct is the original investigator provided they have the time. This is what we need to fix. If we were to move to a lottery process then there would be no need to limit anyone.
- PepeGomez 11y ago4. Require that papers report all possible correlations in order to make p-hacking impossible, to allow "stacking" the results of multiple studies and to make negative results more interesting. 5. Don't limit review to peer review, but involve people from different fields. What can be a well known source of bias in one field may be overlooked in another.
- iamsohungry 11y agoI don't entirely understand why the pressure to publish at all costs is the issue here. The article says: > This particular plot is a statistical smoking gun, and suggests that the positive results from the original studies (black dots) were probably the result of p-hacking. They were chance findings, selectively published because they were positive. From that, it seems to me like the problem isn't what the researchers published, it's what they didn't publish. I'm a programmer, not a researcher, but in college chemistry we had marble notebooks that we wrote everything down in. The notebooks made it obvious if a page was torn out, everything had to be written in pen, and if we crossed something out, we had to cross it out in such a way that it was still legible. If we didn't include data in our results, we were expected to explain why. The most valuable thing I learned in those classes was the importance and practice of auditability. I doubt that my classes were unique in this respect and I wonder if this might be the solution to the problem. I've long been a proponent of people simply publishing literally all their data now that internet hosting is so cheap, and only publishing syntheses of the data in journals, giving anyone the opportunity to verify the author's conclusions. Psychology is different due to the need for confidentiality for the people being studied, but the full data should be available to at least the peer reviewers before a study can be published. EDIT: To add to this, elsewhere on this thread there's a lot of discussion about people not publishing negative results because they're not interesting. But reality doesn't have a bias towards results that are interesting, only humans do. I don't know how to solve this problem, but at least making publication of your non-interesting results a condition of publishing your interesting results would be a start.
- danieltillett 11y agoNot publishing negative results is a symptom, not a cause, of the current extreme competition in science. Fix the competition issue and we will fix problems like the non-publication of negative results.
- iamsohungry 11y agoI'm not sure I see the causal relationship here. Why does increased competition mean that people are not publishing? I think the problem is that the metrics which academics are competing to maximize are poor. If your metric is "How much interesting science has this person published?" you're going to get very different results from "How much accurate science has this person published?" Reducing competition doesn't change which question is being asked, it just reduces the importance of any metric at all. It seems likely to me that removing competition would give no incentive to publish anything measurable at all; after all, why publish if your career isn't connected to it? Obviously some would publish for intrinsic reasons, and it's possible that stress levels due to competition are leading to shoddy work, but none of this amounts to an argument that less competition would lead to people publishing negative results.