24 ms·

As a working scientist, I feel both sides of the problem every day. Most papers I come across turn out to be difficult if not impossible to reproduce, and I'm s

As a working scientist, I feel both sides of the problem every day. Most papers I come across turn out to be difficult if not impossible to reproduce, and I'm sure some of my own papers have fallen into that group at some points (although given that I'm a theorist, what "reproducible" means can be a bit fuzzy sometimes). At the same time, I'm confused every time I see people wondering about the scale of the problem or what to do about it. There is absolutely no mystery here whatsoever. Scientists are generally fairly smart people. Put smart people in a hypercompetitive environment and they will quickly identify the traits that are being rewarded and optimize for those. What does academia reward? Large numbers of publications with lots of citations, nothing else matters. So, naturally, scientists flock to hip buzzword-filled topics and churn out papers as quickly as they can. Not only is there absolutely no reward for easy reproducibility, it can actually harm you if someone manages to copy you quickly enough and then beat you to the next incremental publishable discovery in the neverending race. This is absurd and counterproductive to science in general, but such is the state of the academic job market. Anyone who purports to somehow address the situation without substantially changing the fundamental predicates for hiring and funding in the sciences is just wasting their breath. As long as the incentives are what they are, reproducibility will be a token most scientists pay tribute to in name only.

haecceity 7y ago
How to fix?
- aalleavitch 7y ago
  Science shouldn't be funded this way
- keldaris 7y ago
  The answer is simple, but it's also one that's completely useless for anyone reading this post. What needs to happen is a gradual realignment in both the funding and hiring criteria across all of academia. There needs to be less emphasis on quantitative metrics like citation counts, h-indices and other trivially gameable nonsense, and more emphasis on human judgment along with a set of basic criteria to provide a floor to research quality across the board (things that every good scientist should do, as opposed to things that you should relentlessly maximize). This would require the non-scientific managerial class to abrogate some of the power they now wield in the academia and relevant funding institutions, therefore is very unlikely to happen. And - probably, most annoyingly to the HN crowd - there is nothing meaningful that technological or business disruption can accomplish here. These are systemic social problems that have been continuously and systemically building up since (approximately) WW2. At this point the only entities large enough to make a difference are probably the NIH and NSF in the U.S. and the relevant EU funding agencies in Europe. In my judgment, these are precisely the agencies least likely to institute such change, so here we are.
  - nradov 7y ago
    If we put more weight on human judgment then we'll get more favoritism, nepotism, and focus on politically popular topics. So we might end up just exchanging one set of problems for another.
    - inimino 7y ago
      At least favoritism has some variability in it. The problem with metrics, besides being gamed, is that they are so uniform.
    - keldaris 7y ago
      You won't get much more focus on politically (in the context of the academia) popular topics, the existing system already imposes that particular bias exceedingly well (how do you think you get citations?). But yes, you'll get more everyday human issues along with the judgment. In my view, that's far preferable to the existing situation, if only for the simple reason that the human problems will vary from place to place, whereas the existing system distorts virtually the entirety of academic research in roughly the same ways everywhere, to the uniform detriment of all.
    - TeMPOraL 7y ago
      I used to be more worried about favoritism, nepotism et al. but these days, I'm more worried about quantifiable metrics. The former are situational, and there's only so much nepotism you can engage with before your peers in other places start considering it to be in bad taste. The latter, however, can be ruthlessly and efficiently optimized, to the full exclusion of any value that's not captured in the metrics.
  - cgiles 7y ago
    At my institute, there is only one metric that matters: dollars you bring in through grants. There are a variety of ways to get to the cash: you can do it with a small number of high-impact papers, a larger number of low-impact papers, by finding a valuable niche, or by chasing fads. So I would say that fundamentally the problem is with study sections and how they think. They are incredibly conservative, but in all the wrong ways. They highly value institution of origin, preliminary data from pilots, no matter how shitty, and the sort of hypothesis that seems reasonable from prior literature. There are also a lot of political games on study sections. This means that if you repeat a lie often enough in the literature, and you know people, then voila, you now have support for the same hypothesis in future grants. Peer review is not working for funding. I would say the NIH (in my case) needs to hire completely independent full-time reviewers, and needs to place a real, high emphasis on reproducibility. It is in the NIH's interest to do so, because as I said, its funding will be cut if this continues.
- geoalchimista 7y ago
  Not the OP, but as a working scientist, it seems to me there are two important issues, among many others. (i) Reproducibility is not a criterion for tenure evaluation, therefore it bears little relevance for career advancement for young scientists. This is an overworked and underpaid group in academia. (In the bay area, they earn 1/3--1/2 the salary of an entry level software engineer but probably work 50% more hours.) They simply don't have the luxury to take time off to reproduce someone else's work. (ii) No major funding agency would be willing to support the kind of work that reproduces published studies. When something is published, it is considered "done" and not worth spending more money on it. There needs to be a sea change. But sadly, the academia is at best paying lip services to the structural problems in the "reproducibility crisis". I'd expect it to continue like this for most fields.
- inimino 7y ago
  Making a significant contribution to your field used to be a reasonable requirement for a Ph.D. However, we are churning out more and more Ph.Ds, and there just aren't that many new discoveries to be made. This is creating a pressure to create more and more splintered fields, and more and more useless publications that nobody will ever read. Furthermore the academic system uses publication to measure the value of professors or researchers, creating yet more useless crap published every year. Only changing the incentives can improve the situation and it seems academia left to its own devices never will. Maybe it's time for governments to take a more active role in basic research again. Maybe we need more "patent clerk" jobs where people have time to think without having to chase structured research grants.
- vanniv 7y ago
  That's the hundred-billion-dollar question, isn't it. My radical proposal is to separate the research institutions from the universities -- especially the public universities. If you want public research institutes, that's fine -- but they should be their own entities. The university system, with its mix of research and instruction, and its tenure-vs-non-tenure system creates terrible incentives. Top researchers are rarely also the best instructors -- and so it makes little sense for the two occupations to be so intimately tied. The university funding model is terribly destructive to top science labs as well -- the university sees them as a source of prestige, sure, but more importantly as a source of that all-too-critical grant money (entirely too much of which is bogarted by the rest of the university) -- putting many of the labs in this place where they have to keep securing grants at an ever-increasing pace no matter what, to keep the rest of the university flush with cash to spend elsewhere. There's really no good reason why research needs to be done at a university, or why a research institution needs a body of lecturers and students.
  - CrazyStat 7y ago
    I absolutely agree. My wife did a PhD at an elite private university, undergrad tuition north of $50k/year not including room and board and other expenses. At one point three senior professors in her department went to the department chair and the dean with a proposal to revamp a notoriously unpopular intro class in their department. Rather than having it taught by adjuncts or postdocs, the three of them would team teach it. They would change the syllabus to bring it up to date with the current methods in the field--it had hardly been touched in two decades. They were told to forget about it. The University said it was a waste of time for them to teach undergrads. Presumably the kids applying to these types of universities are not aware of how little concern the institution has for the quality of their education (the perceived quality, on the other hand, is extremely important--but only loosely connected to the actual quality).
    - vanniv 7y ago
      The current undergraduate university market is almost entirely a reputational marketplace -- people are shopping for reputation/prestige. It turns out that prestige has very little to do with undergraduate education quality. This used to make lots of sense -- for centuries, college was more important as a place to meet other people as it was a place to actually learn useful skills -- and so you were primarily trying to go to the highest-class place you could get yourself in to. In this day and age, though, I don't think that's the right way to approach choosing a school.
- BenoitEssiambre 7y ago
  Until university positions and research grants stop being given out based on prior research _results_ (which is what journals tend to look for), we won't be able to trust the research performed there. There are millions of dollars on the line for researchers involved. It is the difference between a well paid career and a life of destitution while being a slave to huge student debt. Professorship and research grants should be given out based on criteria that are only incidental to research results. Evaluate profs and grants based on: 1. Domain knowledge (test the applicants) 2. Math skills (test the applicants, makes sure they know about how to avoid p-hacking, preregistration etc.) 3. Motivation and leadership. 4. Prior and current research _proposals_ (but without looking at the results or whether they have been published). 5. Other skills such as communication, interpersonal skills, outstanding achievements etc. Universities should not rely on journals to evaluate their professors. This corrupts the whole system. Journals have different goals. They want to publish well done research with interesting results. Universities should hire researchers that do good quality research with interesting _questions_ regardless of the results. If universities keep giving out jobs based on having generated interesting publishable results, they are going to keep getting researchers that ignore biases and and ignore bad science practices in order to generate publishable (but unreliable and often false) results.
- JMTQp8lwXL 7y ago
  Historically, the sciences were limited to people that had the wealth to independently research and verify their work. These days, people perform scientific research for a living wage (which in itself, I am not saying is a bad thing --scientific progress is a good thing-- no doubt), but people have to optimize for income producing potential, which means pumping out papers, even if they are of dubious quality, which may or may not intentionally be done. Not all scientists are optimizing for number of publications, but a good number are.
- scarejunba 7y ago
  It's already fixed to some degree. Any good scientist who can capture economic value spin-off to private labs. It's the good scientist who produces economic value but cannot capture it that we have to support but maybe that person cannot easily be found. The state just has to overfund and accept that some large percent goes to nonsense. In my mind, that's okay, though, since the guys who can capture economic value can do a lot with what we have.
- knuteson 7y ago
  Reward being right, penalize being wrong, and transfer information in a manner that facilitates determining what is right and what is wrong. One page (https://www.kn-x.com/static/PWFeb17forum.pdf https://www.kn-x.com/static/PWFeb17forum.pdf), two page (https://ssrn.com/abstract=2835131 https://ssrn.com/abstract=2835131), and three page (https://ssrn.com/abstract=2713998 https://ssrn.com/abstract=2713998) versions of this are available. Further details are provided at http://kn-x.com http://kn-x.com. I doubt anyone on this thread is going to like this solution ... but I think most on this thread agree that the underlying problem is the incentives, that fixing the problem requires changing the incentives, and that changing the incentives within the existing system is very hard.
eecc 7y ago
Ah the Bubka trick, thanks for pointing it out :) https://en.wikipedia.org/wiki/Sergey_Bubka https://en.wikipedia.org/wiki/Sergey_Bubka
cgiles 7y ago
Biology postdoc here, seconded. But what really gets me is the disconnect between "most scientists agree there is a reproducibility crisis" and "most scientists believe most of the papers they read are basically true and reproducible". This was mentioned in the survey and it conforms to my informal experience of attitudes. I do not know how you square that circle. Maybe, because of the pressures you mention, we are all supposed to engage in an informal convention of pretending to believe most previously published work is true, was done correctly, and is reproducible even if we know damn well how unlikely that is. I find it hard to do. One day the public is going to cotton on to all of this. I cringe every time I hear extremely authoritative lectures on "what scientists say" about highly politicized public policy matters. These are not my fields, but if they are as prone to error, bias, irreproducibility, etc, as my own, I'd exercise some humility. It is one thing for we scientists to lie -- errr, project unwarranted confidence -- to each other for the sake of getting grants, but it is quite another to do it directly to the public. But when the public does figure it out, what do you think will happen to funding? It will get tighter, and make the problem even worse. We need reform from within before the hammer falls, and quickly.
- itcrowd 7y ago
  I agree with the last two paragraphs of your post viz. humility and reform. > the disconnect between "most scientists agree there is a reproducibility crisis" and "most scientists believe most of the papers they read are basically true and reproducible" Could it be that many scientists think: "there is a crisis, but not in my field!"? To be honest, I personally think this quite often (and then realize I am being naive).
- keldaris 7y ago
  I don't know that we can reform - if you spend too much time trying, the system will automatically cull you. Personally, I'm lucky enough to be at least slightly insulated from the true scale of the problem by working in theoretical/computational soft matter physics, where the costs/grants/impact factors are comparatively on the small end of things. Medicine and biology seem to be affected the most, for good reason - this is where the messiest problems intersect with the most interest from society at large, so you get the most acutely misaligned incentives. That being said, communicating limits of applicability and degrees of certainty to a popular audience is hellishly difficult even if you're trying to be perfectly honest. Even in a hypothetical world where we've somehow fixed academia, this will always be a hard problem most scientists are ill equipped to tackle.
vanniv 7y ago
Outside of academia, the patent situation looks much like this. In theory, patents have two parts -- the claims (what specific attributes of your invention are getting patent protection) and the teaching (the bulk of the text/diagrams, which are supposed to teach one of "ordinary skill in the art" what would be needed to duplicate the invention). Turns out that you want claims as broad as possible and teachings as useless as possible -- so that nobody can read your patent and then patent other surrounding things. It's gotten so bad that most companies now instruct their employees never to read any patents -- the potential liability increase because they "knew" that something potentially-patented was out there so fantastically dwarfs what you could learn by reading the "teachings" It's all about incentives, as you say. And science today has more or less just as perverse incentives as the IP marketplace.
buboard 7y ago
This situation is also absolutely devastating for modeling science. Pretty much any model can be "based on empirical studies" since just about anything has been found to be statistically "significant". Unconstrained models are useless, as are unconstrained theories. It's also not enough to say "this study is not reproducible". Why isn't it? Towards what direction should the experimenters move next. It's not enough for biology to do "trial and error" studies , but they should be continued and provide more depth to their findings. Sadly , this is not happening largely because of lack of faith: the scientists themselves don't have enough faith in their own results to be making long bets on them.
wallace_f 7y ago
>smart people in a hypercompetitive environment and they will quickly identify the traits that are being rewarded and optimize for those. What does academia reward? Large numbers of publications with lots of citations, nothing else matters. So, naturally, scientists flock to hip buzzword-filled topics and churn out papers as quickly as they can. For some other opinions, consider: 1) Paul Graham who states this is the biggesg lesson to unlearn in academia. He states academia selects for people who tend to 'hack,' not learn, to the easiest grades/favors: https://news.ycombinator.com/item?id=21729619 https://news.ycombinator.com/item?id=21729619 2) Feynman who often stated things like "the pleasure is in finding things out" and that "the whole academic department was rotten." I'm not convinced academia is selecting primarily the attribute: intelligence.
Waterluvian 7y ago
Thanks for sharing, but if I might nitpick for a moment: > Put smart people in a hypercompetitive environment and they will quickly identify the traits that are being rewarded and optimize for those. This is true for anybody, not just "smart" people. It's basically human nature.
chiefalchemist 7y ago
> Scientists are generally fairly smart people. Put smart people in a hypercompetitive environment and they will quickly identify the traits that are being rewarded and optimize for those. What does academia reward? Large numbers of publications with lots of citations, nothing else matters. So, naturally, scientists flock to hip buzzword-filled topics and churn out papers as quickly as they can. Until the fundamentals change - which might never happen - it would help if the science-centric communities showed some humility, as well as some transparency. Even with proper incentives, the process is flaw. It's human-based so it will always be flawed. There's nothing wrong with that. It's the best we got. But perfection it is not. It troubles me when I see those who question science shut down and marginalizes. As if science has some perfect track record. Per the OP, even science has question about science. Fair enough. But does it have to be wrapped in denial?
naveen99 7y ago
Progress has to be incremental because reality is local. It’s the speed of progress (energy of the system) that is important. If you are too slow, people will pass you. If you are too fast, your competitors will catch up faster. That’s why the champions and hustlers alike don’t show all their tricks unless absolutely necessary to their challengers. Even industry / monopolies do incremental releases if they are ahead of their competitors... but if intel or nvidia slow down too much with their incremental progress amd can sneak in.
agumonkey 7y ago
is short termish smart smart at all?
wisty 7y ago
> Put smart people in a hypercompetitive environment and they will quickly identify the traits that are being rewarded and optimize for those. Or you select for hypercompetitive people who don't mind bending the rules. Smart people also have options outside of college.
ericd 7y ago
It’s kind of weird, though, I’d imagine that most of them didn’t get into academia to try to spend their time gaming a system, and they’re giving up quite a lot in material terms to be in academia. So why do it? Are they mostly in it for the prestige rather than the ability to actually move the state of the world forward?
- lrem 7y ago
  Ideals clashing with reality. Some, like yours truly, give up and go to the industry. Some try to make it work, game the system a bit, do some good. Then they get kids, get older, generally run out of energy to do two things in one job. And guess which one can be sacrificed without upending your life.
75dvtwin 7y ago
I have seen your point of view, shared over and over again. Lack of intensives for reproducible research (and for verification of research) is the fundamental problem. If you are in the shoes of policy makers (eg politicians). What changes would you propose (through both laws and funding/grants models)?
- lrem 7y ago
  "Multiply the impact of empirical findings by (#positive reproductions - #negative reproductions)" sounds nice, probably bound it to some interval like [-4, +3]. But this means your measurement pipeline needs to understand more of the paper than just the bibliography section.
- keldaris 7y ago
  I briefly addressed this here: https://news.ycombinator.com/item?id=21964840 https://news.ycombinator.com/item?id=21964840
optimiz3 7y ago
Maybe we should be asking the question: "Why are academics citing papers that can't be reproduced?" A lack of reproducibility of cited papers IMO undermines the credibility of citing papers.
- xamuel 7y ago
  >undermines the credibility of citing papers That credibility is already undermined by the fact that for citation-measuring purposes, there's no real difference between any of the following citations: (1) "Introduction. This paper is a journey into the amazing consequences made possible by (Smith et al, 2019)" (2) "Introduction. In this paper we show that (Smith et al, 2019) is a steaming pile of crap" (3) "Footnote. This minor remark is vaguely reminiscent of (Smith et al, 2019)" All contribute the same citation juice to Smith et al, 2019.
kerkeslager 7y ago
Well, there are two simple steps to change those incentives: 1. Journals should accept hypotheses/procedures, and commit to publish whatever conclusion results, before the experiment is even started. If the experiment is not completed for whatever reason, a transparency document should be published explaining why. 2. Journals should accept only a small percentage[1] of new research. The rest should be attempts to reproduce old research, again only evaluating the hypotheses/procedures, before the attempt to reproduce has started. The challenge here is that there's a bit of a chicken and egg problem: journals won't want to commit to publish a result if no one has committed to fund the experiment, but funding sources won't want to fund experiments if the no one has committed to publish the result. So there would need to be some collaboration between [1] Choosing this percentage is the proper usage of P values. The goal is to attempt to reproduce experiments until the product of the P-values of the results reaches a target aggregate P. Note that this target applies to P and P'. Example 1: Your target P is P=0.01. You perform an experiment, get a P=0.3 result. Then you attempt to reproduce, and get a P=0.4 result, for an aggregate P of 0.12. You then attempt to reproduce again, and get a P=0.2 result, for an aggregate P of 0.024. Finally, you attempt to reproduce again, and get a P=0.4 result, for an aggregate P of 0.0096, below your target P. This proves the alternate hypothesis with the target confidence. Example 2: Your target P is P=0.01. You perform an experiment, get a P=0.7 result (P'=0.3). You then attempt to reproduce, with a P=0.9 result, for an aggregate P' of P'=(0.3 x 0.1)=0.03. You then attempt to reproduce, with a P=0.9 result, for an aggregate P' of P'=(0.03 x 0.1)=0.003, below your target P. This proves the null hypothesis with the target confidence. The example P values were chosen for a few reasons. First, it demonstrates that you can find fairly conclusive confidence values eventually from aggregating experiments with fairly inconclusive results. Second, it demonstrates that the P=0.05 that's frequently used now is actually a very low bar of confidence, when you consider that reproducing even very unsurprising results a few times gives you a much higher confidence.
amelius 7y ago
> Scientists are generally fairly smart people. Put smart people in a hypercompetitive environment and they will quickly identify the traits that are being rewarded and optimize for those. I don't understand this. Scientists could make a lot more money in industry, so this makes me wonder why these "corner-cutting" scientists are doing the work they are doing. Surely not for idealistic reasons.
- funklute 7y ago
  > Scientists could make a lot more money in industry That may be true, as well as accurate, for a field like machine learning. But for most of science, this is a highly misleading claim. Some science (e.g. drug development, materials science) can be done also in industry, although with a lot less freedom than in academia. But most science simply doesn't have an equivalent option in industry. Scientists are doing the work they are doing largely because they are interested researching specific questions, which can not be pursued anywhere else.
- keldaris 7y ago
  Actually, it is mostly for idealistic reasons. Scientists compartmentalize and often don't view it as corner cutting with respect to the science. I'll try to explain. Scientists generally like doing science. Nowadays, the higher you get in science the more you get buried under things that are not science - endless project proposals, reports, reviews, applications and yes, publications. Most scientists cut corners as much as they can get away with on most of these because if you don't, you will never get to actually doing science (others just work 100 hour weeks, but you can't really sustain that). Now unlike all of the bureaucratic garbage publications are supposed to be about communicating your work to your peers, and integral part of science, but in the existing system it's really hard to actually retain that view. Why? Because publications are now intrinsically linked to all the metrics - you need X publications for project Y, containing the right set of buzzwords, submitted to journals that satisfy the right quantitative metrics and are listed in the right databases, with the right set of coauthors, references and acknowledgments and you need to submit them before the right deadline, or else. Spend a decade or two in that environment and it's easy to lump writing publications along with the endless proposals and reports - as mindless drudgery you have to dispense with in order to squeeze out some time for actual science. Of course, you also need to devote enough attention to all the things I listed and much more in order to actually retain your ability to do science and not starve for it, so it's an eternal balancing act that breeds a lot of resentment.
air7 7y ago
> Most papers I come across turn out to be difficult if not impossible to reproduce How is this even possible? Isn't Reproducibility one the pillars of Science? What gives such a paper credibility over say, an eyewitness description of a UFO sighting? It's one thing to not perform the actual reproduction experiments, but publishing claims that (almost) can't be reproduced is another.
yahwrong 7y ago
i.e. take the money out of it then. Pay researchers the same regardless of number of publications. Cut %30, at least, of the administration of a major research University and use that money to fund research, raise professor pay, and lower undergrad tuition. Then put a block on admin raises unless researchers and professors get an equal % bump. Then elect better politicians that actually understand at least a little bit of science and how that correlates to bettering society so they're get bills past to fund higher education and research. We've got the money for it, but it's all in bombs at the moment.

haecceity 7y ago

How to fix?

aalleavitch 7y ago
Science shouldn't be funded this way
keldaris 7y ago
The answer is simple, but it's also one that's completely useless for anyone reading this post. What needs to happen is a gradual realignment in both the funding and hiring criteria across all of academia. There needs to be less emphasis on quantitative metrics like citation counts, h-indices and other trivially gameable nonsense, and more emphasis on human judgment along with a set of basic criteria to provide a floor to research quality across the board (things that every good scientist should do, as opposed to things that you should relentlessly maximize). This would require the non-scientific managerial class to abrogate some of the power they now wield in the academia and relevant funding institutions, therefore is very unlikely to happen. And - probably, most annoyingly to the HN crowd - there is nothing meaningful that technological or business disruption can accomplish here. These are systemic social problems that have been continuously and systemically building up since (approximately) WW2. At this point the only entities large enough to make a difference are probably the NIH and NSF in the U.S. and the relevant EU funding agencies in Europe. In my judgment, these are precisely the agencies least likely to institute such change, so here we are.
- nradov 7y ago
  If we put more weight on human judgment then we'll get more favoritism, nepotism, and focus on politically popular topics. So we might end up just exchanging one set of problems for another.
  - inimino 7y ago
    At least favoritism has some variability in it. The problem with metrics, besides being gamed, is that they are so uniform.
  - keldaris 7y ago
    You won't get much more focus on politically (in the context of the academia) popular topics, the existing system already imposes that particular bias exceedingly well (how do you think you get citations?). But yes, you'll get more everyday human issues along with the judgment. In my view, that's far preferable to the existing situation, if only for the simple reason that the human problems will vary from place to place, whereas the existing system distorts virtually the entirety of academic research in roughly the same ways everywhere, to the uniform detriment of all.
  - TeMPOraL 7y ago
    I used to be more worried about favoritism, nepotism et al. but these days, I'm more worried about quantifiable metrics. The former are situational, and there's only so much nepotism you can engage with before your peers in other places start considering it to be in bad taste. The latter, however, can be ruthlessly and efficiently optimized, to the full exclusion of any value that's not captured in the metrics.
- cgiles 7y ago
  At my institute, there is only one metric that matters: dollars you bring in through grants. There are a variety of ways to get to the cash: you can do it with a small number of high-impact papers, a larger number of low-impact papers, by finding a valuable niche, or by chasing fads. So I would say that fundamentally the problem is with study sections and how they think. They are incredibly conservative, but in all the wrong ways. They highly value institution of origin, preliminary data from pilots, no matter how shitty, and the sort of hypothesis that seems reasonable from prior literature. There are also a lot of political games on study sections. This means that if you repeat a lie often enough in the literature, and you know people, then voila, you now have support for the same hypothesis in future grants. Peer review is not working for funding. I would say the NIH (in my case) needs to hire completely independent full-time reviewers, and needs to place a real, high emphasis on reproducibility. It is in the NIH's interest to do so, because as I said, its funding will be cut if this continues.
geoalchimista 7y ago
Not the OP, but as a working scientist, it seems to me there are two important issues, among many others. (i) Reproducibility is not a criterion for tenure evaluation, therefore it bears little relevance for career advancement for young scientists. This is an overworked and underpaid group in academia. (In the bay area, they earn 1/3--1/2 the salary of an entry level software engineer but probably work 50% more hours.) They simply don't have the luxury to take time off to reproduce someone else's work. (ii) No major funding agency would be willing to support the kind of work that reproduces published studies. When something is published, it is considered "done" and not worth spending more money on it. There needs to be a sea change. But sadly, the academia is at best paying lip services to the structural problems in the "reproducibility crisis". I'd expect it to continue like this for most fields.
inimino 7y ago
Making a significant contribution to your field used to be a reasonable requirement for a Ph.D. However, we are churning out more and more Ph.Ds, and there just aren't that many new discoveries to be made. This is creating a pressure to create more and more splintered fields, and more and more useless publications that nobody will ever read. Furthermore the academic system uses publication to measure the value of professors or researchers, creating yet more useless crap published every year. Only changing the incentives can improve the situation and it seems academia left to its own devices never will. Maybe it's time for governments to take a more active role in basic research again. Maybe we need more "patent clerk" jobs where people have time to think without having to chase structured research grants.
vanniv 7y ago
That's the hundred-billion-dollar question, isn't it. My radical proposal is to separate the research institutions from the universities -- especially the public universities. If you want public research institutes, that's fine -- but they should be their own entities. The university system, with its mix of research and instruction, and its tenure-vs-non-tenure system creates terrible incentives. Top researchers are rarely also the best instructors -- and so it makes little sense for the two occupations to be so intimately tied. The university funding model is terribly destructive to top science labs as well -- the university sees them as a source of prestige, sure, but more importantly as a source of that all-too-critical grant money (entirely too much of which is bogarted by the rest of the university) -- putting many of the labs in this place where they have to keep securing grants at an ever-increasing pace no matter what, to keep the rest of the university flush with cash to spend elsewhere. There's really no good reason why research needs to be done at a university, or why a research institution needs a body of lecturers and students.
- CrazyStat 7y ago
  I absolutely agree. My wife did a PhD at an elite private university, undergrad tuition north of $50k/year not including room and board and other expenses. At one point three senior professors in her department went to the department chair and the dean with a proposal to revamp a notoriously unpopular intro class in their department. Rather than having it taught by adjuncts or postdocs, the three of them would team teach it. They would change the syllabus to bring it up to date with the current methods in the field--it had hardly been touched in two decades. They were told to forget about it. The University said it was a waste of time for them to teach undergrads. Presumably the kids applying to these types of universities are not aware of how little concern the institution has for the quality of their education (the perceived quality, on the other hand, is extremely important--but only loosely connected to the actual quality).
  - vanniv 7y ago
    The current undergraduate university market is almost entirely a reputational marketplace -- people are shopping for reputation/prestige. It turns out that prestige has very little to do with undergraduate education quality. This used to make lots of sense -- for centuries, college was more important as a place to meet other people as it was a place to actually learn useful skills -- and so you were primarily trying to go to the highest-class place you could get yourself in to. In this day and age, though, I don't think that's the right way to approach choosing a school.
BenoitEssiambre 7y ago
Until university positions and research grants stop being given out based on prior research _results_ (which is what journals tend to look for), we won't be able to trust the research performed there. There are millions of dollars on the line for researchers involved. It is the difference between a well paid career and a life of destitution while being a slave to huge student debt. Professorship and research grants should be given out based on criteria that are only incidental to research results. Evaluate profs and grants based on: 1. Domain knowledge (test the applicants) 2. Math skills (test the applicants, makes sure they know about how to avoid p-hacking, preregistration etc.) 3. Motivation and leadership. 4. Prior and current research _proposals_ (but without looking at the results or whether they have been published). 5. Other skills such as communication, interpersonal skills, outstanding achievements etc. Universities should not rely on journals to evaluate their professors. This corrupts the whole system. Journals have different goals. They want to publish well done research with interesting results. Universities should hire researchers that do good quality research with interesting _questions_ regardless of the results. If universities keep giving out jobs based on having generated interesting publishable results, they are going to keep getting researchers that ignore biases and and ignore bad science practices in order to generate publishable (but unreliable and often false) results.
JMTQp8lwXL 7y ago
Historically, the sciences were limited to people that had the wealth to independently research and verify their work. These days, people perform scientific research for a living wage (which in itself, I am not saying is a bad thing --scientific progress is a good thing-- no doubt), but people have to optimize for income producing potential, which means pumping out papers, even if they are of dubious quality, which may or may not intentionally be done. Not all scientists are optimizing for number of publications, but a good number are.
scarejunba 7y ago
It's already fixed to some degree. Any good scientist who can capture economic value spin-off to private labs. It's the good scientist who produces economic value but cannot capture it that we have to support but maybe that person cannot easily be found. The state just has to overfund and accept that some large percent goes to nonsense. In my mind, that's okay, though, since the guys who can capture economic value can do a lot with what we have.
knuteson 7y ago
Reward being right, penalize being wrong, and transfer information in a manner that facilitates determining what is right and what is wrong. One page (https://www.kn-x.com/static/PWFeb17forum.pdf https://www.kn-x.com/static/PWFeb17forum.pdf), two page (https://ssrn.com/abstract=2835131 https://ssrn.com/abstract=2835131), and three page (https://ssrn.com/abstract=2713998 https://ssrn.com/abstract=2713998) versions of this are available. Further details are provided at http://kn-x.com http://kn-x.com. I doubt anyone on this thread is going to like this solution ... but I think most on this thread agree that the underlying problem is the incentives, that fixing the problem requires changing the incentives, and that changing the incentives within the existing system is very hard.

eecc 7y ago

Ah the Bubka trick, thanks for pointing it out :) https://en.wikipedia.org/wiki/Sergey_Bubka https://en.wikipedia.org/wiki/Sergey_Bubka

cgiles 7y ago

Biology postdoc here, seconded. But what really gets me is the disconnect between "most scientists agree there is a reproducibility crisis" and "most scientists believe most of the papers they read are basically true and reproducible". This was mentioned in the survey and it conforms to my informal experience of attitudes. I do not know how you square that circle. Maybe, because of the pressures you mention, we are all supposed to engage in an informal convention of pretending to believe most previously published work is true, was done correctly, and is reproducible even if we know damn well how unlikely that is. I find it hard to do. One day the public is going to cotton on to all of this. I cringe every time I hear extremely authoritative lectures on "what scientists say" about highly politicized public policy matters. These are not my fields, but if they are as prone to error, bias, irreproducibility, etc, as my own, I'd exercise some humility. It is one thing for we scientists to lie -- errr, project unwarranted confidence -- to each other for the sake of getting grants, but it is quite another to do it directly to the public. But when the public does figure it out, what do you think will happen to funding? It will get tighter, and make the problem even worse. We need reform from within before the hammer falls, and quickly.

itcrowd 7y ago
I agree with the last two paragraphs of your post viz. humility and reform. > the disconnect between "most scientists agree there is a reproducibility crisis" and "most scientists believe most of the papers they read are basically true and reproducible" Could it be that many scientists think: "there is a crisis, but not in my field!"? To be honest, I personally think this quite often (and then realize I am being naive).
keldaris 7y ago
I don't know that we can reform - if you spend too much time trying, the system will automatically cull you. Personally, I'm lucky enough to be at least slightly insulated from the true scale of the problem by working in theoretical/computational soft matter physics, where the costs/grants/impact factors are comparatively on the small end of things. Medicine and biology seem to be affected the most, for good reason - this is where the messiest problems intersect with the most interest from society at large, so you get the most acutely misaligned incentives. That being said, communicating limits of applicability and degrees of certainty to a popular audience is hellishly difficult even if you're trying to be perfectly honest. Even in a hypothetical world where we've somehow fixed academia, this will always be a hard problem most scientists are ill equipped to tackle.

vanniv 7y ago

Outside of academia, the patent situation looks much like this. In theory, patents have two parts -- the claims (what specific attributes of your invention are getting patent protection) and the teaching (the bulk of the text/diagrams, which are supposed to teach one of "ordinary skill in the art" what would be needed to duplicate the invention). Turns out that you want claims as broad as possible and teachings as useless as possible -- so that nobody can read your patent and then patent other surrounding things. It's gotten so bad that most companies now instruct their employees never to read any patents -- the potential liability increase because they "knew" that something potentially-patented was out there so fantastically dwarfs what you could learn by reading the "teachings" It's all about incentives, as you say. And science today has more or less just as perverse incentives as the IP marketplace.

buboard 7y ago

This situation is also absolutely devastating for modeling science. Pretty much any model can be "based on empirical studies" since just about anything has been found to be statistically "significant". Unconstrained models are useless, as are unconstrained theories. It's also not enough to say "this study is not reproducible". Why isn't it? Towards what direction should the experimenters move next. It's not enough for biology to do "trial and error" studies , but they should be continued and provide more depth to their findings. Sadly , this is not happening largely because of lack of faith: the scientists themselves don't have enough faith in their own results to be making long bets on them.

wallace_f 7y ago

>smart people in a hypercompetitive environment and they will quickly identify the traits that are being rewarded and optimize for those. What does academia reward? Large numbers of publications with lots of citations, nothing else matters. So, naturally, scientists flock to hip buzzword-filled topics and churn out papers as quickly as they can. For some other opinions, consider: 1) Paul Graham who states this is the biggesg lesson to unlearn in academia. He states academia selects for people who tend to 'hack,' not learn, to the easiest grades/favors: https://news.ycombinator.com/item?id=21729619 https://news.ycombinator.com/item?id=21729619 2) Feynman who often stated things like "the pleasure is in finding things out" and that "the whole academic department was rotten." I'm not convinced academia is selecting primarily the attribute: intelligence.

Waterluvian 7y ago

Thanks for sharing, but if I might nitpick for a moment: > Put smart people in a hypercompetitive environment and they will quickly identify the traits that are being rewarded and optimize for those. This is true for anybody, not just "smart" people. It's basically human nature.

chiefalchemist 7y ago

> Scientists are generally fairly smart people. Put smart people in a hypercompetitive environment and they will quickly identify the traits that are being rewarded and optimize for those. What does academia reward? Large numbers of publications with lots of citations, nothing else matters. So, naturally, scientists flock to hip buzzword-filled topics and churn out papers as quickly as they can. Until the fundamentals change - which might never happen - it would help if the science-centric communities showed some humility, as well as some transparency. Even with proper incentives, the process is flaw. It's human-based so it will always be flawed. There's nothing wrong with that. It's the best we got. But perfection it is not. It troubles me when I see those who question science shut down and marginalizes. As if science has some perfect track record. Per the OP, even science has question about science. Fair enough. But does it have to be wrapped in denial?

naveen99 7y ago

Progress has to be incremental because reality is local. It’s the speed of progress (energy of the system) that is important. If you are too slow, people will pass you. If you are too fast, your competitors will catch up faster. That’s why the champions and hustlers alike don’t show all their tricks unless absolutely necessary to their challengers. Even industry / monopolies do incremental releases if they are ahead of their competitors... but if intel or nvidia slow down too much with their incremental progress amd can sneak in.

agumonkey 7y ago

is short termish smart smart at all?

wisty 7y ago

> Put smart people in a hypercompetitive environment and they will quickly identify the traits that are being rewarded and optimize for those. Or you select for hypercompetitive people who don't mind bending the rules. Smart people also have options outside of college.

ericd 7y ago

It’s kind of weird, though, I’d imagine that most of them didn’t get into academia to try to spend their time gaming a system, and they’re giving up quite a lot in material terms to be in academia. So why do it? Are they mostly in it for the prestige rather than the ability to actually move the state of the world forward?

lrem 7y ago
Ideals clashing with reality. Some, like yours truly, give up and go to the industry. Some try to make it work, game the system a bit, do some good. Then they get kids, get older, generally run out of energy to do two things in one job. And guess which one can be sacrificed without upending your life.

75dvtwin 7y ago

I have seen your point of view, shared over and over again. Lack of intensives for reproducible research (and for verification of research) is the fundamental problem. If you are in the shoes of policy makers (eg politicians). What changes would you propose (through both laws and funding/grants models)?

lrem 7y ago
"Multiply the impact of empirical findings by (#positive reproductions - #negative reproductions)" sounds nice, probably bound it to some interval like [-4, +3]. But this means your measurement pipeline needs to understand more of the paper than just the bibliography section.
keldaris 7y ago
I briefly addressed this here: https://news.ycombinator.com/item?id=21964840 https://news.ycombinator.com/item?id=21964840

optimiz3 7y ago

Maybe we should be asking the question: "Why are academics citing papers that can't be reproduced?" A lack of reproducibility of cited papers IMO undermines the credibility of citing papers.

xamuel 7y ago
>undermines the credibility of citing papers That credibility is already undermined by the fact that for citation-measuring purposes, there's no real difference between any of the following citations: (1) "Introduction. This paper is a journey into the amazing consequences made possible by (Smith et al, 2019)" (2) "Introduction. In this paper we show that (Smith et al, 2019) is a steaming pile of crap" (3) "Footnote. This minor remark is vaguely reminiscent of (Smith et al, 2019)" All contribute the same citation juice to Smith et al, 2019.

kerkeslager 7y ago

Well, there are two simple steps to change those incentives: 1. Journals should accept hypotheses/procedures, and commit to publish whatever conclusion results, before the experiment is even started. If the experiment is not completed for whatever reason, a transparency document should be published explaining why. 2. Journals should accept only a small percentage[1] of new research. The rest should be attempts to reproduce old research, again only evaluating the hypotheses/procedures, before the attempt to reproduce has started. The challenge here is that there's a bit of a chicken and egg problem: journals won't want to commit to publish a result if no one has committed to fund the experiment, but funding sources won't want to fund experiments if the no one has committed to publish the result. So there would need to be some collaboration between [1] Choosing this percentage is the proper usage of P values. The goal is to attempt to reproduce experiments until the product of the P-values of the results reaches a target aggregate P. Note that this target applies to P and P'. Example 1: Your target P is P=0.01. You perform an experiment, get a P=0.3 result. Then you attempt to reproduce, and get a P=0.4 result, for an aggregate P of 0.12. You then attempt to reproduce again, and get a P=0.2 result, for an aggregate P of 0.024. Finally, you attempt to reproduce again, and get a P=0.4 result, for an aggregate P of 0.0096, below your target P. This proves the alternate hypothesis with the target confidence. Example 2: Your target P is P=0.01. You perform an experiment, get a P=0.7 result (P'=0.3). You then attempt to reproduce, with a P=0.9 result, for an aggregate P' of P'=(0.3 x 0.1)=0.03. You then attempt to reproduce, with a P=0.9 result, for an aggregate P' of P'=(0.03 x 0.1)=0.003, below your target P. This proves the null hypothesis with the target confidence. The example P values were chosen for a few reasons. First, it demonstrates that you can find fairly conclusive confidence values eventually from aggregating experiments with fairly inconclusive results. Second, it demonstrates that the P=0.05 that's frequently used now is actually a very low bar of confidence, when you consider that reproducing even very unsurprising results a few times gives you a much higher confidence.

amelius 7y ago

> Scientists are generally fairly smart people. Put smart people in a hypercompetitive environment and they will quickly identify the traits that are being rewarded and optimize for those. I don't understand this. Scientists could make a lot more money in industry, so this makes me wonder why these "corner-cutting" scientists are doing the work they are doing. Surely not for idealistic reasons.

funklute 7y ago
> Scientists could make a lot more money in industry That may be true, as well as accurate, for a field like machine learning. But for most of science, this is a highly misleading claim. Some science (e.g. drug development, materials science) can be done also in industry, although with a lot less freedom than in academia. But most science simply doesn't have an equivalent option in industry. Scientists are doing the work they are doing largely because they are interested researching specific questions, which can not be pursued anywhere else.
keldaris 7y ago
Actually, it is mostly for idealistic reasons. Scientists compartmentalize and often don't view it as corner cutting with respect to the science. I'll try to explain. Scientists generally like doing science. Nowadays, the higher you get in science the more you get buried under things that are not science - endless project proposals, reports, reviews, applications and yes, publications. Most scientists cut corners as much as they can get away with on most of these because if you don't, you will never get to actually doing science (others just work 100 hour weeks, but you can't really sustain that). Now unlike all of the bureaucratic garbage publications are supposed to be about communicating your work to your peers, and integral part of science, but in the existing system it's really hard to actually retain that view. Why? Because publications are now intrinsically linked to all the metrics - you need X publications for project Y, containing the right set of buzzwords, submitted to journals that satisfy the right quantitative metrics and are listed in the right databases, with the right set of coauthors, references and acknowledgments and you need to submit them before the right deadline, or else. Spend a decade or two in that environment and it's easy to lump writing publications along with the endless proposals and reports - as mindless drudgery you have to dispense with in order to squeeze out some time for actual science. Of course, you also need to devote enough attention to all the things I listed and much more in order to actually retain your ability to do science and not starve for it, so it's an eternal balancing act that breeds a lot of resentment.

air7 7y ago

> Most papers I come across turn out to be difficult if not impossible to reproduce How is this even possible? Isn't Reproducibility one the pillars of Science? What gives such a paper credibility over say, an eyewitness description of a UFO sighting? It's one thing to not perform the actual reproduction experiments, but publishing claims that (almost) can't be reproduced is another.

yahwrong 7y ago

i.e. take the money out of it then. Pay researchers the same regardless of number of publications. Cut %30, at least, of the administration of a major research University and use that money to fund research, raise professor pay, and lower undergrad tuition. Then put a block on admin raises unless researchers and professors get an equal % bump. Then elect better politicians that actually understand at least a little bit of science and how that correlates to bettering society so they're get bills past to fund higher education and research. We've got the money for it, but it's all in bombs at the moment.