12 ms·

Stagnation and Scientific Incentives

184 points by hhs 7y ago

awinter-py 7y ago
> This emphasis on citations in the measurement of scientific productivity shifted scientist rewards and behavior on the margin toward incremental science this sounds like the science equivalent of how search engines lead to the creation of content farms. citation scores are to spam science as pagerank is to wikihow? Imagine going through life with a high-tech spam filter from the future that can filter out wikihow and boring science. Impossible to build, I suspect, but that would be the life.
- peterhj 7y ago
  In fact, one of the motivating applications of PageRank was to bridge academic citation analysis to the web: http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf
- tryptophan 7y ago
  About the search aspect, I wonder if there is any way to create a custom search engine or filter out crap from google. I would love a browser addon that auto-filters out quora, wikihow, w3schools, techcrunch, etc... None of those garbage sites are what I'm looking for, ever. I think that, amusingly, filtering out the top 5% of best SEO optimized sites would make results much better.
  - awinter-py 7y ago
    w3schools loads faster than MDN but I'm mostly with you reevaluating our social attitudes towards 'trust in experts' will be an aftermath of the covid crisis, and it may bleed into search & social media as well -- is there value in 'peer-reviewed everything'? (for some definition of peer)
    - petra 7y ago
      What is the alternative for the "trust in experts" model, for most people ? Is there any ?
      - awinter-py 7y ago
        oh wait I didn't mean to say that we were trusting experts in the past and we won't in the future more that we'll ask a lot of questions about it. when are we trusting them, when should we, are governing bodies making proper space for experts I think most people in their personal lives don't have access to good expert advisors, maybe outside medicine & car mechanics, maybe not even there
      - deleted 7y ago
        [deleted]
      - op03 7y ago
        Maybe more multi-disciplined group based (given how much more complex most problems are for individual experts to grapple with by themselves). Probably rated like team sports.
      - TeMPOraL 7y ago
        Actually trust actual experts, instead of mindlessly listening to people with best marketing? This applies to both politics and w3schools.
    - evmar 7y ago
      devdocs.io is a dramatically faster alternative to MDN, with the same content.
  - arkitaip 7y ago
    You can already do that with a Chrome plugin, like Google Search Filter: https://chrome.google.com/webstore/detail/google-search-filter/eidhkmnbiahhgbgpjpiimdogfidfikgf https://chrome.google.com/webstore/detail/google-search-filt...
- dkarl 7y ago
  It's doubly tragic if it simultaneously incentivized the mass production of boring, low-value science and punished the boring but valuable work of replication.
  - awinter-py 7y ago
    yes -- this is kind of the 'deep work' argument the distracting stuff is ultimately low value we've cranked up the gain on spam -- spam is our culture's moai
- TeMPOraL 7y ago
  > this sounds like the science equivalent of how search engines lead to the creation of content farms. citation scores are to spam science as pagerank is to wikihow? Or a science equivalent to what money and market competition does with all human ventures. You need more and faster than your competitors to progress, else you'll fall into obscurity. "First to publish" is isomorphic to "first to market". If you treat research as a videogame, citations fit perfectly as an in-game currency. That is to say, it's an example of a general problem of optimizing short-term metrics, which are a decent proxy of the actual goal up to a point. Science could definitely use a spam filter. Journals unfortunately do a poor way of being one; they're thoroughly gamed, the same way Google is via SEO.
  - buitreVirtual 7y ago
    At least in the market a junk product doesn't survive despite being first to market. The product needs to be good or at least inspire others to be regarded as a real contribution. This additional requirement is what's missing from the incentive system in science today.
    - laughinghan 7y ago
      Tons of junk survives and thrives in the market. Your great grandparent specifically brought up content farms as an example. Homeopathic remedies, counterfeit SD cards, the list goes on. Nothing about markets magically solves the problem that if it's hard to evaluate quality, there will be junk masquerading as quality.
- grabbalacious 7y ago
  >Imagine going through life with a high-tech spam filter from the future that can filter out wikihow and boring science. Happily it's available now and it's called intuition. If something is boring then avoid; if something is exciting then pursue. Problems being that it's purely anecdotal and you have to know and trust yourself. If you're too attracted to prestige, money or job security then it's going to return a distorted signal. Which is why organised science is now bureaucratic and slow despite the fact that there are more scientists than ever before.
bognition 7y ago
There have been converastions for years about the slow down in scientific discovery. Often times the explanation is that science is getting more expensive, something i've often felt was a bit of a straw man argument. Instead pointing at the current incentive structure (that everyone already agrees is broken) makes a lot more sense. So lets start a conversation about a better incentive structure. What do we want to incentivize? How do we do that? How do we fix science?
- bjelkeman-again 7y ago
  What is broken with the current system? (I have an idea what is broken, but it would be good to see it explained.)
  - wegs 7y ago
    A lot of things, but this paper points to one issue: * In order to get an academic position, you need to have letters from a research community. * In order to get citations, you need your papers to be used by a research community. This gives a strong advantage to work done within existing lines of research. Virtually anything outside of one of the existing "academic cottage industries" or "mutual adoration societies" is at a huge disadvantage. You end up with research communities going down long rabbit holes with tunnel vision, and big broad areas of potential research are never explored.
- hef19898 7y ago
  Incentives, and the corresponding KPIs, are always a good place to start looking if one is trying to find out why systems and people behave the way they do. When I read the title I thought the paper was about business ideas and similar things. Having quickly read the paper, citations is a likely reason for, what the authors called, me too science. Maybe it's time to be more bold.
  - jjoonathan 7y ago
    Well, yeah, but bold is nearly impossible to tell from crazy, and people really don't like being told that "actually, the most efficient management technique in this case is to throw money randomly at a lightly-filtered set of projects." People want to believe that blue-sky research can be predicted, managed, and optimized, no matter the staggering mountains of evidence to the contrary.
DubiousPusher 7y ago
There is so much about papers like this that make me skeptical. 1) It assumes we've been good at measuring growth. Which to me is dubious. Our current system counts the production, deployment and detonation of a bomb all as positive production meanwhile not counting unpaid domestic work. I know economists don't like to out a value to different types of services and goods because they feel they are putting a finger on the scale. But they are doing just as much by using a blind metric which ends up counting some work and not other. 2) It never cost adjusts growth. For example, all that wonderful growth in the midst of the 20th cwntury incurred significant externalities. The system we have now tries much harder to make producers realize their externalities. This curbs growth which is not necessarily a bad thing. 3) It assumes we can atomize historic growth and single out what things contributes to what. But this seems ridiculous especially how these factors interact and are not necessarily seperable. For example, we happen to live in a universe in which the movement of electrons can be used transmit energy and appropriate it to many tasks. The story of the 20th cwntury's economy is very much the story of how we mastered this one property of nature. Electrification doesn't just provide heat and light. It enables the transportation of water. It enables the construction of more and larger structures. It enables the creation of aluminum and plated metals. Aluminum itself allows many features of our world we take for granted. From airplanes to electronics. But does abundant aluminum lay at the feet of scientific innovations? In its infancy for sure but after some basic science it is widespread cheap electrification which allows us to mass produce aluminum. How much of that economy do we count to cutting edge science? It's not easily disentangled from popular politics which allowed the mass construction of hydroelectric damns throughout the U.S. and other countries. There is a possible future in which for centuries, growth has slowed to something more than Renaissance levels but well below mid 20th century levels. In such a future we might look back at this period as the time we mastered the single most useful physical property of our universe and so of course it was an era of unprecedented growth. I'm not saying they're wrong. But every time someone from Bill Gates to NBER talks about this issue of growth, whether it has slowed, why it as slowed, how it can be increased, I just get this feeling that people are failing to realized what a unique time we live in. How unique the 150 years preceding it really are. And how little we know about why it happened.
parentheses 7y ago
same has happened to software engineering: things that have a chance to be quantum leaps for a software project get shot down. instead we choose incremental improvements with less risk of failure. incentives driving this include performance review cycle length, focus on immediate impact, and the startup/vc fundraising norms.
- m_j_g 7y ago
  > things that have a chance to be quantum leaps for a software project get shot down Just genuinely cuorius, what examples do you have in mind ? Personally i have hope that in 50years homotopy type theory (ane generally formal methods) will find real applications in the industry
  - StandardFuture 7y ago
    "Real applications" start with someone applying them.
city17 7y ago
The most difficult thing of allowing for more novel science is accepting that more 'useless' research will be done. If you read stories about the old days, you'll see how often professors/researchers used the freedom they had to do absolutely useless things, just based on their own personal preferences. The same freedom of course allowed others to do great things. I think nowadays there is much more awareness about what researchers are doing, and they will be held accountable, not just by the people who fund them, but also by the general public (Why are we funding this research about levitating toads?!?!). Shielding the researchers from such outrage and building acceptance for (seemingly) useless research will be just as important as the article's suggested new metrics for novelty.
- paulryanrogers 7y ago
  While this is probably true there's also the possibility that in the past there was more to be discovered or invented. Edit: typo
  - naasking 7y ago
    That actually justifies giving researchers more freedom. New knowledge will more likely come from unexpected angles.
    - paulryanrogers 7y ago
      Good point. That said I imagine another facet is the risk of fraud and graft as some would love to be paid for navel gazing 'studies' or generally slap-dash work.
- mjfl 7y ago
  People say you need to let scientists do useless exploration in order to promote innovation. I think that is true to a degree, but definitely should not be the spirit of any policy where innovation is the goal. Many of the scientific innovations that pushed the 20th century forward were in fact purposefully done. The Von Neumann architecture was not invented to have fun, it was invented to aim artillery and design nukes. Similarly, much of the tech infrastructure in Silicon Valley descends directly from radio engineers coming out of WWII theater. Bardeen, Brittain, and Shockley didn't invent the solid state transistor because it would be interesting, they invented it because it would allow miniaturization of vacuum tube computers. The laser was invented for telecommunications purposes. There is now an entire field of "quantum electronics" describing the theory behind them. Even going back further, the invention of the steam engine preceded the development of thermodynamics, which initially sought to describe the limitations of these engines. IF "science leads technology", then one would expect steam engines to be rationally designed from the results of thermodynamics. The opposite is true. I think there are enough examples of scientific fields emerging from technological innovation that "uselessness" should not be considered correlated with innovation.
INGELRII 7y ago
Same authors have interesting related papers: * Neophilia ranking of scientific journals https://www.nber.org/papers/w21579 https://www.nber.org/papers/w21579 * Age and the Trying Out Of New Ideas, Journal of Human Capital (2019) https://www.nber.org/papers/w20920 https://www.nber.org/papers/w20920 http://www.nature.com/news/young-scientists-lead-the-way-on-fresh-ideas-1.16934 http://www.nature.com/news/young-scientists-lead-the-way-on-...
dluan 7y ago
Economists, policymakers, and scientists studying how to improve science output have themselves been hamstrung for decades because economists have to measure something, and for so long it's been nothing but bibliometrics. Until we can come up with a way to measure the other outputs of science productivity, we're stuck with this citation-based machine that has all of this institutional and cultural inertia behind it. Which is why I've come to believe this kind of change or new introduction of value isn't going to come from within academia. E.g. novelty is in some fields directly at odds with what makes you a "productive" professional scientist. Last year I was in the NBER's Science of Science Funding working paper session, and most of the datasets discussed are still heavily focused on patents, citations, and bibliometrics (https://projects.nber.org/drupal/SOSF/data https://projects.nber.org/drupal/SOSF/data).
- gumby 7y ago
  It’s the same as using lines of code as a productivity metric.
- jackcosgrove 7y ago
  I think the answer is that trust should be part of the equation. "If you can't measure it, you can't manage it" is a description of how to keep track of a low-trust system. Scientists are generally conscientious people, and bean counting is demoralizing. If there is some organizational solution, I think it should be to keep organization size small enough that it can be governed by personal relationships and trust. Otherwise social trust takes decades to build up, and I don't think there is a quick fix.
knzhou 7y ago
As an academic I will again make the dull but necessary point: figuring out the fundamental laws of nature is hard, stagnation is the norm, and the advances of the mid-20th century were the aftershocks of the great revolutions of quantum mechanics and relativity. You can't just make that happen again on demand. Fundamental laws are in short supply. Whenever people tell me we're stagnating I ask them to name an alternative. So far, from best to worst case, every answer has fallen into one of the following buckets: - a subfield that already exists and already has plenty of people working on it - an idea that was extensively investigated and carefully ruled out over 50 years ago - something that requires more money than the field will receive in total over the next 50 years - a mathematical formalism which simply refactors existing laws in a way that makes them unreadable to almost everyone, without a chance of leading to any new predictions - a complicated, ad hoc model that isn't any more predictive than simpler models, but gives the modeller hundreds of shiny knobs to tune - a metaphysical suggestion about how to view what nature "truly" is, which amounts at best to rewriting existing laws with bigger, more "profound" words I can't see how this is going to be fixed by changing how citations work. If anything, in my experience the particularly bad stuff has been correctly punished by low citations.
- tasogare 7y ago
  What you wrote make total sense for fundamental sciences. However, a lot if not most scientific fields are removed from that, and the supply of ideas or things to investigate is way larger. In this context, gaming the system is easier. I see it in my field, where lot of published stuff is junk written for publication count.
- laughinghan 7y ago
  I strongly disagree. You make a great argument for why stagnation in physics is to be expected, but advances in physics were a minority factor in the technological advances of the 20th century. Advances in chemistry were by far the primary driver, followed by biology, and then by physics. (I love physics, I minored in it in undergrad, and didn't minor or major in chemistry, lest you think I'm biased.) Chemistry brought us: - the Haber-Bosch process, artificial nitrogen fixation that revolutionized food production - the Bessemer process, modern steel production, necessary not just for modern buildings but also engines, modern guns, cars/boats/planes, modern supply chain via shipping and trucks, industrial factory and farm equipment, etc - modern plastics. Seriously, look around your room, wherever you are. How many things don't have some amount of plastic in them? Less than half, probably? - petroleum extraction and processing, which powers those steel engines, powers most of our electricity, and is the raw materials for plastic - modern explosives, especially smokeless gunpowder (did you know that people totally built Gatling-style rotary machine guns in the Victorian Era? The were useless because black powder creates too much smoke for you to be able to aim a black powder-powered machine gun) Biology brought us germ theory and modern medicine, and of course these fields are all interrelated because engines required thermodynamics and electricity is mostly physics, but none of that required relativity or quantum mechanics. Quantum mechanics is the foundation for the physical chemistry subfield of chemistry, is crucial to modern silicon integrated circuit manufacturing, and of course—the atomic age. But none of those have had the impact on society as oil alone, for example. As mentioned, I love physics, but the fact is, details about the fundamental laws of nature just did not have the impact on society that higher-level, less fundamental scientific advancements have had. The Wright brothers created aviation without any quantitative understanding of aerodynamics; and 100% of the societally impactful advancements in aerodynamics since then has had no relation to relativity or quantum mechanics.
Gatsky 7y ago
Part of it is also the research workforce composition. Research students are increasingly people without other good options or who feel like they need to do it for status reasons. It is a form of low paid labour that they take in preference to working low status jobs, or moving away from where they are currently living. Of course it is hard for them to take on bold projects, they are too busy trying to survive and ultimately aren’t that interested in the material. The recent protests from UCSC students saying they can’t afford the rent, and the constant blog posts from long suffering adjuncts are classic examples of this - why would anyone put up with these ridiculous arrangements? PIs don’t care about this stuff, they need warm bodies to churn out papers, and the pay is so low it doesn’t matter. This transition is enabled by the massive expansion of university footprint, staffing and student count. The higher degree has become the current generation’s desperate attempt to show they are making progress after the boomers.
H8crilA 7y ago
There's a much much simpler explanation for the stagnation of real GDP. You have to look at energy consumption (joules / year). The correlations are uncanny and seem to explain just about everything. https://youtu.be/wGt4XwBbCvA https://youtu.be/wGt4XwBbCvA https://www.resilience.org/stories/2015-02-06/charts-showing-the-long-term-gdp-energy-tie/ https://www.resilience.org/stories/2015-02-06/charts-showing... http://illusionofprosperity.blogspot.com/2012/04/total-energy-consumption-vs-real-gdp.html http://illusionofprosperity.blogspot.com/2012/04/total-energ... It's so simple and accurate that I have no idea why doesn't everybody know about this. It better than discovering central banking.
alpineidyll3 7y ago
Simple changes like ditching conference travel and ending grade inflation would totally repair academia. I gave up on it myself. I'm happier in finance. Life's too short.
maxfan8 7y ago
Perhaps initially, citation indexes were useful (i.e. they were a good metric to solve certain productivity issues), but lost their usefulness since people started to optimize and target that specific metric (along the lines of Goodhart's law).
laughinghan 7y ago
This premise that novelty should be better-rewarded seems odd to me, because I thought a common complaint about scientific journals was overly rewarding novelty instead of robustness? Isn't that considered one of the contributing factors to the replication crisis? For example: "Novelty in science – real necessity or distracting obsession?" https://phys.org/news/2018-01-novelty-science-real-necessity-distracting.html https://phys.org/news/2018-01-novelty-science-real-necessity... "Facts Are More Important Than Novelty: Replication in the Education Sciences" https://journals.sagepub.com/stoken/rbtfl/w5mrNxPVD8zSg/full https://journals.sagepub.com/stoken/rbtfl/w5mrNxPVD8zSg/full In the text: "Funding bodies and academic journals that value “novelty” over replication deserve blame too." https://theconversation.com/science-is-in-a-reproducibility-crisis-how-do-we-resolve-it-16998 https://theconversation.com/science-is-in-a-reproducibility-... Etc. I haven't made it through the entire 43-page paper yet, but a quick search for "replic" and "repro" suggests they don't address this point at all.
Haga 7y ago
Scientic disruption is a threat to corruption Contracts. You can not guarantee the votes of coachman union unless you subvert automobile introduction. Only way to circumvent these large entities rigging systems against innovation is war or competition of many small external entities beeing more threatening then runaway power politics.
nullc 7y ago
I think it's interesting to think about the incentives in scientific publication by comparison with the incentives in HN (or Reddit) comment posting. Someone could spend days of time writing a well researched HN comment that was exceptionally informative and accurate. There are occasionally comments that represent an hour or two of work on them... But multiday-effort comments are non-existent: the incentives of the venue don't reward that effort. And if you do take the time the discussion will have moved on before you get it published. If you published them they would likely be lost in a see of other low effort comments, or if they were acknowledged-- not likely much more than comments that merely took an hour. You don't just see fewer multiday comments, you see essentially none at all. Nothing technical about HN or Reddit prevents people from writing some epic work of commentary or research in a comment, and there would often be value from such works existing... The same pattern exists in academic publishing but with the effort levels shifted up one or two orders of magnitude: the maximum moves from an hour to (say) weeks (exact threshold varies by field). Works taking more effort than some field specific cutoff are extremely seldom done. Why make one 10x effort paper when you will serve your interests much better and with lower risks making 10 1x effort papers? This would be fine if all of science in that field could be done in the window of efforts allowed by those incentives, but that isn't the case... especially since a lot of the low hanging fruit-- results that can be obtained below the cutoff-- is already picked. So this is part of why you see things like immunologists pointing out that there has been relatively little academic investigation of virus seasonality-- though its an apparent, interesting, and seemingly important phenomena. Studying it in any depth would require experiments spanning years and likely dealing with human subjects on top of the possibility that your ideas don't turn up anything new... a lot of risk for someone who actually needs to get things published.
- knzhou 7y ago
  The extent to which this is an issue depends on the field, though. In my field, you can get massively rewarded, in both soft and hard criteria, if you put effort into writing a good review paper. And many of the giants of the field have sunk even more effort into writing textbooks, which yield even greater rewards. People can do this precisely because the incentives are better than for internet comments; a good textbook can be celebrated for decades.
StandardFuture 7y ago
Related: https://news.ycombinator.com/item?id=18493019 https://news.ycombinator.com/item?id=18493019
netwanderer3 7y ago
First of all, I believe the stagnation today in scientific discoveries is resulted from the lack of big grand visions of a future that can draw inspirations. Over the years, society has shifted into favoring financial languages and metrics in most of today communications instead of telling stories that are often associated with lifetime generational experiences. Most things are calculated based on precise risks and probabilities so naturally we would opt for the least risky path. As a result, the system has evolved into favoring incremental improvements rather than explorations of uncharted territories that are much more riskier. In scientific publishing, this metric is represented by an over-emphasize in citations which has become the main criteria those publications are now being evaluated based on. Novelty or a desire for new experience that can generate large and meaningful impact, or even simply playful experimental ideas are no longer valued as much. Citations quantity has become the main currency in scientific publishing, and understandably has also led the community to prioritize incremental improvements. In the paper, it mentioned that many seemingly irrelevant or uninteresting new scientific discoveries initially took a long time for the community to understand its potential, but those very same discoveries would later lead to much bigger and more meaningful inventions, such as the gene-editing tool CRISPR we have today. It took 20 years for this to happen counting from the initial discovery, so this is where the disconnection occurred. In that sense, there is a great need to help propagating those initial discoveries both in its magnitudes and speeds so that it can receive more attentions from other scientists and community. The novelty should once again be the main focus to drive motivations and inspirations. Scientific publications shouldn't just prioritize on hard cold metrics like citations, but instead attaching more metaphors and new visions of future possibilities that can excite and propel both science community and public interests. More than ever, people today are craving for that common naivety which used to connect everyone together into believing making the impossible possible. That's precisely what has made Elon Musk and his companies so successful.

184 points by hhs 7y ago

> This emphasis on citations in the measurement of scientific productivity shifted scientist rewards and behavior on the margin toward incremental science this sounds like the science equivalent of how search engines lead to the creation of content farms. citation scores are to spam science as pagerank is to wikihow? Imagine going through life with a high-tech spam filter from the future that can filter out wikihow and boring science. Impossible to build, I suspect, but that would be the life.

peterhj 7y ago
In fact, one of the motivating applications of PageRank was to bridge academic citation analysis to the web: http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf
tryptophan 7y ago
About the search aspect, I wonder if there is any way to create a custom search engine or filter out crap from google. I would love a browser addon that auto-filters out quora, wikihow, w3schools, techcrunch, etc... None of those garbage sites are what I'm looking for, ever. I think that, amusingly, filtering out the top 5% of best SEO optimized sites would make results much better.
- awinter-py 7y ago
  w3schools loads faster than MDN but I'm mostly with you reevaluating our social attitudes towards 'trust in experts' will be an aftermath of the covid crisis, and it may bleed into search & social media as well -- is there value in 'peer-reviewed everything'? (for some definition of peer)
  - petra 7y ago
    What is the alternative for the "trust in experts" model, for most people ? Is there any ?
    - awinter-py 7y ago
      oh wait I didn't mean to say that we were trusting experts in the past and we won't in the future more that we'll ask a lot of questions about it. when are we trusting them, when should we, are governing bodies making proper space for experts I think most people in their personal lives don't have access to good expert advisors, maybe outside medicine & car mechanics, maybe not even there
    - deleted 7y ago
      [deleted]
    - op03 7y ago
      Maybe more multi-disciplined group based (given how much more complex most problems are for individual experts to grapple with by themselves). Probably rated like team sports.
    - TeMPOraL 7y ago
      Actually trust actual experts, instead of mindlessly listening to people with best marketing? This applies to both politics and w3schools.
  - evmar 7y ago
    devdocs.io is a dramatically faster alternative to MDN, with the same content.
- arkitaip 7y ago
  You can already do that with a Chrome plugin, like Google Search Filter: https://chrome.google.com/webstore/detail/google-search-filter/eidhkmnbiahhgbgpjpiimdogfidfikgf https://chrome.google.com/webstore/detail/google-search-filt...
dkarl 7y ago
It's doubly tragic if it simultaneously incentivized the mass production of boring, low-value science and punished the boring but valuable work of replication.
- awinter-py 7y ago
  yes -- this is kind of the 'deep work' argument the distracting stuff is ultimately low value we've cranked up the gain on spam -- spam is our culture's moai
TeMPOraL 7y ago
> this sounds like the science equivalent of how search engines lead to the creation of content farms. citation scores are to spam science as pagerank is to wikihow? Or a science equivalent to what money and market competition does with all human ventures. You need more and faster than your competitors to progress, else you'll fall into obscurity. "First to publish" is isomorphic to "first to market". If you treat research as a videogame, citations fit perfectly as an in-game currency. That is to say, it's an example of a general problem of optimizing short-term metrics, which are a decent proxy of the actual goal up to a point. Science could definitely use a spam filter. Journals unfortunately do a poor way of being one; they're thoroughly gamed, the same way Google is via SEO.
- buitreVirtual 7y ago
  At least in the market a junk product doesn't survive despite being first to market. The product needs to be good or at least inspire others to be regarded as a real contribution. This additional requirement is what's missing from the incentive system in science today.
  - laughinghan 7y ago
    Tons of junk survives and thrives in the market. Your great grandparent specifically brought up content farms as an example. Homeopathic remedies, counterfeit SD cards, the list goes on. Nothing about markets magically solves the problem that if it's hard to evaluate quality, there will be junk masquerading as quality.
grabbalacious 7y ago
>Imagine going through life with a high-tech spam filter from the future that can filter out wikihow and boring science. Happily it's available now and it's called intuition. If something is boring then avoid; if something is exciting then pursue. Problems being that it's purely anecdotal and you have to know and trust yourself. If you're too attracted to prestige, money or job security then it's going to return a distorted signal. Which is why organised science is now bureaucratic and slow despite the fact that there are more scientists than ever before.

bognition 7y ago

There have been converastions for years about the slow down in scientific discovery. Often times the explanation is that science is getting more expensive, something i've often felt was a bit of a straw man argument. Instead pointing at the current incentive structure (that everyone already agrees is broken) makes a lot more sense. So lets start a conversation about a better incentive structure. What do we want to incentivize? How do we do that? How do we fix science?

bjelkeman-again 7y ago
What is broken with the current system? (I have an idea what is broken, but it would be good to see it explained.)
- wegs 7y ago
  A lot of things, but this paper points to one issue: * In order to get an academic position, you need to have letters from a research community. * In order to get citations, you need your papers to be used by a research community. This gives a strong advantage to work done within existing lines of research. Virtually anything outside of one of the existing "academic cottage industries" or "mutual adoration societies" is at a huge disadvantage. You end up with research communities going down long rabbit holes with tunnel vision, and big broad areas of potential research are never explored.
hef19898 7y ago
Incentives, and the corresponding KPIs, are always a good place to start looking if one is trying to find out why systems and people behave the way they do. When I read the title I thought the paper was about business ideas and similar things. Having quickly read the paper, citations is a likely reason for, what the authors called, me too science. Maybe it's time to be more bold.
- jjoonathan 7y ago
  Well, yeah, but bold is nearly impossible to tell from crazy, and people really don't like being told that "actually, the most efficient management technique in this case is to throw money randomly at a lightly-filtered set of projects." People want to believe that blue-sky research can be predicted, managed, and optimized, no matter the staggering mountains of evidence to the contrary.

DubiousPusher 7y ago

There is so much about papers like this that make me skeptical. 1) It assumes we've been good at measuring growth. Which to me is dubious. Our current system counts the production, deployment and detonation of a bomb all as positive production meanwhile not counting unpaid domestic work. I know economists don't like to out a value to different types of services and goods because they feel they are putting a finger on the scale. But they are doing just as much by using a blind metric which ends up counting some work and not other. 2) It never cost adjusts growth. For example, all that wonderful growth in the midst of the 20th cwntury incurred significant externalities. The system we have now tries much harder to make producers realize their externalities. This curbs growth which is not necessarily a bad thing. 3) It assumes we can atomize historic growth and single out what things contributes to what. But this seems ridiculous especially how these factors interact and are not necessarily seperable. For example, we happen to live in a universe in which the movement of electrons can be used transmit energy and appropriate it to many tasks. The story of the 20th cwntury's economy is very much the story of how we mastered this one property of nature. Electrification doesn't just provide heat and light. It enables the transportation of water. It enables the construction of more and larger structures. It enables the creation of aluminum and plated metals. Aluminum itself allows many features of our world we take for granted. From airplanes to electronics. But does abundant aluminum lay at the feet of scientific innovations? In its infancy for sure but after some basic science it is widespread cheap electrification which allows us to mass produce aluminum. How much of that economy do we count to cutting edge science? It's not easily disentangled from popular politics which allowed the mass construction of hydroelectric damns throughout the U.S. and other countries. There is a possible future in which for centuries, growth has slowed to something more than Renaissance levels but well below mid 20th century levels. In such a future we might look back at this period as the time we mastered the single most useful physical property of our universe and so of course it was an era of unprecedented growth. I'm not saying they're wrong. But every time someone from Bill Gates to NBER talks about this issue of growth, whether it has slowed, why it as slowed, how it can be increased, I just get this feeling that people are failing to realized what a unique time we live in. How unique the 150 years preceding it really are. And how little we know about why it happened.

parentheses 7y ago

same has happened to software engineering: things that have a chance to be quantum leaps for a software project get shot down. instead we choose incremental improvements with less risk of failure. incentives driving this include performance review cycle length, focus on immediate impact, and the startup/vc fundraising norms.

m_j_g 7y ago
> things that have a chance to be quantum leaps for a software project get shot down Just genuinely cuorius, what examples do you have in mind ? Personally i have hope that in 50years homotopy type theory (ane generally formal methods) will find real applications in the industry
- StandardFuture 7y ago
  "Real applications" start with someone applying them.

city17 7y ago

The most difficult thing of allowing for more novel science is accepting that more 'useless' research will be done. If you read stories about the old days, you'll see how often professors/researchers used the freedom they had to do absolutely useless things, just based on their own personal preferences. The same freedom of course allowed others to do great things. I think nowadays there is much more awareness about what researchers are doing, and they will be held accountable, not just by the people who fund them, but also by the general public (Why are we funding this research about levitating toads?!?!). Shielding the researchers from such outrage and building acceptance for (seemingly) useless research will be just as important as the article's suggested new metrics for novelty.

paulryanrogers 7y ago
While this is probably true there's also the possibility that in the past there was more to be discovered or invented. Edit: typo
- naasking 7y ago
  That actually justifies giving researchers more freedom. New knowledge will more likely come from unexpected angles.
  - paulryanrogers 7y ago
    Good point. That said I imagine another facet is the risk of fraud and graft as some would love to be paid for navel gazing 'studies' or generally slap-dash work.
mjfl 7y ago
People say you need to let scientists do useless exploration in order to promote innovation. I think that is true to a degree, but definitely should not be the spirit of any policy where innovation is the goal. Many of the scientific innovations that pushed the 20th century forward were in fact purposefully done. The Von Neumann architecture was not invented to have fun, it was invented to aim artillery and design nukes. Similarly, much of the tech infrastructure in Silicon Valley descends directly from radio engineers coming out of WWII theater. Bardeen, Brittain, and Shockley didn't invent the solid state transistor because it would be interesting, they invented it because it would allow miniaturization of vacuum tube computers. The laser was invented for telecommunications purposes. There is now an entire field of "quantum electronics" describing the theory behind them. Even going back further, the invention of the steam engine preceded the development of thermodynamics, which initially sought to describe the limitations of these engines. IF "science leads technology", then one would expect steam engines to be rationally designed from the results of thermodynamics. The opposite is true. I think there are enough examples of scientific fields emerging from technological innovation that "uselessness" should not be considered correlated with innovation.

INGELRII 7y ago

Same authors have interesting related papers: * Neophilia ranking of scientific journals https://www.nber.org/papers/w21579 https://www.nber.org/papers/w21579 * Age and the Trying Out Of New Ideas, Journal of Human Capital (2019) https://www.nber.org/papers/w20920 https://www.nber.org/papers/w20920 http://www.nature.com/news/young-scientists-lead-the-way-on-fresh-ideas-1.16934 http://www.nature.com/news/young-scientists-lead-the-way-on-...

dluan 7y ago

Economists, policymakers, and scientists studying how to improve science output have themselves been hamstrung for decades because economists have to measure something, and for so long it's been nothing but bibliometrics. Until we can come up with a way to measure the other outputs of science productivity, we're stuck with this citation-based machine that has all of this institutional and cultural inertia behind it. Which is why I've come to believe this kind of change or new introduction of value isn't going to come from within academia. E.g. novelty is in some fields directly at odds with what makes you a "productive" professional scientist. Last year I was in the NBER's Science of Science Funding working paper session, and most of the datasets discussed are still heavily focused on patents, citations, and bibliometrics (https://projects.nber.org/drupal/SOSF/data https://projects.nber.org/drupal/SOSF/data).

gumby 7y ago
It’s the same as using lines of code as a productivity metric.
jackcosgrove 7y ago
I think the answer is that trust should be part of the equation. "If you can't measure it, you can't manage it" is a description of how to keep track of a low-trust system. Scientists are generally conscientious people, and bean counting is demoralizing. If there is some organizational solution, I think it should be to keep organization size small enough that it can be governed by personal relationships and trust. Otherwise social trust takes decades to build up, and I don't think there is a quick fix.

knzhou 7y ago

As an academic I will again make the dull but necessary point: figuring out the fundamental laws of nature is hard, stagnation is the norm, and the advances of the mid-20th century were the aftershocks of the great revolutions of quantum mechanics and relativity. You can't just make that happen again on demand. Fundamental laws are in short supply. Whenever people tell me we're stagnating I ask them to name an alternative. So far, from best to worst case, every answer has fallen into one of the following buckets: - a subfield that already exists and already has plenty of people working on it - an idea that was extensively investigated and carefully ruled out over 50 years ago - something that requires more money than the field will receive in total over the next 50 years - a mathematical formalism which simply refactors existing laws in a way that makes them unreadable to almost everyone, without a chance of leading to any new predictions - a complicated, ad hoc model that isn't any more predictive than simpler models, but gives the modeller hundreds of shiny knobs to tune - a metaphysical suggestion about how to view what nature "truly" is, which amounts at best to rewriting existing laws with bigger, more "profound" words I can't see how this is going to be fixed by changing how citations work. If anything, in my experience the particularly bad stuff has been correctly punished by low citations.

tasogare 7y ago
What you wrote make total sense for fundamental sciences. However, a lot if not most scientific fields are removed from that, and the supply of ideas or things to investigate is way larger. In this context, gaming the system is easier. I see it in my field, where lot of published stuff is junk written for publication count.
laughinghan 7y ago
I strongly disagree. You make a great argument for why stagnation in physics is to be expected, but advances in physics were a minority factor in the technological advances of the 20th century. Advances in chemistry were by far the primary driver, followed by biology, and then by physics. (I love physics, I minored in it in undergrad, and didn't minor or major in chemistry, lest you think I'm biased.) Chemistry brought us: - the Haber-Bosch process, artificial nitrogen fixation that revolutionized food production - the Bessemer process, modern steel production, necessary not just for modern buildings but also engines, modern guns, cars/boats/planes, modern supply chain via shipping and trucks, industrial factory and farm equipment, etc - modern plastics. Seriously, look around your room, wherever you are. How many things don't have some amount of plastic in them? Less than half, probably? - petroleum extraction and processing, which powers those steel engines, powers most of our electricity, and is the raw materials for plastic - modern explosives, especially smokeless gunpowder (did you know that people totally built Gatling-style rotary machine guns in the Victorian Era? The were useless because black powder creates too much smoke for you to be able to aim a black powder-powered machine gun) Biology brought us germ theory and modern medicine, and of course these fields are all interrelated because engines required thermodynamics and electricity is mostly physics, but none of that required relativity or quantum mechanics. Quantum mechanics is the foundation for the physical chemistry subfield of chemistry, is crucial to modern silicon integrated circuit manufacturing, and of course—the atomic age. But none of those have had the impact on society as oil alone, for example. As mentioned, I love physics, but the fact is, details about the fundamental laws of nature just did not have the impact on society that higher-level, less fundamental scientific advancements have had. The Wright brothers created aviation without any quantitative understanding of aerodynamics; and 100% of the societally impactful advancements in aerodynamics since then has had no relation to relativity or quantum mechanics.

Gatsky 7y ago

Part of it is also the research workforce composition. Research students are increasingly people without other good options or who feel like they need to do it for status reasons. It is a form of low paid labour that they take in preference to working low status jobs, or moving away from where they are currently living. Of course it is hard for them to take on bold projects, they are too busy trying to survive and ultimately aren’t that interested in the material. The recent protests from UCSC students saying they can’t afford the rent, and the constant blog posts from long suffering adjuncts are classic examples of this - why would anyone put up with these ridiculous arrangements? PIs don’t care about this stuff, they need warm bodies to churn out papers, and the pay is so low it doesn’t matter. This transition is enabled by the massive expansion of university footprint, staffing and student count. The higher degree has become the current generation’s desperate attempt to show they are making progress after the boomers.

H8crilA 7y ago

There's a much much simpler explanation for the stagnation of real GDP. You have to look at energy consumption (joules / year). The correlations are uncanny and seem to explain just about everything. https://youtu.be/wGt4XwBbCvA https://youtu.be/wGt4XwBbCvA https://www.resilience.org/stories/2015-02-06/charts-showing-the-long-term-gdp-energy-tie/ https://www.resilience.org/stories/2015-02-06/charts-showing... http://illusionofprosperity.blogspot.com/2012/04/total-energy-consumption-vs-real-gdp.html http://illusionofprosperity.blogspot.com/2012/04/total-energ... It's so simple and accurate that I have no idea why doesn't everybody know about this. It better than discovering central banking.

alpineidyll3 7y ago

Simple changes like ditching conference travel and ending grade inflation would totally repair academia. I gave up on it myself. I'm happier in finance. Life's too short.

maxfan8 7y ago

Perhaps initially, citation indexes were useful (i.e. they were a good metric to solve certain productivity issues), but lost their usefulness since people started to optimize and target that specific metric (along the lines of Goodhart's law).

laughinghan 7y ago

This premise that novelty should be better-rewarded seems odd to me, because I thought a common complaint about scientific journals was overly rewarding novelty instead of robustness? Isn't that considered one of the contributing factors to the replication crisis? For example: "Novelty in science – real necessity or distracting obsession?" https://phys.org/news/2018-01-novelty-science-real-necessity-distracting.html https://phys.org/news/2018-01-novelty-science-real-necessity... "Facts Are More Important Than Novelty: Replication in the Education Sciences" https://journals.sagepub.com/stoken/rbtfl/w5mrNxPVD8zSg/full https://journals.sagepub.com/stoken/rbtfl/w5mrNxPVD8zSg/full In the text: "Funding bodies and academic journals that value “novelty” over replication deserve blame too." https://theconversation.com/science-is-in-a-reproducibility-crisis-how-do-we-resolve-it-16998 https://theconversation.com/science-is-in-a-reproducibility-... Etc. I haven't made it through the entire 43-page paper yet, but a quick search for "replic" and "repro" suggests they don't address this point at all.

Haga 7y ago

Scientic disruption is a threat to corruption Contracts. You can not guarantee the votes of coachman union unless you subvert automobile introduction. Only way to circumvent these large entities rigging systems against innovation is war or competition of many small external entities beeing more threatening then runaway power politics.

nullc 7y ago

I think it's interesting to think about the incentives in scientific publication by comparison with the incentives in HN (or Reddit) comment posting. Someone could spend days of time writing a well researched HN comment that was exceptionally informative and accurate. There are occasionally comments that represent an hour or two of work on them... But multiday-effort comments are non-existent: the incentives of the venue don't reward that effort. And if you do take the time the discussion will have moved on before you get it published. If you published them they would likely be lost in a see of other low effort comments, or if they were acknowledged-- not likely much more than comments that merely took an hour. You don't just see fewer multiday comments, you see essentially none at all. Nothing technical about HN or Reddit prevents people from writing some epic work of commentary or research in a comment, and there would often be value from such works existing... The same pattern exists in academic publishing but with the effort levels shifted up one or two orders of magnitude: the maximum moves from an hour to (say) weeks (exact threshold varies by field). Works taking more effort than some field specific cutoff are extremely seldom done. Why make one 10x effort paper when you will serve your interests much better and with lower risks making 10 1x effort papers? This would be fine if all of science in that field could be done in the window of efforts allowed by those incentives, but that isn't the case... especially since a lot of the low hanging fruit-- results that can be obtained below the cutoff-- is already picked. So this is part of why you see things like immunologists pointing out that there has been relatively little academic investigation of virus seasonality-- though its an apparent, interesting, and seemingly important phenomena. Studying it in any depth would require experiments spanning years and likely dealing with human subjects on top of the possibility that your ideas don't turn up anything new... a lot of risk for someone who actually needs to get things published.

knzhou 7y ago
The extent to which this is an issue depends on the field, though. In my field, you can get massively rewarded, in both soft and hard criteria, if you put effort into writing a good review paper. And many of the giants of the field have sunk even more effort into writing textbooks, which yield even greater rewards. People can do this precisely because the incentives are better than for internet comments; a good textbook can be celebrated for decades.

StandardFuture 7y ago

Related: https://news.ycombinator.com/item?id=18493019 https://news.ycombinator.com/item?id=18493019

netwanderer3 7y ago

First of all, I believe the stagnation today in scientific discoveries is resulted from the lack of big grand visions of a future that can draw inspirations. Over the years, society has shifted into favoring financial languages and metrics in most of today communications instead of telling stories that are often associated with lifetime generational experiences. Most things are calculated based on precise risks and probabilities so naturally we would opt for the least risky path. As a result, the system has evolved into favoring incremental improvements rather than explorations of uncharted territories that are much more riskier. In scientific publishing, this metric is represented by an over-emphasize in citations which has become the main criteria those publications are now being evaluated based on. Novelty or a desire for new experience that can generate large and meaningful impact, or even simply playful experimental ideas are no longer valued as much. Citations quantity has become the main currency in scientific publishing, and understandably has also led the community to prioritize incremental improvements. In the paper, it mentioned that many seemingly irrelevant or uninteresting new scientific discoveries initially took a long time for the community to understand its potential, but those very same discoveries would later lead to much bigger and more meaningful inventions, such as the gene-editing tool CRISPR we have today. It took 20 years for this to happen counting from the initial discovery, so this is where the disconnection occurred. In that sense, there is a great need to help propagating those initial discoveries both in its magnitudes and speeds so that it can receive more attentions from other scientists and community. The novelty should once again be the main focus to drive motivations and inspirations. Scientific publications shouldn't just prioritize on hard cold metrics like citations, but instead attaching more metaphors and new visions of future possibilities that can excite and propel both science community and public interests. More than ever, people today are craving for that common naivety which used to connect everyone together into believing making the impossible possible. That's precisely what has made Elon Musk and his companies so successful.