4 ms·

To understand the fundamental problem with this paper and the "CD" measure of "disruptiveness", one need only look at Figure 1a, where three Nobel Prize winning

To understand the fundamental problem with this paper and the "CD" measure of "disruptiveness", one need only look at Figure 1a, where three Nobel Prize winning papers and three patents are ranked on the "CD5" scale. Apparently, there are more disruptive discoveries than Watson and Crick's discovery of the double-stranded structure of DNA (CD5 0.62 on a scale of -1.0 to 1.0), and Baltimore's discovery of reverse transcription is -0.55. One might argue the ranks of these two discoveries should be reversed, since DNA had already been shown to be the genetic material, but no one imagined that RNA could be converted to DNA. Likewise, the Wigler patent for transformation into eukaryotic cells (CD5=0.70) has certainly has had less effect on the world economy than Monsanto's patent on glyphosphate resistant plants (CD= -0.85). It's easy to make up a measure and then develop a story that explains why that measure is useful. Was Newton's theory of gravity disruptive, or consolidating. I'm thinking consolidating, since it allowed a lot of other things to make sense. Likewise, perhaps quantum mechanics was disruptive -- it sounds disruptive -- but it also consolidated a lot of confusing observations. (And lets not even mention the problem that when there are 1,000-times as many papers between 1940 and 2010, we might not expect 1,000-times as many disruptions.) This paper has a measure that sounds useful, and is certainly good for headlines, but it is very unclear that it provides any insight into the progress of science.

baxtr 4y ago
Agreed. I think this is fundamentally a sampling rate observation. Pace of publishing and observations have dramatically as you rightly point out.
dekhn 4y ago
You misunderstand why W&C was important (although, given your username, I assume you've already sat through the lectures): in elucidating (not discovering) the structure of DNA, they had a biophysical mechanism that explained heredity (specifically, that it was an antiparallel double helix with complementary bases, which immediately suggests a mechanism for replication). That recognition immediately led to a flurry of follow-on work confirming the mechanism and birthing molecular biology as we know it, which opened up vast new areas of science.
- fastaguy88 4y ago
  Yes, I know why it was disruptive. The question (for this paper) is, what other biological papers were more disruptive (since, based on the range of the scale, they barely made the first quintile). I think the more important question is, how do we validate the usefulness of the CD5 metric. And, why is it better to be "disruptive" than "consolidating"? Based on the examples shown (and others that I can imagine), it is unclear that it measures anything related to scientific significance.
  - dekhn 4y ago
    I had already discounted the measure as being too prone to sampling error to be useful. I don't think these things are really measurable, and even if you did measure it, "disruption" and "consolidation" are such subjective terms it would be arguable to have a single-dimensional slider that went between those extremes. Probably best to just say some of the greatest discoveries are both disruptive, and consolidating.
notahacker 4y ago
Yep. The trend over time could equally be capturing changes in referencing practice (either that really important, influential papers/inventions in the past didn't reference other really important, influential papers/inventions nearly as often as equally novel papers/inventions today, or that modern referencing is much more exhaustive about referencing minor and tangentially relevant papers/inventions. There is, of course, also more significant prior art in journals and patent indices to actually cite in later years...). They carry out some robustness checks, but I'm not sure any of them really dent that hypothesis. It's also a conclusion based on proportions (ie. the effect mostly comes from by there being a lot more "non-disruptive" papers published rather than fewer "disruptive" papers) and there's actually a trend rise in the raw number of "most disruptive" academic papers from the 1950s through to the 1990s, at which point the numbers drop so sharply it looks like a discontinuity in the data (but would probably coincide with lots of journal content becoming easily accessible online...)
crocowhile 4y ago
Whatever CD is measuring, is going down as a trend. So one cannot blame the obviously ethereal nature of CD for this.
dguest 4y ago
Boring title: "Scientific authors follow more citations when reading modern papers". When I read a modern paper, I skim a lot more of the references. Getting to a cited paper is often just two clicks: there's a reference to the bibliography, and from there a link to arXiv (or some other public repository). As papers get older the bibliography might not link to arXiv, or might not link to a digitized paper at all. Those older papers are more time consuming to use as a reference. Scientists aren't lazy but they also don't have infinite time. Put another way: papers form a graph, this result suggests that older nodes are traversed less frequently. I think the metric they propose is useful, but we could rephrase this all as measure of how useful papers are as a node (where CD = -1 is most useful, CD = 1 is least useful). In short, this result might just mean paper discovery is getting easier. That's a good thing. We can wring our hands endlessly about how there's less low hanging fruit (there is) or about how academia is rotting away (as though it was ever different), but those conclusions are big jumps from what this paper says.

baxtr 4y ago

Agreed. I think this is fundamentally a sampling rate observation. Pace of publishing and observations have dramatically as you rightly point out.

dekhn 4y ago

You misunderstand why W&C was important (although, given your username, I assume you've already sat through the lectures): in elucidating (not discovering) the structure of DNA, they had a biophysical mechanism that explained heredity (specifically, that it was an antiparallel double helix with complementary bases, which immediately suggests a mechanism for replication). That recognition immediately led to a flurry of follow-on work confirming the mechanism and birthing molecular biology as we know it, which opened up vast new areas of science.

fastaguy88 4y ago
Yes, I know why it was disruptive. The question (for this paper) is, what other biological papers were more disruptive (since, based on the range of the scale, they barely made the first quintile). I think the more important question is, how do we validate the usefulness of the CD5 metric. And, why is it better to be "disruptive" than "consolidating"? Based on the examples shown (and others that I can imagine), it is unclear that it measures anything related to scientific significance.
- dekhn 4y ago
  I had already discounted the measure as being too prone to sampling error to be useful. I don't think these things are really measurable, and even if you did measure it, "disruption" and "consolidation" are such subjective terms it would be arguable to have a single-dimensional slider that went between those extremes. Probably best to just say some of the greatest discoveries are both disruptive, and consolidating.

notahacker 4y ago

Yep. The trend over time could equally be capturing changes in referencing practice (either that really important, influential papers/inventions in the past didn't reference other really important, influential papers/inventions nearly as often as equally novel papers/inventions today, or that modern referencing is much more exhaustive about referencing minor and tangentially relevant papers/inventions. There is, of course, also more significant prior art in journals and patent indices to actually cite in later years...). They carry out some robustness checks, but I'm not sure any of them really dent that hypothesis. It's also a conclusion based on proportions (ie. the effect mostly comes from by there being a lot more "non-disruptive" papers published rather than fewer "disruptive" papers) and there's actually a trend rise in the raw number of "most disruptive" academic papers from the 1950s through to the 1990s, at which point the numbers drop so sharply it looks like a discontinuity in the data (but would probably coincide with lots of journal content becoming easily accessible online...)

crocowhile 4y ago

Whatever CD is measuring, is going down as a trend. So one cannot blame the obviously ethereal nature of CD for this.

dguest 4y ago

Boring title: "Scientific authors follow more citations when reading modern papers". When I read a modern paper, I skim a lot more of the references. Getting to a cited paper is often just two clicks: there's a reference to the bibliography, and from there a link to arXiv (or some other public repository). As papers get older the bibliography might not link to arXiv, or might not link to a digitized paper at all. Those older papers are more time consuming to use as a reference. Scientists aren't lazy but they also don't have infinite time. Put another way: papers form a graph, this result suggests that older nodes are traversed less frequently. I think the metric they propose is useful, but we could rephrase this all as measure of how useful papers are as a node (where CD = -1 is most useful, CD = 1 is least useful). In short, this result might just mean paper discovery is getting easier. That's a good thing. We can wring our hands endlessly about how there's less low hanging fruit (there is) or about how academia is rotting away (as though it was ever different), but those conclusions are big jumps from what this paper says.