3 ms·

> In my personal experience the main problem with scientific research today is caused by publication counts representing the sole metric for academic success.

> In my personal experience the main problem with scientific research today is caused by publication counts representing the sole metric for academic success. It also leads to bloat in the literature: People learn the "least publishable increment" and go publish that.

e_g 15y ago
Exactly. A study by Tuliving and Madigan (http://alicekim.ca/AnnRev70.pdf http://alicekim.ca/AnnRev70.pdf) outlines this nicely. In the words of David T. Lykken (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.118.2655&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.118...): In their 1970 Annual review chapter on Memory and Verbal learning, Tulving and madigan reported that they had independently rated 540 published articles in terms of their "contribution to knowledge". With "remarkable agreement", they found that they had sorted two-thirds of the articles into a category labelled "utterly inconsequential". "The primary function these papers serve is to give something to do to people who count papers instead of reading them. Future research and understanding of verbal learning and memory would not be affected at all if none of the papers in this category seen the light of day" About 25 percent of the articles were classified as: "run-of-the-mill .. these articles also do not add anything really new to knowledge... [such articles] make one wish that at least some writers, faced with the decision of whether to publish or perish, should have seriously considered the latter alternative" Only about 10% of the entire set of published papers received the modest compliment of being classified as "worthwhile". Given that memory and verbal learning was then a popular and relatively 'hard' area of psychological research, attracting some of the brightest students, this is a devastating assessment of the end product. Hence, of the research ideas generated by these psychologists, who are all card-carrying scientists and who liked these ideas well enough to invest weeks or months of their lives working on them, less than 25% of 40% of 10% = 1% actually appear to make some sort of contribution to the discipline. Given that this work was published in the 1970s when the "publish or perish" pressure was less strongly developed I expect today's number to be worse. Concerning Information Retrieval the field of research of my PhD I would certainly attest that this applies.
- pnathan 15y ago
  I have been trawling through large numbers of papers recently as part of my computer science Master's background research, and some papers are quite frankly unintelligible garble... I can't even understand how they passed peer review. I would say that of the articles I've read, maybe 5% are meaningful contributions that advance the science; maybe 25% contribute something to the current understanding, perhaps 60% are "we did something cool", and 10% are "are these reviewers awake".
- jpdoctor 15y ago
  Thanks for those links. During my own PhD, I was pretty sure that the "least publishable increment" became negative. The underlying problem is that there is no penalty for publishing non-increments. For some inexplicable reason, citation impact assessments do not divide by the total number of publications. I have no doubt that such a normalization factor would go a long way to fixing the problem.

e_g 15y ago

Exactly. A study by Tuliving and Madigan (http://alicekim.ca/AnnRev70.pdf http://alicekim.ca/AnnRev70.pdf) outlines this nicely. In the words of David T. Lykken (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.118.2655&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.118...): In their 1970 Annual review chapter on Memory and Verbal learning, Tulving and madigan reported that they had independently rated 540 published articles in terms of their "contribution to knowledge". With "remarkable agreement", they found that they had sorted two-thirds of the articles into a category labelled "utterly inconsequential". "The primary function these papers serve is to give something to do to people who count papers instead of reading them. Future research and understanding of verbal learning and memory would not be affected at all if none of the papers in this category seen the light of day" About 25 percent of the articles were classified as: "run-of-the-mill .. these articles also do not add anything really new to knowledge... [such articles] make one wish that at least some writers, faced with the decision of whether to publish or perish, should have seriously considered the latter alternative" Only about 10% of the entire set of published papers received the modest compliment of being classified as "worthwhile". Given that memory and verbal learning was then a popular and relatively 'hard' area of psychological research, attracting some of the brightest students, this is a devastating assessment of the end product. Hence, of the research ideas generated by these psychologists, who are all card-carrying scientists and who liked these ideas well enough to invest weeks or months of their lives working on them, less than 25% of 40% of 10% = 1% actually appear to make some sort of contribution to the discipline. Given that this work was published in the 1970s when the "publish or perish" pressure was less strongly developed I expect today's number to be worse. Concerning Information Retrieval the field of research of my PhD I would certainly attest that this applies.

pnathan 15y ago
I have been trawling through large numbers of papers recently as part of my computer science Master's background research, and some papers are quite frankly unintelligible garble... I can't even understand how they passed peer review. I would say that of the articles I've read, maybe 5% are meaningful contributions that advance the science; maybe 25% contribute something to the current understanding, perhaps 60% are "we did something cool", and 10% are "are these reviewers awake".
jpdoctor 15y ago
Thanks for those links. During my own PhD, I was pretty sure that the "least publishable increment" became negative. The underlying problem is that there is no penalty for publishing non-increments. For some inexplicable reason, citation impact assessments do not divide by the total number of publications. I have no doubt that such a normalization factor would go a long way to fixing the problem.