4 ms·
On a related note, some years ago I read an academic paper -- alas, it was printed on a dead tree, and I can't find a link to a digital version of it -- which p
by nkoren 15y ago
On a related note, some years ago I read an academic paper -- alas, it was printed on a dead tree, and I can't find a link to a digital version of it -- which pointed out that the rate at which papers were cited was driven primarily by the rate at which papers were cited. This is not as much of a tautology as it sounds like.
Think about the process of writing a paper: you do some keyword searches for recent articles on related subject. You then look at the bibliographies for those articles, pick out whatever looks relavent to your topic, look at the bibliographies of those articles, etc. What this means is that apart from your initial keyword search, the primary criteria for including an article in your research is: "has it been cited already?". Relevance is merely a secondary filter.
This paper pointed out the effects of this phenomena: the vast majority of published scientific papers are never cited again; a moderate number are cited only a few times, and the remaining few -- having reached a bibliographic critical mass -- are cited thousands of times. The authors of the paper made a strong case that this was not a good reflection of the quality of research. In many cases, the process of reaching bibliographic critical was simply based on the almost random chance of acquiring those first few citations. The authors provided several examples of important scientific ideas which had been lost for decades, arguably because they had not attracted a critical mass of citations in the years immediately after publication.
In other words, humans suck at pagerank.
Anyhow, it occurred to me that this is a problem which could be solved with technology. Imagine an online word processor which -- in a sidebar -- suggests potentially related articles from ArXiv and Google Scholar. This would be based not on crawling bibliographies, but rather on semantic analysis of the adjacent paragraphs.
I think this would create some real benefits. It would remove much of the problems with citation bias, ensuring that important ideas aren't lost, and also that prior research isn't unwittingly duplicated. Wish I had time to implement something like this!
- lhnz 15y agoSomebody needs to make a relevance algorithm which takes into consideration, (1) noise at low levels of citation activity, and (2) social proof being misleading at high levels of citation.
- gliese1337 15y agoI am always looking for ideas for software that I could work on that seems more useful than cat pictures, and this gets pretty high up in my estimation. Most of my minimal NLP expertise is being used up in developing software for foreign language instruction in the near future, but I'm definitely bookmarking this just in case I can ever take it up later.
- sb 15y agoI am not sure that this will work anytime soon, even if NLP gets better substantially. I think the problem reduces to examination of patents and their novelty. I think that having algorithms to find related work is going to be hard to the point of it being unreliable. For example, in CS there is a trend of changing nomenclature every n years, which I think makes it hard to find related work.
- eli_gottlieb 15y agoGoogle manages to serve advertisements despite changing nomenclature. We're just talking about "advertising" papers based on various data about the paper under submission.
- nkoren 15y agoIt would be really great to see somebody take this on! Although if this idea makes you into a millionaire, then you'll owe me a burrito, okay?
- sb 15y agoI think that your impression of doing related work is overly simplified. Since you are submitting to a conference/journal in your field, chances are high that the reviewers are knowledgable in the subject area and will point out errors in attributing due credits to related work. While I agree that there are systemic problems with peer review and how the science "enterprise" works, there is fitting analogy from politics by Winston Churchill: "The worst form of government except for all the others."
- ktizo 15y agoI think you may mean the worst form of government except all the others that have been tried. We need to try more.
- RichardPrice 15y agoI think that is a great sentiment. To find systems that work a lot better than the current one, which is incredibly slow, we are going to need to try a lot of different ideas. When you are venturing into the unknown, innovation is tough and challenging, but it's also very rewarding.
- ajuc 15y agoUnfortunately trying new ideas when it comes to government, sometimes produces milions of victims. But I agree - we should experiment more. One idea - we should be A/B testing any new law, before introducing it in the whole country.
- ktizo 15y agotrying new ideas when it comes to government, sometimes produces milions of victims. Usually only with the forms of government that don't care about millions of victims. Is really a no-brainer that any totalitarian system that intends to make an enemy of a large section of its population is probably going to be pretty awful. So I would strongly suggest weeding those forms out in advance, rather than using the common historical method of allowing them to be implemented through force by the paranoid and insane. This is less of an issue when it comes to systems of improving feedback in science, unless it somehow enables an aspiring evil genius, or something.