7 ms·
GitWikXiv: Toward a successor to the academic paper
- jessriedel 12y ago(I expanded the title to give a bit more context for HN.)
- TheCowboy 12y agoI've thought a lot about this as well and think what you presented is good. I'd like to add the one problem that has usually motivated me to think about this problem the way you have: Academic works can contain ideas or data that becomes outdated or is found to be incorrect. If you're not an expert in a niche, it's hard to sift this out when you come across it. It seems intellectually wasteful to have works that are, for example, 90% accurate and relevant, but have an idea that needs to be updated or tossed. An example of a field where I believe this happens too often is in economics. Alone, I don't have the time to keep up and fact check everything I read, but collaborative editing could help a lot in this area.
- ikeboy 12y agoIs this the idea hinted at in http://hpmor.com/notes/119 http://hpmor.com/notes/119? >First, I’ve designed an attempted successor to Wikipedia and/or Tumblr and/or peer review, and a friend of mine is working full-time on implementing it.
- jessriedel 12y agoNope. I'm certainly not working full time on this. (And I've only briefly met Eliezer once.) Would be very interested to know what that friend is up to, though...
- deleted 12y ago[deleted]
- whistlerbrk 12y agoI've been working on the same thing for quite some time, I hope to launch this year now that I have time again. Love to see this theme continuing to appear in HN.
- jsweojtj 12y agoI've always hoped that the PaperBricks, http://arxiv.org/abs/1102.3523 http://arxiv.org/abs/1102.3523 model would catch on. My summary of the idea: There is an awful lot of redundancy and wasted effort that goes into most papers. From introductions that need to be rewritten every time (when linking to a solid introduction would be both better and less time consuming). Each piece of a piece of a full paper (intro, data, analysis, ...) could be peer-reviewed and published individually. A full paper could then be built from these paper-bricks. Anyway, recommend reading the paper as it's well written and clear. There's also a YouTube video by the author explaining it: http://www.youtube.com/watch?v=4sorEcLjN04 http://www.youtube.com/watch?v=4sorEcLjN04.
- jessriedel 12y agoThanks for this pointer. I added it to the bottom of the post.
- gerty 12y agoI think a positive side-effect of PaperBricks idea would be full standardization of notation. It's tiring to look up meaning of variables as they differ with each author.
- akavel 12y agoTL;DR for PaperBricks: "Formalize the structure of papers, such that each paper is composed of one or many (clearly marked) of the following "sections" ("bricks"): symbol description my description "I" Introduction ("domain intro") "PS" Problem Statement ("specific problem") "HLSI" High-Level Solution Idea ("solution vision") "D" Details ("solution implementation") "PE" Performance Evaluation ("benchmarks") and each "brick" must reference one or many bricks of the same or "earlier" level, forming a global graph." -------- >8 ---------- >8 ---------- Some of the advantages: * no need to rephrase the same "intro" in every domain paper, just reference an existing "I" brick; * a benchmark (PE) can just reference many "D"s; * one can easily work "backwards" -- e.g. start with a benchmark (PE) of existing implementations and already publish it, then propose a new implementation (D); * if someone publishes a similar paper before you, with similar "vision/idea" (HLSI), this doesn't totally destroy your publication, as you can still publish the part with an alternative implementation (D); * "I+PS+HLSI or I+HLSI: This is what some communities call a "vision paper" [...]" * & many more listed in the linked arxiv paper http://arxiv.org/abs/1102.3523 http://arxiv.org/abs/1102.3523. Very nice, short and readable one, this.
- nartz 12y agoI dont think 'collaboration' is the problem, I think the main problem is the presentation. Academics want the credit for their work, which I think they should retain. However, dissemination of work is lacking - there are tons of supersmart ideas locked in academic papers; they act as a barrier for many. Instead of a dense academic paper in LaTEX or whichever, what if the standard were to provide the simplest explanation possible, that required a graphic that demonstrated the idea. Its sort of a tragedy though, when revolutionary papers present ideas simply and are overlooked, often because of the misguided notion that 'simple to understand' equates to a 'trivial insight' - intoerhwords, if you read it and it makes sense immediately, sometimes you think that it musn't be revolutionary. This doesn't always happen, but definitely to an extent. All of the 'correctness' and 'proofs' are useful for a separate crowd, and should also be included, but in a separate section because they are for a different user, namely, for other academics who are well versed in the domain. Also, a place for code/data/materials, as well as a checklist or script or otherwise to strictly reproduce results.
- chrisseaton 12y agoI think that might be in danger of becoming like infographics - things that people share and coo over but without really understanding what it means and what the implications are. I could draw you a picture of my latest papers if you wanted, and it may look pretty but I think you'd be cheating yourself if you thought you were getting anything of out it.
- gayprogrammer 12y agoI think you made a valid point, but it is not an argument against doing things another way.
- jessriedel 12y agoYou're talking about a different issue. Your suggestions don't solve the problems I used to motivate my post.
- nbourbaki 12y ago> All of the 'correctness' and 'proofs' are useful for a separate crowd, and should also be included, but in a separate section because they are for a different user, namely, for other academics who are well versed in the domain. Isn't the audience of an academic paper other academics? Placing proofs and details in a separate section is convenient for the layman but bothersome for the intended audience. Often, it is precisely these details that are important. In mathematics for example, using new methods to give simpler proofs of old theorems is very useful; here, other researchers would care about how particular details are resolved.
- pickle27 12y agogreat post. I've also spent a lot of time thinking about this problem and would like to eventually put some work towards it. A couple of additional ideas that I've had: * A paper could rely on a critical reference to build upon and the referenced paper could be disproven down the line but this is not immediately obvious from the paper that used it. * Currently it doesn't seem like any merit is given to researchers who are very good at reviewing papers. Compare this to software where a good code review is celebrated. Editing and cleaning up the state of science should be valued when scientists are looking for work so I think that something along the line of a Github CV for scientists would be valuable.
- jessriedel 12y agoThanks for this. I now quote those points at the bottom of the post.
- ISL 12y agoGood journals honor great referees by listing them on an annual accounting of Outstanding Referees: http://journals.aps.org/OutstandingReferees http://journals.aps.org/OutstandingReferees
- Jace_Harker 12y agoVery cool article. I like that it lays out the key features that would be needed on any academic paper platform. This actually sounds similar to Authorea (https://www.authorea.com https://www.authorea.com). It's an online collaborative academic word processor and publishing platform. Uses LaTeX, but more powerful and efficient. I added a comment on the post with some more details. (Disclaimer: I work at Authorea, so I'm biased.)
- jessriedel 12y agoLike I say in my reply, I think Authorea is a nice example of a useful tool to learn from like StackExchange, but even if successful wouldn't answer the central question I brought up. http://blog.jessriedel.com/2015/04/16/beyond-papers-gitwikxiv/#comment-1980409937 http://blog.jessriedel.com/2015/04/16/beyond-papers-gitwikxi...
- bkcooper 12y agoI'm with you on the lack of review articles, and I think they are probably the aspect of conventional publishing that this could most likely replace. I think you'd need a very radical overhaul of how science works to replace the scorekeeping aspect. I don't think a Github CV is a good analogy to how this could work, because academic science is interested in hiring leaders (i.e. people who can get funded), not contributors. I think realistically you would need to change how science rewards/emphasizes certain activities first, and then publishing would follow. That would be a good outcome for science, but I think it'll be awfully hard to get out of this equilibrium. edit: I'm curious what you think of the PubPeer model. That's obviously different from what you're envisioning, but thematically I think there are some similarities.
- jessriedel 12y ago> I think you'd need a very radical overhaul of how science works to replace the scorekeeping aspect. I don't think a Github CV is a good analogy to how this could work, because academic science is interested in hiring leaders (i.e. people who can get funded), not contributors. I think realistically you would need to change how science rewards/emphasizes certain activities first, and then publishing would follow. I have a lot of criticisms of academic incentives, and I agree there is something of a chicken-or-the-egg problem, but at least in my field there are plenty of people who don't command big grants but have large citations. The problem comes more because people are hired based on metrics that are unusually bad at tracking what we want. > I'm curious what you think of the PubPeer model. It could be useful. Only seems to differ strongly from blogs in its centrality, but (surprisingly) I'm not sure this is actually a big issue. My immediate concern with PubPeer is that it expands and diffuses the literature rather than distilling it to excellent concise articles.
- hyperion2010 12y agoThere are lots of people working in this space. See for example https://www.force11.org/ https://www.force11.org/. Some of the issues that the article does not touch on involve the difference between annotating an article and editing an article. The idea of forking an article doesn't work very well because what you want is annotation not editing. Changing someone's words isn't really very useful unless you are editing and collaborating. Collaboration isn't really an issue, google docs basically solves this issue (or git+tex or whatever). Wiki provides a reasonable way to deal with this as well but doesn't really solve the annotation problem. In a practical sense we are still looking for something that can work like http://www.xanadu.com/ http://www.xanadu.com/ (amazingly there is now a demo). Nothing is going to replace the paper, we can make them better and make it easier to annotate and talk about them. One major issue is that to date the citation graph for all papers and works is still trapped behind paywalls. There is no magical solution for this, it requires us to build lots of infrastructure and work to bring the publishers along. One thing the article does not mention is the need for better ways of documenting provenance for data.
- jessriedel 12y agoThanks for the link to force11. Any other suggestions? I specifically don't like the idea of annotations rather than editing, because annotations expand and diffuse the literature rather than distilling it to excellent concise articles. In particular, annotations only address two of my six motivations (#4 and #6). Within the hard sciences, what do you think annotation accomplishes that can't be accomplished through good editing? > Nothing is going to replace the paper, Ahh, a defeatist. You may say I'm a dreamer... But seriously, I think this is a much more realistic goal than Wikipedia and ArXiv looked like when they were launched.
- coliveira 12y agoThe problem is that wikipedia editors are not creating new knowledge, they are just organizing it. A sort of wikipedia for research would be very valuable as a separate project, but papers are necessary because they enable a technical conversation. One or more authors present their particular view of a topic, and other researchers join the discussion by writing their own papers. On the other hand, a shared page is an amorphous piece of information that is very difficult to use as a discussion medium. In this sense, wiki articles are much more primitive than traditional research papers.
- xvilka 12y agoI'd suggest to integrate more semantics in this system, like e.g. Knowledge Graph in Google or SemanticMediaWiki. This can help for automated processing years later. Also, adding more interactivity with ipython/webppl notebooks embedded directly in the article and/or comments would be a huge step up for the community. Or even integration with systems like Coq, Agda, Why/Alt-Ergo.
- jessriedel 12y agoI agree. However, I think that the medium of the work is mostly (although obviously not completely) separable from whether papers are collaborative and continuously evolving.
- jedbrown 12y agoNice proposal for living documents, but how do you suggest measuring importance? Journals are a low bar -- it's too easy to publish crap and yet many of the most important papers were rejected by the prestigious journals they were first submitted to. In my opinion, decoupling dissemination (what arXiv provides) from endorsement is desirable. Endorsements should be revokable and non-exclusive.
- jessriedel 12y agoAgreed. I think collaborative documents will ease the transition away from journals as endorsements.
- coliveira 12y agoI think you're mistaken about the quality of journals. They work most of the time, although the anomalies you mention do occur. If it was so easy to publish crap on journals as you suggest, these same journals would not be so prestigious and used as an evaluation tool by university departments worldwide.
- amelius 12y agoI would already consider it an improvement if we could just comment publicly on papers. I'm surprised Google scholar isn't offering something like this already.
- jessriedel 12y agoAnother commenter suggested PubPeer. https://pubpeer.com/ https://pubpeer.com/ I think this might take off in physics if the ArXiv interfaced with, or copied, PubPeer. Doesn't address most of my concerns, though.
- jspaulding 12y agoWe're also working on this as part of ThinkLab http://thinklab.com/publications http://thinklab.com/publications
- capnrefsmmat 12y agoThere's also PubMed Commons, for publications in journals index by PubMed: http://www.ncbi.nlm.nih.gov/pubmedcommons/ http://www.ncbi.nlm.nih.gov/pubmedcommons/ Wouldn't be helpful for physics and math, though.
- christudor 12y agoI think about this a lot. How would you deal with peer review in all this? We don't want a system where academic papers haven't been peer reviewed, do we?
- sjtrny 12y agoAbout arXiv licensing: most people choose the minimal licensing because they are submitting the paper to a conference of journal. If you put the paper in the public domain or under a cc license then you risk the journal or conference rejecting your paper. If journals didn't care about having the exclusive publication rights then I suspect a lot more academics would select more flexible licenses.
- panax 12y agoIt would be very useful if we had access to not only the papers but the actual data behind what is presented in the paper. Like have everybody store their research in the cloud and allow everyone else access to it. Right now much data sits on hard drives in labs at universities and are never seen by anyone but the researchers who write the papers. Eventually the data gets deleted and all that data is lost and all you are left with is the paper. That is a lot of money/effort/information lost. See: http://dx.doi.org/10.1016/j.cub.2013.11.014 http://dx.doi.org/10.1016/j.cub.2013.11.014 ‘We are nonchalantly throwing all of our data into what could become an information black hole.’ -Google's Vincent Cerf http://www.theguardian.com/technology/2015/feb/13/google-boss-warns-forgotten-century-email-photos-vint-cerf http://www.theguardian.com/technology/2015/feb/13/google-bos... Open access to the data would allow others to corroborate it, determine the correctness of the analysis in the paper, perform meta analysis, aggregate it with other similar data. Though there are bound to be privacy concerns, especially with medical research. Science itself has still not undergone the digital revolution and it desperately needs it.
- abecedarius 12y ago"Hypertext Publishing and the Evolution of Knowledge" had ideas in this general area back before the web existed. An excerpt: "One result of all this activity is what amounts to a review article, developed incrementally, thoroughly critiqued, and regularly updated. It takes the form of a hierarchy of topics bottoming out in a hierarchy of result-summaries; disagreements appear as argumentation structures [17]. When new results are accepted, their authors propose modifications to the summary-document; they become visible (to a typical reader) to the extent that they become accepted." http://e-drexler.com/d/06/00/Hypertext/HPEK3.html#anchor316503 http://e-drexler.com/d/06/00/Hypertext/HPEK3.html#anchor3165... (to link to a relatively concrete scenario; the paper as a whole is interesting but some of it had to be just to address the very desirability of something like the web.) Much more recent and also good: Reinventing Discovery by Michael Nielsen.