13 ms·

Retire This Idea: Scientific Knowledge Structured as “Literature”

158 points by jordigh 12y ago

dalke 12y ago
While not essential to the argument, it uses software development as an analogy for scientific publications. Unfortunately, that analogy isn't correct. > Just as the software industry has moved from a "waterfall" process to an "agile" process—from monolithic releases shipped from warehouses of mass-produced disks to over-the-air differential updates—so must academic publishing move from its current read-only model and embrace a process as dynamic, up-to-date, and collaborative as science itself. "Agile" is one name for a long line of software processes, which includes XP, RAD, and Unified Process. Iterative development stretches back to the dawn of software engineering. In modern practice, "agile" seems used for anything which is even a touch more flexible than the strictest of waterfall. Take your process, add a standup meeting or story cards, and poof, you're agile. Nor is "agile" development tied so closely to 'over-the-air differential updates'. The development methodology has very little to do with deployment. Perhaps the easiest example is to look at the various language implementations (Python, Ruby, gcc, etc.). They don't have auto-update. So I guess the question is, are they agile? > But academic literature makes no distinction between citations merely considered significant and ones additionally considered true. What academic literature needs goes deeper than the view of citations as kudos and shout-outs. It needs what software engineers have used for decades: dependency management. > A dependency graph would tell us, at a click, which of the pillars of scientific theory are truly load-bearing. I have no idea of how to make this work. I give as an example the Kabsch algorithm algorithm for RMSD minimization of two molecular structures (see http://en.wikipedia.org/wiki/Kabsch_algorithm http://en.wikipedia.org/wiki/Kabsch_algorithm ). The original paper contained an error that sometimes resulted in a sign error. This was fixed in a subsequent paper. Most cite the first paper, because it introduced the core concepts. Some cite the first paper but actually implement the correct solution from the second paper without even realizing there was a problem, because they combined the principles with their own mathematical understanding rather than duplicating the exact steps in the paper. Still others cite both papers. How does this hypothetical dependency graph incorporate those details? That paper is a simple one, because it really is a single algorithm for a single goal. Many papers introduce multiple concepts, solutions, or observations. It can well be that one of them was wrong while the others were correct, interesting, and influential. For example, in one paper I read the authors did not include a test case because, they said, the source material was invalid. I investigated, and discovered that it was due to a transcription error by the authors of the paper, and not a problem in the source material. The addition of that one data point would have no material impact on the overall paper. But it's still wrong. How does his hypothetical dependency graph capture the multitude of disprovable points in the average paper? How does it capture what the "pillars" are? Bear in mind that there's been over 50 years of attempts to do this, so I think there's no easy answer.
- jordigh 12y ago
  You make some very good points. I was hoping HN would pick up the story, but it didn't. It reminded me a lot of http://marciovm.com/i-want-a-github-of-science/ http://marciovm.com/i-want-a-github-of-science/ edit: Oh, never mind. The story did pick up! I don't get HN's rankings at all. Usually if you don't pick up within the first 30 minutes, the story is forgotten forever.
- jordigh 12y ago
  > How does his hypothetical dependency graph capture the multitude of disprovable points in the average paper? How does it capture what the "pillars" are? How about bug trackers? Papers that are continuously revised as they're published? Scientific prestige based on how often your papers are cited and how active their bug trackers are? That's how we do it in programming. I think the analogies could be stretched to scientific publishing.
  - dalke 12y ago
    Could you be a bit more concrete as I am having a difficult time trying to understand how this would work? To start with, if there is no activity in the bug tracker, doesn't that indicate success in the paper, because it contained no problems? Or should I put in a few silly errors (misplaced commas, typos in the citation list, etc.) in order to increase the bug activity? You suggest that programming prestige is based on the activity of the bug tracker. Can you give some examples? As an admittedly extreme case, TeX has has very few bugs, and has no bug tracker, but is a well known project. Bug trackers only work for active projects. If the main author writes a paper in solid state physics, is awarded a PhD, and gets a job working for Seagate on high density magnetic compounds for storage tapes, then why should the author care about maintaining the paper's bug tracker? But getting back to the topic, the dependency graph for programming is extremely coarse grained. For example, here is a recent bug for Python - "integer overflow in itertools.permutations" - http://bugs.python.org/issue23363 http://bugs.python.org/issue23363 . Given your software dependency graph, can you tell me which programs that depend on Python are affected by that bug? Because that's the sort of thing that Brian Christian (the author of this piece) wants for scientific publications: > A dependency graph would tell us, at a click, which of the pillars of scientific theory are truly load-bearing. And it would tell us, at a click, which other ideas are likely to get swept away with the rubble of a particular theory. An academic publisher worth their salt would, for instance, not only be able to flag articles that have been retracted—that this is not currently standard practice is, again, inexcusable—but would be able to flag articles that depend in some meaningful way on the results of retracted work. How do I indicate "in some meaningful" that my program depends on itertools.permutations() not having an integer overflow?
jgmjgm 12y ago
I what is missing from the article is an understanding of the incentives of academics. Quite often putting a piece of research "to bed" is the goal. Why? Because any good academic has a long list of things they want to get to. Very few people want to get mired in the (inevitable) problems that arise in a paper for eternity. This is why there are review papers that summarize the state of the knowledge at a given point. These articles are useful summaries of what came before and what (at least appear to be) dead ends were found. Maybe I'm missing some point with the author's article. But getting units of production out and finished (i.e. papers) is a useful process. I'm not sure what would be gained by keeping documents editable forever. Scientific literature is not code.
- return0 12y ago
  Most of the great scientists worked on a single idea their entire life and expanded upon it, improved it, corrected it and were never really done with it (einstein comes to mind). Scientific knowledge is never final I personally have a huge beef with the way life scientists publish their results in tiny tiny bits, that makes it extremely hard to cross-check them with other studies, to find out if they were later disproved or even to figure out the average of some quantity. Try making a concrete model of something and you will find yourself searching for experimentally measured values in tomes among the blabber of introductions and discussions, like monks did in the middle ages... Databases sometimes exist, but they are not mandatory so you never know if they are to be trusted. We need a structured way to catalog scientific results , and journals are not it (esp. with politics which lead scientists to publish in marginally related journals ).
  - dluan 12y ago
    That's not necessarily true, and I'd be incredibly worried if it were true. I think it's more an artifact of the design of this version of 'science' as an institution. Freeman Dyson was a brilliant drop out mathematician who happened to meet Richard Feynman and demonstrated a proof for Feynman's work (which led him to winning a Nobel). He says, "I think it's almost true without exception if you want to win a Nobel Prize, you should have a long attention span, get hold of some deep and important problem and stay with it for ten years. That wasn't my style." Here's a list of Dyson's awards: Heineman Prize (1965), Lorentz Medal (1966), Hughes Medal (1968), Harvey Prize (1977), Wolf Prize (1981), Andrew Gemant Award (1988), Matteucci Medal (1989), Oersted Medal (1991), Fermi Award (1993), Templeton Prize (2000), Pomeranchuk Prize (2003), Poincaré Prize (2012).
peter303 12y ago
The title made think this article was going to be about how to write science. Writing of all kind benefits if you incorporate elements of a story.
xiaq 12y ago
Cool ideas. We will probably end up with a better world if someone implements them, but I would argue that these "agile" things are much less important in scientific publications than in software development. Science and technology are different. Software artifacts are huge and uncontrollable. Today it is very rare for a software developer to know all the ins and outs of a software he/she develops, along with all its dependencies. There is simply way too much code, and it is impossible to control them manually. On the other hand, scientific papers tend to contain very few core ideas. It is still reasonable today for a researcher to understand everything he/she publishes, along with every idea in all the papers he/she cites. Once you get the core idea, you get the whole paper. They merely act as mediums for conveying ideas and there are not that many of them. Often you can just "manage" them with your own brain. Anyway, I resonate with the first few paragraphs about retraction a lot. It is frustrating to read a paper for a whole day, find out that it is wrong, and that someone already refuted it in another paper :-/
- dalke 12y ago
  What about scientific papers that depend on software artifacts? Either scientists somehow understands all the "ins and outs of a software he/she develops, along with all its dependencies" in order to publish a paper, where a programmer doesn't have the same understanding, or you are comparing two entirely different concepts as if they were the same.
Trombone12 12y ago
First of all, there exists "living review" journals, where authors are mandated to regularly update their texts every now and then to reflect new developments in the field covered. This seems like something the author ought to really comment on. Secondly, while research and software development have many similarities, the idea that academic research is best conducted with methodologies created to allow efficient development of CRUD applications which nice interfaces is not very credible.
ThomPete 12y ago
Yes. Scientific knowledge structured as literature means scientific knowledge as subject fields which basically means a lot of lacking knowledge between these fields. It's not that we should not have specific categories but rather that we should have a lot more than we have right now. "Removing literature" as a framework for knowledge will mean the flourishing of knowledge that can only be created by combining several fields.
mhuffman 12y ago
There has also been a big push to only count "recent" scientific literature as valid for reference in some scientific fields. I know that this helps keep any new papers "fresh", but I have always had a bitter aftertaste about the way that still-valid, but older, research is looked down upon.
tikhonj 12y ago
Personally, what really gets me with the current publishing regime is how fashion-driven it is (at least in many parts of CS). Oftentimes, it feels like the most challenging aspect of producing publishable research at first is not rigor or novelty or writing a sufficiently readable complete paper but hitting something sufficiently "interesting"—largely a function of what happens to be popular at the moment. Publishing has other issues, but they can be mitigated by releasing results in forms besides simple papers. (Again, I'm mostly talking about CS.) A lot of successful projects end up releasing blog posts, enhanced online resources, libraries or complete, living open source projects. At least in CS, this is happening enough to be useful even if it is by no means universal. This even comes up in more theoretical fields—somebody recently pointed me to a website[1] on tree edit distance, which is a great example. (I really wish it had existed a few years ago when I needed the algorithm originally!) [1]: http://www.inf.unibz.it/dis/projects/tree-edit-distance/ http://www.inf.unibz.it/dis/projects/tree-edit-distance/
- jonlucc 12y ago
  In biology, there are a large number of very specific niche journals such that a paper on a topic out of vogue is likely to land in a more specific journal. This is not without drawbacks, but it sort of works.
nopinsight 12y ago
One possibility is to keep the current publishing model but add an additional "meta" layer in a central repository to indicate the relations between the contributions of each paper. The repository should be independent of an individual journal or academic field so that multidisciplinary collaboration would be easier and more visible. If designed well, cross-pollination of ideas between fields may multiply. Each node in the dependency graph should not represent an entire paper, but a chunk of contribution (knowledge, information, or experimental data), with an option to delve deeper to see its source code/raw data. A hierarchy of compositional structures can represent broader units of knowledge. Citation links should allow for labels to indicate the reason(s) for citation: prior work, refutation, tool, methodology, etc. This is a more practical transition than revamping the whole system and should still add much valuable information for researchers and consumers of research. If realized, an automated system may be able to analyze the graph for additional insights into our knowledge structure and point to interesting research directions as well.
- tormeh 12y ago
  Sciencepedia, with hyperlinks.
justbecause 12y ago
I like the concept, however I'm afraid "Going Agile" is not a solution for science, it's just more jargon. Would be nice to see science testing several different methods though. Also, it feels like some areas of discipline might work better with different workflows. For instance theoretical physics may need a different process than structural engineering or chemistry. I think it's also weird that we currently use very similar processes across disciplines when there are surely better tools that could be made specifically for each discipline with interoperability between disciples handled on another level. The right tools for the job would hopefully lead to more science faster.
entee 12y ago
Speaking from the perspective of life science, I think the idea of the "article" and "review" as two separate complimentary parts of constructing knowledge actually makes a lot of sense. An individual article is an argument, with evidence for how some process works. A review looks at all the papers, getting the more global view of "knowledge" the OP seems to want. Perhaps in the spirit of the article we should have reviews that are ongoing, sort of like wiki pages for various research areas. These might capture a living, complete state of the field better than current review articles. It's really unclear how this would be done though because: 1.) How fields and areas interact is constantly changing, how do you manage the merging and splitting of review articles? You end up with a confusing view of "what we know" once again. 2.) As others have noted, who gets to write these things? The cutting edge literature is full of big personalities in conflict with one another. It takes years to figure out who is actually right. That said, at a minimum retractions should be much easier to track. I'll bet we'd save a whole bunch of wasted effort.
- tormeh 12y ago
  Well, now Wikipedia is the review. Depending on the field, you may just do all the real research on Wikipedia and then add in citations in your paper to look professional. How about a Wikipedia with original research? "Sciencepedia"? Each article gets a section for each editor. If you want to contribute you can either submit a pull request to an editor you view favourably or you can write your own section. Sections can be voted up or down.
  - jszymborski 12y ago
    Wikipedia is not the review, not in the life sciences sense of the word. A review, despite "reviewing" the literature, offers a unique perspective, and does have an argument, although one that is based solely on the aggregation of existing evidence and literature. Wikipedia just describes parts of the field, reviews offer insight and perspective.
  - return0 12y ago
    There is scholarpedia, which was initially crippled by the fact that each article (which was similar to a review) would often be limited to the (invited and authoritative) author's point of view. In fact they have iterated their policies . Still their contents are often adapted copies of review articles published in "real" journals.
brchr 12y ago
Author here. Delighted to see this near the top of HN this afternoon, and to see the healthy discussion it's provoked. Happy to answer any questions as best I can if you've got 'em.
- tokai 12y ago
  Are you familiar with the field of bibliometrics?[0] It is a not-that smalish research area that works on quantitative research studies. Qualifying types of citations have been a pipe dream there for decades. There are a lot of scientific literature on the subject out there. [0] Or sciencemetrics, informetrics and other assorted subfields
  - brchr 12y ago
    I'm aware of portions of that literature, but haven't studied it explicitly. What do you think are the most interesting things happening in that space at the moment? For me one of the most salient things in that space -- albeit a different point than the one raised in the essay -- is the debate about what sorts of metrics are most appropriate (if any) for informing decisions about hiring, tenure, and the like. In some ways it's quite like the be-careful-what-you-wish-for problems any business has in choosing which metrics to care about. For instance, the h-index might shape a researcher's choice of whether to publish something as one larger paper or two smaller ones. As Sam Altman put it, "It really is true that the company will build whatever the CEO decides to measure." I think a similar principle holds for research and scholarship.
- roel_v 12y ago
  Are you a scientist? If so, in what capacity? If not, where do your assumptions come from? My problem with your post is that it seems to be based on a faulty understanding of the actual way things work (compared to some theory on it), and furthermore is a case of 'I'm a software engineer, I can solve anything' without actually understanding the issues. Much like engineer's armchair lawyering, which is always so far off the mark that it's impossible to even intelligently debate, because it's based on so much misinformation.
tehchromic 12y ago
I see his point. What's true of scientific literature is true of all literature. But the solution doesn't match the ambition of the article's title. Charts and graphs, and all the high tech tools available are certainly potentially beneficial, but what's needed is a new product and center of culture, and that's a function of ethics. Those are always defined in terms of the challenge of their day. But tools with out ethics are dangerous indeed.
kriro 12y ago
The two things that are most ridiculous about science in this day and age (imo, I work in the field) 1) Tons of papers get published without the underlying data. Why is there no easy system in place where I have to upload all data from experiments etc. and if the data isn't there to be verified -> questionable research at best (or in the current framework of science autoreject) 2) Paywalls and the entire publishing system. Most science is state funded in some way, shape or form and as such it should be available for free for all citizens. I think publish and improve is an ok model but there's some issues like the author says it's pretty strange how the whole redacting a paper system works. It's also nontrivial to say "well I was wrong in my paper X as shown in papers Y and Z" (some sort of HTTP redirect to X/Z would be in order) Maybe I'm just not aware of it but it would also be interesting to have a huge Dung-style argumentation system of papers for fields (X attacks Y, Z defends Y etc.). Might be a neat project if it doesn't exist.
- cafebeen 12y ago
  These things take time and money to build and maintain though, and there's very little of that to go around unfortunately.
cyphunk 12y ago
> A dependency graph would tell us, at a click, which of the pillars of scientific theory are truly load-bearing. This is the problem with the academic form of knowledge transfer as a whole. It's too dependent on feeding of ego. And the "improvements" discussed here-in only reenforce that. If you boil knowledge down to meme, the necessity to "weight" and define pillars becomes less relevant. Without the ego's you get a remix culture where the Amen Brothers are more highly valued than say some one hit wonder... but not because we've measured them, and because we know that sample was from the Amen Brothers. It's more highly valued not because we know the source. In-fact few would know the music they are listening to has the Amen Brothers sample in it. It is more highly valued because it has taken so many forms and created actual value. We don't need tools to allow for better attribution and knowledge measurement. We need to remove ownership from knowledge as ownership is a hindrance to the transfer and consumption of it.
gojomo 12y ago
Baby steps. Can we just ban using two-columns in PDFs? That single change will speed the rate at which new results can be read and understood.
- IndianAstronaut 12y ago
  Forget PDFs. JOVE and other video scientific journals should become the norm. Much less chance of fabrication and much better replication if you can demonstrate on video.
scythe 12y ago
When all you have is a hammer, everything looks like a nail. The hubris on display here is: "it works for me, it must be right for everyone". What the author is missing is that the article-reference system is actually more flexible than a tree of dependencies. Consider the obvious example: citing Wikipedia. It's a pain, because Wikipedia changes, so you have to link to a certain revision and if you want to know the author you have to trawl the contributions, which nobody does. And this is true of Wikipedia because Wikipedia is a work in progress: it's not meant to be a presentation of knowledge but a repository of knowledge. Science works like that too, but on the blackboards and notebooks and today blogs and comments of the investigators, which lack real start and end dates and are regularly crossed out or replaced. The work is refined for publication because publications are essentially learning materials, they're written and edited to be as comprehensible as possible for any active researcher in the field to read for education or leisure; likewise in order for citations to be useful the cited works must include some background information which goes a long way in keeping the sciences connected, because it's not rare for analogous problems to pop up in unrelated fields, and it is crucial that researchers from outside each others' fields can read each others' work in order for interdisciplinary collaboration of this form to be possible. But all of this background information is just background noise to the people actively working on a problem and that's why it isn't on the blackboard or in the blog post, and it's why terse notations like dummy indices became popular in scratch work. The divide between the didactic and generative arrangements of scientific work is better explained thus: papers are binaries, not source code. Binaries don't go in source control.
fiatjaf 12y ago
Stop writing filler text.
anifow 12y ago
It seems the problem is that the author is trying to create a representation of knowledge, whereas research papers are more of a logging of work done. His approach may be a perfect fit for an experiment in meta-research. You could run periodic reviews of articles released in specific topical areas and create summaries of the findings. Any time those findings change, its a git commit, and you can track this change over time. It's something like Wikipedia, but designed for research. I would not be surprised if this already exists. I see a challenge in figuring out the document structure which will be the most conducive to distributed version control (pull requests, etc), and can provide some insights on a historicalbasis. For example, you could run these summaries retrospectively, say looking at DNA research in the 1960s, and do a separate commit for every key finding through the years. It seems to me that you would pretty much have to specify a specialized coding language for scientific knowledge for this structure to work.
- jonlucc 12y ago
  This sounds like an update to the format of textbooks. They sort of work like this already: as new versions come out, they include the latest revisions in the field. On the other hand, they also serve their publishers by putting out new versions with little or no improvement to continue earning money.
- kijin 12y ago
  Exactly. The problem is that papers aren't worth what the rest of the world think they are. Nobody who is working on the cutting edge of a given field give a damn about the papers that are published in the field's major journals. Papers are little more than permanent records of what people talked about at some conference several months before, and through other informal channels even earlier. By the time they're published, they're already old news. They may be worth some archival value, but that's about it. Unfortunately, the rest of the world thinks of papers as the primary method by which scientists exchange ideas. This is a myth. Sure, there some scientists (mostly in developing countries, or those coming from a different field) who rely on papers to figure out what their colleagues are up to, but if so, that's only a reason to improve real-time communication, not a reason to turn papers into real-time communication tools.
stellographer 12y ago
This solution is like trying to stir a pot of chili with an absinthe spoon. I'm not sure why there's this holy grail of "the unified master version of" whatever. Let me give an example. Say I write a paper on the shape of the non-dark-matter (stellar) density in the milky way by looking at y-type stars and I get an answer x. Now Bob comes along and looks at y2-type stars and gets answer x2. People have the idea that you just go to the first paper, add a footnote to y and x showing alternate values for y2 and x2... But what that doesn't take into account is the fact that I used telescope a (a 10 meter hawaiian behemoth) and pointed it in one beam of the sky for 8 hours to get an ultra-deep pencil; but bob used telescope a2 (a modest 1.8 meter in la palma) that took an all sky survey and only goes very shallow. Now we add this in a footnote. Next, there's a critical difference in the stars we studied. My y-type stars take 8 Gyr just to form, but Bob was using y2-type stars which live anywhere from 100 Myr to 15 Gyr. So I'm looking at the old stars and he's looking at all the stars. Since we know that different age stars live in different parts of the galaxy (old in the halo, young in the disk and bulge), our results are starting to look not as comparable as we thought... but it's minor, we'll add a footnote. But then we realize that, since my old stars are giants and his all age stars are dwarfs, my stars are way brighter than his. Since my telescope is monstrous, and his is a small surveyor, my stars actually end up being observed to a distance 10 times that of his sample. In fact at these distances, the original model is a bad fit and we need to change from a power law to an Einasto profile. Bob can do that too, so our answers are easily comparable, but the Einasto law has more parameters so it would give a worse fit per parameter value than the power law he wanted to use originally... We add an appendix to the paper to explain this bit. Then I notice that Bob's been using infrared data, and in the infrared there's a well known problem separating stars and galaxies in the data on telescope a2. In fact, Bob has to write a whole new section on some probabilistic tests and models he uses to adequately remove these galaxies from his y2 star sample. My telescope, observing in the optical at high resolution, has no such problem, so that section doesn't exist in my paper. Bob looks around awkwardly and stickies a hyperlink to his meta-analysis somewhere in my data section. Then Jill comes along and says she doesn't agree at all with us; she got value x3 using the distribution of dwarf galaxies and if you believe in theory z, then _hers_ is the most accurate answer. And we tell Jill to go write her own fucking paper.
ExpiredLink 12y ago
The author clearly doesn't understand academia and scientific work. > It is time for science to go agile. Goodness gracious!
sam_lowry_ 12y ago
I had a similar article written once on lawmaking. In general, many fields will benefit from the advances in software-defined collaboration. https://www.linkedin.com/pulse/20140814084517-815484-lawmakers-should-learn-from-software-programmers-to-improve-transparency?trk=mp-author-card https://www.linkedin.com/pulse/20140814084517-815484-lawmake...

158 points by jordigh 12y ago

dalke 12y ago

While not essential to the argument, it uses software development as an analogy for scientific publications. Unfortunately, that analogy isn't correct. > Just as the software industry has moved from a "waterfall" process to an "agile" process—from monolithic releases shipped from warehouses of mass-produced disks to over-the-air differential updates—so must academic publishing move from its current read-only model and embrace a process as dynamic, up-to-date, and collaborative as science itself. "Agile" is one name for a long line of software processes, which includes XP, RAD, and Unified Process. Iterative development stretches back to the dawn of software engineering. In modern practice, "agile" seems used for anything which is even a touch more flexible than the strictest of waterfall. Take your process, add a standup meeting or story cards, and poof, you're agile. Nor is "agile" development tied so closely to 'over-the-air differential updates'. The development methodology has very little to do with deployment. Perhaps the easiest example is to look at the various language implementations (Python, Ruby, gcc, etc.). They don't have auto-update. So I guess the question is, are they agile? > But academic literature makes no distinction between citations merely considered significant and ones additionally considered true. What academic literature needs goes deeper than the view of citations as kudos and shout-outs. It needs what software engineers have used for decades: dependency management. > A dependency graph would tell us, at a click, which of the pillars of scientific theory are truly load-bearing. I have no idea of how to make this work. I give as an example the Kabsch algorithm algorithm for RMSD minimization of two molecular structures (see http://en.wikipedia.org/wiki/Kabsch_algorithm http://en.wikipedia.org/wiki/Kabsch_algorithm ). The original paper contained an error that sometimes resulted in a sign error. This was fixed in a subsequent paper. Most cite the first paper, because it introduced the core concepts. Some cite the first paper but actually implement the correct solution from the second paper without even realizing there was a problem, because they combined the principles with their own mathematical understanding rather than duplicating the exact steps in the paper. Still others cite both papers. How does this hypothetical dependency graph incorporate those details? That paper is a simple one, because it really is a single algorithm for a single goal. Many papers introduce multiple concepts, solutions, or observations. It can well be that one of them was wrong while the others were correct, interesting, and influential. For example, in one paper I read the authors did not include a test case because, they said, the source material was invalid. I investigated, and discovered that it was due to a transcription error by the authors of the paper, and not a problem in the source material. The addition of that one data point would have no material impact on the overall paper. But it's still wrong. How does his hypothetical dependency graph capture the multitude of disprovable points in the average paper? How does it capture what the "pillars" are? Bear in mind that there's been over 50 years of attempts to do this, so I think there's no easy answer.

jordigh 12y ago
You make some very good points. I was hoping HN would pick up the story, but it didn't. It reminded me a lot of http://marciovm.com/i-want-a-github-of-science/ http://marciovm.com/i-want-a-github-of-science/ edit: Oh, never mind. The story did pick up! I don't get HN's rankings at all. Usually if you don't pick up within the first 30 minutes, the story is forgotten forever.
jordigh 12y ago
> How does his hypothetical dependency graph capture the multitude of disprovable points in the average paper? How does it capture what the "pillars" are? How about bug trackers? Papers that are continuously revised as they're published? Scientific prestige based on how often your papers are cited and how active their bug trackers are? That's how we do it in programming. I think the analogies could be stretched to scientific publishing.
- dalke 12y ago
  Could you be a bit more concrete as I am having a difficult time trying to understand how this would work? To start with, if there is no activity in the bug tracker, doesn't that indicate success in the paper, because it contained no problems? Or should I put in a few silly errors (misplaced commas, typos in the citation list, etc.) in order to increase the bug activity? You suggest that programming prestige is based on the activity of the bug tracker. Can you give some examples? As an admittedly extreme case, TeX has has very few bugs, and has no bug tracker, but is a well known project. Bug trackers only work for active projects. If the main author writes a paper in solid state physics, is awarded a PhD, and gets a job working for Seagate on high density magnetic compounds for storage tapes, then why should the author care about maintaining the paper's bug tracker? But getting back to the topic, the dependency graph for programming is extremely coarse grained. For example, here is a recent bug for Python - "integer overflow in itertools.permutations" - http://bugs.python.org/issue23363 http://bugs.python.org/issue23363 . Given your software dependency graph, can you tell me which programs that depend on Python are affected by that bug? Because that's the sort of thing that Brian Christian (the author of this piece) wants for scientific publications: > A dependency graph would tell us, at a click, which of the pillars of scientific theory are truly load-bearing. And it would tell us, at a click, which other ideas are likely to get swept away with the rubble of a particular theory. An academic publisher worth their salt would, for instance, not only be able to flag articles that have been retracted—that this is not currently standard practice is, again, inexcusable—but would be able to flag articles that depend in some meaningful way on the results of retracted work. How do I indicate "in some meaningful" that my program depends on itertools.permutations() not having an integer overflow?

jgmjgm 12y ago

I what is missing from the article is an understanding of the incentives of academics. Quite often putting a piece of research "to bed" is the goal. Why? Because any good academic has a long list of things they want to get to. Very few people want to get mired in the (inevitable) problems that arise in a paper for eternity. This is why there are review papers that summarize the state of the knowledge at a given point. These articles are useful summaries of what came before and what (at least appear to be) dead ends were found. Maybe I'm missing some point with the author's article. But getting units of production out and finished (i.e. papers) is a useful process. I'm not sure what would be gained by keeping documents editable forever. Scientific literature is not code.

return0 12y ago
Most of the great scientists worked on a single idea their entire life and expanded upon it, improved it, corrected it and were never really done with it (einstein comes to mind). Scientific knowledge is never final I personally have a huge beef with the way life scientists publish their results in tiny tiny bits, that makes it extremely hard to cross-check them with other studies, to find out if they were later disproved or even to figure out the average of some quantity. Try making a concrete model of something and you will find yourself searching for experimentally measured values in tomes among the blabber of introductions and discussions, like monks did in the middle ages... Databases sometimes exist, but they are not mandatory so you never know if they are to be trusted. We need a structured way to catalog scientific results , and journals are not it (esp. with politics which lead scientists to publish in marginally related journals ).
- dluan 12y ago
  That's not necessarily true, and I'd be incredibly worried if it were true. I think it's more an artifact of the design of this version of 'science' as an institution. Freeman Dyson was a brilliant drop out mathematician who happened to meet Richard Feynman and demonstrated a proof for Feynman's work (which led him to winning a Nobel). He says, "I think it's almost true without exception if you want to win a Nobel Prize, you should have a long attention span, get hold of some deep and important problem and stay with it for ten years. That wasn't my style." Here's a list of Dyson's awards: Heineman Prize (1965), Lorentz Medal (1966), Hughes Medal (1968), Harvey Prize (1977), Wolf Prize (1981), Andrew Gemant Award (1988), Matteucci Medal (1989), Oersted Medal (1991), Fermi Award (1993), Templeton Prize (2000), Pomeranchuk Prize (2003), Poincaré Prize (2012).

peter303 12y ago

The title made think this article was going to be about how to write science. Writing of all kind benefits if you incorporate elements of a story.

xiaq 12y ago

Cool ideas. We will probably end up with a better world if someone implements them, but I would argue that these "agile" things are much less important in scientific publications than in software development. Science and technology are different. Software artifacts are huge and uncontrollable. Today it is very rare for a software developer to know all the ins and outs of a software he/she develops, along with all its dependencies. There is simply way too much code, and it is impossible to control them manually. On the other hand, scientific papers tend to contain very few core ideas. It is still reasonable today for a researcher to understand everything he/she publishes, along with every idea in all the papers he/she cites. Once you get the core idea, you get the whole paper. They merely act as mediums for conveying ideas and there are not that many of them. Often you can just "manage" them with your own brain. Anyway, I resonate with the first few paragraphs about retraction a lot. It is frustrating to read a paper for a whole day, find out that it is wrong, and that someone already refuted it in another paper :-/

dalke 12y ago
What about scientific papers that depend on software artifacts? Either scientists somehow understands all the "ins and outs of a software he/she develops, along with all its dependencies" in order to publish a paper, where a programmer doesn't have the same understanding, or you are comparing two entirely different concepts as if they were the same.

Trombone12 12y ago

First of all, there exists "living review" journals, where authors are mandated to regularly update their texts every now and then to reflect new developments in the field covered. This seems like something the author ought to really comment on. Secondly, while research and software development have many similarities, the idea that academic research is best conducted with methodologies created to allow efficient development of CRUD applications which nice interfaces is not very credible.

ThomPete 12y ago

Yes. Scientific knowledge structured as literature means scientific knowledge as subject fields which basically means a lot of lacking knowledge between these fields. It's not that we should not have specific categories but rather that we should have a lot more than we have right now. "Removing literature" as a framework for knowledge will mean the flourishing of knowledge that can only be created by combining several fields.

mhuffman 12y ago

There has also been a big push to only count "recent" scientific literature as valid for reference in some scientific fields. I know that this helps keep any new papers "fresh", but I have always had a bitter aftertaste about the way that still-valid, but older, research is looked down upon.

tikhonj 12y ago

Personally, what really gets me with the current publishing regime is how fashion-driven it is (at least in many parts of CS). Oftentimes, it feels like the most challenging aspect of producing publishable research at first is not rigor or novelty or writing a sufficiently readable complete paper but hitting something sufficiently "interesting"—largely a function of what happens to be popular at the moment. Publishing has other issues, but they can be mitigated by releasing results in forms besides simple papers. (Again, I'm mostly talking about CS.) A lot of successful projects end up releasing blog posts, enhanced online resources, libraries or complete, living open source projects. At least in CS, this is happening enough to be useful even if it is by no means universal. This even comes up in more theoretical fields—somebody recently pointed me to a website[1] on tree edit distance, which is a great example. (I really wish it had existed a few years ago when I needed the algorithm originally!) [1]: http://www.inf.unibz.it/dis/projects/tree-edit-distance/ http://www.inf.unibz.it/dis/projects/tree-edit-distance/

jonlucc 12y ago
In biology, there are a large number of very specific niche journals such that a paper on a topic out of vogue is likely to land in a more specific journal. This is not without drawbacks, but it sort of works.

nopinsight 12y ago

One possibility is to keep the current publishing model but add an additional "meta" layer in a central repository to indicate the relations between the contributions of each paper. The repository should be independent of an individual journal or academic field so that multidisciplinary collaboration would be easier and more visible. If designed well, cross-pollination of ideas between fields may multiply. Each node in the dependency graph should not represent an entire paper, but a chunk of contribution (knowledge, information, or experimental data), with an option to delve deeper to see its source code/raw data. A hierarchy of compositional structures can represent broader units of knowledge. Citation links should allow for labels to indicate the reason(s) for citation: prior work, refutation, tool, methodology, etc. This is a more practical transition than revamping the whole system and should still add much valuable information for researchers and consumers of research. If realized, an automated system may be able to analyze the graph for additional insights into our knowledge structure and point to interesting research directions as well.

tormeh 12y ago
Sciencepedia, with hyperlinks.

justbecause 12y ago

I like the concept, however I'm afraid "Going Agile" is not a solution for science, it's just more jargon. Would be nice to see science testing several different methods though. Also, it feels like some areas of discipline might work better with different workflows. For instance theoretical physics may need a different process than structural engineering or chemistry. I think it's also weird that we currently use very similar processes across disciplines when there are surely better tools that could be made specifically for each discipline with interoperability between disciples handled on another level. The right tools for the job would hopefully lead to more science faster.

entee 12y ago

Speaking from the perspective of life science, I think the idea of the "article" and "review" as two separate complimentary parts of constructing knowledge actually makes a lot of sense. An individual article is an argument, with evidence for how some process works. A review looks at all the papers, getting the more global view of "knowledge" the OP seems to want. Perhaps in the spirit of the article we should have reviews that are ongoing, sort of like wiki pages for various research areas. These might capture a living, complete state of the field better than current review articles. It's really unclear how this would be done though because: 1.) How fields and areas interact is constantly changing, how do you manage the merging and splitting of review articles? You end up with a confusing view of "what we know" once again. 2.) As others have noted, who gets to write these things? The cutting edge literature is full of big personalities in conflict with one another. It takes years to figure out who is actually right. That said, at a minimum retractions should be much easier to track. I'll bet we'd save a whole bunch of wasted effort.

tormeh 12y ago
Well, now Wikipedia is the review. Depending on the field, you may just do all the real research on Wikipedia and then add in citations in your paper to look professional. How about a Wikipedia with original research? "Sciencepedia"? Each article gets a section for each editor. If you want to contribute you can either submit a pull request to an editor you view favourably or you can write your own section. Sections can be voted up or down.
- jszymborski 12y ago
  Wikipedia is not the review, not in the life sciences sense of the word. A review, despite "reviewing" the literature, offers a unique perspective, and does have an argument, although one that is based solely on the aggregation of existing evidence and literature. Wikipedia just describes parts of the field, reviews offer insight and perspective.
- return0 12y ago
  There is scholarpedia, which was initially crippled by the fact that each article (which was similar to a review) would often be limited to the (invited and authoritative) author's point of view. In fact they have iterated their policies . Still their contents are often adapted copies of review articles published in "real" journals.

brchr 12y ago

Author here. Delighted to see this near the top of HN this afternoon, and to see the healthy discussion it's provoked. Happy to answer any questions as best I can if you've got 'em.

tokai 12y ago
Are you familiar with the field of bibliometrics?[0] It is a not-that smalish research area that works on quantitative research studies. Qualifying types of citations have been a pipe dream there for decades. There are a lot of scientific literature on the subject out there. [0] Or sciencemetrics, informetrics and other assorted subfields
- brchr 12y ago
  I'm aware of portions of that literature, but haven't studied it explicitly. What do you think are the most interesting things happening in that space at the moment? For me one of the most salient things in that space -- albeit a different point than the one raised in the essay -- is the debate about what sorts of metrics are most appropriate (if any) for informing decisions about hiring, tenure, and the like. In some ways it's quite like the be-careful-what-you-wish-for problems any business has in choosing which metrics to care about. For instance, the h-index might shape a researcher's choice of whether to publish something as one larger paper or two smaller ones. As Sam Altman put it, "It really is true that the company will build whatever the CEO decides to measure." I think a similar principle holds for research and scholarship.
roel_v 12y ago
Are you a scientist? If so, in what capacity? If not, where do your assumptions come from? My problem with your post is that it seems to be based on a faulty understanding of the actual way things work (compared to some theory on it), and furthermore is a case of 'I'm a software engineer, I can solve anything' without actually understanding the issues. Much like engineer's armchair lawyering, which is always so far off the mark that it's impossible to even intelligently debate, because it's based on so much misinformation.

tehchromic 12y ago

I see his point. What's true of scientific literature is true of all literature. But the solution doesn't match the ambition of the article's title. Charts and graphs, and all the high tech tools available are certainly potentially beneficial, but what's needed is a new product and center of culture, and that's a function of ethics. Those are always defined in terms of the challenge of their day. But tools with out ethics are dangerous indeed.

kriro 12y ago

The two things that are most ridiculous about science in this day and age (imo, I work in the field) 1) Tons of papers get published without the underlying data. Why is there no easy system in place where I have to upload all data from experiments etc. and if the data isn't there to be verified -> questionable research at best (or in the current framework of science autoreject) 2) Paywalls and the entire publishing system. Most science is state funded in some way, shape or form and as such it should be available for free for all citizens. I think publish and improve is an ok model but there's some issues like the author says it's pretty strange how the whole redacting a paper system works. It's also nontrivial to say "well I was wrong in my paper X as shown in papers Y and Z" (some sort of HTTP redirect to X/Z would be in order) Maybe I'm just not aware of it but it would also be interesting to have a huge Dung-style argumentation system of papers for fields (X attacks Y, Z defends Y etc.). Might be a neat project if it doesn't exist.

cafebeen 12y ago
These things take time and money to build and maintain though, and there's very little of that to go around unfortunately.

cyphunk 12y ago

> A dependency graph would tell us, at a click, which of the pillars of scientific theory are truly load-bearing. This is the problem with the academic form of knowledge transfer as a whole. It's too dependent on feeding of ego. And the "improvements" discussed here-in only reenforce that. If you boil knowledge down to meme, the necessity to "weight" and define pillars becomes less relevant. Without the ego's you get a remix culture where the Amen Brothers are more highly valued than say some one hit wonder... but not because we've measured them, and because we know that sample was from the Amen Brothers. It's more highly valued not because we know the source. In-fact few would know the music they are listening to has the Amen Brothers sample in it. It is more highly valued because it has taken so many forms and created actual value. We don't need tools to allow for better attribution and knowledge measurement. We need to remove ownership from knowledge as ownership is a hindrance to the transfer and consumption of it.

gojomo 12y ago

Baby steps. Can we just ban using two-columns in PDFs? That single change will speed the rate at which new results can be read and understood.

IndianAstronaut 12y ago
Forget PDFs. JOVE and other video scientific journals should become the norm. Much less chance of fabrication and much better replication if you can demonstrate on video.

scythe 12y ago

When all you have is a hammer, everything looks like a nail. The hubris on display here is: "it works for me, it must be right for everyone". What the author is missing is that the article-reference system is actually more flexible than a tree of dependencies. Consider the obvious example: citing Wikipedia. It's a pain, because Wikipedia changes, so you have to link to a certain revision and if you want to know the author you have to trawl the contributions, which nobody does. And this is true of Wikipedia because Wikipedia is a work in progress: it's not meant to be a presentation of knowledge but a repository of knowledge. Science works like that too, but on the blackboards and notebooks and today blogs and comments of the investigators, which lack real start and end dates and are regularly crossed out or replaced. The work is refined for publication because publications are essentially learning materials, they're written and edited to be as comprehensible as possible for any active researcher in the field to read for education or leisure; likewise in order for citations to be useful the cited works must include some background information which goes a long way in keeping the sciences connected, because it's not rare for analogous problems to pop up in unrelated fields, and it is crucial that researchers from outside each others' fields can read each others' work in order for interdisciplinary collaboration of this form to be possible. But all of this background information is just background noise to the people actively working on a problem and that's why it isn't on the blackboard or in the blog post, and it's why terse notations like dummy indices became popular in scratch work. The divide between the didactic and generative arrangements of scientific work is better explained thus: papers are binaries, not source code. Binaries don't go in source control.

fiatjaf 12y ago

Stop writing filler text.

anifow 12y ago

It seems the problem is that the author is trying to create a representation of knowledge, whereas research papers are more of a logging of work done. His approach may be a perfect fit for an experiment in meta-research. You could run periodic reviews of articles released in specific topical areas and create summaries of the findings. Any time those findings change, its a git commit, and you can track this change over time. It's something like Wikipedia, but designed for research. I would not be surprised if this already exists. I see a challenge in figuring out the document structure which will be the most conducive to distributed version control (pull requests, etc), and can provide some insights on a historicalbasis. For example, you could run these summaries retrospectively, say looking at DNA research in the 1960s, and do a separate commit for every key finding through the years. It seems to me that you would pretty much have to specify a specialized coding language for scientific knowledge for this structure to work.

jonlucc 12y ago
This sounds like an update to the format of textbooks. They sort of work like this already: as new versions come out, they include the latest revisions in the field. On the other hand, they also serve their publishers by putting out new versions with little or no improvement to continue earning money.
kijin 12y ago
Exactly. The problem is that papers aren't worth what the rest of the world think they are. Nobody who is working on the cutting edge of a given field give a damn about the papers that are published in the field's major journals. Papers are little more than permanent records of what people talked about at some conference several months before, and through other informal channels even earlier. By the time they're published, they're already old news. They may be worth some archival value, but that's about it. Unfortunately, the rest of the world thinks of papers as the primary method by which scientists exchange ideas. This is a myth. Sure, there some scientists (mostly in developing countries, or those coming from a different field) who rely on papers to figure out what their colleagues are up to, but if so, that's only a reason to improve real-time communication, not a reason to turn papers into real-time communication tools.

stellographer 12y ago

This solution is like trying to stir a pot of chili with an absinthe spoon. I'm not sure why there's this holy grail of "the unified master version of" whatever. Let me give an example. Say I write a paper on the shape of the non-dark-matter (stellar) density in the milky way by looking at y-type stars and I get an answer x. Now Bob comes along and looks at y2-type stars and gets answer x2. People have the idea that you just go to the first paper, add a footnote to y and x showing alternate values for y2 and x2... But what that doesn't take into account is the fact that I used telescope a (a 10 meter hawaiian behemoth) and pointed it in one beam of the sky for 8 hours to get an ultra-deep pencil; but bob used telescope a2 (a modest 1.8 meter in la palma) that took an all sky survey and only goes very shallow. Now we add this in a footnote. Next, there's a critical difference in the stars we studied. My y-type stars take 8 Gyr just to form, but Bob was using y2-type stars which live anywhere from 100 Myr to 15 Gyr. So I'm looking at the old stars and he's looking at all the stars. Since we know that different age stars live in different parts of the galaxy (old in the halo, young in the disk and bulge), our results are starting to look not as comparable as we thought... but it's minor, we'll add a footnote. But then we realize that, since my old stars are giants and his all age stars are dwarfs, my stars are way brighter than his. Since my telescope is monstrous, and his is a small surveyor, my stars actually end up being observed to a distance 10 times that of his sample. In fact at these distances, the original model is a bad fit and we need to change from a power law to an Einasto profile. Bob can do that too, so our answers are easily comparable, but the Einasto law has more parameters so it would give a worse fit per parameter value than the power law he wanted to use originally... We add an appendix to the paper to explain this bit. Then I notice that Bob's been using infrared data, and in the infrared there's a well known problem separating stars and galaxies in the data on telescope a2. In fact, Bob has to write a whole new section on some probabilistic tests and models he uses to adequately remove these galaxies from his y2 star sample. My telescope, observing in the optical at high resolution, has no such problem, so that section doesn't exist in my paper. Bob looks around awkwardly and stickies a hyperlink to his meta-analysis somewhere in my data section. Then Jill comes along and says she doesn't agree at all with us; she got value x3 using the distribution of dwarf galaxies and if you believe in theory z, then _hers_ is the most accurate answer. And we tell Jill to go write her own fucking paper.

ExpiredLink 12y ago

The author clearly doesn't understand academia and scientific work. > It is time for science to go agile. Goodness gracious!

sam_lowry_ 12y ago

I had a similar article written once on lawmaking. In general, many fields will benefit from the advances in software-defined collaboration. https://www.linkedin.com/pulse/20140814084517-815484-lawmakers-should-learn-from-software-programmers-to-improve-transparency?trk=mp-author-card https://www.linkedin.com/pulse/20140814084517-815484-lawmake...