9 ms·

A Github for scientific code doesn't go nearly far enough. The transition from paper journals to electronic publications has only converted dead-paper into "ele

by rgejman 15y ago

A Github for scientific code doesn't go nearly far enough. The transition from paper journals to electronic publications has only converted dead-paper into "electronic" paper. With some exceptions, e.g. video recordings, animations and supplementary figures/documents/spreadsheets/code, the document that you download from any major science publisher is a PDF that looks almost exactly like the printed publication. Most don't even include links to referenced publications[1]! Today, we know a lot about how to make documents that have complex formatting (think micro formats, links) and even more about making abstract document formats that can be presented and styled in different ways (think XML and stylesheet type separation of data and presentation). Having a standardized scientific publication format (with open-source user or publisher generated extensions as needed) would completely change the way we produce and consume the literature. Imagine the possibilities for meta-analysis! Yes, code should (in most cases) be released together with a paper. But even better would be if the code were released as part of a standardized data format that would allow you to, for instance, selectively download raw data and re-run the computational experiments on your own computer (think: re-running simulations in Mekentosj's Papers as you read the paper). Even simpler (and possibly more useful): provide both low and original (high) resolution versions of figures that can be examined separately from the main document. I can't tell you how many times I've been annoyed by the low quality of the published images and wished I could zoom in to the level of detail I know was in the original image. Even more frustrating: why should I have to take screenshots of images in Preview to add to a figure in my lab meeting. Separate the presentation and the data! [1]although some now include intra-document links from a citation in the text to the reference in the coda

bo1024 15y ago
I completely agree that these things need to be open, but it's difficult to solve all these problems at once. I'd tend to think that the original published pdf should remain as-is, a stand alone document. However, I think the journal (and more importantly, the author's own website) ought to additionally provide a .zip download containing any supplemental materials. It would be nice if these generally included all figures in high-res, all nontrivial source code, and either the data or a link to the data online if it is large.
- rgejman 15y ago
  Right, but you can provide a platform for solving these problems over time if you make publishing work off of an extensible, data-presentation separate document model. PDF isn't it. More directly to your point: I agree that the "telling a story" via a x-number page document is important. I'm not arguing for changing the general method by which scientific results are conveyed from an author to the audience (i.e. description of problem, description of approach, description of results, analysis of results and conclusion). It's just that once I've digested your results, I want to analyze them critically, re-use them, integrate them into my work, etc. Publishers should facilitate that. Why can't I easily copy a list of genes from a table in YOUR paper so that I can check if they are present in MY work. Why don't terms and gene names and genic loci show up as links that take me to the genome browser of my choice when I click on them? These are simple problems to solve once you have a richer document format.
  - bo1024 15y ago
    I guess I just much prefer the idea of a "paper" as published being self-contained, and "supplementary" material being separate. A paper shouldn't become incomplete if it is printed on paper or if the links are broken. > It's just that once I've digested your results, I want to analyze them critically, re-use them, integrate them into my work, etc. Publishers should facilitate that. I completely agree! But I think this is a separate process from just reading the results, and is best done with separate tools. I think all the things in your last paragraph should be easy to do, but I'm not sure if the best way to do them is to embed them in the same document that's published in the journal. For example, suppose I publish the results of an analysis on genomes of 100 fruit flies I raised in the lab. Here are some things you might be interested in acquiring: - the text I published (say as pdf, or some other format) with included figures, references, etc (the thing that appears in the journal) -- say 4 megabytes - the document used to generate the above document (e.g. LaTeX file) -- less than 1 MB - high-resolution pictures of the flies' eyes up close -- say 10 more megabytes - high-resolution plots of data -- say 2 more megabytes - raw data used for these plots and analogous plots not presented -- lists of genes and statistics -- call it a megabyte. - source code used to generate this data from the genomes and plot it -- not more than a megabyte. - raw genomes of all flies in my study -- in the 10s of gigabytes. Now how much of this data do we want to bundle in as part of the original document? Keep in mind that 95% of readers are only interested in the first item on the list -- the 4-MB document that was published in the journal. Should we also embed in it an additional 15 MB of data for those few who might be interested, and make the document somehow interactive so that this data is accessible by clicking? (Of course, we definitely can't embed the gigabytes of raw genome data, so we'll need a separate solution for distributing that anyway.) I'd argue that a better solution is to simply bundle all of this "supplementary" stuff separately in a .zip and make it available for download separately. Again, very large files or datasets will still need their own solution -- for instance, I might host the genomes on my website and just provide you with the link. This isn't to say that pdf itself is the be-all and end-all of portable document formats (although I think it's very good), but I do want to argue against bloating the published report document with what I view as supplementary information, because it's hard to tell where to draw the line, and this imposes a large memory cost on lots of people who don't need it. (As a sidenote, if I want someone's code and data, getting it from within the document by clicking links also seems a bit odd to me -- where in a published article should these links go?) So I really feel like separate downloads is the best solution.
w1ntermute 15y ago
This is a great idea, but one of the issues is that scientists don't like to release raw data - it puts their conclusions at risk of reinterpretation by other scientists. They'd much rather release figures that are formatted such that they only allow one to draw conclusions that support their research.
- kijin 15y ago
  Also, the data could have been collected by unsound methods in the first place, so running the same data through the same computer program won't really amount to reproducing a result.
  - zerostar07 15y ago
    So, it appears this is a fundamental dishonesty with "publishable" science: that it becomes an ego trip for some scientists who like to overstate their results. While a groundbreaking discovery is sure to be scrutinized to the most minute detail by the community, an average "incremental science" paper will not, and will probably be published even if it contains errors. There are many published papers with unintented but grave computational errors that render even their main findings invalid, and oftentimes they are not retracted. This attitude should change, with people admitting that sometimes mistakes are only human, and they do not diminish the contribution of the research. Complete data sharing would be a huge leap for science: Imagine all scientists digging into every experiment ever made for, say, cancer or HIV research and discovering hitherto unknown correlations, new interpretations of the data etc. It would provide huge shortcuts, given how many experiments get essentially repeated over the years.
    - w1ntermute 15y ago
      > it becomes an ego trip for some scientists who like to overstate their results I think you're looking at this the wrong way - while some scientists might indeed be doing it for the "ego trip," the vast majority are just trying to avoid being swallowed whole by the vicious academic research environment. Reaching the position of tenured professor at a major research university is extremely challenging. Now, I'm not suggesting that what they're doing is right. I'm just saying that one needs to dig further to find the root causes of these problems.
toufka 15y ago
biochemist here... I would love to release my data in an open format. But the software I'm using is proprietary - and the format is therefore closed. Here's the problem: Scientific software sucks. It's stuck 20 years ago. Usually a pain in the ass. Barely works. Crashes. Proprietary. Hacked together. And worst of all - ridiculously expensive. And if it's not proprietary, then it was written by me. The script is just good enough to do exactly what I need it to. I've hardcoded file directories and tab-delimiters. If I use a microscope to take images it can be in a Nikon file, a Canon file, a Zeiss file, or a number of other files. None of which are interoperable. TIF, in general, doesn't cut it - it doesn't communicate with the scope and generally can't store all the meta-data that must be kept with the image itself. If I were to dump my raw Nikon files it would help nothing - no one could read them without the $10k software that comes with the scope. Open standards, even for the more commonly used scientific formats, simply don't really exist. And when they do, they're still hacky and ugly. If someone would write some good open-source microscopy, genetic, and basic mathematical software it would make thousands of grad-students' lives easier. Someone PLEASE write a good plasmid viewer (an editor if you're feeling kind) for genbank files. An iPhoto/iTunes for microscopy images. A GUI for basic/common perl/python scripts. Some mathematical software that's not impenetrable (a la MatLab/Mathematica). If these softwares were out there and used, it'd be trivial to dump the raw data and allow anyone to use them. (I'd love to have you guys make me some sweet software. It would take you a relatively short time - these things are standard and already spec'ed out. But sorry - we have no money - we can't really pay you.)
- rflrob 15y ago
  >I would love to release my data in an open format. But the software I'm using is proprietary - and the format is therefore closed. > >Here's the problem: Scientific software sucks. It's stuck 20 years ago. Usually a pain in the ass. Barely works. Crashes. Proprietary. Hacked together. And worst of all - ridiculously expensive. And if it's not proprietary, then it was written by me. The script is just good enough to do exactly what I need it to. I've hardcoded file directories and tab-delimiters. Release the data and code anyways. I feel your pain with the proprietary formats, but ImageJ is pretty good about dealing with bizarre formats (and other scientists are more likely to have access to the $10k software, or at least know someone who does). I feel your pain with the hacked together scripts. After sharing an analysis program with some collaborators, I had to go through about three rounds of testing to figure out all* of the unreliable assumptions I had made based on one particularly well-formed dataset. This process will make your code better, though, and it will make you a better coder. It will also help other people who are, like you and me, muddling through, trying to find any kind of usable starting point. [*] Okay, "some"

by rgejman 15y ago

bo1024 15y ago

I completely agree that these things need to be open, but it's difficult to solve all these problems at once. I'd tend to think that the original published pdf should remain as-is, a stand alone document. However, I think the journal (and more importantly, the author's own website) ought to additionally provide a .zip download containing any supplemental materials. It would be nice if these generally included all figures in high-res, all nontrivial source code, and either the data or a link to the data online if it is large.

rgejman 15y ago
Right, but you can provide a platform for solving these problems over time if you make publishing work off of an extensible, data-presentation separate document model. PDF isn't it. More directly to your point: I agree that the "telling a story" via a x-number page document is important. I'm not arguing for changing the general method by which scientific results are conveyed from an author to the audience (i.e. description of problem, description of approach, description of results, analysis of results and conclusion). It's just that once I've digested your results, I want to analyze them critically, re-use them, integrate them into my work, etc. Publishers should facilitate that. Why can't I easily copy a list of genes from a table in YOUR paper so that I can check if they are present in MY work. Why don't terms and gene names and genic loci show up as links that take me to the genome browser of my choice when I click on them? These are simple problems to solve once you have a richer document format.
- bo1024 15y ago
  I guess I just much prefer the idea of a "paper" as published being self-contained, and "supplementary" material being separate. A paper shouldn't become incomplete if it is printed on paper or if the links are broken. > It's just that once I've digested your results, I want to analyze them critically, re-use them, integrate them into my work, etc. Publishers should facilitate that. I completely agree! But I think this is a separate process from just reading the results, and is best done with separate tools. I think all the things in your last paragraph should be easy to do, but I'm not sure if the best way to do them is to embed them in the same document that's published in the journal. For example, suppose I publish the results of an analysis on genomes of 100 fruit flies I raised in the lab. Here are some things you might be interested in acquiring: - the text I published (say as pdf, or some other format) with included figures, references, etc (the thing that appears in the journal) -- say 4 megabytes - the document used to generate the above document (e.g. LaTeX file) -- less than 1 MB - high-resolution pictures of the flies' eyes up close -- say 10 more megabytes - high-resolution plots of data -- say 2 more megabytes - raw data used for these plots and analogous plots not presented -- lists of genes and statistics -- call it a megabyte. - source code used to generate this data from the genomes and plot it -- not more than a megabyte. - raw genomes of all flies in my study -- in the 10s of gigabytes. Now how much of this data do we want to bundle in as part of the original document? Keep in mind that 95% of readers are only interested in the first item on the list -- the 4-MB document that was published in the journal. Should we also embed in it an additional 15 MB of data for those few who might be interested, and make the document somehow interactive so that this data is accessible by clicking? (Of course, we definitely can't embed the gigabytes of raw genome data, so we'll need a separate solution for distributing that anyway.) I'd argue that a better solution is to simply bundle all of this "supplementary" stuff separately in a .zip and make it available for download separately. Again, very large files or datasets will still need their own solution -- for instance, I might host the genomes on my website and just provide you with the link. This isn't to say that pdf itself is the be-all and end-all of portable document formats (although I think it's very good), but I do want to argue against bloating the published report document with what I view as supplementary information, because it's hard to tell where to draw the line, and this imposes a large memory cost on lots of people who don't need it. (As a sidenote, if I want someone's code and data, getting it from within the document by clicking links also seems a bit odd to me -- where in a published article should these links go?) So I really feel like separate downloads is the best solution.

w1ntermute 15y ago

This is a great idea, but one of the issues is that scientists don't like to release raw data - it puts their conclusions at risk of reinterpretation by other scientists. They'd much rather release figures that are formatted such that they only allow one to draw conclusions that support their research.

kijin 15y ago
Also, the data could have been collected by unsound methods in the first place, so running the same data through the same computer program won't really amount to reproducing a result.
- zerostar07 15y ago
  So, it appears this is a fundamental dishonesty with "publishable" science: that it becomes an ego trip for some scientists who like to overstate their results. While a groundbreaking discovery is sure to be scrutinized to the most minute detail by the community, an average "incremental science" paper will not, and will probably be published even if it contains errors. There are many published papers with unintented but grave computational errors that render even their main findings invalid, and oftentimes they are not retracted. This attitude should change, with people admitting that sometimes mistakes are only human, and they do not diminish the contribution of the research. Complete data sharing would be a huge leap for science: Imagine all scientists digging into every experiment ever made for, say, cancer or HIV research and discovering hitherto unknown correlations, new interpretations of the data etc. It would provide huge shortcuts, given how many experiments get essentially repeated over the years.
  - w1ntermute 15y ago
    > it becomes an ego trip for some scientists who like to overstate their results I think you're looking at this the wrong way - while some scientists might indeed be doing it for the "ego trip," the vast majority are just trying to avoid being swallowed whole by the vicious academic research environment. Reaching the position of tenured professor at a major research university is extremely challenging. Now, I'm not suggesting that what they're doing is right. I'm just saying that one needs to dig further to find the root causes of these problems.

toufka 15y ago

biochemist here... I would love to release my data in an open format. But the software I'm using is proprietary - and the format is therefore closed. Here's the problem: Scientific software sucks. It's stuck 20 years ago. Usually a pain in the ass. Barely works. Crashes. Proprietary. Hacked together. And worst of all - ridiculously expensive. And if it's not proprietary, then it was written by me. The script is just good enough to do exactly what I need it to. I've hardcoded file directories and tab-delimiters. If I use a microscope to take images it can be in a Nikon file, a Canon file, a Zeiss file, or a number of other files. None of which are interoperable. TIF, in general, doesn't cut it - it doesn't communicate with the scope and generally can't store all the meta-data that must be kept with the image itself. If I were to dump my raw Nikon files it would help nothing - no one could read them without the $10k software that comes with the scope. Open standards, even for the more commonly used scientific formats, simply don't really exist. And when they do, they're still hacky and ugly. If someone would write some good open-source microscopy, genetic, and basic mathematical software it would make thousands of grad-students' lives easier. Someone PLEASE write a good plasmid viewer (an editor if you're feeling kind) for genbank files. An iPhoto/iTunes for microscopy images. A GUI for basic/common perl/python scripts. Some mathematical software that's not impenetrable (a la MatLab/Mathematica). If these softwares were out there and used, it'd be trivial to dump the raw data and allow anyone to use them. (I'd love to have you guys make me some sweet software. It would take you a relatively short time - these things are standard and already spec'ed out. But sorry - we have no money - we can't really pay you.)

rflrob 15y ago
>I would love to release my data in an open format. But the software I'm using is proprietary - and the format is therefore closed. > >Here's the problem: Scientific software sucks. It's stuck 20 years ago. Usually a pain in the ass. Barely works. Crashes. Proprietary. Hacked together. And worst of all - ridiculously expensive. And if it's not proprietary, then it was written by me. The script is just good enough to do exactly what I need it to. I've hardcoded file directories and tab-delimiters. Release the data and code anyways. I feel your pain with the proprietary formats, but ImageJ is pretty good about dealing with bizarre formats (and other scientists are more likely to have access to the $10k software, or at least know someone who does). I feel your pain with the hacked together scripts. After sharing an analysis program with some collaborators, I had to go through about three rounds of testing to figure out all* of the unreliable assumptions I had made based on one particularly well-formed dataset. This process will make your code better, though, and it will make you a better coder. It will also help other people who are, like you and me, muddling through, trying to find any kind of usable starting point. [*] Okay, "some"