9 ms·

Practices in source code sharing in astrophysics

73 points by ngoldbaum 13y ago

ngoldbaum 13y ago
As an astronomer, I hope this idea takes off. Unfortunately, judging by the reactions my colleagues have had when I've brought up this idea in the past, I don't think it's very likely.
- jared314 13y ago
  Is it a code quality or a code portability issue?
  - ngoldbaum 13y ago
    As discussed in the linked article, it's mostly because of code quality but with a healthy dose of percieved threat to job security. Unfortunately the academic citation economy isn't very nurturing of community codes, so a lot of work gets repeated when secretive research groups compete. That's not the whole story and the fact that this article is getting published speaks to that, however I still don't think it's likely that things will change in the short term. My friend Matt Turk wrote about a similar topic recently: http://arxiv.org/abs/1301.7064 http://arxiv.org/abs/1301.7064
    - zbyszek 13y ago
      Repetition isn't necessarily a bad thing. Whether compatible results can be obtained with independent analyses of the same data is surely an important indicator of success. Of course that should not preclude the possibility of replication as described here. I find the idea of threats to job security misguided. In the particle physics world some significant code (not everything, granted) has been made available for some time; e.g. Monte-Carlo simulators like Geant or lattice gauge theory codes like those from MILC, SciDAC or FermiQCD. Even highly optimised code has been available. Users are requested to cite the authors. No careers have been harmed, as far as I can tell. Also, the code itself is not much use for replicating results without making the data available too. This can be more problematic politically and technically, but even here there are good precedents in the particle physics world.
    - raverbashing 13y ago
      Unfortunately the world of scientific collaboration follows a strict pattern: The Article Which is good as a way of 'settling information' into a medium but an awful way for today. It's the equivalent of stone tablets compared to what's available today It would be much better if the product of research could be shared in a more flexible format. For example, don't get me started on reading two columns PDFs in a computer screen. Flexible formatting, attachable metadata, reusable "source code" (that could be an excel spreadsheet, a txt list of data, or source code per se), redrawable graphs, etc We need a faster, more flexible, albeit official way of sharing data. Yes, there are ownership issues, IP issues, etc. But maybe having a more easily searchable/traceable system may help with this.
      - nilx 13y ago
        > We need a faster, more flexible, albeit official way of sharing data. Metajournals are trying to do it, with an "Article" overlay to data/software publication : http://metajnl.com/ http://metajnl.com/. IPOL also tries to publish image processing software: http://www.ipol.im/ http://www.ipol.im/. And there is also some work to interlink science assets (more than articles) on http://linkedscience.org/ http://linkedscience.org/. \ I welcome publication models not based on "PDF standard" (ie an electronic clone of the centuries-old ink&paper text article), but simply sharing data/software is not sufficient. I think it is important to have an editorial line and a peer review, otherwise anything online would qualify as scientific communication. Moreover, software and data is hardly sufficient by itself; it needs to be explained, documented, illustrated and discussed to be more than a "harddisk dump".
    - zer0gravity 13y ago
      I think this is a deeper problem that affects the whole scientific academic community and maybe the solution to this, is the growth of a new parallel movement of amateur scientists that are pursuing research from passion and the desire to advance human knowledge, not because their job/position depends on it. In such an environment sharing your tools would probably be the norm.
- A_Allen 13y ago
  I hope the idea takes off, too. Providing codes is imperative; they are part of the methods and should be available for examination to ensure the integrity of the science. Those conducting research funded with public monies should be required by the funding agencies to release the products of that research, not just the results, but the data and codes, too (absent truly compelling reasons, such as national security). Eventually, I expect funding agencies will indeed require this for astronomy, just as they do for some other sciences; journals could help the field along, could improve the transparency and reproducibility of research, by requiring code release upon publication. Absent funding agencies and journals insisting on code release and the moral argument of reproducibility, what incentives would help convince code authors to release their software?
  - nilx 13y ago
    > Absent funding agencies and journals insisting on code release and the moral argument of reproducibility, what incentives would help convince code authors to release their software? Impact Factor? -> Patrick Vandewalle, "Code Sharing Is Associated with Research Impact in Image Processing", CiSE 2012 http://doi.ieeecomputersociety.org/10.1109/MCSE.2012.63 http://doi.ieeecomputersociety.org/10.1109/MCSE.2012.63 It could also be the pressure of colleagues who, as anonymous reviewers, would always ask for the code whenever a paper depends on computation. Journal policies will not switch to REQUIRING the code anytime soon, but peer-review can add some pressure.
crntaylor 13y ago
Have you also seen the Open Exoplanet Catalogue proposal by Hanno Rein? (http://arxiv.org/abs/1211.7121 http://arxiv.org/abs/1211.7121) I have no idea if you know him or not, but if not, you should consider getting in touch!
- teuben 13y ago
  Isn't that exactly what the virtual observatory already has? They base their tables on XML, and call it a VOTable. Lots of fabulous software can read and write those.
anon_barcode 13y ago
As an astronomer I hope this does not take off. The threat to job security is not just "perceived". Astronomers are frequently kept in a state of constant job-induced anxiety by the prevalent practice of fighting for 6 month - 2 year "postdocs" for the first 15-20 years of their careers (seriously, go to an astronomical conference, you would not believe the amount of people greying under 35). To "keep" your job (in reality, get another postdoc) you must do two things: author papers and get papers cited. This practice makes collaboration actually detrimental to a researcher's career unless one of two things happens: 1) they lead the collaboration and get their name as first author 2) the collaboration allows some kind of recompense for the time invested into the collaboration. For this reason, many collaborations have a period of proprietarity, a time when the collaboration data is available exclusively to the people who have sunk their time into the collaboration. Imagine, if the people behind the Millenium or Aquarius simulations were forced to publish their code as soon as they had run the simulations (this is true for any theorist)-- now these people have spent the past years of their lives, tweaking and perfecting code _for which they get no publications or citations_ and before they can even begin to analyze the easiest results ("low hanging fruit", the simplistic papers that usually go to collaboration members), the simulations are being run and analyzed all over the world. They have gained nothing by their efforts in actually developing the code (our world does not work like industry, we don't get a pat on the back and a raise for being team players). For theorists, often entire careers revolve around codes that have been developed over the researcher's whole career-- to be forced to hand it over to all the first year PhDs in the world is a tremendous slap in the face. For me, as an "experimentalist" (read, data-analyst), I also maintain job attractiveness by having sets of code that no one else has. I'm experienced in a variety of pattern finding algorithms, and have even invented a few myself for very specific problems-- and within my little sphere, people know this about me, I'm the person people come to for certain things. I spent years of my life perfecting these, learning about algorithms, learning the intricacies and quirks of the various datasets we use-- if someone wants to take my place in this community as "that guy" then I expect them to devote as much time as I have to learning these techniques inside out and then to do something better than me. I have no desire to pass my code off to a masters student and let him naively plug some dataset into it-- firstly because no one should ever rely on a black box in this field, and secondly because these codes all need to be tweaked to account for the different instruments and data structures being used. Perhaps if my contract didn't have a built in expiration I would be willing to care-bear-share my fractal search methods and spend my Thursday afternoons showing you how to Savitsky-Golay smoothing works, but as long as I'm out of a job next July, and you're the guy applying for my job, you can keep your filthy hands off my code.
deleted 13y ago
[deleted]
gituliar 13y ago
I'm a particle physics theorist. In our field there are a lot of open-source (see http://www.hepforge.org/projects http://www.hepforge.org/projects), as well as some private projects. I believe that source code should be published/shared in tandem with results it produces. Otherwise, those results have no scientific value and I personaly tend not to trust them, even though they are in agreement with well-known predictions. What matters is an algorithm results are obtained with, not results themselves. In other words, scientists should form a hacker-like community and don't get into the trap of a developer-user relationships where some develop while others use software. I actually wanted to know your opinion concerning the migration of open-source to private software. The most common open-source and free software licenses, like MIT or GNU, allow this kind of migration. That is Bob could modify Alice's public project, produce results, and publish them without sharing sources with anybody. That looks unfair, since Alice did a great job and most likely would like to see the changes. How to protect Alice from such situations? Are there any kind of license which forces to share modified software when some results it produces are published or available to the general public?
- tripzilch 13y ago
  Isn't that what GPL does?
  - gituliar 13y ago
    I think that no. GPL permits to keep one's changes private and doesn't force to make them public, that is bad in the case of scientific software. However, it forbids to distribute changes under the terms of other licenses, i.e. non-free, that is good. I believe that an enforcement to share source code should be imposed on provate modifications of open-source scientific software.
Blahah 13y ago
I'm a computational biologist. If I read a paper where the results of an analysis are presented, but the code used is not, that analysis is worthless. Thankfully, in computational biology & bioinformatics at least, open source is widespread and growing. Every programmer knows that every programmer makes mistakes. Lots of them. Mistakes that can completely change the outcome of an analysis. And if the results of your analysis are to be taken seriously, people have to be able to trust them, which means they have to be able to check your working. Not publishing your code is anti-science, selfish, and IMHO should be disallowed in the literature.
- anon_barcode 13y ago
  In astronomy we are still fighting for public data. Verification of results comes from another researcher collecting his own data and running his own analysis. Out of curiosity, do you work for industry or academia?
  - Blahah 13y ago
    That's an extremely inefficient and costly way to validate a result! It would be rendered unnecessary in cases where just looking at the code invalidated the result. I'm in academia (see my profile for details).
    - J_Thomas 13y ago
      It's good and useful for independent researchers to collect their own data and analyse it. That's a check on the data collection methods. It's also useful for others to check the analysis of trusted data. If they do it with their own methods and get a different result, then it's time to actually compare the methods. It's a good thing for the original researchers to release their code too. But looking for bugs in the code is no substitute for doing independent analysis and independent data collection. It's an important supplement, but it isn't enough.
      - Blahah 13y ago
        I agree with your points. The saving in the case of released source code is that, if the original analysis is flawed in such a way that there's no need to repeat it because it's wrong, then a large amount of expense and effort can be saved. Of course, if the original analysis holds up given the data and code, independent replication is a necessary thing.
lutusp 13y ago
> While software and algorithms have become increasingly important in astronomy, the majority of authors who publish computational astronomy research do not share the source code they develop, making it difficult to replicate and reuse the work. This is troubling. There's no field involving computation in which withholding source code is routinely accepted. The four-color map theorem proof (Appel & Haken, 1976) would never have been accepted without source code. Modern mathematics, to the degree to which it relies on computer results, also relies on the publication of source. Another example is the recent revelation involving an error in an Excel spreadsheet and its effect on an economic analysis -- the correction wouldn't have been possible without publication of the spreadsheet alongside the conclusions drawn from it. Also, replication is a cornerstone of serious science. Without replication, astrophysics becomes psychology, where replication is rare. I hope this paper has the effect of correcting this systematic flaw in astrophysics publication.

73 points by ngoldbaum 13y ago

As an astronomer, I hope this idea takes off. Unfortunately, judging by the reactions my colleagues have had when I've brought up this idea in the past, I don't think it's very likely.

jared314 13y ago
Is it a code quality or a code portability issue?
- ngoldbaum 13y ago
  As discussed in the linked article, it's mostly because of code quality but with a healthy dose of percieved threat to job security. Unfortunately the academic citation economy isn't very nurturing of community codes, so a lot of work gets repeated when secretive research groups compete. That's not the whole story and the fact that this article is getting published speaks to that, however I still don't think it's likely that things will change in the short term. My friend Matt Turk wrote about a similar topic recently: http://arxiv.org/abs/1301.7064 http://arxiv.org/abs/1301.7064
  - zbyszek 13y ago
    Repetition isn't necessarily a bad thing. Whether compatible results can be obtained with independent analyses of the same data is surely an important indicator of success. Of course that should not preclude the possibility of replication as described here. I find the idea of threats to job security misguided. In the particle physics world some significant code (not everything, granted) has been made available for some time; e.g. Monte-Carlo simulators like Geant or lattice gauge theory codes like those from MILC, SciDAC or FermiQCD. Even highly optimised code has been available. Users are requested to cite the authors. No careers have been harmed, as far as I can tell. Also, the code itself is not much use for replicating results without making the data available too. This can be more problematic politically and technically, but even here there are good precedents in the particle physics world.
  - raverbashing 13y ago
    Unfortunately the world of scientific collaboration follows a strict pattern: The Article Which is good as a way of 'settling information' into a medium but an awful way for today. It's the equivalent of stone tablets compared to what's available today It would be much better if the product of research could be shared in a more flexible format. For example, don't get me started on reading two columns PDFs in a computer screen. Flexible formatting, attachable metadata, reusable "source code" (that could be an excel spreadsheet, a txt list of data, or source code per se), redrawable graphs, etc We need a faster, more flexible, albeit official way of sharing data. Yes, there are ownership issues, IP issues, etc. But maybe having a more easily searchable/traceable system may help with this.
    - nilx 13y ago
      > We need a faster, more flexible, albeit official way of sharing data. Metajournals are trying to do it, with an "Article" overlay to data/software publication : http://metajnl.com/ http://metajnl.com/. IPOL also tries to publish image processing software: http://www.ipol.im/ http://www.ipol.im/. And there is also some work to interlink science assets (more than articles) on http://linkedscience.org/ http://linkedscience.org/. \ I welcome publication models not based on "PDF standard" (ie an electronic clone of the centuries-old ink&paper text article), but simply sharing data/software is not sufficient. I think it is important to have an editorial line and a peer review, otherwise anything online would qualify as scientific communication. Moreover, software and data is hardly sufficient by itself; it needs to be explained, documented, illustrated and discussed to be more than a "harddisk dump".
  - zer0gravity 13y ago
    I think this is a deeper problem that affects the whole scientific academic community and maybe the solution to this, is the growth of a new parallel movement of amateur scientists that are pursuing research from passion and the desire to advance human knowledge, not because their job/position depends on it. In such an environment sharing your tools would probably be the norm.
A_Allen 13y ago
I hope the idea takes off, too. Providing codes is imperative; they are part of the methods and should be available for examination to ensure the integrity of the science. Those conducting research funded with public monies should be required by the funding agencies to release the products of that research, not just the results, but the data and codes, too (absent truly compelling reasons, such as national security). Eventually, I expect funding agencies will indeed require this for astronomy, just as they do for some other sciences; journals could help the field along, could improve the transparency and reproducibility of research, by requiring code release upon publication. Absent funding agencies and journals insisting on code release and the moral argument of reproducibility, what incentives would help convince code authors to release their software?
- nilx 13y ago
  > Absent funding agencies and journals insisting on code release and the moral argument of reproducibility, what incentives would help convince code authors to release their software? Impact Factor? -> Patrick Vandewalle, "Code Sharing Is Associated with Research Impact in Image Processing", CiSE 2012 http://doi.ieeecomputersociety.org/10.1109/MCSE.2012.63 http://doi.ieeecomputersociety.org/10.1109/MCSE.2012.63 It could also be the pressure of colleagues who, as anonymous reviewers, would always ask for the code whenever a paper depends on computation. Journal policies will not switch to REQUIRING the code anytime soon, but peer-review can add some pressure.

crntaylor 13y ago

Have you also seen the Open Exoplanet Catalogue proposal by Hanno Rein? (http://arxiv.org/abs/1211.7121 http://arxiv.org/abs/1211.7121) I have no idea if you know him or not, but if not, you should consider getting in touch!

teuben 13y ago
Isn't that exactly what the virtual observatory already has? They base their tables on XML, and call it a VOTable. Lots of fabulous software can read and write those.

anon_barcode 13y ago

As an astronomer I hope this does not take off. The threat to job security is not just "perceived". Astronomers are frequently kept in a state of constant job-induced anxiety by the prevalent practice of fighting for 6 month - 2 year "postdocs" for the first 15-20 years of their careers (seriously, go to an astronomical conference, you would not believe the amount of people greying under 35). To "keep" your job (in reality, get another postdoc) you must do two things: author papers and get papers cited. This practice makes collaboration actually detrimental to a researcher's career unless one of two things happens: 1) they lead the collaboration and get their name as first author 2) the collaboration allows some kind of recompense for the time invested into the collaboration. For this reason, many collaborations have a period of proprietarity, a time when the collaboration data is available exclusively to the people who have sunk their time into the collaboration. Imagine, if the people behind the Millenium or Aquarius simulations were forced to publish their code as soon as they had run the simulations (this is true for any theorist)-- now these people have spent the past years of their lives, tweaking and perfecting code _for which they get no publications or citations_ and before they can even begin to analyze the easiest results ("low hanging fruit", the simplistic papers that usually go to collaboration members), the simulations are being run and analyzed all over the world. They have gained nothing by their efforts in actually developing the code (our world does not work like industry, we don't get a pat on the back and a raise for being team players). For theorists, often entire careers revolve around codes that have been developed over the researcher's whole career-- to be forced to hand it over to all the first year PhDs in the world is a tremendous slap in the face. For me, as an "experimentalist" (read, data-analyst), I also maintain job attractiveness by having sets of code that no one else has. I'm experienced in a variety of pattern finding algorithms, and have even invented a few myself for very specific problems-- and within my little sphere, people know this about me, I'm the person people come to for certain things. I spent years of my life perfecting these, learning about algorithms, learning the intricacies and quirks of the various datasets we use-- if someone wants to take my place in this community as "that guy" then I expect them to devote as much time as I have to learning these techniques inside out and then to do something better than me. I have no desire to pass my code off to a masters student and let him naively plug some dataset into it-- firstly because no one should ever rely on a black box in this field, and secondly because these codes all need to be tweaked to account for the different instruments and data structures being used. Perhaps if my contract didn't have a built in expiration I would be willing to care-bear-share my fractal search methods and spend my Thursday afternoons showing you how to Savitsky-Golay smoothing works, but as long as I'm out of a job next July, and you're the guy applying for my job, you can keep your filthy hands off my code.

deleted 13y ago

[deleted]

gituliar 13y ago

I'm a particle physics theorist. In our field there are a lot of open-source (see http://www.hepforge.org/projects http://www.hepforge.org/projects), as well as some private projects. I believe that source code should be published/shared in tandem with results it produces. Otherwise, those results have no scientific value and I personaly tend not to trust them, even though they are in agreement with well-known predictions. What matters is an algorithm results are obtained with, not results themselves. In other words, scientists should form a hacker-like community and don't get into the trap of a developer-user relationships where some develop while others use software. I actually wanted to know your opinion concerning the migration of open-source to private software. The most common open-source and free software licenses, like MIT or GNU, allow this kind of migration. That is Bob could modify Alice's public project, produce results, and publish them without sharing sources with anybody. That looks unfair, since Alice did a great job and most likely would like to see the changes. How to protect Alice from such situations? Are there any kind of license which forces to share modified software when some results it produces are published or available to the general public?

tripzilch 13y ago
Isn't that what GPL does?
- gituliar 13y ago
  I think that no. GPL permits to keep one's changes private and doesn't force to make them public, that is bad in the case of scientific software. However, it forbids to distribute changes under the terms of other licenses, i.e. non-free, that is good. I believe that an enforcement to share source code should be imposed on provate modifications of open-source scientific software.

Blahah 13y ago

I'm a computational biologist. If I read a paper where the results of an analysis are presented, but the code used is not, that analysis is worthless. Thankfully, in computational biology & bioinformatics at least, open source is widespread and growing. Every programmer knows that every programmer makes mistakes. Lots of them. Mistakes that can completely change the outcome of an analysis. And if the results of your analysis are to be taken seriously, people have to be able to trust them, which means they have to be able to check your working. Not publishing your code is anti-science, selfish, and IMHO should be disallowed in the literature.

anon_barcode 13y ago
In astronomy we are still fighting for public data. Verification of results comes from another researcher collecting his own data and running his own analysis. Out of curiosity, do you work for industry or academia?
- Blahah 13y ago
  That's an extremely inefficient and costly way to validate a result! It would be rendered unnecessary in cases where just looking at the code invalidated the result. I'm in academia (see my profile for details).
  - J_Thomas 13y ago
    It's good and useful for independent researchers to collect their own data and analyse it. That's a check on the data collection methods. It's also useful for others to check the analysis of trusted data. If they do it with their own methods and get a different result, then it's time to actually compare the methods. It's a good thing for the original researchers to release their code too. But looking for bugs in the code is no substitute for doing independent analysis and independent data collection. It's an important supplement, but it isn't enough.
    - Blahah 13y ago
      I agree with your points. The saving in the case of released source code is that, if the original analysis is flawed in such a way that there's no need to repeat it because it's wrong, then a large amount of expense and effort can be saved. Of course, if the original analysis holds up given the data and code, independent replication is a necessary thing.

lutusp 13y ago

> While software and algorithms have become increasingly important in astronomy, the majority of authors who publish computational astronomy research do not share the source code they develop, making it difficult to replicate and reuse the work. This is troubling. There's no field involving computation in which withholding source code is routinely accepted. The four-color map theorem proof (Appel & Haken, 1976) would never have been accepted without source code. Modern mathematics, to the degree to which it relies on computer results, also relies on the publication of source. Another example is the recent revelation involving an error in an Excel spreadsheet and its effect on an economic analysis -- the correction wouldn't have been possible without publication of the spreadsheet alongside the conclusions drawn from it. Also, replication is a cornerstone of serious science. Without replication, astrophysics becomes psychology, where replication is rare. I hope this paper has the effect of correcting this systematic flaw in astrophysics publication.