3 ms·

As an astronomer, I hope this idea takes off. Unfortunately, judging by the reactions my colleagues have had when I've brought up this idea in the past, I don'

As an astronomer, I hope this idea takes off. Unfortunately, judging by the reactions my colleagues have had when I've brought up this idea in the past, I don't think it's very likely.

jared314 13y ago
Is it a code quality or a code portability issue?
- ngoldbaum 13y ago
  As discussed in the linked article, it's mostly because of code quality but with a healthy dose of percieved threat to job security. Unfortunately the academic citation economy isn't very nurturing of community codes, so a lot of work gets repeated when secretive research groups compete. That's not the whole story and the fact that this article is getting published speaks to that, however I still don't think it's likely that things will change in the short term. My friend Matt Turk wrote about a similar topic recently: http://arxiv.org/abs/1301.7064 http://arxiv.org/abs/1301.7064
  - zbyszek 13y ago
    Repetition isn't necessarily a bad thing. Whether compatible results can be obtained with independent analyses of the same data is surely an important indicator of success. Of course that should not preclude the possibility of replication as described here. I find the idea of threats to job security misguided. In the particle physics world some significant code (not everything, granted) has been made available for some time; e.g. Monte-Carlo simulators like Geant or lattice gauge theory codes like those from MILC, SciDAC or FermiQCD. Even highly optimised code has been available. Users are requested to cite the authors. No careers have been harmed, as far as I can tell. Also, the code itself is not much use for replicating results without making the data available too. This can be more problematic politically and technically, but even here there are good precedents in the particle physics world.
  - raverbashing 13y ago
    Unfortunately the world of scientific collaboration follows a strict pattern: The Article Which is good as a way of 'settling information' into a medium but an awful way for today. It's the equivalent of stone tablets compared to what's available today It would be much better if the product of research could be shared in a more flexible format. For example, don't get me started on reading two columns PDFs in a computer screen. Flexible formatting, attachable metadata, reusable "source code" (that could be an excel spreadsheet, a txt list of data, or source code per se), redrawable graphs, etc We need a faster, more flexible, albeit official way of sharing data. Yes, there are ownership issues, IP issues, etc. But maybe having a more easily searchable/traceable system may help with this.
    - nilx 13y ago
      > We need a faster, more flexible, albeit official way of sharing data. Metajournals are trying to do it, with an "Article" overlay to data/software publication : http://metajnl.com/ http://metajnl.com/. IPOL also tries to publish image processing software: http://www.ipol.im/ http://www.ipol.im/. And there is also some work to interlink science assets (more than articles) on http://linkedscience.org/ http://linkedscience.org/. \ I welcome publication models not based on "PDF standard" (ie an electronic clone of the centuries-old ink&paper text article), but simply sharing data/software is not sufficient. I think it is important to have an editorial line and a peer review, otherwise anything online would qualify as scientific communication. Moreover, software and data is hardly sufficient by itself; it needs to be explained, documented, illustrated and discussed to be more than a "harddisk dump".
  - zer0gravity 13y ago
    I think this is a deeper problem that affects the whole scientific academic community and maybe the solution to this, is the growth of a new parallel movement of amateur scientists that are pursuing research from passion and the desire to advance human knowledge, not because their job/position depends on it. In such an environment sharing your tools would probably be the norm.
A_Allen 13y ago
I hope the idea takes off, too. Providing codes is imperative; they are part of the methods and should be available for examination to ensure the integrity of the science. Those conducting research funded with public monies should be required by the funding agencies to release the products of that research, not just the results, but the data and codes, too (absent truly compelling reasons, such as national security). Eventually, I expect funding agencies will indeed require this for astronomy, just as they do for some other sciences; journals could help the field along, could improve the transparency and reproducibility of research, by requiring code release upon publication. Absent funding agencies and journals insisting on code release and the moral argument of reproducibility, what incentives would help convince code authors to release their software?
- nilx 13y ago
  > Absent funding agencies and journals insisting on code release and the moral argument of reproducibility, what incentives would help convince code authors to release their software? Impact Factor? -> Patrick Vandewalle, "Code Sharing Is Associated with Research Impact in Image Processing", CiSE 2012 http://doi.ieeecomputersociety.org/10.1109/MCSE.2012.63 http://doi.ieeecomputersociety.org/10.1109/MCSE.2012.63 It could also be the pressure of colleagues who, as anonymous reviewers, would always ask for the code whenever a paper depends on computation. Journal policies will not switch to REQUIRING the code anytime soon, but peer-review can add some pressure.

As an astronomer, I hope this idea takes off. Unfortunately, judging by the reactions my colleagues have had when I've brought up this idea in the past, I don't think it's very likely.

jared314 13y ago

Is it a code quality or a code portability issue?

ngoldbaum 13y ago
As discussed in the linked article, it's mostly because of code quality but with a healthy dose of percieved threat to job security. Unfortunately the academic citation economy isn't very nurturing of community codes, so a lot of work gets repeated when secretive research groups compete. That's not the whole story and the fact that this article is getting published speaks to that, however I still don't think it's likely that things will change in the short term. My friend Matt Turk wrote about a similar topic recently: http://arxiv.org/abs/1301.7064 http://arxiv.org/abs/1301.7064
- zbyszek 13y ago
  Repetition isn't necessarily a bad thing. Whether compatible results can be obtained with independent analyses of the same data is surely an important indicator of success. Of course that should not preclude the possibility of replication as described here. I find the idea of threats to job security misguided. In the particle physics world some significant code (not everything, granted) has been made available for some time; e.g. Monte-Carlo simulators like Geant or lattice gauge theory codes like those from MILC, SciDAC or FermiQCD. Even highly optimised code has been available. Users are requested to cite the authors. No careers have been harmed, as far as I can tell. Also, the code itself is not much use for replicating results without making the data available too. This can be more problematic politically and technically, but even here there are good precedents in the particle physics world.
- raverbashing 13y ago
  Unfortunately the world of scientific collaboration follows a strict pattern: The Article Which is good as a way of 'settling information' into a medium but an awful way for today. It's the equivalent of stone tablets compared to what's available today It would be much better if the product of research could be shared in a more flexible format. For example, don't get me started on reading two columns PDFs in a computer screen. Flexible formatting, attachable metadata, reusable "source code" (that could be an excel spreadsheet, a txt list of data, or source code per se), redrawable graphs, etc We need a faster, more flexible, albeit official way of sharing data. Yes, there are ownership issues, IP issues, etc. But maybe having a more easily searchable/traceable system may help with this.
  - nilx 13y ago
    > We need a faster, more flexible, albeit official way of sharing data. Metajournals are trying to do it, with an "Article" overlay to data/software publication : http://metajnl.com/ http://metajnl.com/. IPOL also tries to publish image processing software: http://www.ipol.im/ http://www.ipol.im/. And there is also some work to interlink science assets (more than articles) on http://linkedscience.org/ http://linkedscience.org/. \ I welcome publication models not based on "PDF standard" (ie an electronic clone of the centuries-old ink&paper text article), but simply sharing data/software is not sufficient. I think it is important to have an editorial line and a peer review, otherwise anything online would qualify as scientific communication. Moreover, software and data is hardly sufficient by itself; it needs to be explained, documented, illustrated and discussed to be more than a "harddisk dump".
- zer0gravity 13y ago
  I think this is a deeper problem that affects the whole scientific academic community and maybe the solution to this, is the growth of a new parallel movement of amateur scientists that are pursuing research from passion and the desire to advance human knowledge, not because their job/position depends on it. In such an environment sharing your tools would probably be the norm.

A_Allen 13y ago

I hope the idea takes off, too. Providing codes is imperative; they are part of the methods and should be available for examination to ensure the integrity of the science. Those conducting research funded with public monies should be required by the funding agencies to release the products of that research, not just the results, but the data and codes, too (absent truly compelling reasons, such as national security). Eventually, I expect funding agencies will indeed require this for astronomy, just as they do for some other sciences; journals could help the field along, could improve the transparency and reproducibility of research, by requiring code release upon publication. Absent funding agencies and journals insisting on code release and the moral argument of reproducibility, what incentives would help convince code authors to release their software?

nilx 13y ago
> Absent funding agencies and journals insisting on code release and the moral argument of reproducibility, what incentives would help convince code authors to release their software? Impact Factor? -> Patrick Vandewalle, "Code Sharing Is Associated with Research Impact in Image Processing", CiSE 2012 http://doi.ieeecomputersociety.org/10.1109/MCSE.2012.63 http://doi.ieeecomputersociety.org/10.1109/MCSE.2012.63 It could also be the pressure of colleagues who, as anonymous reviewers, would always ask for the code whenever a paper depends on computation. Journal policies will not switch to REQUIRING the code anytime soon, but peer-review can add some pressure.