3 ms·

Yeah! Having spent much of the last year plowing through computer vision papers, it's pretty rare to encounter published code or even datasets. Why is this? Do

by rboyd 11y ago

Yeah! Having spent much of the last year plowing through computer vision papers, it's pretty rare to encounter published code or even datasets. Why is this? Do people spend all their time polishing prose and neglecting the code? Or maybe they keep it closed for commercial intentions? Or is it just a cultural thing in all of academia?

jmcmahon443 11y ago
Science = Nature. Code = Applying Nature.
- rboyd 11y ago
  This would be a lot easier to jibe with if the paper was based on proof alone. A lot of the work in this realm isn't that. Plenty of these researchers are also fine engineers and I'm sure have great code. It just seems crazy to develop a new technique so completely and stop short of releasing the code. I think it's because the academic culture encourages this as if the code detracts from the purity of the research.
  - jmcmahon443 11y ago
    I see where you're coming from. I think the truth is actually dependent on the particular institution and particular academic. For example, OpenRave is the result of academics open sourcing their research. Update: Someone posted a link to the code.
Xcelerate 11y ago
Speaking from experience, the code is normally functional but unsightly, thus it would be embarrassing to place it online. Most researchers are happy to email you the code if you ask for it though. Most of my code is for scientific computing and consists of a bunch of one-off Jupyter notebooks and Julia scripts that are written with two objectives in mind: get-the-task-done-before-my-advisor-meeting and make-it-as-fast-as-possible. Good coding practices and code cleanliness are not priorities. And particularly for code that needs to run on HPC clusters or supercomputers, it's really hard to make it look nice — one could easily spend weeks designing a proper, modular system only to throw it all away in favor of using a better algorithm that was just published. Anecdotally, the vast majority of my code has produced results that were later discarded. For example, I have a function named YPXt. What does such a woefully named subroutine do? It multiplies a matrix Y by a permutation matrix P, and then multiplies that by another (transposed) matrix X. The matrices are small, so the overhead of using BLAS for the task is wasteful. It's better to use Julia's macro system to generate the necessary for loops on the fly and let the compiler SIMD-ify the result. Also, permutation matrix multiplies can be done in a way that doesn't actually involve creating a full permutation matrix. And of course the subroutine has side effects — in order to conserve memory I need to overwrite the input matrices instead of allocating new ones. Immutable data structures may work great for functional programming, but the overhead of red-black trees and stateless programming is too high for numerical data crunching. Add that to the fact that a lot of scientists in academia are unfamiliar with tools like Git, and so manually versioning the code and emailing it back and forth is unfortunately a necessary part of the collaboration process.
- dr_zoidberg 11y ago
  I completely agree with you and I can relate with your "objectives in mind". But you really need to push to get others using Git. GitHub's GUI client is very easy to use and with a bit of nagging around you can get even people who don't understand what VCS are to use it. The one time they ruin the code and get the chance to easily go back to how things were before they'll see its value and endlessly thank you for it.
deleted 11y ago
[deleted]
zxv 11y ago
In my experience, blind reviewers for peer reviewed journal articles in certain cases may argue that publicly available binary or sources are a prior publication of the technique or method, and a reason for rejecting the paper. This risk is hard to assess in advance. Many journals do not clarify the policy regarding advance publishing of source code or even binaries alone. Given the impact this could have on a student's career, this is worth considering carefully, and if possible, clarifying in advance with the journal's editor or program chair.

by rboyd 11y ago

jmcmahon443 11y ago

Science = Nature. Code = Applying Nature.

rboyd 11y ago
This would be a lot easier to jibe with if the paper was based on proof alone. A lot of the work in this realm isn't that. Plenty of these researchers are also fine engineers and I'm sure have great code. It just seems crazy to develop a new technique so completely and stop short of releasing the code. I think it's because the academic culture encourages this as if the code detracts from the purity of the research.
- jmcmahon443 11y ago
  I see where you're coming from. I think the truth is actually dependent on the particular institution and particular academic. For example, OpenRave is the result of academics open sourcing their research. Update: Someone posted a link to the code.

Xcelerate 11y ago

Speaking from experience, the code is normally functional but unsightly, thus it would be embarrassing to place it online. Most researchers are happy to email you the code if you ask for it though. Most of my code is for scientific computing and consists of a bunch of one-off Jupyter notebooks and Julia scripts that are written with two objectives in mind: get-the-task-done-before-my-advisor-meeting and make-it-as-fast-as-possible. Good coding practices and code cleanliness are not priorities. And particularly for code that needs to run on HPC clusters or supercomputers, it's really hard to make it look nice — one could easily spend weeks designing a proper, modular system only to throw it all away in favor of using a better algorithm that was just published. Anecdotally, the vast majority of my code has produced results that were later discarded. For example, I have a function named YPXt. What does such a woefully named subroutine do? It multiplies a matrix Y by a permutation matrix P, and then multiplies that by another (transposed) matrix X. The matrices are small, so the overhead of using BLAS for the task is wasteful. It's better to use Julia's macro system to generate the necessary for loops on the fly and let the compiler SIMD-ify the result. Also, permutation matrix multiplies can be done in a way that doesn't actually involve creating a full permutation matrix. And of course the subroutine has side effects — in order to conserve memory I need to overwrite the input matrices instead of allocating new ones. Immutable data structures may work great for functional programming, but the overhead of red-black trees and stateless programming is too high for numerical data crunching. Add that to the fact that a lot of scientists in academia are unfamiliar with tools like Git, and so manually versioning the code and emailing it back and forth is unfortunately a necessary part of the collaboration process.

dr_zoidberg 11y ago
I completely agree with you and I can relate with your "objectives in mind". But you really need to push to get others using Git. GitHub's GUI client is very easy to use and with a bit of nagging around you can get even people who don't understand what VCS are to use it. The one time they ruin the code and get the chance to easily go back to how things were before they'll see its value and endlessly thank you for it.

deleted 11y ago

[deleted]

zxv 11y ago

In my experience, blind reviewers for peer reviewed journal articles in certain cases may argue that publicly available binary or sources are a prior publication of the technique or method, and a reason for rejecting the paper. This risk is hard to assess in advance. Many journals do not clarify the policy regarding advance publishing of source code or even binaries alone. Given the impact this could have on a student's career, this is worth considering carefully, and if possible, clarifying in advance with the journal's editor or program chair.