5 ms·

I wish these groups would make their code available so others could play with it and test it out. Some do, but I wish more did. I would be curious to see the p

by pen2l 11y ago

I wish these groups would make their code available so others could play with it and test it out. Some do, but I wish more did. I would be curious to see the performance of this 3d depth reconstruction technique for non-rigid environments.

jmcmahon443 11y ago
I love open source, but there is a very good reason why academics generally don't release their code. They are scientists: they are best at testing, explaining, and documenting nature. Code is an engineer's job: making it as fast and robust as possible for the given application or with the latest tools. They want to make you work for it :)
- pen2l 11y ago
  With all due respect, that is a stupid reason. I don't have time to implement every last thing. And I don't require robust code all the time -- it's very understandable for the code to not be robust if it's being provided by a small group with not a lot of funding or resources. Hell, if they make it open source I will contribute to it expecting nothing in return to make it better in any way I can. I really think that the reason the code isn't shared is journals don't encourage this type of thing (or make it easy). Lab websites live and die, so hosting it there is not a good idea. Very simply, the code should should be packed with the paper somehow. I hope some journal takes the lead soon in creating a system that encourages a bundled presentation of paper + code.
  - jmcmahon443 11y ago
    Academics aren't always concerned with making your job easy, they have lives too. They may have struggled a lot for their results and their code is still a living document, or they sell it to a company. Maybe their grant ran out and they got pulled into something else. Maybe they died. There's a lot of factors. Look at OpenRave. It's the result of an academic open sourcing his research. I just think at the end of the day, a research paper exists to validate human understanding of nature - not give you a boilerplate for implementing it.
rboyd 11y ago
Yeah! Having spent much of the last year plowing through computer vision papers, it's pretty rare to encounter published code or even datasets. Why is this? Do people spend all their time polishing prose and neglecting the code? Or maybe they keep it closed for commercial intentions? Or is it just a cultural thing in all of academia?
- jmcmahon443 11y ago
  Science = Nature. Code = Applying Nature.
  - rboyd 11y ago
    This would be a lot easier to jibe with if the paper was based on proof alone. A lot of the work in this realm isn't that. Plenty of these researchers are also fine engineers and I'm sure have great code. It just seems crazy to develop a new technique so completely and stop short of releasing the code. I think it's because the academic culture encourages this as if the code detracts from the purity of the research.
    - jmcmahon443 11y ago
      I see where you're coming from. I think the truth is actually dependent on the particular institution and particular academic. For example, OpenRave is the result of academics open sourcing their research. Update: Someone posted a link to the code.
- Xcelerate 11y ago
  Speaking from experience, the code is normally functional but unsightly, thus it would be embarrassing to place it online. Most researchers are happy to email you the code if you ask for it though. Most of my code is for scientific computing and consists of a bunch of one-off Jupyter notebooks and Julia scripts that are written with two objectives in mind: get-the-task-done-before-my-advisor-meeting and make-it-as-fast-as-possible. Good coding practices and code cleanliness are not priorities. And particularly for code that needs to run on HPC clusters or supercomputers, it's really hard to make it look nice — one could easily spend weeks designing a proper, modular system only to throw it all away in favor of using a better algorithm that was just published. Anecdotally, the vast majority of my code has produced results that were later discarded. For example, I have a function named YPXt. What does such a woefully named subroutine do? It multiplies a matrix Y by a permutation matrix P, and then multiplies that by another (transposed) matrix X. The matrices are small, so the overhead of using BLAS for the task is wasteful. It's better to use Julia's macro system to generate the necessary for loops on the fly and let the compiler SIMD-ify the result. Also, permutation matrix multiplies can be done in a way that doesn't actually involve creating a full permutation matrix. And of course the subroutine has side effects — in order to conserve memory I need to overwrite the input matrices instead of allocating new ones. Immutable data structures may work great for functional programming, but the overhead of red-black trees and stateless programming is too high for numerical data crunching. Add that to the fact that a lot of scientists in academia are unfamiliar with tools like Git, and so manually versioning the code and emailing it back and forth is unfortunately a necessary part of the collaboration process.
derf_ 11y ago
I re-implemented the techniques described here for a former employer (which sadly means I also cannot release the code). I was able to reproduce the quoted performance figures (both accuracy according to the paper's metrics and execution time), but the actual quality of the output was not very good. Something like > 90% (number hazy based on vague recollection) of images produced results that had gross, glaring errors that would be unacceptable to a human. Errors like, "We added a horizon line where there wasn't one in the actual image," etc. But a few images produced compellingly plausible results. It's still very interesting stuff, and some of his later work on using this as an additional prior for more normal structure-from-motion pipelines looked pretty promising. In the form reported in this paper it wasn't something ready to be productized. Although it's also completely possible that I just screwed something up.
web007 11y ago
The YouTube video in the HN comments links to one of the authors' Stanford page, which redirects to this page at Cornell: http://www.cs.cornell.edu/~asaxena/rccar/ http://www.cs.cornell.edu/~asaxena/rccar/ There you can find the source for both the vision code in MatLAB, and a link to http://make3d.cs.cornell.edu/code.html http://make3d.cs.cornell.edu/code.html for the full source.
- jmcmahon443 11y ago
  Thanks!

by pen2l 11y ago

jmcmahon443 11y ago

I love open source, but there is a very good reason why academics generally don't release their code. They are scientists: they are best at testing, explaining, and documenting nature. Code is an engineer's job: making it as fast and robust as possible for the given application or with the latest tools. They want to make you work for it :)

pen2l 11y ago
With all due respect, that is a stupid reason. I don't have time to implement every last thing. And I don't require robust code all the time -- it's very understandable for the code to not be robust if it's being provided by a small group with not a lot of funding or resources. Hell, if they make it open source I will contribute to it expecting nothing in return to make it better in any way I can. I really think that the reason the code isn't shared is journals don't encourage this type of thing (or make it easy). Lab websites live and die, so hosting it there is not a good idea. Very simply, the code should should be packed with the paper somehow. I hope some journal takes the lead soon in creating a system that encourages a bundled presentation of paper + code.
- jmcmahon443 11y ago
  Academics aren't always concerned with making your job easy, they have lives too. They may have struggled a lot for their results and their code is still a living document, or they sell it to a company. Maybe their grant ran out and they got pulled into something else. Maybe they died. There's a lot of factors. Look at OpenRave. It's the result of an academic open sourcing his research. I just think at the end of the day, a research paper exists to validate human understanding of nature - not give you a boilerplate for implementing it.

rboyd 11y ago

Yeah! Having spent much of the last year plowing through computer vision papers, it's pretty rare to encounter published code or even datasets. Why is this? Do people spend all their time polishing prose and neglecting the code? Or maybe they keep it closed for commercial intentions? Or is it just a cultural thing in all of academia?

jmcmahon443 11y ago
Science = Nature. Code = Applying Nature.
- rboyd 11y ago
  This would be a lot easier to jibe with if the paper was based on proof alone. A lot of the work in this realm isn't that. Plenty of these researchers are also fine engineers and I'm sure have great code. It just seems crazy to develop a new technique so completely and stop short of releasing the code. I think it's because the academic culture encourages this as if the code detracts from the purity of the research.
  - jmcmahon443 11y ago
    I see where you're coming from. I think the truth is actually dependent on the particular institution and particular academic. For example, OpenRave is the result of academics open sourcing their research. Update: Someone posted a link to the code.
Xcelerate 11y ago
Speaking from experience, the code is normally functional but unsightly, thus it would be embarrassing to place it online. Most researchers are happy to email you the code if you ask for it though. Most of my code is for scientific computing and consists of a bunch of one-off Jupyter notebooks and Julia scripts that are written with two objectives in mind: get-the-task-done-before-my-advisor-meeting and make-it-as-fast-as-possible. Good coding practices and code cleanliness are not priorities. And particularly for code that needs to run on HPC clusters or supercomputers, it's really hard to make it look nice — one could easily spend weeks designing a proper, modular system only to throw it all away in favor of using a better algorithm that was just published. Anecdotally, the vast majority of my code has produced results that were later discarded. For example, I have a function named YPXt. What does such a woefully named subroutine do? It multiplies a matrix Y by a permutation matrix P, and then multiplies that by another (transposed) matrix X. The matrices are small, so the overhead of using BLAS for the task is wasteful. It's better to use Julia's macro system to generate the necessary for loops on the fly and let the compiler SIMD-ify the result. Also, permutation matrix multiplies can be done in a way that doesn't actually involve creating a full permutation matrix. And of course the subroutine has side effects — in order to conserve memory I need to overwrite the input matrices instead of allocating new ones. Immutable data structures may work great for functional programming, but the overhead of red-black trees and stateless programming is too high for numerical data crunching. Add that to the fact that a lot of scientists in academia are unfamiliar with tools like Git, and so manually versioning the code and emailing it back and forth is unfortunately a necessary part of the collaboration process.

derf_ 11y ago

I re-implemented the techniques described here for a former employer (which sadly means I also cannot release the code). I was able to reproduce the quoted performance figures (both accuracy according to the paper's metrics and execution time), but the actual quality of the output was not very good. Something like > 90% (number hazy based on vague recollection) of images produced results that had gross, glaring errors that would be unacceptable to a human. Errors like, "We added a horizon line where there wasn't one in the actual image," etc. But a few images produced compellingly plausible results. It's still very interesting stuff, and some of his later work on using this as an additional prior for more normal structure-from-motion pipelines looked pretty promising. In the form reported in this paper it wasn't something ready to be productized. Although it's also completely possible that I just screwed something up.

web007 11y ago

The YouTube video in the HN comments links to one of the authors' Stanford page, which redirects to this page at Cornell: http://www.cs.cornell.edu/~asaxena/rccar/ http://www.cs.cornell.edu/~asaxena/rccar/ There you can find the source for both the vision code in MatLAB, and a link to http://make3d.cs.cornell.edu/code.html http://make3d.cs.cornell.edu/code.html for the full source.

jmcmahon443 11y ago
Thanks!