5 ms·

I'm a huge fan of this kind of practice, where the code for a paper is all located in a single public repository with build instructions, along with directions

by qchris 6y ago

I'm a huge fan of this kind of practice, where the code for a paper is all located in a single public repository with build instructions, along with directions for how to cite it. Obviously, it's a little tough to do with some more data-intensive sources (besides GH hosting limits, no one really wants to download 100G of data if they're just trying to clone a repository), but this kind of thing sets a high standard for reproducibility of published results.

dllthomas 6y ago
> but this kind of thing sets a high standard for reproducibility of published results. I think making the code available is good, but I think we should be careful how we use the term "reproducibility". Pulling your repo and running it had better give the same results, but it's not the same sort of thing as building my own experimental setup according to a paper's specification. The latter gives more room for variability such that successful replication speaks more strongly to the robustness of the result, and also puts human brain power next to each step of the process in a way where weirdness might be noticed. Replication should probably involve reimplementation, if it's to carry its traditional weight. In the event that we fail to replicate, though, having the source code for both versions is likely to be hugely informative.
- enchiridion 6y ago
  While I agree, I think reimplementation is high bar for most research, especially in very niche areas. I think think extension also carries similar value. It is less grunt work to do, but still requires a deep understanding of the existing code. "Weirdness" should quickly become apparent.
- dekhn 6y ago
  I think this is a fair point, but in my experience, having a concrete replica that people can start from (and compare to the paper) can make a year's difference in speeding up progress. Many times, I've read a paper, thought something was great, and then implemented the paper and failed to reproduce the author's results. In the cases where I've been able to compare my implementation to a reference on github, I often find the paper doesn't match the code, or a subtle data processing step was left out. Having a replica (a commit hash and a pointed to versioned input data) can often make a huge difference in time.
  - dllthomas 6y ago
    Yeah, I'm certainly not saying it isn't advisable or even important. I'm just saying it's not the same thing as replication.
rozgo 6y ago
It's incredibly satisfying to reproduce these papers. I now make Rust versions of the most interesting projects. And try to make low-latency inference pipelines for those that show potential for real-time use. Some are sketched out here: https://github.com/Simbotic/SimboticTorch https://github.com/Simbotic/SimboticTorch The bulk of the work to get real-time working is to move more of pipeline to GPU. Mostly things handled by numpy and some image/video transformations.
- qchris 6y ago
  That's awesome! If you're interested, there's a group working on machine learning in Rust, including some working on doing GPU pipelines for it at https://github.com/rust-ml/wg https://github.com/rust-ml/wg . I'm not sure if any of the work being done right now is directly applicable to any of the projects that you're reproducing, but it might be worth a look!
  - rozgo 6y ago
    Oh, I'm interested. Thanks for letting me know. Would love to contribute.
    - BracketMaster 6y ago
      This truly is amazing. I've noticed that GPU does help a lot with inference. It would be nice if it were easier to make projects like these mobile. Google and Apple have SDKs for running nets on phones, but its a shame its so hard to do things like this on the Raspberry pi...
      - Miraste 6y ago
        Tensorflow Lite runs on Raspberry Pis. It's even in the official docs: https://www.tensorflow.org/lite/guide/build_rpi https://www.tensorflow.org/lite/guide/build_rpi

by qchris 6y ago

dllthomas 6y ago

> but this kind of thing sets a high standard for reproducibility of published results. I think making the code available is good, but I think we should be careful how we use the term "reproducibility". Pulling your repo and running it had better give the same results, but it's not the same sort of thing as building my own experimental setup according to a paper's specification. The latter gives more room for variability such that successful replication speaks more strongly to the robustness of the result, and also puts human brain power next to each step of the process in a way where weirdness might be noticed. Replication should probably involve reimplementation, if it's to carry its traditional weight. In the event that we fail to replicate, though, having the source code for both versions is likely to be hugely informative.

enchiridion 6y ago
While I agree, I think reimplementation is high bar for most research, especially in very niche areas. I think think extension also carries similar value. It is less grunt work to do, but still requires a deep understanding of the existing code. "Weirdness" should quickly become apparent.
dekhn 6y ago
I think this is a fair point, but in my experience, having a concrete replica that people can start from (and compare to the paper) can make a year's difference in speeding up progress. Many times, I've read a paper, thought something was great, and then implemented the paper and failed to reproduce the author's results. In the cases where I've been able to compare my implementation to a reference on github, I often find the paper doesn't match the code, or a subtle data processing step was left out. Having a replica (a commit hash and a pointed to versioned input data) can often make a huge difference in time.
- dllthomas 6y ago
  Yeah, I'm certainly not saying it isn't advisable or even important. I'm just saying it's not the same thing as replication.

rozgo 6y ago

It's incredibly satisfying to reproduce these papers. I now make Rust versions of the most interesting projects. And try to make low-latency inference pipelines for those that show potential for real-time use. Some are sketched out here: https://github.com/Simbotic/SimboticTorch https://github.com/Simbotic/SimboticTorch The bulk of the work to get real-time working is to move more of pipeline to GPU. Mostly things handled by numpy and some image/video transformations.

qchris 6y ago
That's awesome! If you're interested, there's a group working on machine learning in Rust, including some working on doing GPU pipelines for it at https://github.com/rust-ml/wg https://github.com/rust-ml/wg . I'm not sure if any of the work being done right now is directly applicable to any of the projects that you're reproducing, but it might be worth a look!
- rozgo 6y ago
  Oh, I'm interested. Thanks for letting me know. Would love to contribute.
  - BracketMaster 6y ago
    This truly is amazing. I've noticed that GPU does help a lot with inference. It would be nice if it were easier to make projects like these mobile. Google and Apple have SDKs for running nets on phones, but its a shame its so hard to do things like this on the Raspberry pi...
    - Miraste 6y ago
      Tensorflow Lite runs on Raspberry Pis. It's even in the official docs: https://www.tensorflow.org/lite/guide/build_rpi https://www.tensorflow.org/lite/guide/build_rpi