4 ms·

After seeing plenty of code and projects by people who weren't professional software engineers used to working on teams, part of the problem is likely that code

by ufmace 5y ago

After seeing plenty of code and projects by people who weren't professional software engineers used to working on teams, part of the problem is likely that code written for this sort of thing often depends on a ton of dependencies and system-specific configuration bits that are documented poorly or not at all. Getting such projects to a state where a random person could git pull it and make sense of it and use it is a whole project unto itself that usually the core contributors are poorly equipped to take on. How many really understand the pain of onboarding into a poorly-documented repo and how to use the right tools to make it a smooth process?

Twisol 5y ago
On top of that, missions are heavily incentivized (in an "our success depends on this" way) to solve only the problems they absolutely need to solve, due to constraints on time, budget, and manpower. It's an incredible feat to achieve what they do, but reuse and non-specialist use are non-goals.
mhh__ 5y ago
(Argumentatively) I don't care. If I can't replicate your work you can't test it. If you make it clear it must be reproducible from the start and threaten to not publish if it can't be then they'll get in line - the kind of projects I'm thinking of are < 5k lines usually.
- seoaeu 5y ago
  The only way to properly replicate a CS paper is to re-implement the code from scratch. Simply running someone's code again is most likely just going to give you the same buggy output they got (or just as likely a bunch of unrelated compile errors). But frequently, that isn't even what you want. Many CS papers are of the form "we were able to build software that does XYZ using this design", which isn't really a falsifiable statement in the first place. It just serves to give future researchers and practitioners data points they can use when building their own software
  - mhh__ 5y ago
    That's true but it also allows me to look for selection bias in their data. If I see a paper claiming remarkably good predictive capabilities of (say) the performance of a basic block, and no discussion of it's flaws you bet I'm assuming they didn't test it well enough.

by ufmace 5y ago

Twisol 5y ago

On top of that, missions are heavily incentivized (in an "our success depends on this" way) to solve only the problems they absolutely need to solve, due to constraints on time, budget, and manpower. It's an incredible feat to achieve what they do, but reuse and non-specialist use are non-goals.

mhh__ 5y ago

(Argumentatively) I don't care. If I can't replicate your work you can't test it. If you make it clear it must be reproducible from the start and threaten to not publish if it can't be then they'll get in line - the kind of projects I'm thinking of are < 5k lines usually.

seoaeu 5y ago
The only way to properly replicate a CS paper is to re-implement the code from scratch. Simply running someone's code again is most likely just going to give you the same buggy output they got (or just as likely a bunch of unrelated compile errors). But frequently, that isn't even what you want. Many CS papers are of the form "we were able to build software that does XYZ using this design", which isn't really a falsifiable statement in the first place. It just serves to give future researchers and practitioners data points they can use when building their own software
- mhh__ 5y ago
  That's true but it also allows me to look for selection bias in their data. If I see a paper claiming remarkably good predictive capabilities of (say) the performance of a basic block, and no discussion of it's flaws you bet I'm assuming they didn't test it well enough.