4 ms·
> It’s a weird thing about academics — they like to write papers and “share ideas”, but it’s very hard to get source code from them. I suppose at least part of
by OJFord 5y ago
> It’s a weird thing about academics — they like to write papers and “share ideas”, but it’s very hard to get source code from them.
I suppose at least part of the reason, especially for things like crypto and statistics, is that they don't want to get bogged down in plausibly-correct claims of fatal flaws, and be perhaps repeatedly (over years hence) put in a defensive position of having to prove the criticism incorrect or have everyone assume the work was wrong?
I can sort of understand that. The same's true for the papers themselves of course, but I can see it being more annoying for code. ('But it's never going to actually be in that state', etc.)
- gww 5y agoIn the biological sciences there is a huge push for the code to be released alongside the paper. Many publications (but not enough yet) include a complete pipeline in a workflow language that downloads, processes the data and generates the figures used in the paper.
- toomuchtodo 5y agoWhat’s the language de jour typically leveraged to declare these workflows?
- semi-extrinsic 5y agoSince GP mentions biology, I would guess R is the first choice.
- gww 5y agoNextflow and Snakemake are quite common. I prefer the later as I have a lot of Python experience.
- staticassertion 5y agoAre you saying this based on experience? My assumption was that it was a number of factors. 1. Researchers are often weak coders, and they may be self conscious about their code. 2. My own experience is that researcher code is totally undocument, or broken. I've repeatedly found that the code linked to from a paper has no straightforward way to be built, or appears to have compiler errors etc. 3. It's more work, and researchers have the incentive to put out a compelling paper, not to put out code (because we don't hold them to that standard). My own experiences have led to me to mostly dismiss papers where I can't find the code or can't get it to build.
- kzrdude 5y agoGiven all that, it's understandable that it's not released and completely reasonable - a support structure is needed if the code should be released.
- staticassertion 5y agoI'm setting the bar super low. A `./configure` and a readme with basic info about the project and how to build/run it would be quite exceptional. I don't need them to be triaging github issues and merging in PRs.
- OJFord 5y ago> Are you saying this based on experience? No, just observation & supposition. > My assumption was that it was a number of factors. I'm sure! I did say I suppose part of the reason. Didn't mean to imply any experience/authority, OP just made me think about it. Sorry if it sounded like that.
- staticassertion 5y agoOh, my question was genuine, I wasn't trying to be like "you DONT have experience". I was just curious. Rereading my post it sounds like I'm trying to disagree, I'm no expert either.
- JJMcJ 5y agoDonald Knuth once wrote a paper with some source code. He warned his readers about just copying the code, perhaps half jokingly. He said, approximately, "this code has been proven correct, but not tested."
- bunnie 5y agoYou're probably right. And assuming you are, then I think maybe the core problem may be that academics even tolerate the petty undercutting of reputation based not on the merit of the big ideas, but rather minor flaws in the proofs of concept. There's also a problem in allowing papers to get away with making arbitrary claims that are by definition irreproducible. Every paper always includes some section about how many gates a design took up and how fast it ran -- but there aren't entirely consistent standards for reporting these things; it's often too easy to hide significant setup costs when running benchmarks. This leads to an incentive to overstate results, because without any threat of checks it would be disadvantageous to not cherry-pick the absolute best numbers you could get through a peer review committee. I get the desire to escape petty criticism: but it seems like a weak argument against the myriad of reproducibility and integrity hazards such a lack of transparency begets, especially for a line of work that ostensibly distinguishes itself on high standards for merit and integrity. It also would have been nice to just receive a polite response of the form of "sorry, I don't support this anymore, I've moved on..." from anyone I had reached out to. Before starting on this project, I emailed four or five groups that had all published various papers on Curve25519 accelerators hoping to get even a broken non-compiling repo that could have helped me sanity-check my understanding of the thornier concepts presented in the papers. Not even a single response...hence the motivation to write up notes and share them. Seems a waste of funding to have so many implementations, but no way to actually use any of them...
- OJFord 5y agoActually thinking about it, another aspect is almost certainly 'commercial sensitivity' for want of a better phrase? The university will own the copyright; they might have a policy against just handing it out when they might want to be able to later decide to commercialise it (because it's some whizz bang faster than everyone else's 25519 accelerator)? I spent my first internship at Arm benchmarking various libraries' performance on Cortex-M. Arm subsequently acquired PolarSSL, now MbedTLS (I don't recall my results - no claims here, about PolarSSL nor the influence of my lowly undergrad intern work!) for I assume some non-trivial amount. You could imagine a similar company acquiring WhizzBang25519AccelCorp from the university but perhaps less likely/for less if there's a convenient WhizzBunnie25519AccelOSS re-implementation?
- joppy 5y agoIn a field like crypto you also have the concern that someone might copy your implementation and actually use it - this is fine if your specialty is crypto implementation (resistant to side-channel attacks and so on), but if your specialty is instead coming up with faster algorithm, your implementation is going to be much more proof-of-concept. In the theme of “don’t write your own crypto” is “don’t release your own crypto if you look authoritative (Eg a professor) and are not absolutely sure of it “