3 ms·

You're probably right. And assuming you are, then I think maybe the core problem may be that academics even tolerate the petty undercutting of reputation based

by bunnie 5y ago

You're probably right. And assuming you are, then I think maybe the core problem may be that academics even tolerate the petty undercutting of reputation based not on the merit of the big ideas, but rather minor flaws in the proofs of concept. There's also a problem in allowing papers to get away with making arbitrary claims that are by definition irreproducible. Every paper always includes some section about how many gates a design took up and how fast it ran -- but there aren't entirely consistent standards for reporting these things; it's often too easy to hide significant setup costs when running benchmarks. This leads to an incentive to overstate results, because without any threat of checks it would be disadvantageous to not cherry-pick the absolute best numbers you could get through a peer review committee. I get the desire to escape petty criticism: but it seems like a weak argument against the myriad of reproducibility and integrity hazards such a lack of transparency begets, especially for a line of work that ostensibly distinguishes itself on high standards for merit and integrity. It also would have been nice to just receive a polite response of the form of "sorry, I don't support this anymore, I've moved on..." from anyone I had reached out to. Before starting on this project, I emailed four or five groups that had all published various papers on Curve25519 accelerators hoping to get even a broken non-compiling repo that could have helped me sanity-check my understanding of the thornier concepts presented in the papers. Not even a single response...hence the motivation to write up notes and share them. Seems a waste of funding to have so many implementations, but no way to actually use any of them...

OJFord 5y ago
Actually thinking about it, another aspect is almost certainly 'commercial sensitivity' for want of a better phrase? The university will own the copyright; they might have a policy against just handing it out when they might want to be able to later decide to commercialise it (because it's some whizz bang faster than everyone else's 25519 accelerator)? I spent my first internship at Arm benchmarking various libraries' performance on Cortex-M. Arm subsequently acquired PolarSSL, now MbedTLS (I don't recall my results - no claims here, about PolarSSL nor the influence of my lowly undergrad intern work!) for I assume some non-trivial amount. You could imagine a similar company acquiring WhizzBang25519AccelCorp from the university but perhaps less likely/for less if there's a convenient WhizzBunnie25519AccelOSS re-implementation?
- bunnie 5y ago
  You might be right about that concern, however, a lot of this work is done through public funding. Seems a little bit like double-dipping to get the public to pay for you to make IP that you then later sell to the highest bidder, with no obligation to share to the public. Of course, if the professor was say, the NXP chair for crypto core excellence and they endowed her department with the understanding that all IP would belong exclusively to NXP, then sure, that would make more sense. I did not see anything like that, though, in the research groups' bio.
nullc 5y ago
One reason you sometimes don't get responses on this sort of thing is that the claims in the paper are sometimes just fake. Not totally fake-- e.g. they'll do part of the work so that they feel like they can extrapolate to the rest. Like they might implement a field multiply and benchmark it, then just count how many of those they need for the whole circuit. Then no one wants to waste the column inches explaining what they actually did or dealing with reviewers that want to nitpick it ... at least not if they can help it. I know I've personally regretted including experimental details for that reason. ("oh great, a reviewer wants me to re-run 20 CPU years of number crunching because they've taken issue with some irrelevant detail that I could have just left out. Fine.") The "more approximate than they let on" doesn't just come from sketchy and obscure research groups but this kind of work happens under well known names too. If you read a lot of the lit you can get an impression on which groups/people are actually competent implementers because they publish them or at least also solve sub-problems that you'd only encounter when creating a full implementation. There is a lot of pressure to go this route because of time constraints and because the engineering to actually implement this stuff is often too far outside the authors expertise-- the people who really know about implementations can get highly paid in industry and many do. For anyone who isn't faking their research this should be all the more incentive to publish your code! But so far it seems the incentives haven't lined up that way yet.

by bunnie 5y ago

OJFord 5y ago

Actually thinking about it, another aspect is almost certainly 'commercial sensitivity' for want of a better phrase? The university will own the copyright; they might have a policy against just handing it out when they might want to be able to later decide to commercialise it (because it's some whizz bang faster than everyone else's 25519 accelerator)? I spent my first internship at Arm benchmarking various libraries' performance on Cortex-M. Arm subsequently acquired PolarSSL, now MbedTLS (I don't recall my results - no claims here, about PolarSSL nor the influence of my lowly undergrad intern work!) for I assume some non-trivial amount. You could imagine a similar company acquiring WhizzBang25519AccelCorp from the university but perhaps less likely/for less if there's a convenient WhizzBunnie25519AccelOSS re-implementation?

bunnie 5y ago
You might be right about that concern, however, a lot of this work is done through public funding. Seems a little bit like double-dipping to get the public to pay for you to make IP that you then later sell to the highest bidder, with no obligation to share to the public. Of course, if the professor was say, the NXP chair for crypto core excellence and they endowed her department with the understanding that all IP would belong exclusively to NXP, then sure, that would make more sense. I did not see anything like that, though, in the research groups' bio.

nullc 5y ago

One reason you sometimes don't get responses on this sort of thing is that the claims in the paper are sometimes just fake. Not totally fake-- e.g. they'll do part of the work so that they feel like they can extrapolate to the rest. Like they might implement a field multiply and benchmark it, then just count how many of those they need for the whole circuit. Then no one wants to waste the column inches explaining what they actually did or dealing with reviewers that want to nitpick it ... at least not if they can help it. I know I've personally regretted including experimental details for that reason. ("oh great, a reviewer wants me to re-run 20 CPU years of number crunching because they've taken issue with some irrelevant detail that I could have just left out. Fine.") The "more approximate than they let on" doesn't just come from sketchy and obscure research groups but this kind of work happens under well known names too. If you read a lot of the lit you can get an impression on which groups/people are actually competent implementers because they publish them or at least also solve sub-problems that you'd only encounter when creating a full implementation. There is a lot of pressure to go this route because of time constraints and because the engineering to actually implement this stuff is often too far outside the authors expertise-- the people who really know about implementations can get highly paid in industry and many do. For anyone who isn't faking their research this should be all the more incentive to publish your code! But so far it seems the incentives haven't lined up that way yet.