6 ms·

> Most code written by academics is missing trivial, obvious optimisations I agree with this, but I would pose it the opposite way. Academics shouldn't be prem

> Most code written by academics is missing trivial, obvious optimisations I agree with this, but I would pose it the opposite way. Academics shouldn't be prematurely optimizing their code: they don't have the resources for it, nor will their results be representative across all hardware. I routinely see (and publish) papers with novel cryptographic protocols where the reviewers will say things like "compare the running time of your algorithm to a state-of-the-art production system that Apple spent $50m developing and optimizing for their silicon, using hand-tuned assembly." We're not going to win in that benchmark, and we don't have the resources to even compete. Moreover, the resulting code wouldn't make for good reference code. But that doesn't mean we shouldn't publish our work, so that e.g., Apple (or other resources) can begin the process of optimizing it for production.

josephg 3y ago
> Academics shouldn't be prematurely optimizing their code While I agree with that in principle, the question of whether or not a given algorithm is fast enough to be practically usable is very useful information for the reader of your paper. Do I need to read your paper, understand it, implement it myself, and only then figure out that its 3 orders of magnitude slower than existing algorithms? I know its a lot of work, but as the reader I don't want to do that work either. I've spent the last few years working on a novel collaborative editing algorithm for text editing. If the algorithm was orders of magnitude slower than existing CRDTs, it would never get adopted. And I want my work to be used. I want to use it, but again - only if its fast enough to be practically usable. I ended up writing a very highly optimised implementation of the algorithm - which took a massive amount of time - to answer that question. For the paper, I spent another couple of days writing a much slower reference implementation that people can read to understand how it works. (The optimised version is thousands of lines, and the reference code is a few hundred lines). I understand how much work and expertise is needed to do things like this, but I still want someone to do this work. Even if its "We wrote a straightforward implementation of <competing algorithm> and <our algorithm> in plain C code. They perform at roughly similar levels of performance <see chart>. We suspect highly optimised versions of both approaches would also perform at roughly similar levels of performance."

josephg 3y ago

> Academics shouldn't be prematurely optimizing their code While I agree with that in principle, the question of whether or not a given algorithm is fast enough to be practically usable is very useful information for the reader of your paper. Do I need to read your paper, understand it, implement it myself, and only then figure out that its 3 orders of magnitude slower than existing algorithms? I know its a lot of work, but as the reader I don't want to do that work either. I've spent the last few years working on a novel collaborative editing algorithm for text editing. If the algorithm was orders of magnitude slower than existing CRDTs, it would never get adopted. And I want my work to be used. I want to use it, but again - only if its fast enough to be practically usable. I ended up writing a very highly optimised implementation of the algorithm - which took a massive amount of time - to answer that question. For the paper, I spent another couple of days writing a much slower reference implementation that people can read to understand how it works. (The optimised version is thousands of lines, and the reference code is a few hundred lines). I understand how much work and expertise is needed to do things like this, but I still want someone to do this work. Even if its "We wrote a straightforward implementation of <competing algorithm> and <our algorithm> in plain C code. They perform at roughly similar levels of performance <see chart>. We suspect highly optimised versions of both approaches would also perform at roughly similar levels of performance."