4 ms·

I think this nature of retro-fitting models used for software engineering into research is ill-targeted. Research is non-linear by nature. One of the reasons r

I think this nature of retro-fitting models used for software engineering into research is ill-targeted. Research is non-linear by nature. One of the reasons research code is usually not well written is because of this same fact - there is no clear plan (or else it wouldn’t be research). Restricting research cultures to follow this linear progress model also imposes an artificial block that may be detrimental to productivity. Alternatively, this exercise could demand a second pass over research code which cleans it up and puts it in its “place”. This may or may not work because “clean up” is a non-trivial task which the researcher may find limited time and utility for. By no means I’m against writing clean code. In fact, I’ve had multiple discussions advocating it but it’s hard to keep everyone aligned. The primary objective of research in a fast-paced field like Machine Learning is to get the idea out the door. Unfortunately, the exceptionally compounding benefits of writing clean thoughtful code from the beginning are realized much later in this timeline. By then, it’s too late and too hard to retro-fix code. My understanding so far (having written code for multiple research projects) is that the only way to fix this culture is to deeply ingrain the compounding benefits of clean code in new researchers. By exposing junior researchers to the compounding benefits of “clean” code, we can gradually nudge the community into a more favorable culture with respect to this “commit history” of research.

MiroF 6y ago
Agreed - and I would say this is part of a larger phenomenon of SWE hubris that expertism in software automatically translates to process expertise in every other domain under the sun.
- activatedgeek 6y ago
  To be fair, SWE patterns are insightful. But here's where the problem lies - the proposal in the article is representative of how the community has become so used to this nice side-effect of modularity of neural networks, giving the impression that a clean linear decomposition of contributions is even conceivable. There are examples where this modularity isn't possible (or not optimal even if possible). The poster child, I would say, is probabilistic programming. It is easy to think about building general purpose inference algorithms in probabilistic programming. However, more often than not, inference heavily relies on context. An example for this case is Gaussian Processes. In theory, it is easy to conceive a chain of ideas that connect threads of research within this tiny field. Unfortunately, in practice it turns out inference with GPs is more efficient when we exploit structural assumptions within the context of modeling choices and general purpose few-liner code changes would be terribly inefficient. You'd find the code will look vastly different to an untrained eye even though it would compose similar sounding (or looking) building blocks.
- hprotagonist 6y ago
  In my neck of the woods (biomedical sciences), this hubris' name is "the andy grove fallacy": https://blogs.sciencemag.org/pipeline/archives/2007/11/06/andy_grove_rich_famous_smart_and_wrong https://blogs.sciencemag.org/pipeline/archives/2007/11/06/an... and it is rampant.
  - activatedgeek 6y ago
    I understand the sentiment. Although, the article you link is more of a ad hominem attack than criticism. Communities must be wary of giving into the Semmelweis reflex. Often times, the perceived "outsiders" tend to have interesting perspectives. Another example to consider is Fermat vs Descartes - Fermat was a relatively unpopular figure at a time when mathematics was confined to closed-group elitist figures like Descartes. Nevertheless, he provided more elegant perspectives to problems which Descartes thought couldn't be done better.
    - hprotagonist 6y ago
      Not Invented Hereism is different from "look, you outsiders fundamentally just don't understand the score, and until you do, i'm going to ignore you". Swanking into the joint, assuming spherical cows, and berating the locals for being lazy roustabouts who just need to see the light is not going to make you friends, because it's demonstrably not useful. To the extent that there's human politics and gatekeeping, it's resolved by getting ones hands dirty in the lab before one holds forth ex cathedra about things one obviously doesn't understand. It's certainly possible to take valuable ideas from one field and apply them in a productive cross-disciplinary way. The people who do it well don't play this game. Intuitions and practices developed to handle designed and often linearizable systems for which spec sheets of some form are available just aren't applicable without heavy modification in the biosciences, which routinely deal with eking out victory in systems which are not designed, are wildly nonlinear, and for which the partial specs which do exist are known to be at best lies of omission and often just complete misunderstandings which can be, and are, revised at a moment's notice.
- 6gvONxR4sf7o 6y ago
  Or it's the converse. If you're an expert in a thing you do with code, thinking that makes your code good.

MiroF 6y ago

Agreed - and I would say this is part of a larger phenomenon of SWE hubris that expertism in software automatically translates to process expertise in every other domain under the sun.

activatedgeek 6y ago
To be fair, SWE patterns are insightful. But here's where the problem lies - the proposal in the article is representative of how the community has become so used to this nice side-effect of modularity of neural networks, giving the impression that a clean linear decomposition of contributions is even conceivable. There are examples where this modularity isn't possible (or not optimal even if possible). The poster child, I would say, is probabilistic programming. It is easy to think about building general purpose inference algorithms in probabilistic programming. However, more often than not, inference heavily relies on context. An example for this case is Gaussian Processes. In theory, it is easy to conceive a chain of ideas that connect threads of research within this tiny field. Unfortunately, in practice it turns out inference with GPs is more efficient when we exploit structural assumptions within the context of modeling choices and general purpose few-liner code changes would be terribly inefficient. You'd find the code will look vastly different to an untrained eye even though it would compose similar sounding (or looking) building blocks.
hprotagonist 6y ago
In my neck of the woods (biomedical sciences), this hubris' name is "the andy grove fallacy": https://blogs.sciencemag.org/pipeline/archives/2007/11/06/andy_grove_rich_famous_smart_and_wrong https://blogs.sciencemag.org/pipeline/archives/2007/11/06/an... and it is rampant.
- activatedgeek 6y ago
  I understand the sentiment. Although, the article you link is more of a ad hominem attack than criticism. Communities must be wary of giving into the Semmelweis reflex. Often times, the perceived "outsiders" tend to have interesting perspectives. Another example to consider is Fermat vs Descartes - Fermat was a relatively unpopular figure at a time when mathematics was confined to closed-group elitist figures like Descartes. Nevertheless, he provided more elegant perspectives to problems which Descartes thought couldn't be done better.
  - hprotagonist 6y ago
    Not Invented Hereism is different from "look, you outsiders fundamentally just don't understand the score, and until you do, i'm going to ignore you". Swanking into the joint, assuming spherical cows, and berating the locals for being lazy roustabouts who just need to see the light is not going to make you friends, because it's demonstrably not useful. To the extent that there's human politics and gatekeeping, it's resolved by getting ones hands dirty in the lab before one holds forth ex cathedra about things one obviously doesn't understand. It's certainly possible to take valuable ideas from one field and apply them in a productive cross-disciplinary way. The people who do it well don't play this game. Intuitions and practices developed to handle designed and often linearizable systems for which spec sheets of some form are available just aren't applicable without heavy modification in the biosciences, which routinely deal with eking out victory in systems which are not designed, are wildly nonlinear, and for which the partial specs which do exist are known to be at best lies of omission and often just complete misunderstandings which can be, and are, revised at a moment's notice.
6gvONxR4sf7o 6y ago
Or it's the converse. If you're an expert in a thing you do with code, thinking that makes your code good.