17 ms·

Research software code is likely to remain a tangled mess

182 points by hacksilver 6y ago

einpoklum 6y ago
One of the things which has helped derail my own research career [1] is the tendency to not write tangled-mess code, and to publish and maintain much of my research code after I was supposed to be done with it. Annoyingly, more people now know of me due to those pieces of software than for my research agenda. :-( [1] : Not the only thing mind you.
- jimmyvalmer 6y ago
  C'mon, no good deed goes unpunished. Everyone knows that.
chilukrn 6y ago
I agree the post makes valid points, but is there anything new in that? It had been discussed several times here and on other forums as well. "RSE" is just another made-up position with a very average pay structure -- even this is not new. However, RSEs (or just general software training) may help research groups establish a structure on how to format code, put some standards in place, and at least have some basic tests. This way, more people can read/modify the code efficiently (more = not necessarily general public, but it at least helps incoming grad students/postdocs to pick up the project easily).
svalorzen 6y ago
I don't really agree with the reasons given, even though my conclusions are the same. The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus giving no opportunity to structure the code in advance so it is easy to change. To make a concrete example, imagine writing an application where requirements changed unpredictably every day, and where the scope of those changes is unbounded. The closest to "orderly" I think research code can become would be akin to Enterprise style coding, where literally everything is an interface and all implementation details can be changed in all possible ways. We already know how those codebases tend to end..
- j-pb 6y ago
  There's only one way to solve this: Simplicity. Ironically this is also what occams razor would demand from good Science, so you'd have a win win scenario, where you both create good software and good research, because you focus on the simplest most minimal approach that could possibly work.
  - gmueckl 6y ago
    How do you keep a codebase simple when you need have things in it like implementations of state of the art algorithms to compare against and the previous iterations of your own method so that you can test whether you're actually improving? Then, depending on what you're doing, there's also all the extra nontrivial code for tests and sanity checks of all these implementations. Simplicity is a nice dream. The realities of research are very often stacked against it.
    - childintime 6y ago
      It seems Julia has the answer: https://arstechnica.com/science/2020/10/the-unreasonable-effectiveness-of-the-julia-programming-language/ https://arstechnica.com/science/2020/10/the-unreasonable-eff...
      - gmueckl 6y ago
        I can't quite follow what the article is trying to describe because of the heavy use of analogies. A Google search makes it look like Julia has a mechanism where you can extent the sets of overloads of a function or method outside the original module. The terminology is different (functions have methods instead of overloads in their speak). I don't see how that feature solves the problem in practice.
        cdsousa 6y ago
        Julia multiple dispatch allows dynamic polymorphism. Check https://stackoverflow.com/questions/1801216/what-is-the-difference-between-multiple-dispatch-and-method-overloading https://stackoverflow.com/questions/1801216/what-is-the-diff... and https://www.youtube.com/watch?v=kc9HwsxE1OY https://www.youtube.com/watch?v=kc9HwsxE1OY
ellimilial 6y ago
I am actually quite surprised at the figure of 73% research-related code packages not being updated after the publication, was expecting it to be higher.
- rovr138 6y ago
  Same. But it could be an issue with the sample. 213 in a span of 14 years is not a lot. Also, a question. If you publish a paper with a repo, what would be the best way to handle the version in the paper matching the repo in the future? An opinion, there is such a thing as software being ‘done’ and ‘as is’. Software solves a need. After that’s meet, that’s it. There’s also this part that strikes me, >Given a tangled mess of source code, I think I could reproduce the results in the associated paper (assuming the author was shipping the code associated with the paper; I have encountered cases where this was not true). And it strikes me as weird. The main issue to reproduce results is usually data. And depending on the dataset, it’s very hard to get. To be able to reproduce the code, I just need the paper. The code may have bugs, may stop working, may be in a different language/framework. The source of truth is the paper. This is why the paper was published.
  - speters 6y ago
    > Also, a question. If you publish a paper with a repo, what would be the best way to handle the version in the paper matching the repo in the future? You can include the hash of the commit used for your paper.
    - rovr138 6y ago
      oh, that's good. Or even a tag
  - medstrom 6y ago
    >The source of truth is the paper. This is why the paper was published. Speaking as someone who's not the best at math, I find it easier to understand what a paper is saying after I run the code and see all the intermediate results. When the code doesn't work, it takes me 20 times longer to digest a paper. They could do with only uploading code -- to me it's the shortest and most effective way to express the ideas in the paper.
dariosalvi78 6y ago
I think that incentives play a big role here. Software has near to zero value in academic evaluation and even less its update and maintenance. The only way to make research software survive is to offer packages that other researchers can also use. Maybe.
- Frost1x 6y ago
  This is changing drastically. The issue is that more and more science relies heavily on computation. Analytic platforms, computational science, modeling/simulation, etc. There's less "bench" science and more of the scientific process is being embedded in software. There's a certain degree of naivity in this process that SMEs think it's a trivial step translating their research into software. It's not, not if you demand the rigor science should be operating at. As such, many budgets are astronomically lower than they should be. This has worked in the past but as more science moves into software and it becomes more critical to the process, you must invest in the software and it's not going to be cheap. The shortcuts taken in the past won't cut it. There's a bigger issue in that as a society we don't want to invest in basic research so it's already cash strapped. Combine that with research scientists who already have to cut corners with the massive cost quality software will take and you're creating a storm where science will either produce garbage or well need to reevaluate how we invest in software systems for science.
bsenftner 6y ago
Not all research software is a tangled mass. I have extensively worked as a "quant" (before the term was popular) for math, medical, network, media, and physics researchers as my side gig for decades. I'd say about 1/3 of the home brewed research software is constructed with fairly reasonable assumptions, the authors are scientists after all, and I am able to grow their basic setup into a framework they intimately understand and prefer to use. More than once I've found brilliantly engineered software not unlike what I'd find at a pro software development firm.
k__ 6y ago
I did some research projects, but the problem is that they are a mix of regular projects and experiments. Things like Nix worked out great, but other stuff I saw is a tangled mess of Java grown over the last 10 years, written by 30 different students that didn't talk or let alone knew each other.
fabian2k 6y ago
There is no real incentive to organize and clean up the code, even if the scientists involved have the skills to write well-organized software. And organizing this kind of code that often starts in a more exploratory way is a pretty large amount of additional effort. This kind of effort is simply not appreciated, and if spending time on it means you publish fewer papers it's a net negative for your career. I'd settle for just publishing the code at all, even if it is a tangled mess. This is still not all that common in the natural sciences, though I have a bit of hope this will change.
- hnedeotes 6y ago
  Yeah I mean, if your study is implying the apocalypse (or even if not, but more so if that's the case) you better put the code there, because that's what the scientific method requires, how should I believe your conclusions or and cute graphs if I can't see how you arrived at it? Maybe it was drawn in Narnia for all I know, maybe it has significant errors, or it's so tailored to produce those results that it's irrelevant. And if the tools and methods you used for arriving at them are so messy that you dare not publish them what does that tell me about: - your process; - the organisation of your ideas; - the conclusions or points made in the paper? I don't mean it has to be idiomatic well written code, but it should be readable enough to be followed.
deleted 6y ago
[deleted]
mkl95 6y ago
If you want to write good research software, a good way is to have professional developers implement it. I worked closely with an NLP researcher for a while on a project that had received a hefty state grant. She knew more or less what her team needed, but she needed someone to implement it cleanly and in a way that would not make users step on each others toes. The chances of that project being a buggy mess would have been pretty high if it had been written by people who don't write software for a living. And maybe that's OK.
- mattkrause 6y ago
  Here's the problem with hiring a pro. The workhorse NIH grant[0] is a R01 with a $250,000/year x 5 years "modular" budget. Most labs have, at most, one. Some have two, and a very few have more than that. This covers everything involved in the research: salaries (including the prof's), supplies, publication fees, etc. Suppose you find a programmer for $75k. With benefits/fringe (~31% for us, all-in), that's nearly $100k/year. If the principal investigator (prof, usually) takes a similar amount out of the grant, there's very little money left to do the (often very expensive) work. In contrast, you can get a student or postdoc for far less--and they might even be eligible for a training grant slot, TAship, or their own fellowship, making their net cost to the lab ~$0. This would be easy to fix: the NIH already has a program for staff scientists, the R50. However, they fund like two dozen per year; that number should be way higher. [0] Other mechanisms exist at the NIH--and elsewhere--but NSF (etc) grants are often much smaller.
  - qudat 6y ago
    > In contrast, you can get a student or postdoc for far less Yeah I totally agree on this part. The academic system relies not on monetary compensation for its labor, rather it provides them reputation by getting their names on a paper. I worked essentially for free for a lab in my spare time for 4 years. They get to the result they want, even if its built on a shaky foundation, and for basically free (it doesn't cost anything to put a name on a paper). At the end of the 4 years the dream of getting my name on a paper didn't even pan out (lab was ramping down and was essentially a teaching research lab by the time I showed up).
bjarneh 6y ago
I agree that academia produces its fare share of spaghetti code, but I don't think all of his arguments are correct. > writing software is a low status academic activity This is just not true. People like: Stallman, Knuth, Ritchie, Kernighan, Norvig or Torvalds are not considered as people of low status in the academic world. Writing horrible spaghetti code in academia may be considered "low status"; but that's another story. He should compare apples to apples. I.e. do people who work in academia write better or worse code there; compared to when they work for a business? I.e. they should be compared to themselves in different situations, not to some imaginary high coding standard that I've never seen anywhere. In my own experience from academia at least I'd say that the lack of deadlines; the possibility to do whatever I want, plus the lack of management, creates much higher quality software in academia. When you work commercially, you will churn out embarrassing stuff just to make some stuff work before a deadline.
- pca006132 6y ago
  I think rather than low status academic activity, it is just not valued in the academia... The code is usually just the by-product of the paper, and higher quality code does not translate to higher quality paper. When your code works, you probably already developed the main part of your paper, and you would have no incentives to improve on your program if what you want is just publication... At least this is what I think.
  - bjarneh 6y ago
    I've seen many research papers where all they disclose about the software is some pseudo code + some tables with timing results to prove their "performance gains". For those types of papers I agree with your statement. But in many academic scenarios others will want to inspect the source, and the quality of that code is certainly something that will add or subtract from your "status" in the academic world so to speak :-)
    - pca006132 6y ago
      OK, perhaps I should read more papers :)
neffy 6y ago
I don't think research code in aggregate is any worse than any other source for code. If we had the same kind of visibility into all the commercially written code, it would be the same pattern of some well structured, and some a complete mess, without any correlation with the companies concerned, but with a lot of correlation to the ability of the author. The recent example of Citibank´s loan payment interface comes immediately to mind. So does Imperial's Covid model (the one that had timing issues when run on different computers.)
- AshamedCaptain 6y ago
  Exactly. You can imagine most engineering software to be in a similar state as research code. It's just that people get to see research code.
asdf_snar 6y ago
This article seems to cover research software that even can be built. I claim the majority of _code_ written to support research articles is a collection of scripts written to produce figures to put in the paper. Even when the article is about an algorithm, the script that runs this algorithm is just good enough to produce the theoretically expected results; it is never tested, reproduced, or published, never mind being updated after publication. While others here point out that researchers = bad programmers is a lazy excuse, I think it is important to point out just how steep the learning curve of computer environments can be for the layperson that uses Excel or MATLAB for all their computational work. It can be a huge time investment to get started with tools, such as git or Docker, that we take for granted. I think recognizing this dearth of computer skills is a first step towards training researchers to be computer-competent. Currently, I find the attitude among academics (especially theorists) to be dismissive of the importance of such competencies.
- Dumblydorr 6y ago
  I am a research scientist published via R, Stata, and Excel analyses. My code documents wouldn't be helpful since the data is all locked up due to HIPAA concerns. We're talking names, health conditions, scrambled SSN, this isn't reproducible because the data is locked to those without security clearance. The code itself is a ton of munging and then some basic stat functions. This information can be gleaned from the methods section of the article anyway. So, really, my field of public health doesn't use GitHub or sharing much, there's simply too little benefit to the researcher to share their code. There's an unwarranted fear of getting your work poached. In modern science, publications are everything, they determine your career. Enabling your direct competitors, those who want the same grants and students and glories, is not common in science.
  - asdf_snar 6y ago
    I don't disagree with you on any points. I have some academic friends who mostly do "a ton of munging and then some basic stat functions", as you say (but with less sensitive data). The problem is that their workflow is prone to human error. Even though the stat functions are simple, the proper labeling of inputs and outputs is less reliable. I have some research published for which I wrote MATLAB code years ago. I trust the fundamental results but not the values displayed in the tables. I would have personally benefited from rudimentary version control and unit testing.
milliams 6y ago
Disclaimer: I am one of the trustees of the mentioned charity, The Society of Research Software Engineering. You say that you don't see it having much "difference with regard status and salary". The problem here is two-fold. Firstly, salaries at UK universities are set on a band structure and so an RSE will earn a comparable amount to a postdoc or lecturer. These aren't positions that are known for high wages and historically the reason that people work in research is not for a higher salary. As for status, I can see that the creation of the Research Software Engineer title (since about 2012) has done great good for improving the status of people with those skills. Before they were "just" postdocs with not many papers but now they can focus on doing what they do best and have career paths which recognise their skills. My role (at the University of Bristol - https://www.bristol.ac.uk/acrc/research-software-engineering/ https://www.bristol.ac.uk/acrc/research-software-engineering...) is focused almost entirely on teaching. I'm not trying to create a new band of specialists who would identify as RSEs but rather provide technical competency for people working in research so that the code they write is better. There is a spectrum of RSEs from primarily research-focused postcode who write code to support their work along to full-time RSEs whose job is to support others with their research (almost a contractor-type model). We need to have impact all the way along that spectrum, from training at one end to careers and status at the other. For more info on the history of the role, there's a great article at https://www.software.ac.uk/blog/2016-08-17-not-so-brief-history-research-software-engineers-0 https://www.software.ac.uk/blog/2016-08-17-not-so-brief-hist... written by one of the founding members of the Society of Research Software Engineering.
orange_tee 6y ago
Well as somebody who has written research software, I don't agree that research software is a "tangled mess". A couple of points, 1. often when I read read software written by profession programmers I find it very hard to read because it is too abstract, almost every time I try to figure out how something works, it turns out I need to learn a new framework and api, by contrast research code tends to be very self contained 2. when I first wrote research software I applied all the programming best practices and was told these weren't any good; turns out using lots of abstraction to increase modularity makes the code much slower, this is language dependent of course 3. you will find it much harder to read research code if you don't understand the math+science behind it > many of those writing software know very little about how to do it This is just not true. I found in my experience that people writing research software have a very specific skillset that very very few industry programmers are likely to have. They know how to write good numerics code, and they know how to write fast code for super computers. Not to mention, interpreting the numerics theory correctly in the first place is not a trivial matter either.
- exdsq 6y ago
  Point 1 is so true, I think it’s why I like Golang without generics so people can’t go crazy with abstractions.
- taeric 6y ago
  Your points apply to industry, too. I heretically push flatter code all the time. I'm not against abstraction, but it is easy to fall into the trap of building a solution machine, but missing the solution you need.
- acmj 6y ago
  Quite a few professional programmers evaluate the quality of code by "look": presence of tests, variable length, function length etc. However, what makes great code is really the code structure and logical flows behind. In my experience, good industrial programmers are as rare as good academic programmers. Many industrial programmers make a fuss about coding styles but are not really good at organizing structured code for a medium sized project.
- 6y ago
ArtWomb 6y ago
There's a huge digital divide forming as well. Between the hardware a junior software engineer at a well funded research institution such as DeepMind has access to. Compared to the postdoc in Theoretical Physics at Princeton. Who is expected not only to write software. But maintain hardware for a proprietary "supercomputer" that was probably cast off ages ago from a government lab or wall street. We don't expect Aerospace / Mechanical engineering students to learn metalworking. They typically have access to shop technicians for that work. Why not persuade university administrators to similarly invest in in-house software engineering talent. Generalists who can provide services to any problem domain: from digital humanities to deep reinforcement learning?
- einpoklum 6y ago
  > We don't expect Aerospace / Mechanical engineering students to learn metalworking. They typically have access to shop technicians for that work. You'd be surprised, but that is often not the case. Lack of sufficient funding, or technicians being dicks, or mis-management by PIs, often result in graduate students having to do the technical work of metalwork, welding, lab equipment calibration, and a bunch of other tasks. Sometimes they even have to operate heavier machinery, or lasers etc without the minimum reasonable technical staff support. I know this from my time on the executive committee of my old university's Grad Student Organization.
- mattkrause 6y ago
  > We don't expect Aerospace / Mechanical engineering students to learn metalworking. Umm...we sorta do. As a neuroscience postdoc, I have done virtually everything from analysis to zookeeping, including some (light) fabrication. We outsource really difficult or mass-production stuff to pro, and there's a single, very overworked machinist who can sometimes help you, but most of the time it's DIY.
lmilcin 6y ago
So here is simple fact. It does not make sense to judge any piece of code that does not meet "highest standard" to be a tangled mess. There are valid reasons to have varying quality of code and also the idea of quality might be changing from problem to problem and project to project. A quality of code that governs your car's ECU should be different from quality of code that some research team threw together to demonstrate an idea. A coding project should achieve some kind of goal or set of goals as efficiently as possible and in many valid cases quality is just not high on the list and for a good reason. Right now I am working on a PoC to verify an idea that will take a longer time to implement. We do this because we don't want to spend weeks on development just to see it doesn't work or that we want to change something. So spending 2-3 days to avoid significant part of the risk of the rest of the project is fine. It does not need to be spelled out that the code is going to be incomplete, messy and maybe buggy. There is also something to be said for research people to be actually focusing on something else. Professional developers focus their careers on a single problem -- how to write well (or at least they should). But not all people do. Some people actually focus on something else (physics maybe?) and writing code is just a tool to achieve some other goals. If you think about people working on UIs and why UI code tends to be so messy, this is also probably why. Because these guys focus on something else entirely and the code is there just to animate their graphical design.
- sdwvit 6y ago
  Yeah but you spend more time debugging, that if you write it once with a good architecture and unittests let’s say
knuthsat 6y ago
I think there's not enough researchers that publish code. For example, discrete optimization research (nurse rostering, travelling salesman, vehicle routing problem, etc.) is filled with papers where people are evaluating their methods on public benchmarks but code never sees the day. There's a lot of state-of-the-art methods that never have their code released. I'm pretty sure it's like that elsewhere. Machine learning and deep learning for some reason has a lot of code in the open but that's not the norm. I'd prefer the code to be open first. Once that's abundant then I might prefer the code to also be well designed.
- lou1306 6y ago
  > I think there's not enough researchers that publish code. I agree, although lately there's been some effort by academia to make authors publish their code, or at least disclose it to the reviewers. Several conferences have an artifact evaluation committee, which tries to reproduce the experimental part of submitted papers. Some conferences actually require a successful artifact evaluation to be accepted (see, for instance, the tool tracks at CAV [1] and TACAS [2]). Others, while not requiring an artifact evaluation, may encourage it by other means. The ACM, for instance, marks accepted papers with special badges [3] reflecting how well the alleged findings can be reproduced and whether the code is publicly available. [1] http://i-cav.org/2021/artifact-evaluation/ http://i-cav.org/2021/artifact-evaluation/ [2] https://etaps.org/2021/call-for-papers https://etaps.org/2021/call-for-papers [3] https://www.acm.org/publications/policies/artifact-review-and-badging-current https://www.acm.org/publications/policies/artifact-review-an...
  - cratermoon 6y ago
    This feels like the right approach. If peer review were to include artifact evaluation, including some kind of code review, and require certain standards be met for acceptance, things would change. As others have noted here, the mechanisms of grant-funded work strongly discourage attention to code quality, and that would have to change as well. I'm not in academia now, but I started out my career doing sysops and programming in a lab at a medical school and have worked with academics a bit since. I don't do it much because it's basically volunteer work, and it's almost impossible to contribute meaningfully unless you are also well-versed in the field.
screye 6y ago
> writing software is a low status academic activity Yep, that's the one liner right there. The incentives simply do not match the complaints. Researchers already work upwards of 60 hrs/wk on most occasions. Alongside writing code, they also have to do actual research, write papers, give talks and write grants. All of the latter tasks are primary aspects of their jobs and are commensurately rewarded. The only situation where a well coded tool is rewarded, is when a package blows up, which is quite rare. Like all fields, the high-level answer to such questions is rather straightforward. The individual contributors align their efforts to the incentives. Find a way to incentivize good research code, and we will see changes overnight.
nirse 6y ago
What doesn't really get mentioned in the article, is that a lot of academic software was written by a single developer. All bigger software projects, academic or not, that were only built and maintained by a single person tends to become messier and messier with time. Perhaps most software suffers from that, that over time it becomes a mess, but having more developers look at code (and enough time, and many other factors) can certainly help to keep things in better shape.
- Robotbeat 6y ago
  Plus it becomes impossible to get multiple developers to work on the code if they can’t understand it because of its messiness, so there’s a bit of survivor’s bias and stronger motivation to clean the code up to be comprehensible to others when you have multiple people working on it. Also, I feel personally attacked by the headline. :) It is for this reason I try to keep my code and models pretty simple, only two or three pages of code (or ideally a single page), and I don’t try to do too many things with one program, and I choose implementations and algorithms that are simpler to implement to make concise code feasible (sometimes at the expense of speed or generality).
SavageBeast 6y ago
I'm currently refactoring a fairly large piece of research code myself. It was written with lean startup thinking in that a little code ought to produce some value in its results. If i was able to eeek some usefulness out of this code, then Id put more energy into it. Otherwise I was perfectly happy to Fail Fast and Fail Cheap. How did it become such a mess in the first place? Simple - I didn't know my requirements when I started writing it. I built it to do one thing. In running it I learned more things (this is good - why you build stuff like this in the first place). The code changed rapidly to accommodate these lessons. It wasn't long before I was running into limitations in the design of the underlying libs I was using etc. Of course I could find a way to make it work but it wasn't going to win any Software Design Awards. Im happy to report that despite ending up a tangled mess, it actually helped me to come to understand and conquer a very specific kind of problem. In doing so I learned the limitations of commercially available tooling, the limitations of commercially available data, not to mention a great deal about the problem domain itself. This research software has earned its keep and is now being cleaned up into a more organized, near commercial quality kind of project. Im glad I threw out "architecture" when I first started with this. It could have gone the other way where I had a very well built piece of code that didn't in fact perform any useful function.
- analog31 6y ago
  I believe that of all the lessons to come from contemporary software development, constant refactoring may be the most valuable. The spaghetti monster looms large when you're in the heat of battle. But we've all got some idle time for whatever reason. I spend some time every week doing a couple of things: 1) Reading about good techniques. 2) Working through old code and cleaning it up. Because changing your code could always break it, refactoring also reinforces the habit of writing code that can be readily tested -- also a good thing.
  - amelius 6y ago
    Makes you wonder why most languages don't come with good refactoring tools.
currymj 6y ago
as people are saying, the typical software engineering advice simply wouldn't work in a research context. one exception is the most basic stuff - people should use version control, do light unit testing, and explicitly track dependencies. These weren't really done in the past but are becoming more and more common, fortunately. I think if software engineering experts actually sat down, looked at how researchers work with computers, and figured out a set of practices to follow that would work well in the research context, they could do a lot of good. This is really needed. But the standard software engineering advice won't work as it is, it has to be adapted somehow.
- pydry 6y ago
  Another issue is that the standard software engineering advice doesn't guarantee clean code either.
TomMasz 6y ago
I once interviewed for a programming job that was a bit of bait and switch. The hiring manager showed me a foot-high stack of green bar paper that was Fortran code written by optical scientists that I was expected to convert to C. He was somewhat surprised when I declined and ended the interview. I pity whoever got stuck with that task.
analog31 6y ago
>>> writing software is a low status academic activity; it is a low status activity in some companies, but those involved don’t commonly have other higher status tasks available to work on. If measured by compensation, then research is a low status activity. Perhaps more precisely, researchers have low bargaining power. But I don't think that academics actually analyze activities in such detail. The PI might not even know how much programming is being done. The researcher is programming, not because they see it as a way to raise (or lower) their status, but because it's a force multiplier for making themselves more productive overall. Though I work in industry, I'm a "research" programmer for all intents and purposes. I program because I need stuff right away, and I do the kind of work that the engineers hate. Reacting to rapidly changing requirements on a moment's notice disrupts their long term planning. Communicating requirements to an engineer who doesn't possess domain knowledge or math skills is painful. Often, a working piece of spaghetti code that demonstrates a process is the best way to communicate what I need. They can translate it into fully developed software if it threatens to go into a shipping product. That's a good use of their time and not of mine. >>> Why would a researcher want to invest in becoming proficient in a low status activity? To get a better job. I sometimes suspect that anybody who is good enough at programming to get paid for it, is already doing so. >>> Why would the principal investigator spend lots of their grant money hiring a proficient developer to work on a low status activity? Because they don't know how to manage a developer. Software development is costly in terms of both time and effort, and nobody knows how to manage it. Entire books have been written in this topic, and it has been discussed at length on HN. A software project that becomes an end unto itself or goes entirely off the rails can eat you alive. Finding a developer who can do quantitative engineering is hard, and they're already in high demand. It may be that the PI has a better chance managing a researcher who happens to know how to translate their own needs into "good enough" code, than to manage a software project.
nowardic 6y ago
A nice counter example of research software code that adheres to general software engineering best practices and is easy to pick up and use is the OSMNX project: https://github.com/gboeing/osmnx https://github.com/gboeing/osmnx Props to Geoff for setting a nice standard.
f6v 6y ago
Keep in mind that there're different kinds of research software. Take Seurat[1] as an example. There's CI, issue tracking, etc. It might not be the prettiest code you ever seen, but it absolutely has to be maintainable as it's being actively developed. Such projects are rare, but the low quality is often an indication of a software that isn't used by anyone. 1. https://github.com/satijalab/seurat https://github.com/satijalab/seurat
- cratermoon 6y ago
  Also things like EISPAC, BLAS, LINPACK and so on, for FORTRAN. Back in the 70s my dad worked a bit with them when he was employed for UTHERCC: The University of Texas Health, Education, and Research Computer Center. You can find references to UTHERCC in papers from that era. Come to think of it, something like UTHERCC might be exactly what is needed to help the current situation.
shadowgovt 6y ago
One of the biggest eye-openers for me as an undergrad was when, upon getting to the point where I'd have to decide whether to pursue graduate education or exit academia and join the workforce, I began to look at the process for publishing novel computer science. To be clear, novel computer science is valuable and the lifeblood of the software engineering industries. But the actual product? I discovered of myself that I like quality code more than I like novel discovery, and the output of the academic world ain't it. Examples I saw were damn near pessimized... not just a lack of comments, but single-letter variables (attempting to represent the Greek letters in the underlying mathematical formulae) and five-letter abbreviated function names. I walked away and never looked back. If there's one thing I wish I could have told freshman-year me, it's that software as a discipline is extremely wide. If you find yourself hating it and you're surprised you're hating it, you may just be doing the kind that doesn't mesh with your interests.
cryptica 6y ago
Math and physics are a tangled mess so it's not surprising that mathematicians and physicists write code which looks like a tangled mess. Mathematicians and physicists are trained to handle ambiguous concepts and they can work with weird abstractions which are far detached from reality. Unlike programming languages, the language of math is full of gaps - This requires the reader to make assumptions using past knowledge and conventions. Computers, on the other hand cannot make assumptions so the code must be extremely precise and unambiguous. Writing good code requires a different mindset; firstly, it requires acknowledging that communication is extremely ambiguous and that it takes a great deal of effort to communicate clearly and to choose the right abstractions. A lot of the best coders I've met struggle with math and a lot of the best mathematicians I've met struggle with writing good code.
- porker 6y ago
  Thank you. Having trained as a physicist, this matches my experience.
civilized 6y ago
I see people here saying research is like writing software with fast-changing requirements. I can see how that could seem like an adequate analogy to a software engineer, but it's not. Researchers use code as a tool of thought to make progress on very ambiguous, high-level problems that lack pre-existing methodology. Like, how could I detect this theoretical astrophysical phenomenon in this dataset? What would it take to predict disease transmission dynamics in a complex environment like a city? Could a neural network leveraging this bag of tricks in some way improve on the state-of-the-art? If you have JIRA tickets like that in your queue, maybe you can compare your job to that of a researcher.
deeeeplearning 6y ago
Why is this surprising? Has anyone been inside a Chemistry or Bio lab? You think that what happens in those labs to get research done is industrial grade?
sumanthvepa 6y ago
What I find really surprising about research software, is that even people in Computer Science write poorly designed code as part of their research. I would have imagined that they would be better qualified to create good code. Just goes to show that Software Engineering != Computer Science.
ejz 6y ago
This is actually the opportunity for our startup. I think there is generally a great opportunity to be the Databricks of a lot of academic software. We're starting in a big research area in biology :)
optiklab 6y ago
I’m an engineer from the other side of researches or science, but somehow interested in the topic. Recently, I’ve learned about great work done by Grigori Fursin and entire community of reserach engineers with the goal to make research software more applicable to the industries by doing it with some kind of framework inside. I want to leave some links here, if you don’t mind to watch it – the talk is called ” Reproducing 150 Research Papers and Testing Them in the Real World”:ACM page with webcast https://event.on24.com/wcc/r/2942043/9C904C7AE045B5C92AAB2CF216826732 https://event.on24.com/wcc/r/2942043/9C904C7AE045B5C92AAB2CF... Also, source docs available here: https://zenodo.org/record/4005773?fbclid=IwAR1JGaAj4lwCJDrkJdgJQHoWroUR6zqIW1STS0D2BRkeaFf6iD0U-KakZSM#.YDPz_nlRVHZ https://zenodo.org/record/4005773?fbclid=IwAR1JGaAj4lwCJDrkJ... And, their solution product https://cknowledge.io/ https://cknowledge.io/ and source code https://github.com/ctuning/ck https://github.com/ctuning/ck I guess it should be helpful to the researchers community.
cratermoon 6y ago
I was reminded that there are research packages like LINPACK, BLAS, and EISPACK for FORTRAN (and some other languages) that have been maintained since the 70s and are still in use. Back in the 70s my dad was working for an organization called UTHERCC, the University of Texas Health, Education, and Research Computer Center, and these libraries were some of the code he worked with. You can find references to UTHERCC in papers from the time, although I don't think it exists under that name. Maybe institutions need something like UTHERCC as an ongoing department now.
hntrader 6y ago
These are some concepts that I believe in for research code. Research code shouldn't be a monolith. Each hypothesis should be a script that follows a data pipeline pattern. If you have a big research question, think about what the most modular progression of steps would be along the path from raw data to final output, and write small scripts that perform each step (input is the output from the previous step). Glue them all together with the data pipeline, which itself is a standalone, disposable script. If step N has already been run, then running the pipeline script once again shouldn't resubmit step N (as long as the input hasn't changed since the last run). This "intermediate data" approach is useful because we can check for errors each step on the way and we don't need to redo calculations if a particular step is shared by multiple research questions. I was taught this by a good mentor and I've been using this approach for many years for various ML projects and couldn't recommend it more highly.
- porker 6y ago
  This, absolutely this. I looked over a friend's PhD program because the results were unstable. I knew nothing about the domain which was a large disadvantage, but on the code front it was a monolith following a vague data pipeline approach. Unfortunately components wouldn't run separately and there were only a single end to end tests taking hours to run. Had each section had its own tests, diagnosing which algorithm(s) were malfunctioning would have been easier. We never did.
brakus127 6y ago
Structure is great for a well understood problem space, but this is not usually the case when working working something novel. As a researcher your focus should be on learning and problem solving, not creating a beautiful code base. Imposing too many constraints early on can negatively impact your project later on. In the worse case, your code starts to limit the way you think about your research. I agree that there are some general best practices that should be applied to nearly all forms of coding, but beyond that it's a balance. The same thinking should be used when adding regulation to an industry. Heavy regulation on a rapid developing industry can stifle innovation. Regulation (if needed), should be applied as our understanding of the industry increases.
- bordercases 6y ago
  Results need to be refined so that the way they were first formulated doesn't get in the way of their replication. At scale, this too becomes a cost to industry. In the small, this isn't different from taking a lab notebook and making it clearer and better summarized so that it can be passed on to the poor sucker who has to do what you did after you move on to another project. Furthermore, software projects that are put under the same iterative stress you imply for R&D inevitably go through a refactoring phase so that performance isn't affected in the long run.
  - brakus127 6y ago
    Agreed that there should be a minimum bar for "completed" research code such as reproducibility and a clear summary, but engineers shouldn't expect the first version of a new algorithm to be easy to understand without additional material or ready for production without a complete rewrite.
tryonenow 6y ago
Not all "research" code is equal. I would imagine that research code closer to hard sciences and cutting edges is more difficult to keep clean. The trouble is when you're breaking new ground in applied science, you don't necessarily know how well the new tech will work, and exactly what you'll be able to do with it. As development progresses your expectations must be adjusted, and it is impossible to forecast the direction of highly experimental research since progress is typically incremental and constantly dependent on the most recent results. There are many paths to scaling a mountain, so to speak, and sometimes for any of a multitude of reasons you end up on another peak long after you've started climbing.
yudlejoza 6y ago
This topic is near and dear to my heart and at a quick glance, I pretty much agree with all/most of this post. I gained multiple years of industry software engineering experience before joining academia (non-CS, graduate-level). And I was flabbergasted at the way software and programming is treated in research setting where the "domain" is not CS or software itself. It took me a few years just to get a hint of what on earth these people (my collaborators who program side-by-side with me) are thinking, and what kind of mindset do they come from. Then I took a short break and went to the industry. Software engineering, hardcore CS; no domain, no BS. I was expecting that it would feel like an oasis. It didn't. Apart from a handful of process improvements, like use of version control, issue tracking, deadline-management, the quality of the tangled mess of the code was only slightly better. Initially I took away the lesson that it's the same in academia and industry. But on further reflection there are two big differences: - The codebase I worked on in the industry was at least 10x bigger. Despite that, the quality was noticeably better. - More importantly, I could connect with the my coworkers in the industry. If I raised a point about some SwE terminology like test-driven dev, agile, git, whatever, I could have a meaningful discussion. Whereas in academia, not only most domain experts knew jack about 90% of software-engineering concepts and terminology, they were expert at hiding their ignorance, and would steer the conversation in a way that you wouldn't know if they really didn't know or knew too much. I never got over that deceitful ignorance mixed with elitist arrogance. In the end, I do think that, despite enormous flaws, the industry is doing way better than academia when it comes to writing and collaborating on software and programming, and that the side-by-side comparison of actual codebases is a very small aspect of it.
RocketSyntax 6y ago
jupyterlab github issue advocating for documenting research: https://github.com/jupyterlab/team-compass/issues/121 https://github.com/jupyterlab/team-compass/issues/121
innerlee 6y ago
Checkout the openmmlab project [1], where messy research codes in computer vision are rewritten an reorganized into a coherent whole. If there are more researchers join this, then not only research are fully reproducible, much more accessible to everyone, but also be compared fairly. (I'm the maintainer of mmpose [2], mmaction2 [3], and mmediting [4]) [1] https://github.com/open-mmlab https://github.com/open-mmlab [2] https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose [3] https://github.com/open-mmlab/mmaction2 https://github.com/open-mmlab/mmaction2 [4] https://github.com/open-mmlab/mmediting https://github.com/open-mmlab/mmediting
jonbronson 6y ago
I worked at a productive computing research institute for a number of a years. I cannot count the number of times I found research teams duplicating critical algorithms. Research Scientists not only pay the price of the spaghetti nature of the code, they pay it over and over again by not sharing and improving on what has already been built by previous research groups. The software industry has its own share of problems, but from what I've seen the research community is still largely operating on an outdated software model that shuns open collaboration out of fear of being "scooped".
naebother 6y ago
Good. Please, keep on writing more spaghetti code. Dumb people like myself need it to make a living "cleaning" it up. Circle of life.

182 points by hacksilver 6y ago

One of the things which has helped derail my own research career [1] is the tendency to not write tangled-mess code, and to publish and maintain much of my research code after I was supposed to be done with it. Annoyingly, more people now know of me due to those pieces of software than for my research agenda. :-( [1] : Not the only thing mind you.

jimmyvalmer 6y ago
C'mon, no good deed goes unpunished. Everyone knows that.

chilukrn 6y ago

I agree the post makes valid points, but is there anything new in that? It had been discussed several times here and on other forums as well. "RSE" is just another made-up position with a very average pay structure -- even this is not new. However, RSEs (or just general software training) may help research groups establish a structure on how to format code, put some standards in place, and at least have some basic tests. This way, more people can read/modify the code efficiently (more = not necessarily general public, but it at least helps incoming grad students/postdocs to pick up the project easily).

svalorzen 6y ago

I don't really agree with the reasons given, even though my conclusions are the same. The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus giving no opportunity to structure the code in advance so it is easy to change. To make a concrete example, imagine writing an application where requirements changed unpredictably every day, and where the scope of those changes is unbounded. The closest to "orderly" I think research code can become would be akin to Enterprise style coding, where literally everything is an interface and all implementation details can be changed in all possible ways. We already know how those codebases tend to end..

j-pb 6y ago
There's only one way to solve this: Simplicity. Ironically this is also what occams razor would demand from good Science, so you'd have a win win scenario, where you both create good software and good research, because you focus on the simplest most minimal approach that could possibly work.
- gmueckl 6y ago
  How do you keep a codebase simple when you need have things in it like implementations of state of the art algorithms to compare against and the previous iterations of your own method so that you can test whether you're actually improving? Then, depending on what you're doing, there's also all the extra nontrivial code for tests and sanity checks of all these implementations. Simplicity is a nice dream. The realities of research are very often stacked against it.
  - childintime 6y ago
    It seems Julia has the answer: https://arstechnica.com/science/2020/10/the-unreasonable-effectiveness-of-the-julia-programming-language/ https://arstechnica.com/science/2020/10/the-unreasonable-eff...
    - gmueckl 6y ago
      I can't quite follow what the article is trying to describe because of the heavy use of analogies. A Google search makes it look like Julia has a mechanism where you can extent the sets of overloads of a function or method outside the original module. The terminology is different (functions have methods instead of overloads in their speak). I don't see how that feature solves the problem in practice.
      - cdsousa 6y ago
        Julia multiple dispatch allows dynamic polymorphism. Check https://stackoverflow.com/questions/1801216/what-is-the-difference-between-multiple-dispatch-and-method-overloading https://stackoverflow.com/questions/1801216/what-is-the-diff... and https://www.youtube.com/watch?v=kc9HwsxE1OY https://www.youtube.com/watch?v=kc9HwsxE1OY

ellimilial 6y ago

I am actually quite surprised at the figure of 73% research-related code packages not being updated after the publication, was expecting it to be higher.

rovr138 6y ago
Same. But it could be an issue with the sample. 213 in a span of 14 years is not a lot. Also, a question. If you publish a paper with a repo, what would be the best way to handle the version in the paper matching the repo in the future? An opinion, there is such a thing as software being ‘done’ and ‘as is’. Software solves a need. After that’s meet, that’s it. There’s also this part that strikes me, >Given a tangled mess of source code, I think I could reproduce the results in the associated paper (assuming the author was shipping the code associated with the paper; I have encountered cases where this was not true). And it strikes me as weird. The main issue to reproduce results is usually data. And depending on the dataset, it’s very hard to get. To be able to reproduce the code, I just need the paper. The code may have bugs, may stop working, may be in a different language/framework. The source of truth is the paper. This is why the paper was published.
- speters 6y ago
  > Also, a question. If you publish a paper with a repo, what would be the best way to handle the version in the paper matching the repo in the future? You can include the hash of the commit used for your paper.
  - rovr138 6y ago
    oh, that's good. Or even a tag
- medstrom 6y ago
  >The source of truth is the paper. This is why the paper was published. Speaking as someone who's not the best at math, I find it easier to understand what a paper is saying after I run the code and see all the intermediate results. When the code doesn't work, it takes me 20 times longer to digest a paper. They could do with only uploading code -- to me it's the shortest and most effective way to express the ideas in the paper.

dariosalvi78 6y ago

I think that incentives play a big role here. Software has near to zero value in academic evaluation and even less its update and maintenance. The only way to make research software survive is to offer packages that other researchers can also use. Maybe.

Frost1x 6y ago
This is changing drastically. The issue is that more and more science relies heavily on computation. Analytic platforms, computational science, modeling/simulation, etc. There's less "bench" science and more of the scientific process is being embedded in software. There's a certain degree of naivity in this process that SMEs think it's a trivial step translating their research into software. It's not, not if you demand the rigor science should be operating at. As such, many budgets are astronomically lower than they should be. This has worked in the past but as more science moves into software and it becomes more critical to the process, you must invest in the software and it's not going to be cheap. The shortcuts taken in the past won't cut it. There's a bigger issue in that as a society we don't want to invest in basic research so it's already cash strapped. Combine that with research scientists who already have to cut corners with the massive cost quality software will take and you're creating a storm where science will either produce garbage or well need to reevaluate how we invest in software systems for science.

bsenftner 6y ago

Not all research software is a tangled mass. I have extensively worked as a "quant" (before the term was popular) for math, medical, network, media, and physics researchers as my side gig for decades. I'd say about 1/3 of the home brewed research software is constructed with fairly reasonable assumptions, the authors are scientists after all, and I am able to grow their basic setup into a framework they intimately understand and prefer to use. More than once I've found brilliantly engineered software not unlike what I'd find at a pro software development firm.

k__ 6y ago

I did some research projects, but the problem is that they are a mix of regular projects and experiments. Things like Nix worked out great, but other stuff I saw is a tangled mess of Java grown over the last 10 years, written by 30 different students that didn't talk or let alone knew each other.

fabian2k 6y ago

There is no real incentive to organize and clean up the code, even if the scientists involved have the skills to write well-organized software. And organizing this kind of code that often starts in a more exploratory way is a pretty large amount of additional effort. This kind of effort is simply not appreciated, and if spending time on it means you publish fewer papers it's a net negative for your career. I'd settle for just publishing the code at all, even if it is a tangled mess. This is still not all that common in the natural sciences, though I have a bit of hope this will change.

hnedeotes 6y ago
Yeah I mean, if your study is implying the apocalypse (or even if not, but more so if that's the case) you better put the code there, because that's what the scientific method requires, how should I believe your conclusions or and cute graphs if I can't see how you arrived at it? Maybe it was drawn in Narnia for all I know, maybe it has significant errors, or it's so tailored to produce those results that it's irrelevant. And if the tools and methods you used for arriving at them are so messy that you dare not publish them what does that tell me about: - your process; - the organisation of your ideas; - the conclusions or points made in the paper? I don't mean it has to be idiomatic well written code, but it should be readable enough to be followed.

deleted 6y ago

[deleted]

mkl95 6y ago

If you want to write good research software, a good way is to have professional developers implement it. I worked closely with an NLP researcher for a while on a project that had received a hefty state grant. She knew more or less what her team needed, but she needed someone to implement it cleanly and in a way that would not make users step on each others toes. The chances of that project being a buggy mess would have been pretty high if it had been written by people who don't write software for a living. And maybe that's OK.

mattkrause 6y ago
Here's the problem with hiring a pro. The workhorse NIH grant[0] is a R01 with a $250,000/year x 5 years "modular" budget. Most labs have, at most, one. Some have two, and a very few have more than that. This covers everything involved in the research: salaries (including the prof's), supplies, publication fees, etc. Suppose you find a programmer for $75k. With benefits/fringe (~31% for us, all-in), that's nearly $100k/year. If the principal investigator (prof, usually) takes a similar amount out of the grant, there's very little money left to do the (often very expensive) work. In contrast, you can get a student or postdoc for far less--and they might even be eligible for a training grant slot, TAship, or their own fellowship, making their net cost to the lab ~$0. This would be easy to fix: the NIH already has a program for staff scientists, the R50. However, they fund like two dozen per year; that number should be way higher. [0] Other mechanisms exist at the NIH--and elsewhere--but NSF (etc) grants are often much smaller.
- qudat 6y ago
  > In contrast, you can get a student or postdoc for far less Yeah I totally agree on this part. The academic system relies not on monetary compensation for its labor, rather it provides them reputation by getting their names on a paper. I worked essentially for free for a lab in my spare time for 4 years. They get to the result they want, even if its built on a shaky foundation, and for basically free (it doesn't cost anything to put a name on a paper). At the end of the 4 years the dream of getting my name on a paper didn't even pan out (lab was ramping down and was essentially a teaching research lab by the time I showed up).

bjarneh 6y ago

I agree that academia produces its fare share of spaghetti code, but I don't think all of his arguments are correct. > writing software is a low status academic activity This is just not true. People like: Stallman, Knuth, Ritchie, Kernighan, Norvig or Torvalds are not considered as people of low status in the academic world. Writing horrible spaghetti code in academia may be considered "low status"; but that's another story. He should compare apples to apples. I.e. do people who work in academia write better or worse code there; compared to when they work for a business? I.e. they should be compared to themselves in different situations, not to some imaginary high coding standard that I've never seen anywhere. In my own experience from academia at least I'd say that the lack of deadlines; the possibility to do whatever I want, plus the lack of management, creates much higher quality software in academia. When you work commercially, you will churn out embarrassing stuff just to make some stuff work before a deadline.

pca006132 6y ago
I think rather than low status academic activity, it is just not valued in the academia... The code is usually just the by-product of the paper, and higher quality code does not translate to higher quality paper. When your code works, you probably already developed the main part of your paper, and you would have no incentives to improve on your program if what you want is just publication... At least this is what I think.
- bjarneh 6y ago
  I've seen many research papers where all they disclose about the software is some pseudo code + some tables with timing results to prove their "performance gains". For those types of papers I agree with your statement. But in many academic scenarios others will want to inspect the source, and the quality of that code is certainly something that will add or subtract from your "status" in the academic world so to speak :-)
  - pca006132 6y ago
    OK, perhaps I should read more papers :)

neffy 6y ago

I don't think research code in aggregate is any worse than any other source for code. If we had the same kind of visibility into all the commercially written code, it would be the same pattern of some well structured, and some a complete mess, without any correlation with the companies concerned, but with a lot of correlation to the ability of the author. The recent example of Citibank´s loan payment interface comes immediately to mind. So does Imperial's Covid model (the one that had timing issues when run on different computers.)

AshamedCaptain 6y ago
Exactly. You can imagine most engineering software to be in a similar state as research code. It's just that people get to see research code.

asdf_snar 6y ago

This article seems to cover research software that even can be built. I claim the majority of _code_ written to support research articles is a collection of scripts written to produce figures to put in the paper. Even when the article is about an algorithm, the script that runs this algorithm is just good enough to produce the theoretically expected results; it is never tested, reproduced, or published, never mind being updated after publication. While others here point out that researchers = bad programmers is a lazy excuse, I think it is important to point out just how steep the learning curve of computer environments can be for the layperson that uses Excel or MATLAB for all their computational work. It can be a huge time investment to get started with tools, such as git or Docker, that we take for granted. I think recognizing this dearth of computer skills is a first step towards training researchers to be computer-competent. Currently, I find the attitude among academics (especially theorists) to be dismissive of the importance of such competencies.

Dumblydorr 6y ago
I am a research scientist published via R, Stata, and Excel analyses. My code documents wouldn't be helpful since the data is all locked up due to HIPAA concerns. We're talking names, health conditions, scrambled SSN, this isn't reproducible because the data is locked to those without security clearance. The code itself is a ton of munging and then some basic stat functions. This information can be gleaned from the methods section of the article anyway. So, really, my field of public health doesn't use GitHub or sharing much, there's simply too little benefit to the researcher to share their code. There's an unwarranted fear of getting your work poached. In modern science, publications are everything, they determine your career. Enabling your direct competitors, those who want the same grants and students and glories, is not common in science.
- asdf_snar 6y ago
  I don't disagree with you on any points. I have some academic friends who mostly do "a ton of munging and then some basic stat functions", as you say (but with less sensitive data). The problem is that their workflow is prone to human error. Even though the stat functions are simple, the proper labeling of inputs and outputs is less reliable. I have some research published for which I wrote MATLAB code years ago. I trust the fundamental results but not the values displayed in the tables. I would have personally benefited from rudimentary version control and unit testing.

milliams 6y ago

Disclaimer: I am one of the trustees of the mentioned charity, The Society of Research Software Engineering. You say that you don't see it having much "difference with regard status and salary". The problem here is two-fold. Firstly, salaries at UK universities are set on a band structure and so an RSE will earn a comparable amount to a postdoc or lecturer. These aren't positions that are known for high wages and historically the reason that people work in research is not for a higher salary. As for status, I can see that the creation of the Research Software Engineer title (since about 2012) has done great good for improving the status of people with those skills. Before they were "just" postdocs with not many papers but now they can focus on doing what they do best and have career paths which recognise their skills. My role (at the University of Bristol - https://www.bristol.ac.uk/acrc/research-software-engineering/ https://www.bristol.ac.uk/acrc/research-software-engineering...) is focused almost entirely on teaching. I'm not trying to create a new band of specialists who would identify as RSEs but rather provide technical competency for people working in research so that the code they write is better. There is a spectrum of RSEs from primarily research-focused postcode who write code to support their work along to full-time RSEs whose job is to support others with their research (almost a contractor-type model). We need to have impact all the way along that spectrum, from training at one end to careers and status at the other. For more info on the history of the role, there's a great article at https://www.software.ac.uk/blog/2016-08-17-not-so-brief-history-research-software-engineers-0 https://www.software.ac.uk/blog/2016-08-17-not-so-brief-hist... written by one of the founding members of the Society of Research Software Engineering.

orange_tee 6y ago

Well as somebody who has written research software, I don't agree that research software is a "tangled mess". A couple of points, 1. often when I read read software written by profession programmers I find it very hard to read because it is too abstract, almost every time I try to figure out how something works, it turns out I need to learn a new framework and api, by contrast research code tends to be very self contained 2. when I first wrote research software I applied all the programming best practices and was told these weren't any good; turns out using lots of abstraction to increase modularity makes the code much slower, this is language dependent of course 3. you will find it much harder to read research code if you don't understand the math+science behind it > many of those writing software know very little about how to do it This is just not true. I found in my experience that people writing research software have a very specific skillset that very very few industry programmers are likely to have. They know how to write good numerics code, and they know how to write fast code for super computers. Not to mention, interpreting the numerics theory correctly in the first place is not a trivial matter either.

exdsq 6y ago
Point 1 is so true, I think it’s why I like Golang without generics so people can’t go crazy with abstractions.
taeric 6y ago
Your points apply to industry, too. I heretically push flatter code all the time. I'm not against abstraction, but it is easy to fall into the trap of building a solution machine, but missing the solution you need.
acmj 6y ago
Quite a few professional programmers evaluate the quality of code by "look": presence of tests, variable length, function length etc. However, what makes great code is really the code structure and logical flows behind. In my experience, good industrial programmers are as rare as good academic programmers. Many industrial programmers make a fuss about coding styles but are not really good at organizing structured code for a medium sized project.
6y ago

ArtWomb 6y ago

There's a huge digital divide forming as well. Between the hardware a junior software engineer at a well funded research institution such as DeepMind has access to. Compared to the postdoc in Theoretical Physics at Princeton. Who is expected not only to write software. But maintain hardware for a proprietary "supercomputer" that was probably cast off ages ago from a government lab or wall street. We don't expect Aerospace / Mechanical engineering students to learn metalworking. They typically have access to shop technicians for that work. Why not persuade university administrators to similarly invest in in-house software engineering talent. Generalists who can provide services to any problem domain: from digital humanities to deep reinforcement learning?

einpoklum 6y ago
> We don't expect Aerospace / Mechanical engineering students to learn metalworking. They typically have access to shop technicians for that work. You'd be surprised, but that is often not the case. Lack of sufficient funding, or technicians being dicks, or mis-management by PIs, often result in graduate students having to do the technical work of metalwork, welding, lab equipment calibration, and a bunch of other tasks. Sometimes they even have to operate heavier machinery, or lasers etc without the minimum reasonable technical staff support. I know this from my time on the executive committee of my old university's Grad Student Organization.
mattkrause 6y ago
> We don't expect Aerospace / Mechanical engineering students to learn metalworking. Umm...we sorta do. As a neuroscience postdoc, I have done virtually everything from analysis to zookeeping, including some (light) fabrication. We outsource really difficult or mass-production stuff to pro, and there's a single, very overworked machinist who can sometimes help you, but most of the time it's DIY.

lmilcin 6y ago

So here is simple fact. It does not make sense to judge any piece of code that does not meet "highest standard" to be a tangled mess. There are valid reasons to have varying quality of code and also the idea of quality might be changing from problem to problem and project to project. A quality of code that governs your car's ECU should be different from quality of code that some research team threw together to demonstrate an idea. A coding project should achieve some kind of goal or set of goals as efficiently as possible and in many valid cases quality is just not high on the list and for a good reason. Right now I am working on a PoC to verify an idea that will take a longer time to implement. We do this because we don't want to spend weeks on development just to see it doesn't work or that we want to change something. So spending 2-3 days to avoid significant part of the risk of the rest of the project is fine. It does not need to be spelled out that the code is going to be incomplete, messy and maybe buggy. There is also something to be said for research people to be actually focusing on something else. Professional developers focus their careers on a single problem -- how to write well (or at least they should). But not all people do. Some people actually focus on something else (physics maybe?) and writing code is just a tool to achieve some other goals. If you think about people working on UIs and why UI code tends to be so messy, this is also probably why. Because these guys focus on something else entirely and the code is there just to animate their graphical design.

sdwvit 6y ago
Yeah but you spend more time debugging, that if you write it once with a good architecture and unittests let’s say

knuthsat 6y ago

I think there's not enough researchers that publish code. For example, discrete optimization research (nurse rostering, travelling salesman, vehicle routing problem, etc.) is filled with papers where people are evaluating their methods on public benchmarks but code never sees the day. There's a lot of state-of-the-art methods that never have their code released. I'm pretty sure it's like that elsewhere. Machine learning and deep learning for some reason has a lot of code in the open but that's not the norm. I'd prefer the code to be open first. Once that's abundant then I might prefer the code to also be well designed.

lou1306 6y ago
> I think there's not enough researchers that publish code. I agree, although lately there's been some effort by academia to make authors publish their code, or at least disclose it to the reviewers. Several conferences have an artifact evaluation committee, which tries to reproduce the experimental part of submitted papers. Some conferences actually require a successful artifact evaluation to be accepted (see, for instance, the tool tracks at CAV [1] and TACAS [2]). Others, while not requiring an artifact evaluation, may encourage it by other means. The ACM, for instance, marks accepted papers with special badges [3] reflecting how well the alleged findings can be reproduced and whether the code is publicly available. [1] http://i-cav.org/2021/artifact-evaluation/ http://i-cav.org/2021/artifact-evaluation/ [2] https://etaps.org/2021/call-for-papers https://etaps.org/2021/call-for-papers [3] https://www.acm.org/publications/policies/artifact-review-and-badging-current https://www.acm.org/publications/policies/artifact-review-an...
- cratermoon 6y ago
  This feels like the right approach. If peer review were to include artifact evaluation, including some kind of code review, and require certain standards be met for acceptance, things would change. As others have noted here, the mechanisms of grant-funded work strongly discourage attention to code quality, and that would have to change as well. I'm not in academia now, but I started out my career doing sysops and programming in a lab at a medical school and have worked with academics a bit since. I don't do it much because it's basically volunteer work, and it's almost impossible to contribute meaningfully unless you are also well-versed in the field.

screye 6y ago

> writing software is a low status academic activity Yep, that's the one liner right there. The incentives simply do not match the complaints. Researchers already work upwards of 60 hrs/wk on most occasions. Alongside writing code, they also have to do actual research, write papers, give talks and write grants. All of the latter tasks are primary aspects of their jobs and are commensurately rewarded. The only situation where a well coded tool is rewarded, is when a package blows up, which is quite rare. Like all fields, the high-level answer to such questions is rather straightforward. The individual contributors align their efforts to the incentives. Find a way to incentivize good research code, and we will see changes overnight.

nirse 6y ago

What doesn't really get mentioned in the article, is that a lot of academic software was written by a single developer. All bigger software projects, academic or not, that were only built and maintained by a single person tends to become messier and messier with time. Perhaps most software suffers from that, that over time it becomes a mess, but having more developers look at code (and enough time, and many other factors) can certainly help to keep things in better shape.

Robotbeat 6y ago
Plus it becomes impossible to get multiple developers to work on the code if they can’t understand it because of its messiness, so there’s a bit of survivor’s bias and stronger motivation to clean the code up to be comprehensible to others when you have multiple people working on it. Also, I feel personally attacked by the headline. :) It is for this reason I try to keep my code and models pretty simple, only two or three pages of code (or ideally a single page), and I don’t try to do too many things with one program, and I choose implementations and algorithms that are simpler to implement to make concise code feasible (sometimes at the expense of speed or generality).

SavageBeast 6y ago

I'm currently refactoring a fairly large piece of research code myself. It was written with lean startup thinking in that a little code ought to produce some value in its results. If i was able to eeek some usefulness out of this code, then Id put more energy into it. Otherwise I was perfectly happy to Fail Fast and Fail Cheap. How did it become such a mess in the first place? Simple - I didn't know my requirements when I started writing it. I built it to do one thing. In running it I learned more things (this is good - why you build stuff like this in the first place). The code changed rapidly to accommodate these lessons. It wasn't long before I was running into limitations in the design of the underlying libs I was using etc. Of course I could find a way to make it work but it wasn't going to win any Software Design Awards. Im happy to report that despite ending up a tangled mess, it actually helped me to come to understand and conquer a very specific kind of problem. In doing so I learned the limitations of commercially available tooling, the limitations of commercially available data, not to mention a great deal about the problem domain itself. This research software has earned its keep and is now being cleaned up into a more organized, near commercial quality kind of project. Im glad I threw out "architecture" when I first started with this. It could have gone the other way where I had a very well built piece of code that didn't in fact perform any useful function.

analog31 6y ago
I believe that of all the lessons to come from contemporary software development, constant refactoring may be the most valuable. The spaghetti monster looms large when you're in the heat of battle. But we've all got some idle time for whatever reason. I spend some time every week doing a couple of things: 1) Reading about good techniques. 2) Working through old code and cleaning it up. Because changing your code could always break it, refactoring also reinforces the habit of writing code that can be readily tested -- also a good thing.
- amelius 6y ago
  Makes you wonder why most languages don't come with good refactoring tools.

currymj 6y ago

as people are saying, the typical software engineering advice simply wouldn't work in a research context. one exception is the most basic stuff - people should use version control, do light unit testing, and explicitly track dependencies. These weren't really done in the past but are becoming more and more common, fortunately. I think if software engineering experts actually sat down, looked at how researchers work with computers, and figured out a set of practices to follow that would work well in the research context, they could do a lot of good. This is really needed. But the standard software engineering advice won't work as it is, it has to be adapted somehow.

pydry 6y ago
Another issue is that the standard software engineering advice doesn't guarantee clean code either.

TomMasz 6y ago

I once interviewed for a programming job that was a bit of bait and switch. The hiring manager showed me a foot-high stack of green bar paper that was Fortran code written by optical scientists that I was expected to convert to C. He was somewhat surprised when I declined and ended the interview. I pity whoever got stuck with that task.

analog31 6y ago

>>> writing software is a low status academic activity; it is a low status activity in some companies, but those involved don’t commonly have other higher status tasks available to work on. If measured by compensation, then research is a low status activity. Perhaps more precisely, researchers have low bargaining power. But I don't think that academics actually analyze activities in such detail. The PI might not even know how much programming is being done. The researcher is programming, not because they see it as a way to raise (or lower) their status, but because it's a force multiplier for making themselves more productive overall. Though I work in industry, I'm a "research" programmer for all intents and purposes. I program because I need stuff right away, and I do the kind of work that the engineers hate. Reacting to rapidly changing requirements on a moment's notice disrupts their long term planning. Communicating requirements to an engineer who doesn't possess domain knowledge or math skills is painful. Often, a working piece of spaghetti code that demonstrates a process is the best way to communicate what I need. They can translate it into fully developed software if it threatens to go into a shipping product. That's a good use of their time and not of mine. >>> Why would a researcher want to invest in becoming proficient in a low status activity? To get a better job. I sometimes suspect that anybody who is good enough at programming to get paid for it, is already doing so. >>> Why would the principal investigator spend lots of their grant money hiring a proficient developer to work on a low status activity? Because they don't know how to manage a developer. Software development is costly in terms of both time and effort, and nobody knows how to manage it. Entire books have been written in this topic, and it has been discussed at length on HN. A software project that becomes an end unto itself or goes entirely off the rails can eat you alive. Finding a developer who can do quantitative engineering is hard, and they're already in high demand. It may be that the PI has a better chance managing a researcher who happens to know how to translate their own needs into "good enough" code, than to manage a software project.

nowardic 6y ago

A nice counter example of research software code that adheres to general software engineering best practices and is easy to pick up and use is the OSMNX project: https://github.com/gboeing/osmnx https://github.com/gboeing/osmnx Props to Geoff for setting a nice standard.

f6v 6y ago

Keep in mind that there're different kinds of research software. Take Seurat[1] as an example. There's CI, issue tracking, etc. It might not be the prettiest code you ever seen, but it absolutely has to be maintainable as it's being actively developed. Such projects are rare, but the low quality is often an indication of a software that isn't used by anyone. 1. https://github.com/satijalab/seurat https://github.com/satijalab/seurat

cratermoon 6y ago
Also things like EISPAC, BLAS, LINPACK and so on, for FORTRAN. Back in the 70s my dad worked a bit with them when he was employed for UTHERCC: The University of Texas Health, Education, and Research Computer Center. You can find references to UTHERCC in papers from that era. Come to think of it, something like UTHERCC might be exactly what is needed to help the current situation.

shadowgovt 6y ago

One of the biggest eye-openers for me as an undergrad was when, upon getting to the point where I'd have to decide whether to pursue graduate education or exit academia and join the workforce, I began to look at the process for publishing novel computer science. To be clear, novel computer science is valuable and the lifeblood of the software engineering industries. But the actual product? I discovered of myself that I like quality code more than I like novel discovery, and the output of the academic world ain't it. Examples I saw were damn near pessimized... not just a lack of comments, but single-letter variables (attempting to represent the Greek letters in the underlying mathematical formulae) and five-letter abbreviated function names. I walked away and never looked back. If there's one thing I wish I could have told freshman-year me, it's that software as a discipline is extremely wide. If you find yourself hating it and you're surprised you're hating it, you may just be doing the kind that doesn't mesh with your interests.

cryptica 6y ago

Math and physics are a tangled mess so it's not surprising that mathematicians and physicists write code which looks like a tangled mess. Mathematicians and physicists are trained to handle ambiguous concepts and they can work with weird abstractions which are far detached from reality. Unlike programming languages, the language of math is full of gaps - This requires the reader to make assumptions using past knowledge and conventions. Computers, on the other hand cannot make assumptions so the code must be extremely precise and unambiguous. Writing good code requires a different mindset; firstly, it requires acknowledging that communication is extremely ambiguous and that it takes a great deal of effort to communicate clearly and to choose the right abstractions. A lot of the best coders I've met struggle with math and a lot of the best mathematicians I've met struggle with writing good code.

porker 6y ago
Thank you. Having trained as a physicist, this matches my experience.

civilized 6y ago

I see people here saying research is like writing software with fast-changing requirements. I can see how that could seem like an adequate analogy to a software engineer, but it's not. Researchers use code as a tool of thought to make progress on very ambiguous, high-level problems that lack pre-existing methodology. Like, how could I detect this theoretical astrophysical phenomenon in this dataset? What would it take to predict disease transmission dynamics in a complex environment like a city? Could a neural network leveraging this bag of tricks in some way improve on the state-of-the-art? If you have JIRA tickets like that in your queue, maybe you can compare your job to that of a researcher.

deeeeplearning 6y ago

Why is this surprising? Has anyone been inside a Chemistry or Bio lab? You think that what happens in those labs to get research done is industrial grade?

sumanthvepa 6y ago

What I find really surprising about research software, is that even people in Computer Science write poorly designed code as part of their research. I would have imagined that they would be better qualified to create good code. Just goes to show that Software Engineering != Computer Science.

ejz 6y ago

This is actually the opportunity for our startup. I think there is generally a great opportunity to be the Databricks of a lot of academic software. We're starting in a big research area in biology :)

optiklab 6y ago

I’m an engineer from the other side of researches or science, but somehow interested in the topic. Recently, I’ve learned about great work done by Grigori Fursin and entire community of reserach engineers with the goal to make research software more applicable to the industries by doing it with some kind of framework inside. I want to leave some links here, if you don’t mind to watch it – the talk is called ” Reproducing 150 Research Papers and Testing Them in the Real World”:ACM page with webcast https://event.on24.com/wcc/r/2942043/9C904C7AE045B5C92AAB2CF216826732 https://event.on24.com/wcc/r/2942043/9C904C7AE045B5C92AAB2CF... Also, source docs available here: https://zenodo.org/record/4005773?fbclid=IwAR1JGaAj4lwCJDrkJdgJQHoWroUR6zqIW1STS0D2BRkeaFf6iD0U-KakZSM#.YDPz_nlRVHZ https://zenodo.org/record/4005773?fbclid=IwAR1JGaAj4lwCJDrkJ... And, their solution product https://cknowledge.io/ https://cknowledge.io/ and source code https://github.com/ctuning/ck https://github.com/ctuning/ck I guess it should be helpful to the researchers community.

cratermoon 6y ago

I was reminded that there are research packages like LINPACK, BLAS, and EISPACK for FORTRAN (and some other languages) that have been maintained since the 70s and are still in use. Back in the 70s my dad was working for an organization called UTHERCC, the University of Texas Health, Education, and Research Computer Center, and these libraries were some of the code he worked with. You can find references to UTHERCC in papers from the time, although I don't think it exists under that name. Maybe institutions need something like UTHERCC as an ongoing department now.

hntrader 6y ago

These are some concepts that I believe in for research code. Research code shouldn't be a monolith. Each hypothesis should be a script that follows a data pipeline pattern. If you have a big research question, think about what the most modular progression of steps would be along the path from raw data to final output, and write small scripts that perform each step (input is the output from the previous step). Glue them all together with the data pipeline, which itself is a standalone, disposable script. If step N has already been run, then running the pipeline script once again shouldn't resubmit step N (as long as the input hasn't changed since the last run). This "intermediate data" approach is useful because we can check for errors each step on the way and we don't need to redo calculations if a particular step is shared by multiple research questions. I was taught this by a good mentor and I've been using this approach for many years for various ML projects and couldn't recommend it more highly.

porker 6y ago
This, absolutely this. I looked over a friend's PhD program because the results were unstable. I knew nothing about the domain which was a large disadvantage, but on the code front it was a monolith following a vague data pipeline approach. Unfortunately components wouldn't run separately and there were only a single end to end tests taking hours to run. Had each section had its own tests, diagnosing which algorithm(s) were malfunctioning would have been easier. We never did.

brakus127 6y ago

Structure is great for a well understood problem space, but this is not usually the case when working working something novel. As a researcher your focus should be on learning and problem solving, not creating a beautiful code base. Imposing too many constraints early on can negatively impact your project later on. In the worse case, your code starts to limit the way you think about your research. I agree that there are some general best practices that should be applied to nearly all forms of coding, but beyond that it's a balance. The same thinking should be used when adding regulation to an industry. Heavy regulation on a rapid developing industry can stifle innovation. Regulation (if needed), should be applied as our understanding of the industry increases.

bordercases 6y ago
Results need to be refined so that the way they were first formulated doesn't get in the way of their replication. At scale, this too becomes a cost to industry. In the small, this isn't different from taking a lab notebook and making it clearer and better summarized so that it can be passed on to the poor sucker who has to do what you did after you move on to another project. Furthermore, software projects that are put under the same iterative stress you imply for R&D inevitably go through a refactoring phase so that performance isn't affected in the long run.
- brakus127 6y ago
  Agreed that there should be a minimum bar for "completed" research code such as reproducibility and a clear summary, but engineers shouldn't expect the first version of a new algorithm to be easy to understand without additional material or ready for production without a complete rewrite.

tryonenow 6y ago

Not all "research" code is equal. I would imagine that research code closer to hard sciences and cutting edges is more difficult to keep clean. The trouble is when you're breaking new ground in applied science, you don't necessarily know how well the new tech will work, and exactly what you'll be able to do with it. As development progresses your expectations must be adjusted, and it is impossible to forecast the direction of highly experimental research since progress is typically incremental and constantly dependent on the most recent results. There are many paths to scaling a mountain, so to speak, and sometimes for any of a multitude of reasons you end up on another peak long after you've started climbing.

yudlejoza 6y ago

This topic is near and dear to my heart and at a quick glance, I pretty much agree with all/most of this post. I gained multiple years of industry software engineering experience before joining academia (non-CS, graduate-level). And I was flabbergasted at the way software and programming is treated in research setting where the "domain" is not CS or software itself. It took me a few years just to get a hint of what on earth these people (my collaborators who program side-by-side with me) are thinking, and what kind of mindset do they come from. Then I took a short break and went to the industry. Software engineering, hardcore CS; no domain, no BS. I was expecting that it would feel like an oasis. It didn't. Apart from a handful of process improvements, like use of version control, issue tracking, deadline-management, the quality of the tangled mess of the code was only slightly better. Initially I took away the lesson that it's the same in academia and industry. But on further reflection there are two big differences: - The codebase I worked on in the industry was at least 10x bigger. Despite that, the quality was noticeably better. - More importantly, I could connect with the my coworkers in the industry. If I raised a point about some SwE terminology like test-driven dev, agile, git, whatever, I could have a meaningful discussion. Whereas in academia, not only most domain experts knew jack about 90% of software-engineering concepts and terminology, they were expert at hiding their ignorance, and would steer the conversation in a way that you wouldn't know if they really didn't know or knew too much. I never got over that deceitful ignorance mixed with elitist arrogance. In the end, I do think that, despite enormous flaws, the industry is doing way better than academia when it comes to writing and collaborating on software and programming, and that the side-by-side comparison of actual codebases is a very small aspect of it.

RocketSyntax 6y ago

jupyterlab github issue advocating for documenting research: https://github.com/jupyterlab/team-compass/issues/121 https://github.com/jupyterlab/team-compass/issues/121

innerlee 6y ago

Checkout the openmmlab project [1], where messy research codes in computer vision are rewritten an reorganized into a coherent whole. If there are more researchers join this, then not only research are fully reproducible, much more accessible to everyone, but also be compared fairly. (I'm the maintainer of mmpose [2], mmaction2 [3], and mmediting [4]) [1] https://github.com/open-mmlab https://github.com/open-mmlab [2] https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose [3] https://github.com/open-mmlab/mmaction2 https://github.com/open-mmlab/mmaction2 [4] https://github.com/open-mmlab/mmediting https://github.com/open-mmlab/mmediting

jonbronson 6y ago

I worked at a productive computing research institute for a number of a years. I cannot count the number of times I found research teams duplicating critical algorithms. Research Scientists not only pay the price of the spaghetti nature of the code, they pay it over and over again by not sharing and improving on what has already been built by previous research groups. The software industry has its own share of problems, but from what I've seen the research community is still largely operating on an outdated software model that shuns open collaboration out of fear of being "scooped".

naebother 6y ago

Good. Please, keep on writing more spaghetti code. Dumb people like myself need it to make a living "cleaning" it up. Circle of life.