4 ms·

I write scientific code for a living and my code is very difficult for anyone else to maintain. It is not because my code is badly documented or written, it is

I write scientific code for a living and my code is very difficult for anyone else to maintain. It is not because my code is badly documented or written, it is because what it does is very complex. Every module is documented why it exists and what it does and the code straightforward to read, yet the interaction of all the modules is very complex as it reflects the underlying complexity of the problem the code is solving. Some code is hard to maintain because it is solving a complex problem that few people understand.

Michaelanjello 8y ago
The way to manage this is to make your code more modular, to factor out subcomponents, to share and have them critiqued separately. For example, in machine learning frameworks, autograd is a separate package for automatic differentiation. I actively post questions and answers for subcomponents on Stack Overflow.
perfmode 8y ago
Managing complexity is one of the chief concerns of software engineering.
- danieltillett 8y ago
  Complexity can be managed, but the required domain knowledge is hard to design around. The problem my colleagues struggle with is the domain knowledge I have that they don't. The positive from this is there are few people in the world that can do my job :)
  - analog31 8y ago
    In my view, there's only a certain extent to which code can teach domain knowledge. People tend to freeze up when they see any kind of math, physics, or quantitative engineering. And managers want to believe that domain knowledge is worthless because it refutes treating people as interchangeable cogs. The only solution is to make sure you keep someone on staff who has a hope of understanding the underlying technology, and failing to do so is a business risk like any other.
- nomel 8y ago
  I think you’re talking about code complexity. He’s seems be talking about problem complexity, which just is.
dekhn 8y ago
I don't buy it. I've worked with very complicated systems (ads at google) and it was pretty straightforward- it's the distributed systems communicating at RPC interfaces that is hard to understand. I've also worked with quantum chemistry codes and large supercomputer codes. The ones that were difficult to maintain were just badly written. There was an article recently about Jeff Dean and Sanjay Ghemawat- one of Sanjay's skills is to make APIs that let you build very complex systems from components such that the resulting system is still comprehensible. This had huge long term positive effects for Google. I strongly recommend reconsidering that your position; I've been able to drop into codes in tens of different domains, whenever they are well-written, regardless of the complexity.
- Derbasti 8y ago
  How would you make, say, a fourier transform or IIR filter design accessible to people who don't know any signal processing or complex numbers? You can spend years studying these topics and their related math. Entire text books have been written on these subjects. Any one algorithm probably has multiple scientific publications, explaining how and why they work. This much information can not fit into a few code comments and a README or two. Especially not in a format accessible to people without domain knowledge.
  - gravypod 8y ago
    You don't need to understand FFTs to understand how to update good software. If there's a segfault generated by a program and I can run it though a debugger I should be able to fix the segfault without explicit domain knowledge of the code. My modifications should be testable via unit tests. There should also be a formalized process to incorporate my changes into the authoritative codebase. That process should contain a code reviewing process where one or more experts on the codebase critique my code. These are all software engineering things that have nothing to do with domain knowledge.
  - dekhn 8y ago
    Nothing about spectral transforms requires complicated software, beyond the need to optimize the algorithms. Sure, the inner components need specialists. No complaint there. But it's not like your stuff is "complicated". I guess the counter argument is something like Goto BLAS, where the author has an extremely complex internal mental model of Intel instruction scheduling and can emit incomprehensible, but absurdly fast instruction sequences. But of course, the BLAS API is dead simple and almost nobody using it needs to know the underlying solver complexity in detail.

Michaelanjello 8y ago

The way to manage this is to make your code more modular, to factor out subcomponents, to share and have them critiqued separately. For example, in machine learning frameworks, autograd is a separate package for automatic differentiation. I actively post questions and answers for subcomponents on Stack Overflow.

perfmode 8y ago

Managing complexity is one of the chief concerns of software engineering.

danieltillett 8y ago
Complexity can be managed, but the required domain knowledge is hard to design around. The problem my colleagues struggle with is the domain knowledge I have that they don't. The positive from this is there are few people in the world that can do my job :)
- analog31 8y ago
  In my view, there's only a certain extent to which code can teach domain knowledge. People tend to freeze up when they see any kind of math, physics, or quantitative engineering. And managers want to believe that domain knowledge is worthless because it refutes treating people as interchangeable cogs. The only solution is to make sure you keep someone on staff who has a hope of understanding the underlying technology, and failing to do so is a business risk like any other.
nomel 8y ago
I think you’re talking about code complexity. He’s seems be talking about problem complexity, which just is.

dekhn 8y ago

I don't buy it. I've worked with very complicated systems (ads at google) and it was pretty straightforward- it's the distributed systems communicating at RPC interfaces that is hard to understand. I've also worked with quantum chemistry codes and large supercomputer codes. The ones that were difficult to maintain were just badly written. There was an article recently about Jeff Dean and Sanjay Ghemawat- one of Sanjay's skills is to make APIs that let you build very complex systems from components such that the resulting system is still comprehensible. This had huge long term positive effects for Google. I strongly recommend reconsidering that your position; I've been able to drop into codes in tens of different domains, whenever they are well-written, regardless of the complexity.

Derbasti 8y ago
How would you make, say, a fourier transform or IIR filter design accessible to people who don't know any signal processing or complex numbers? You can spend years studying these topics and their related math. Entire text books have been written on these subjects. Any one algorithm probably has multiple scientific publications, explaining how and why they work. This much information can not fit into a few code comments and a README or two. Especially not in a format accessible to people without domain knowledge.
- gravypod 8y ago
  You don't need to understand FFTs to understand how to update good software. If there's a segfault generated by a program and I can run it though a debugger I should be able to fix the segfault without explicit domain knowledge of the code. My modifications should be testable via unit tests. There should also be a formalized process to incorporate my changes into the authoritative codebase. That process should contain a code reviewing process where one or more experts on the codebase critique my code. These are all software engineering things that have nothing to do with domain knowledge.
- dekhn 8y ago
  Nothing about spectral transforms requires complicated software, beyond the need to optimize the algorithms. Sure, the inner components need specialists. No complaint there. But it's not like your stuff is "complicated". I guess the counter argument is something like Goto BLAS, where the author has an extremely complex internal mental model of Intel instruction scheduling and can emit incomprehensible, but absurdly fast instruction sequences. But of course, the BLAS API is dead simple and almost nobody using it needs to know the underlying solver complexity in detail.