5 ms·

I do this kind of work. It's frustrating how awful most of the code is. You'd think people smart enough to pursue stem phds would understand basic programming a

I do this kind of work. It's frustrating how awful most of the code is. You'd think people smart enough to pursue stem phds would understand basic programming abstractions, but that is often not the case. Another issue is that researchers don't normally use good software engineering practices. I have yet to be given any code that has even a single unit test. Source control is being used these days, but the repositories are usually unorganized messes with unhelpful commit histories. No one keeps track of system dependencies, and few understand build systems. I can spend a week just trying to get some software to build. I could keep complaining, but I don't want to give people the impression that I don't like what I do. It beats the hell out of writing CRUD apps.

keithpeter 9y ago
So, lots of low hanging fruit. Can you identify (say) five things that academics can do with an existing code base that will make your job easier (and therefore cheaper to the downstream users)? Perhaps pitch the top five pick at the abilities of a computer science student looking for a project? Can you identify (say) three key practices to adopt when starting a new project given the likely skill levels of postgrad students in your domain? You could get an ebook out of those I imagine.
- tome 9y ago
  1. Use version control 2. Have significant test coverage 3. Have at least a rudimentary form of Continuous Integration, even if it's just running a script on every check in.
- tellarin 9y ago
  Proper version control would already be a huge gain.
- vanderZwan 9y ago
  > So, lots of low hanging fruit. Well, yes and no. Speaking as someone who has done this as well: if it's UI stuff, or gluing processes together, it's fine. But sometimes they want you to fix up a custom algorithm they implemented. Which can be one of those cutting-edge scientific things that actually require a PhD-level of knowledge and insight into the topic. So you can only get half-way with untangling the spaghetti mess without sincerely not knowing for sure if you broke something or fixed a bug, because you don't know what the correct output should be.
- greydius 9y ago
  I don't have time to give a fuller answer right now, but if I were to write an ebook as you suggested, the recurring theme would be "design for collaboration." Make your project easy for others to use and contribute to. And, contrary to what many people have stated in this thread, I think this aligns with the incentives of academic research. If someone else can easily extend your project with some new ideas, then you've just co-authored another paper. In addition, projects that gain traction beyond your own research group are more likely to get funding. 1) test suite - this shouldn't require explanation 2) documentation - both top level docs with examples and well-commented code. keep track of units 3) one-step build - I should be able to download the project, type 'make' (or something equivalent), and start using it
  - Balgair 9y ago
    > design for collaboration You have a different view of how academia works than most academics. Most PIs these days do NOT collaborate and are actively hostile towards it. It's a shame, but it's true. Also, the niche-ing of academia is real, in many fields, there may be only 3 other people on the planet that understand what you are actually doing, and many of them may not speak English and you may not know they are out there until 2 years from now (publishing takes a long time). Most code is written by grad-students that barely know what a for loop is, let alone how to use git, and they only write it to do it once for a specific paper. Look into MatLab, that is the most used language in bio, by far. It's basically psuedo-code that compiles, and it's still a mystery to most grad-students.
mikebenfield 9y ago
> You'd think people smart enough to pursue stem phds would understand basic programming abstractions, but that is often not the case. Another way of looking at it: People doing STEM PhDs are extremely busy. They have to master the content in their field, keep up with new research, conduct their own research, write papers, go to conferences, give talks, teach, keep up with whatever department service responsibilities they have, etc. They're smart people so they're going to be able to slap together some code that does what they need it to do. But to expect them to learn git and how to use it well, to learn about managing dependencies, about unit tests, about build systems, about best practices for documenting code, about programming abstractions or OOP whatever else... that's a lot to ask. (And I say this as a STEM PhD who did learn all that stuff, more or less.)
- vidarh 9y ago
  Even in computer science a lot of this holds true, and you're lucky if you find someone who both knows the specifics of their niche of the field well and knows how to write clean code. I once had a quite respectable lecturer working for me, and his code was awful. But that was ok - we knew that. He was hired for his conceptual skills and domain knowledge, and we paired him with a junior developer that could take his raw output and turn them into cleaner code.
- BlackFly 9y ago
  I would say if unit tests, build systems, documentation, abstractions, modern languages, and so on are valuable for productivity, reduced bugginess or ease of access for the next developers then the incentives exist for academic researchers to learn and use these tools. I think the real problem in academia is the same as the problem in some organizations that also don't follow best practices: the people empowered to make the decisions about whether to invest people's time into improving testing, building, technical debt, or documentation do not actually believe that an increase in productivity, increase in access or decrease in bugginess will be achieved.
  - digitalzombie 9y ago
    > I would say if unit tests, build systems, documentation, abstractions, modern languages, and so on are valuable for productivity, reduced bugginess or ease of access for the next developers then the incentives exist for academic researchers to learn and use these tools. The time to learn all of this is almost like a field in itself say Software engineering. I just need to get shit done. I had a project to do bayesian hierarchical modeling. I wrote it from scratch in R without using a MCMC framework. Yeah it's ugly, yeah it's slow. But I got initial results to satisfy my mentor and within the time limit of the summer internship. This is on top of learning two semesters worth of Bayesian statistic in 3-4 weeks. Time isn't expendable and I can't devote my life to every single thing while neglecting other aspect (love, health, etc..). Yeah your theory is nice only if you don't have a deadline, a life outside of work, etc...
britworst 9y ago
Agree with everything except for unit tests, they're something of a cargo-cult fad. Any of my developers caught wasting their time on these would get a stern talking to.
- fsloth 9y ago
  You are kidding, right?
- Boothroid 9y ago
  Interesting - so how else do you test?
  - dasmoth 9y ago
    End-to-end tests comparing output to hand-computed results on small examples (or perhaps the output of an earlier, simpler, prototype)? (For predictive models) evaluate the output on a test set? Depending on what you're trying to do, the contents of the black box may not matter so very much if the results on an agreed test set are good (...and you're confident it's independent from any training data...) Manual eyeballing and sanity-checking of output? (in my experience important however many layers of automated testing you're using, and undervalued by people who focus on software as an engineering process more than an art-form). ...and, circumstantially, probably a bunch of others. I don't think this is a field where making lots of rules is especially helpful (except, perhaps, the "always eyeball" one...)
    - Boothroid 9y ago
      Thanks, useful. I never seem to be allocated enough time to write tests and always worry as a result, since use of automated tests seem to be close to dogma for many. Along with agile it seems to be one of those things that people cling on to for security, appropriateness or otherwise notwithstanding.
- b2811 9y ago
  You must have a superhuman team. The only languages I could even remotely imagine to get by without tests are Haskell/ML. Even then, logic mistakes can and do still happen.
Malice 9y ago
How did you get into it? I think something like that would be my dream job. Yeah, I'm not normal :)
- eric_bullington 9y ago
  I'm with you. This is the second time I'm reading about this niche and would love to find out more info about getting into the field. I used to work as a scientific/medical translator and one of the things I loved about my work then was that with each new project, I had to learn enough about a new topic/subfield to become a bit of an pseudo-expert in it. I greatly enjoyed that research and would love to do some work in software development that would require the same type of constant learning with each new project. Diving into some convoluted, 20-year-old Fortran driving highly-specialized scientific software actually sounds kind of exciting to me (yeah, I'm not normal either :) ). Is there much demand for this kind of work, I wonder? I'm guessing the best way to get your foot in the door is to be in academia, and those days are long passed for me.
  - vanderZwan 9y ago
    I think the issue is more one of funding than of a lack of demand - I suspect there's not that many grants that let you hire a programmer for this kind of thing.
  - Balgair 9y ago
    Fortran story time: As an undergrad, I had a good friend who was doing his PhD in Atmospheric Physics. It turns out, most of that field works in Fortran '88. This is not a very useful language, seeing as it uses GOTO statements to function as a loop. Fortunately, it does have comments. My friend managed to sweet-talk an older PI into giving him the old code for use in nuclear blast atmospherics (Exp: say you nuked all of France, what happens to Greenland's ice). At about 3 am before a project was due in the morning, he was pulling through the spaghetti that was the code, tired, jittery, and over-caffeinated. In this mess of logic diagrams he had to draw out by hand, he finally got to somewhere he thought was going to really cement all the code together for him. He follows a GOTO statement, and there was only a set of another GOTO statements. This went on for about 30 (my recollection of his words) GOTO statements, all 'nested'. Eventually, he gets to one that only has a comment line: 'HAHA MADE YOU LOOK'. His laptop was defenestrated and he had to buy a new one with me about a week later.
torchous 9y ago
I moved from CS into Earth science a decade ago and I was similarly appalled at the state of affairs in the beginning. First, this doesn't have anything to do with how smart people are, it's a question of time and other resources. Our job simply isn't to write pretty code, it's just a means to an end. If you know how to do something in FORTRAN, why would you invest into learning how to do it with <lastest trend>? It's a huge cost! Second, the evolution of such code occurs often over decades; as the article states, with students and post-docs hammering along until it finally does what they need to get their project to move forward. The code usually begins with "Do X" and then the rest of the alphabet is tacked on later. We all know it's bad. But in the end, it's usually only your own lab group using this stuff; maybe a handful of people around the world. The cost of polishing something that started out as hastily written code to get something done in time for {conference,paper,proposal} a decade ago is simply to high. It works. Done. I wish the state of affairs was prettier, but I doubt it will change any time soon. And frankly, I don't think it has to. Software engineers will always find something to complain about others' code (coding style discussions come to mind). If there's value in commercializing scientific prototypes, experts should rush in and do it. There's no need for us to invest in perfect code just in case someone may want to reuse it. Having said that, NSF is starting to fund initiatives to improve the state of affairs. For my field, it's EarthCube: https://www.earthcube.org/ https://www.earthcube.org/ but this seems to be fumbling along; all of their goals would need to be adapted by scientists outside of that community (standards, etc). Without the right incentives (funding, publications), it's just not going to happen.
tbrownaw 9y ago
You'd think people smart enough to pursue stem phds would understand basic programming abstractions, but that is often not the case. Another issue is that researchers don't normally use good software engineering practices. I have yet to be given any code that has even a single unit test. Source control is being used these days, but the repositories are usually unorganized messes with unhelpful commit histories. No one keeps track of system dependencies, and few understand build systems. Good developers get paid enough, that it indicates that maybe our job isn't quite that easy to learn. Our basic programming abstractions and software engineering practices are things that we as a field developed over time, based on experience; it's not like they're something you just automatically know without having to sink time into studying if you're above some IQ level. They also depend to varying degrees on how big your program is, which parts are expected to change more or less rapidly, what kind of program it is, etc. Unit tests assume you know what the code is supposed to do before you write it, that the code will change significantly more often than the required functionality, and that the tests are easier to read and check than the code being tested.
pjmorris 9y ago
You're not alone in feeling this way. Greg Wilson organized 'Software Carpentry' [0] to teach researchers the basics of software engineering. They regularly do workshops (list/schedule at site). [0] https://software-carpentry.org/ https://software-carpentry.org/
PeachPlum 9y ago
> Another issue is that researchers don't normally use good software engineering practices. I have yet to be given any code that has even a single unit test. Source control is being used these days, but the repositories are usually unorganized messes with unhelpful commit histories. Funny, that sounds exactly like my IT Dept.'s approach to coding. I work in Operations and was exposed to their approach when helping them translate some business rules into SQL for the warehouse management system. During the course of interaction the IT Director lost the code we had worked on together twice, keeps the requirements docs in his email etc. etc. Me: your engineering practices are terrible He: We don't have time to do it properly Me: but somehow we have time to do it three times I told my manager but no-one really understands or cares enough. The Warehousing Director calling out the IT Director about his approach to IT - where would that all end ?
sgt101 9y ago
I remember carefully crafting a software framework for my Ph.D. I got it running, it worked perfectly, it didn't deliver any advantages vs state of the art. I think it took me 7 mths to write. I then slapped together another one, and that took me 3 mths, it worked (just about), it produced some decent results, and I could move onto the next thing. The software I was producing now was not good but this was in fact excellent experience for research software development as as my work progressed into more innovative areas I found that many, many ideas didn't work out at all, so the priority was to be able to develop code really really fast. I suspect that this is what is going on with most research code - write it really fast, not really good.
segmondy 9y ago
Give me a break, you are complaining. I don't expect academics to follow enterprise standards. They are more rooted in theory than practice. I expect shitty code, uncommented code, no unit tests. You know that thing SV calls the real MVP? Yup, that's what I expect from academia, the most basic Proof of concept to demonstrate that an idea is possible. Correct is cherished, beauty is not. Note, the correctness is enough for best case/data at hand. What's more important and useful to me and society at large is the papers that follow. Then there's the group of folks who tend to read those papers and turn it into products. Yes, those are the ones I expect to have repos, unit tests, build systems and all that.

keithpeter 9y ago

So, lots of low hanging fruit. Can you identify (say) five things that academics can do with an existing code base that will make your job easier (and therefore cheaper to the downstream users)? Perhaps pitch the top five pick at the abilities of a computer science student looking for a project? Can you identify (say) three key practices to adopt when starting a new project given the likely skill levels of postgrad students in your domain? You could get an ebook out of those I imagine.

tome 9y ago
1. Use version control 2. Have significant test coverage 3. Have at least a rudimentary form of Continuous Integration, even if it's just running a script on every check in.
tellarin 9y ago
Proper version control would already be a huge gain.
vanderZwan 9y ago
> So, lots of low hanging fruit. Well, yes and no. Speaking as someone who has done this as well: if it's UI stuff, or gluing processes together, it's fine. But sometimes they want you to fix up a custom algorithm they implemented. Which can be one of those cutting-edge scientific things that actually require a PhD-level of knowledge and insight into the topic. So you can only get half-way with untangling the spaghetti mess without sincerely not knowing for sure if you broke something or fixed a bug, because you don't know what the correct output should be.
greydius 9y ago
I don't have time to give a fuller answer right now, but if I were to write an ebook as you suggested, the recurring theme would be "design for collaboration." Make your project easy for others to use and contribute to. And, contrary to what many people have stated in this thread, I think this aligns with the incentives of academic research. If someone else can easily extend your project with some new ideas, then you've just co-authored another paper. In addition, projects that gain traction beyond your own research group are more likely to get funding. 1) test suite - this shouldn't require explanation 2) documentation - both top level docs with examples and well-commented code. keep track of units 3) one-step build - I should be able to download the project, type 'make' (or something equivalent), and start using it
- Balgair 9y ago
  > design for collaboration You have a different view of how academia works than most academics. Most PIs these days do NOT collaborate and are actively hostile towards it. It's a shame, but it's true. Also, the niche-ing of academia is real, in many fields, there may be only 3 other people on the planet that understand what you are actually doing, and many of them may not speak English and you may not know they are out there until 2 years from now (publishing takes a long time). Most code is written by grad-students that barely know what a for loop is, let alone how to use git, and they only write it to do it once for a specific paper. Look into MatLab, that is the most used language in bio, by far. It's basically psuedo-code that compiles, and it's still a mystery to most grad-students.

mikebenfield 9y ago

> You'd think people smart enough to pursue stem phds would understand basic programming abstractions, but that is often not the case. Another way of looking at it: People doing STEM PhDs are extremely busy. They have to master the content in their field, keep up with new research, conduct their own research, write papers, go to conferences, give talks, teach, keep up with whatever department service responsibilities they have, etc. They're smart people so they're going to be able to slap together some code that does what they need it to do. But to expect them to learn git and how to use it well, to learn about managing dependencies, about unit tests, about build systems, about best practices for documenting code, about programming abstractions or OOP whatever else... that's a lot to ask. (And I say this as a STEM PhD who did learn all that stuff, more or less.)

vidarh 9y ago
Even in computer science a lot of this holds true, and you're lucky if you find someone who both knows the specifics of their niche of the field well and knows how to write clean code. I once had a quite respectable lecturer working for me, and his code was awful. But that was ok - we knew that. He was hired for his conceptual skills and domain knowledge, and we paired him with a junior developer that could take his raw output and turn them into cleaner code.
BlackFly 9y ago
I would say if unit tests, build systems, documentation, abstractions, modern languages, and so on are valuable for productivity, reduced bugginess or ease of access for the next developers then the incentives exist for academic researchers to learn and use these tools. I think the real problem in academia is the same as the problem in some organizations that also don't follow best practices: the people empowered to make the decisions about whether to invest people's time into improving testing, building, technical debt, or documentation do not actually believe that an increase in productivity, increase in access or decrease in bugginess will be achieved.
- digitalzombie 9y ago
  > I would say if unit tests, build systems, documentation, abstractions, modern languages, and so on are valuable for productivity, reduced bugginess or ease of access for the next developers then the incentives exist for academic researchers to learn and use these tools. The time to learn all of this is almost like a field in itself say Software engineering. I just need to get shit done. I had a project to do bayesian hierarchical modeling. I wrote it from scratch in R without using a MCMC framework. Yeah it's ugly, yeah it's slow. But I got initial results to satisfy my mentor and within the time limit of the summer internship. This is on top of learning two semesters worth of Bayesian statistic in 3-4 weeks. Time isn't expendable and I can't devote my life to every single thing while neglecting other aspect (love, health, etc..). Yeah your theory is nice only if you don't have a deadline, a life outside of work, etc...

britworst 9y ago

Agree with everything except for unit tests, they're something of a cargo-cult fad. Any of my developers caught wasting their time on these would get a stern talking to.

fsloth 9y ago
You are kidding, right?
Boothroid 9y ago
Interesting - so how else do you test?
- dasmoth 9y ago
  End-to-end tests comparing output to hand-computed results on small examples (or perhaps the output of an earlier, simpler, prototype)? (For predictive models) evaluate the output on a test set? Depending on what you're trying to do, the contents of the black box may not matter so very much if the results on an agreed test set are good (...and you're confident it's independent from any training data...) Manual eyeballing and sanity-checking of output? (in my experience important however many layers of automated testing you're using, and undervalued by people who focus on software as an engineering process more than an art-form). ...and, circumstantially, probably a bunch of others. I don't think this is a field where making lots of rules is especially helpful (except, perhaps, the "always eyeball" one...)
  - Boothroid 9y ago
    Thanks, useful. I never seem to be allocated enough time to write tests and always worry as a result, since use of automated tests seem to be close to dogma for many. Along with agile it seems to be one of those things that people cling on to for security, appropriateness or otherwise notwithstanding.
b2811 9y ago
You must have a superhuman team. The only languages I could even remotely imagine to get by without tests are Haskell/ML. Even then, logic mistakes can and do still happen.

Malice 9y ago

How did you get into it? I think something like that would be my dream job. Yeah, I'm not normal :)

eric_bullington 9y ago
I'm with you. This is the second time I'm reading about this niche and would love to find out more info about getting into the field. I used to work as a scientific/medical translator and one of the things I loved about my work then was that with each new project, I had to learn enough about a new topic/subfield to become a bit of an pseudo-expert in it. I greatly enjoyed that research and would love to do some work in software development that would require the same type of constant learning with each new project. Diving into some convoluted, 20-year-old Fortran driving highly-specialized scientific software actually sounds kind of exciting to me (yeah, I'm not normal either :) ). Is there much demand for this kind of work, I wonder? I'm guessing the best way to get your foot in the door is to be in academia, and those days are long passed for me.
- vanderZwan 9y ago
  I think the issue is more one of funding than of a lack of demand - I suspect there's not that many grants that let you hire a programmer for this kind of thing.
- Balgair 9y ago
  Fortran story time: As an undergrad, I had a good friend who was doing his PhD in Atmospheric Physics. It turns out, most of that field works in Fortran '88. This is not a very useful language, seeing as it uses GOTO statements to function as a loop. Fortunately, it does have comments. My friend managed to sweet-talk an older PI into giving him the old code for use in nuclear blast atmospherics (Exp: say you nuked all of France, what happens to Greenland's ice). At about 3 am before a project was due in the morning, he was pulling through the spaghetti that was the code, tired, jittery, and over-caffeinated. In this mess of logic diagrams he had to draw out by hand, he finally got to somewhere he thought was going to really cement all the code together for him. He follows a GOTO statement, and there was only a set of another GOTO statements. This went on for about 30 (my recollection of his words) GOTO statements, all 'nested'. Eventually, he gets to one that only has a comment line: 'HAHA MADE YOU LOOK'. His laptop was defenestrated and he had to buy a new one with me about a week later.

torchous 9y ago

I moved from CS into Earth science a decade ago and I was similarly appalled at the state of affairs in the beginning. First, this doesn't have anything to do with how smart people are, it's a question of time and other resources. Our job simply isn't to write pretty code, it's just a means to an end. If you know how to do something in FORTRAN, why would you invest into learning how to do it with <lastest trend>? It's a huge cost! Second, the evolution of such code occurs often over decades; as the article states, with students and post-docs hammering along until it finally does what they need to get their project to move forward. The code usually begins with "Do X" and then the rest of the alphabet is tacked on later. We all know it's bad. But in the end, it's usually only your own lab group using this stuff; maybe a handful of people around the world. The cost of polishing something that started out as hastily written code to get something done in time for {conference,paper,proposal} a decade ago is simply to high. It works. Done. I wish the state of affairs was prettier, but I doubt it will change any time soon. And frankly, I don't think it has to. Software engineers will always find something to complain about others' code (coding style discussions come to mind). If there's value in commercializing scientific prototypes, experts should rush in and do it. There's no need for us to invest in perfect code just in case someone may want to reuse it. Having said that, NSF is starting to fund initiatives to improve the state of affairs. For my field, it's EarthCube: https://www.earthcube.org/ https://www.earthcube.org/ but this seems to be fumbling along; all of their goals would need to be adapted by scientists outside of that community (standards, etc). Without the right incentives (funding, publications), it's just not going to happen.

tbrownaw 9y ago

You'd think people smart enough to pursue stem phds would understand basic programming abstractions, but that is often not the case. Another issue is that researchers don't normally use good software engineering practices. I have yet to be given any code that has even a single unit test. Source control is being used these days, but the repositories are usually unorganized messes with unhelpful commit histories. No one keeps track of system dependencies, and few understand build systems. Good developers get paid enough, that it indicates that maybe our job isn't quite that easy to learn. Our basic programming abstractions and software engineering practices are things that we as a field developed over time, based on experience; it's not like they're something you just automatically know without having to sink time into studying if you're above some IQ level. They also depend to varying degrees on how big your program is, which parts are expected to change more or less rapidly, what kind of program it is, etc. Unit tests assume you know what the code is supposed to do before you write it, that the code will change significantly more often than the required functionality, and that the tests are easier to read and check than the code being tested.

pjmorris 9y ago

You're not alone in feeling this way. Greg Wilson organized 'Software Carpentry' [0] to teach researchers the basics of software engineering. They regularly do workshops (list/schedule at site). [0] https://software-carpentry.org/ https://software-carpentry.org/

PeachPlum 9y ago

> Another issue is that researchers don't normally use good software engineering practices. I have yet to be given any code that has even a single unit test. Source control is being used these days, but the repositories are usually unorganized messes with unhelpful commit histories. Funny, that sounds exactly like my IT Dept.'s approach to coding. I work in Operations and was exposed to their approach when helping them translate some business rules into SQL for the warehouse management system. During the course of interaction the IT Director lost the code we had worked on together twice, keeps the requirements docs in his email etc. etc. Me: your engineering practices are terrible He: We don't have time to do it properly Me: but somehow we have time to do it three times I told my manager but no-one really understands or cares enough. The Warehousing Director calling out the IT Director about his approach to IT - where would that all end ?

sgt101 9y ago

I remember carefully crafting a software framework for my Ph.D. I got it running, it worked perfectly, it didn't deliver any advantages vs state of the art. I think it took me 7 mths to write. I then slapped together another one, and that took me 3 mths, it worked (just about), it produced some decent results, and I could move onto the next thing. The software I was producing now was not good but this was in fact excellent experience for research software development as as my work progressed into more innovative areas I found that many, many ideas didn't work out at all, so the priority was to be able to develop code really really fast. I suspect that this is what is going on with most research code - write it really fast, not really good.

segmondy 9y ago

Give me a break, you are complaining. I don't expect academics to follow enterprise standards. They are more rooted in theory than practice. I expect shitty code, uncommented code, no unit tests. You know that thing SV calls the real MVP? Yup, that's what I expect from academia, the most basic Proof of concept to demonstrate that an idea is possible. Correct is cherished, beauty is not. Note, the correctness is enough for best case/data at hand. What's more important and useful to me and society at large is the papers that follow. Then there's the group of folks who tend to read those papers and turn it into products. Yes, those are the ones I expect to have repos, unit tests, build systems and all that.