6 ms·

Great article, and fantastic to see a spotlight on an issue that I've thought a lot about. The sad part is that to a lot of scientists and researchers, softwar

Great article, and fantastic to see a spotlight on an issue that I've thought a lot about. The sad part is that to a lot of scientists and researchers, software/software engineers isn't something worth paying for. It's not uncommon to see "programmer" jobs that are looking for 3+ years of experience that offer <$15 dollars an hour in the US. Sometimes they're "volunteer intern" positions. Of course the people who end up filling these positions aren't usually actual developers, so the software gets built poorly, eventually gets scrapped, and the cycle continues. Management also hasn't really evolved past the 90's. Non-technical scientists often want 100% of control and to make each decision, but don't want to spend any time on it. This means developers often have little to no specs to work with, but spend all of their time guessing about what the scientists want, and having to go back and fix everything after. >“That’s really the tragedy of the funding agencies in general,” says Carpenter. “They’ll fund 50 different groups to make 50 different algorithms, but they won’t pay for one software engineer.” This is the crux of my frustration. It's not even 50 different algorithms often. A lot of the time, 50 different research groups will be working on very similar programs, and none will be able to deliver a working version. Though the article mentions that research funding does exist, clicking on one of those funding pages and looking through their examples reveals that only ~1/10 of their websites are actually still active, and they aren't old sites. Again this goes back to the whole "scientists don't value software thing". I've seen scientists happily sign off on spending $20,000+ on hardware components that would usually cost <$100 to make, but balk at contributing $50 yearly to support open source. I got lucky that I managed to find a place where I get paid fairly, and my boss is actually technical and can manage tech projects well, but these places are few and far between.

opportune 7y ago
A lot of the points you bring up re related to cost. Here's the thing about cost, let's say it costs $100k/year to hire a good software engineer capable of writing scientific code (able to program and test complex algorithms, write HPC code, turn whitepapers into code), which might even be an underestimate depending on how benefits are paid out and the area. You can also fund 3 more grad students for that kind of money. The grad students will directly convert a PI's money into authorships while the software engineer's contribution will be only indirect, and likely take years to pay off. Plus, with only a single software engineer, there's a good chance you get unlucky and end up with someone clueless/lazy. You would probably need 3-4 software engineers to make a functioning team with best practices and hedge your bets against accidentally hiring someone who sucks. So now we're talking 10+ grad students. Open source software is a bit different because many labs can band together to fund things they find useful. But again there are still issues with cost-effectiveness. I'm guessing most lab contributors to OSS would want some sort of quid-pro-quo which may not be realistic for all OSS projects. And by funding OSS you are also funding competing labs' abilities to use the same features you use, which is good for science in general but not good for people's careers sometimes
- bluGill 7y ago
  Why hire an engineer? Why not walk over to the CS (might have a different name) and talk to a professor there. They can set you up with plenty of undergrads who need this experience, and they should be able to guide them into something that is maintainable long term. Note that I said should there. How to write maintainable programs seems to be lacking in research area.
  - bp0017 7y ago
    Incidentally, this is exactly what my lab does
  - boron1006 7y ago
    This would be a good solution, but from what I've seen with psychologists and statisticians, it's unlikely to happen for reasons I don't fully understand. Another thing is that undergraduates often learn by adopting the norms of the institution that they're in (e.g. using version control, linters, etc) but when they're brought in as the technical person, they don't have that opportunity to improve (This is my personal anecdote as that cs undergrad at one time).
  - opportune 7y ago
    I was such a CS undergrad working in a lab once upon a time. I don't think you really want that because the undergrad will probably only be working for like 4-15hr/week potentially for only a single semester. For a summer position, sure it's 40hr/week but still only for about 10-15 weeks. And still, you're getting a generic CS undergrad's caliber of work and responsibility which I would say on average is not great. They might not be as familiar with version control, best practices, etc. and could just end up writing code just as bad as the scientists. I think if hiring a team you would need at least one somewhat experienced full-time software engineer to act as team lead/PM for the other developers, whether fulltime or students.
    - Master_Odin 7y ago
      Yeah, picking up undergrads (or even grad students) from the CS department is not a surefire way to end up code that follows best practices, is well maintained/documented, etc. and I'd probably argue if your team leader doesn't have that experience, you're more likely to end up with something that doesn't follow great practices (especially with regards to any sort of test suite).
mehrdadn 7y ago
> The sad part is that to a lot of scientists and researchers, software/software engineers isn't something worth paying for. I have to ask how you came to this conclusion. Did you have anecdotes from them supporting this or did you just conclude "they are not paying for it therefore software is not worth paying for to them"?
- ylem 7y ago
  I think the original poster is applying the "cheap talk" heuristic. People may say they value things, but we see what's really valued by what they pay for (given the resources they have)...
chrisseaton 7y ago
> Non-technical scientists How can you have a ‘non-technical scientist’? All science is inherently technical. Are you using ‘technical’ as a short-hand for ‘can program’? Stop doing that - programming is not the only technology.
- scarejunba 7y ago
  What an incredibly pointless thing to argue about. I think we all knew what he meant and, this being a specialist forum, we benefit from using shorthand in speech that is a lossy representation of ideas to those not initiated. I hate to have to create this conversation fork but I really wish people wouldn't make comments like this. They're so low signal.
  - chrisseaton 7y ago
    I disagree - I think reducing anyone who doesn’t happen to program to being ‘non-technical’, with the implication that they don't know what they're talking about, is insidious.
    - scarejunba 7y ago
      It's clear he means "Scientists who don't want too much input into the engineering decisions". You're reading an implication of competency where there is none.
- boron1006 7y ago
  I'm using technical in a broad sense. One effect of scientific research becoming bigger is that roles are becoming more specialized. Often the people leading the research have primarily spend time writing grants or papers, and less senior professors and post-docs will carry out the actual tasks. When I first started I was a bit shocked at how many of the big names knew basically nothing about the process of how their own research is carried out. The effects are more pronounced depending on what field we're talking about. For example, in physics, I'd imagine most people have at least the fundamentals of programming down, even if software design may be lacking. In the field I work in (Neuroimaging), a lot of PI's are doctors or neuropsychologists, and might barely even know how to use a computer.
ylem 7y ago
Two other points to consider--equipment is a one-off expense and staffing is a continuous expense. The other is that the pots of money for equipment and staff may be different....
- boron1006 7y ago
  Completely right, I've been trying to hype up the "Github sponsorships" program as a way of changing the thinking around software (e.g. tacking OSS onto grants as required equipment), but haven't found much support.
nwallin 7y ago
The thing that terrifies me about the state of affairs is what happens when the software gives the wrong result because it's poorly written? If a scientists has input B and expects output X, and writes code that accepts B, and it happens to output X, chances are they'll write the paper and send it off for publication. Even if there's a bug in the code, and the real output should have been Y. I genuinely believe that this happens more often than most of us would find acceptable. Hell, there have probably been instances of the code outputting Y and the scientist scratches their head like, "Hmmm, that's not right, something's wrong," and will massage the code until it eventually outputs X, and then they'll say, "Fixed it!" and move on. I know for a fact it's happened at least once, and led to a scientific controversy that lasted for decades. Unfortunately, I don't remember the specifics; if someone recognizes my vague description please step in with a citation. But one group of scientists published a paper saying a certain dynamic system behaved in a certain way, and a second group published a paper saying it behaved in a different way, and the two results were completely incompatible with each other. Significant public disagreement ensued. One group published their code, and the other group attacked it saying it was poorly written, etc. The second group did not release their code. Decades later, some other scientist at the second institution released the code, and after a code review, it was found that the data was incorrectly initialized. They needed to initialize the particles with "random" initial velocity vectors, but the scientist who wrote the code didn't know how to do it correctly, and wrote an ad-hoc algorithm that gave the initial velocity vectors significant bias along an axis. But the paper was already written, peer reviewed, published, and cited, so even though the paper was wrong, the result was still accepted by (half of) the scientific community. AFAIK the paper was never retracted.
- roel_v 7y ago
  I have seen and fixed code that was first written as a prototype, then used as is in a follow on project and so on, until it turned out to be used for deciding on whether or not to grand certain subsidies. Except whoops in one spot it interpreted kilometers as meters without dividing by 1000. Along with literally dozens of other outright bugs. But nobody cares, really.
kochthesecond 7y ago
When said scientists want to build some new equipment, they get involved and they use professionals. When said scientists need to build their new lab, they hire professionals. When they need code for their research projects they?

opportune 7y ago

A lot of the points you bring up re related to cost. Here's the thing about cost, let's say it costs $100k/year to hire a good software engineer capable of writing scientific code (able to program and test complex algorithms, write HPC code, turn whitepapers into code), which might even be an underestimate depending on how benefits are paid out and the area. You can also fund 3 more grad students for that kind of money. The grad students will directly convert a PI's money into authorships while the software engineer's contribution will be only indirect, and likely take years to pay off. Plus, with only a single software engineer, there's a good chance you get unlucky and end up with someone clueless/lazy. You would probably need 3-4 software engineers to make a functioning team with best practices and hedge your bets against accidentally hiring someone who sucks. So now we're talking 10+ grad students. Open source software is a bit different because many labs can band together to fund things they find useful. But again there are still issues with cost-effectiveness. I'm guessing most lab contributors to OSS would want some sort of quid-pro-quo which may not be realistic for all OSS projects. And by funding OSS you are also funding competing labs' abilities to use the same features you use, which is good for science in general but not good for people's careers sometimes

bluGill 7y ago
Why hire an engineer? Why not walk over to the CS (might have a different name) and talk to a professor there. They can set you up with plenty of undergrads who need this experience, and they should be able to guide them into something that is maintainable long term. Note that I said should there. How to write maintainable programs seems to be lacking in research area.
- bp0017 7y ago
  Incidentally, this is exactly what my lab does
- boron1006 7y ago
  This would be a good solution, but from what I've seen with psychologists and statisticians, it's unlikely to happen for reasons I don't fully understand. Another thing is that undergraduates often learn by adopting the norms of the institution that they're in (e.g. using version control, linters, etc) but when they're brought in as the technical person, they don't have that opportunity to improve (This is my personal anecdote as that cs undergrad at one time).
- opportune 7y ago
  I was such a CS undergrad working in a lab once upon a time. I don't think you really want that because the undergrad will probably only be working for like 4-15hr/week potentially for only a single semester. For a summer position, sure it's 40hr/week but still only for about 10-15 weeks. And still, you're getting a generic CS undergrad's caliber of work and responsibility which I would say on average is not great. They might not be as familiar with version control, best practices, etc. and could just end up writing code just as bad as the scientists. I think if hiring a team you would need at least one somewhat experienced full-time software engineer to act as team lead/PM for the other developers, whether fulltime or students.
  - Master_Odin 7y ago
    Yeah, picking up undergrads (or even grad students) from the CS department is not a surefire way to end up code that follows best practices, is well maintained/documented, etc. and I'd probably argue if your team leader doesn't have that experience, you're more likely to end up with something that doesn't follow great practices (especially with regards to any sort of test suite).

mehrdadn 7y ago

> The sad part is that to a lot of scientists and researchers, software/software engineers isn't something worth paying for. I have to ask how you came to this conclusion. Did you have anecdotes from them supporting this or did you just conclude "they are not paying for it therefore software is not worth paying for to them"?

ylem 7y ago
I think the original poster is applying the "cheap talk" heuristic. People may say they value things, but we see what's really valued by what they pay for (given the resources they have)...

chrisseaton 7y ago

> Non-technical scientists How can you have a ‘non-technical scientist’? All science is inherently technical. Are you using ‘technical’ as a short-hand for ‘can program’? Stop doing that - programming is not the only technology.

scarejunba 7y ago
What an incredibly pointless thing to argue about. I think we all knew what he meant and, this being a specialist forum, we benefit from using shorthand in speech that is a lossy representation of ideas to those not initiated. I hate to have to create this conversation fork but I really wish people wouldn't make comments like this. They're so low signal.
- chrisseaton 7y ago
  I disagree - I think reducing anyone who doesn’t happen to program to being ‘non-technical’, with the implication that they don't know what they're talking about, is insidious.
  - scarejunba 7y ago
    It's clear he means "Scientists who don't want too much input into the engineering decisions". You're reading an implication of competency where there is none.
boron1006 7y ago
I'm using technical in a broad sense. One effect of scientific research becoming bigger is that roles are becoming more specialized. Often the people leading the research have primarily spend time writing grants or papers, and less senior professors and post-docs will carry out the actual tasks. When I first started I was a bit shocked at how many of the big names knew basically nothing about the process of how their own research is carried out. The effects are more pronounced depending on what field we're talking about. For example, in physics, I'd imagine most people have at least the fundamentals of programming down, even if software design may be lacking. In the field I work in (Neuroimaging), a lot of PI's are doctors or neuropsychologists, and might barely even know how to use a computer.

ylem 7y ago

Two other points to consider--equipment is a one-off expense and staffing is a continuous expense. The other is that the pots of money for equipment and staff may be different....

boron1006 7y ago
Completely right, I've been trying to hype up the "Github sponsorships" program as a way of changing the thinking around software (e.g. tacking OSS onto grants as required equipment), but haven't found much support.

nwallin 7y ago

The thing that terrifies me about the state of affairs is what happens when the software gives the wrong result because it's poorly written? If a scientists has input B and expects output X, and writes code that accepts B, and it happens to output X, chances are they'll write the paper and send it off for publication. Even if there's a bug in the code, and the real output should have been Y. I genuinely believe that this happens more often than most of us would find acceptable. Hell, there have probably been instances of the code outputting Y and the scientist scratches their head like, "Hmmm, that's not right, something's wrong," and will massage the code until it eventually outputs X, and then they'll say, "Fixed it!" and move on. I know for a fact it's happened at least once, and led to a scientific controversy that lasted for decades. Unfortunately, I don't remember the specifics; if someone recognizes my vague description please step in with a citation. But one group of scientists published a paper saying a certain dynamic system behaved in a certain way, and a second group published a paper saying it behaved in a different way, and the two results were completely incompatible with each other. Significant public disagreement ensued. One group published their code, and the other group attacked it saying it was poorly written, etc. The second group did not release their code. Decades later, some other scientist at the second institution released the code, and after a code review, it was found that the data was incorrectly initialized. They needed to initialize the particles with "random" initial velocity vectors, but the scientist who wrote the code didn't know how to do it correctly, and wrote an ad-hoc algorithm that gave the initial velocity vectors significant bias along an axis. But the paper was already written, peer reviewed, published, and cited, so even though the paper was wrong, the result was still accepted by (half of) the scientific community. AFAIK the paper was never retracted.

roel_v 7y ago
I have seen and fixed code that was first written as a prototype, then used as is in a follow on project and so on, until it turned out to be used for deciding on whether or not to grand certain subsidies. Except whoops in one spot it interpreted kilometers as meters without dividing by 1000. Along with literally dozens of other outright bugs. But nobody cares, really.

kochthesecond 7y ago

When said scientists want to build some new equipment, they get involved and they use professionals. When said scientists need to build their new lab, they hire professionals. When they need code for their research projects they?