8 ms·

There is simply no incentive for science to do proper software engineering. It does not directly produce papers or money, so nobody does it. I've seen it severa

There is simply no incentive for science to do proper software engineering. It does not directly produce papers or money, so nobody does it. I've seen it several times in my own career in several countries. PhDs having their own version of the code, incompatible with the one of the Postdoc sitting opposite of them. Each slaving away at their niche project but nobody there to bring it all together. This particular university had really great code and they thought they could sell it. But they did not even use git and when I gave a presentation there about it, I was met with absolute rejection. They don't have an idea on how much effort it takes to maintain a commercial code, unit tests, customer support etc. I'm now working at a company that produces simulation software spun out from university. Fantastic job and we do all of those things mentioned above. But obviously my paper output has been near zero even though we sometimes do cutting edge research. I wish I had a solution to the problem as so much grant money is wasted that produces a paper or two and the corresponding software for that just goes to digital nirvana, it's a real shame.

w0de0 4y ago
Is there any solution to be found through analogy with established fields' relationship between engineering and research disciplines? How, for instance, do physicists share formal models which rocket engineers might apply? Certainly the physicists aren't out developing unit tests for engines which apply their theory, but nor are the engineers exasperated by physicists rejecting their standardized tools (I imagine?).
- Azrael3000 4y ago
  Well from experience the interface is conferences and papers. On conferences you get to know the latest stuff that is going on and the implementation details can be found in papers. What needs to be said though is that reimplementing something from a paper is near impossible. I had to do that a couple of times and I was only fully successful if the paper was accompanied by some open source code as there always are tiny little edge cases or initialization details that won't be covered in the paper.
joshvm 4y ago
RSEs should not be working on, or assisting, research at the PhD level. By that I mean news/skunkworks stuff. They should be taking the output from a PhD and turning it into core software for the group. That doesn't solve the collaboration issue between PhDs/Postdocs, but there is a particular point in a research project lifecycle where it makes sense to hire an RSE. A bigger challenge is that most PIs are not project managers and have no experience as such, so they don't know how to express what they need in a structured way, or how to steer their group to collaborate properly. Outside computer science, many would struggle to budget software development or compute properly on a grant application (and the assessors have zero idea either).
- elementalest 4y ago
  RSE's can and often absolutely should be involved at the PhD level. In my experience, collaboration between the scientist and engineer in the process of research iterations almost always produce better results. Each has insights the other may not, likely leading to better outcomes for the research, final product/tool and time taken. The scientist just wants to focus on their research and once they have a barely working proof of concept, hand it over to the engineer to figure the rest out. The engineer wants a well specified design and prototype that they can lightly refactor to clean up, scale up and turn into a product/tool. The reality is that approach makes it way harder for both, though most often harder for the engineer as they are generally at the end of the chain in Academia and have little power. For example, the code or spec from the scientist is often terrible, so the engineer needs to start from scratch and keep going back to the scientist to spec out the design as they were not involved at any stage prior. They may even find edge cases or flaws the scientist had not considered that are fundamentally problematic to turning it into a viable product/tool. This is why the big corporate/industry research labs often have high level RSE that are involved in the research process and get their names in papers (they sometimes have PhD's themselves). They are not optimising for the scientists time, but for the companies resources
  - joshvm 4y ago
    Yeah let me be clear. PhD students absolutely should get guidance from experienced engineers (so I was a bit over-zealous with "assist" in my parent post). But this should be more like understanding best practices, and they should feel free to ask questions and figure out how to write better code. There are initiatives to do this called Software Carpentry.[0] However, RSEs should not be writing code for students doing PhD level projects in my opinion, for exactly the reasons you mention. I know some of the big research councils do this in the UK. For example STFC has a program where they'll work with universities and companies to production-ise research code. > The scientist just wants to focus on their research and once they have a barely working proof of concept, hand it over to the engineer to figure the rest out. The engineer wants a well specified design and prototype that they can lightly refactor to clean up, scale up and turn into a product/tool. As you say, this is a great idea in principle. In reality I think that it's really difficult to make it work. [0] https://www.software.ac.uk/programmes-events/carpentries/software-carpentry https://www.software.ac.uk/programmes-events/carpentries/sof...
soco 4y ago
I'd say it depends a lot on the environment you'd be landing. I was hired to cleanup some clean room software and boy did it need cleanup. Simply by implementing some best practices and pretty much all what today goes as devops (this was 15 years ago in Switzerland) the thing got so speedy that a company got interested in buying it, so my contract was extended to add the final perks to close that sale. I refused the offer to join the buyer company and that was it for me. Ah and my name landed second on a paper, the only non-PhD on the list. All in all it was an exciting stunt while the pay was in line with what others said - 30% under the market average (they were even surprised why I wanted it).
exdsq 4y ago
This isn't completely true. My wife is a postdoc at a medical lab at Stanford and there's a big drive to push good software practices - containerisation, tests, documentation, etc... and a good friend who's a postdoc in medical imaging at Oxford runs training courses for their labs software (along with following all standard engineering practices you'd see at a company). This will be subjective of where you're working and if you're driven to write good software. With the reproducibility issues in science, writing good clear software is becoming more important.
- Azrael3000 4y ago
  Of course my statement wasn't meant to be absolute. I come from an engineering background. I am not surprised that in a medical setting the situation is somewhat different, after all I guess the stakes there are higher. If your CFD code does not perform well, who cares. If your analysis of some medication is crap, chances somebody cares is going to be much higher. Also from experience, it helps if universities have closer ties to industry. As an industrial player you can't work with software that does not deliver reproducible results.
- goodpoint 4y ago
  > good software practices > containerisation Pick one.
  - roguas 4y ago
    ?
- sea-shunned 4y ago
  I'll agree that there is increasing emphasis on reproducibility and _useable_ software in academia. Writing documentation, unit tests etc. is still not really rewarded properly, but at least within the current paradigm such efforts are often rewarded with more users (and therefore citations) which is rewarded. Soon, hopefully, it'll be recognised more directly. Also, I'm currently a postdoc in medical imaging at UCL, super interested in learning a little more about the group in Oxford you mentioned if you're OK with sharing a link/group name? I may be able to guess but just want to check!
Hendrikto 4y ago
The goal of research scientists is rarely to produce a finished and polished product. Instead, they aim to prove some concept/algorithm/technique/etc. They are almost always time and resource strapped, so it becomes a near necessity to deprioritize factors like readability and maintainability. Still, many RSEs could immensely benefit from applying basic best practices.
myst1 4y ago
One could argue that bad software engineering practices are rewarded by academia. If no one can follow your work you drive away competition so why write comments. If you regularly refactor your work no one can use your API but you. Not unit testing ensures that code is cryptic, and people will have a hard time refuting your claims due to errors. The list goes on and on.

w0de0 4y ago

Is there any solution to be found through analogy with established fields' relationship between engineering and research disciplines? How, for instance, do physicists share formal models which rocket engineers might apply? Certainly the physicists aren't out developing unit tests for engines which apply their theory, but nor are the engineers exasperated by physicists rejecting their standardized tools (I imagine?).

Azrael3000 4y ago
Well from experience the interface is conferences and papers. On conferences you get to know the latest stuff that is going on and the implementation details can be found in papers. What needs to be said though is that reimplementing something from a paper is near impossible. I had to do that a couple of times and I was only fully successful if the paper was accompanied by some open source code as there always are tiny little edge cases or initialization details that won't be covered in the paper.

joshvm 4y ago

RSEs should not be working on, or assisting, research at the PhD level. By that I mean news/skunkworks stuff. They should be taking the output from a PhD and turning it into core software for the group. That doesn't solve the collaboration issue between PhDs/Postdocs, but there is a particular point in a research project lifecycle where it makes sense to hire an RSE. A bigger challenge is that most PIs are not project managers and have no experience as such, so they don't know how to express what they need in a structured way, or how to steer their group to collaborate properly. Outside computer science, many would struggle to budget software development or compute properly on a grant application (and the assessors have zero idea either).

elementalest 4y ago
RSE's can and often absolutely should be involved at the PhD level. In my experience, collaboration between the scientist and engineer in the process of research iterations almost always produce better results. Each has insights the other may not, likely leading to better outcomes for the research, final product/tool and time taken. The scientist just wants to focus on their research and once they have a barely working proof of concept, hand it over to the engineer to figure the rest out. The engineer wants a well specified design and prototype that they can lightly refactor to clean up, scale up and turn into a product/tool. The reality is that approach makes it way harder for both, though most often harder for the engineer as they are generally at the end of the chain in Academia and have little power. For example, the code or spec from the scientist is often terrible, so the engineer needs to start from scratch and keep going back to the scientist to spec out the design as they were not involved at any stage prior. They may even find edge cases or flaws the scientist had not considered that are fundamentally problematic to turning it into a viable product/tool. This is why the big corporate/industry research labs often have high level RSE that are involved in the research process and get their names in papers (they sometimes have PhD's themselves). They are not optimising for the scientists time, but for the companies resources
- joshvm 4y ago
  Yeah let me be clear. PhD students absolutely should get guidance from experienced engineers (so I was a bit over-zealous with "assist" in my parent post). But this should be more like understanding best practices, and they should feel free to ask questions and figure out how to write better code. There are initiatives to do this called Software Carpentry.[0] However, RSEs should not be writing code for students doing PhD level projects in my opinion, for exactly the reasons you mention. I know some of the big research councils do this in the UK. For example STFC has a program where they'll work with universities and companies to production-ise research code. > The scientist just wants to focus on their research and once they have a barely working proof of concept, hand it over to the engineer to figure the rest out. The engineer wants a well specified design and prototype that they can lightly refactor to clean up, scale up and turn into a product/tool. As you say, this is a great idea in principle. In reality I think that it's really difficult to make it work. [0] https://www.software.ac.uk/programmes-events/carpentries/software-carpentry https://www.software.ac.uk/programmes-events/carpentries/sof...

soco 4y ago

I'd say it depends a lot on the environment you'd be landing. I was hired to cleanup some clean room software and boy did it need cleanup. Simply by implementing some best practices and pretty much all what today goes as devops (this was 15 years ago in Switzerland) the thing got so speedy that a company got interested in buying it, so my contract was extended to add the final perks to close that sale. I refused the offer to join the buyer company and that was it for me. Ah and my name landed second on a paper, the only non-PhD on the list. All in all it was an exciting stunt while the pay was in line with what others said - 30% under the market average (they were even surprised why I wanted it).

exdsq 4y ago

This isn't completely true. My wife is a postdoc at a medical lab at Stanford and there's a big drive to push good software practices - containerisation, tests, documentation, etc... and a good friend who's a postdoc in medical imaging at Oxford runs training courses for their labs software (along with following all standard engineering practices you'd see at a company). This will be subjective of where you're working and if you're driven to write good software. With the reproducibility issues in science, writing good clear software is becoming more important.

Azrael3000 4y ago
Of course my statement wasn't meant to be absolute. I come from an engineering background. I am not surprised that in a medical setting the situation is somewhat different, after all I guess the stakes there are higher. If your CFD code does not perform well, who cares. If your analysis of some medication is crap, chances somebody cares is going to be much higher. Also from experience, it helps if universities have closer ties to industry. As an industrial player you can't work with software that does not deliver reproducible results.
goodpoint 4y ago
> good software practices > containerisation Pick one.
- roguas 4y ago
  ?
sea-shunned 4y ago
I'll agree that there is increasing emphasis on reproducibility and _useable_ software in academia. Writing documentation, unit tests etc. is still not really rewarded properly, but at least within the current paradigm such efforts are often rewarded with more users (and therefore citations) which is rewarded. Soon, hopefully, it'll be recognised more directly. Also, I'm currently a postdoc in medical imaging at UCL, super interested in learning a little more about the group in Oxford you mentioned if you're OK with sharing a link/group name? I may be able to guess but just want to check!

Hendrikto 4y ago

The goal of research scientists is rarely to produce a finished and polished product. Instead, they aim to prove some concept/algorithm/technique/etc. They are almost always time and resource strapped, so it becomes a near necessity to deprioritize factors like readability and maintainability. Still, many RSEs could immensely benefit from applying basic best practices.

myst1 4y ago

One could argue that bad software engineering practices are rewarded by academia. If no one can follow your work you drive away competition so why write comments. If you regularly refactor your work no one can use your API but you. Not unit testing ensures that code is cryptic, and people will have a hard time refuting your claims due to errors. The list goes on and on.