6 ms·

I don't think scientists want research software engineers. PIs typically like to assume any and every role on a project, including software design. They don't l

by bmitc 4y ago

I don't think scientists want research software engineers. PIs typically like to assume any and every role on a project, including software design. They don't like handing things off and letting people build modern, best practices software. You start talking about source code control, package managers, a language used in industry that they've never even heard of and don't care enough to learn about, etc., and you just get asked "why can't you just write it in Python or C because those are the only two things I'm remotely familiar with and zip it up and put it on a USB drive?". Heaven forbid you actually want to do something novel, no matter how enabling it could be. I have been continually inspired by Bret Victor's Seeing Spaces talk and have always wanted to build such a system described for a lab, but unfortunately, it was really hard to get anyone interested much less excited by such an idea. Scientists are far more institutionalized, conservative, and myopic than they admit. Physicists don't even use the proper word for code. They call it "codes". (I've seen in first-hand, in research papers, and in published books. It's endemic.) I've literally sat in meetings trying to get access to scientific projects I'm interested in and applied for jobs even at the university I already worked at in similar capacities, and yet, it's filled with PIs just wanting grunt labor (i.e., what they're used to from graduate students) to build things that they understand or you absolutely never hear back or the pay is a third of what you already make. And then there's pockets of software people in this niche world that only want to do things their way, because that's all they've ever done, and they can really be quite obstinate. They're like this massive pillar you must somehow work around. I really, really, really like writing software in scientific contexts, but it's such a quagmire. I'd like to return to that world some day with the lessons I've learned outside of it. Software in support of an interesting domain or context is just really fun. I've also tried getting a job at an architecture firm that had some very neat ideas for building out a tool, but the response was basically the same as described above.

throwawaysleep 4y ago
I recently quit a job as a software engineer on a research project for all the reasons mentioned. They want grunts who don't care about project management or good engineering practices and they will belittle you as not worth the money even when you took a (theoretical, as I am overemployed) massive paycut to go and work for them. I made 1/3 there compared to the jobs I am interviewing for to replace it. These people desperately need software help and have worthwhile projects that need software help. They also need to learn to collaborate over just parsing out obscure goals and being upset when what they get matches their three sentence email and not their vision in their head.
- bmitc 4y ago
  Yea, it's frustrating because there are indeed some super cool projects just waiting to have interesting software written for them, especially in the more hardware/physical oriented R&D areas. There are some areas that take software seriously, but in general, I have found it really difficult to get scientists and research engineers excited about anything that they don't know about. I don't think its malice. I think its part of their education and institutionalization couple a little with arrogance and somewhat blind view towards areas outside their expertise. I have given presentations before, and I usually only got interesting feedback from non-PI level workers. I always tried to paint a picture that we were doing state of the art technology development, so why shouldn't state of the art software be developed to support that?
  - cozzyd 4y ago
    On the flip side of that, many experiments run for decades or longer, and "state of the art" software is often a poor impedance match for that.
    - throwawaysleep 4y ago
      State of the art doesn't necessarily mean React. Just something with unit tests and more than one giant 5000 line file. If the requirement is that it lasts 30 years, you can choose more timeless languages like C, over say Rust.
      - cozzyd 4y ago
        I guess a better word for that might be "not terrible" :). And yes, way too much research code reads like somebody's first coding project.
    - bmitc 4y ago
      I'm not convinced of that. Because what I've seen (several times) on a somewhat small scale perhaps, is someone's code, sometimes written decades ago, never maintained upgraded or properly developed used for years on a system and eventually it gives out, forcing basically a rewrite. And state of the art can mean different things for differently scaled projects.
l0b0 4y ago
I'd encourage you to have a look at a well-known European physics research laboratory's (rhymes with Bern) library and engineering departments (not IT, unless they've started using version control and tests since I worked there). The salary is not as bad as academia, and there's plenty of interesting stuff going on. Most of it is accessible to anyone interested in a way I've not seen outside of academia.
- bmitc 4y ago
  Certainly cool stuff but out of the question due to location. I've worked for or met with people from similar size (at least in ambition) projects. It's a tough world to break into and especially difficult to bring a different outlook to.
  - kyawzazaw 4y ago
    Have you considered US's national labs if you are in the US?
    - myst1 4y ago
      Hard to get into, have to move to one of five places, usually requires security clearances (huge ordeal), extremely clean lifestyle, if your PI is evil expect zero protection, post doc can be considered entry level qualifications, can be asked to work 7 days a week. Ultimately, usually still pays less than entry level software positions offering full time remote work...
  - l0b0 4y ago
    I can understand not wanting to move very far, but the location is in many ways great: French alps within driving distance, Jura within walking distance, the lake, old architecture, lots of fields and forests. World-leading public transport. Geneva itself isn't very exciting, to be fair, but you're in the heart of Europe with easy access to all the rest. As for "breaking in," I dunno. The hiring process seems very fair and international (very few locals work there in a technical capacity AFAICT), and there are plenty of people with flexible ideas. Not everyone, for sure, but then there are so many different groups there you couldn't possibly blanket everyone with that statement.
jjtheblunt 4y ago
I think you accidentally revealed the problem when you said "best practices", because asserting such introduces a presumed total order on expertise, and that introduces conflict. thus ends the discussion between a PI and someone telling them they know better?
- bmitc 4y ago
  It's pretty easy for me to acquiesce all design decisions to their expertise and/or role in addition to being excited by what they do. I'd just like that to be reciprocated.
stackbutterflow 4y ago
See this infamous discussion: https://www.reddit.com/r/MachineLearning/comments/6l2esd/d_why_cant_you_guys_comment_your_fucking_code/ https://www.reddit.com/r/MachineLearning/comments/6l2esd/d_w... Tldr: They view software as just a tool and don't see how they'll benefit from following best practices.
- saghm 4y ago
  Oh wow, looking at that Python code they linked to brings back memories of my ML course in college. We had skeleton code given to us to fill in for the assignments, and often times it took longer for me to understand what the code they gave was even doing so I'd know what to fill in than it took me to actually write the code. Occasionally there would be random semicolons at the end of the lines, but the author seemed to understand that wasn't necessary in Python since they weren't on most lines, so I guess they were just careless. They also used a very amusing "OO" pattern where they would pass in every input needed to the constructor of objects, then have one method pass all of those instance variables directly into other methods which ignored the fact that they had a `self` parameter and just used the copies of the instance variables (with the same names) passed in as parameters. The biggest confusion I ever had was in one method where they used the typical `n, d = matrix.dimensions` (I can't remember the exact incantation, but it was used in all the code to store the dimensions in `n` and `d`, immediately followed by `d = d + 1`, which I took to assume that the leftmost column of the matrix was just filled with 1s and wasn't needed (which happened in some of the algorithms), but then down in the implementation of the algorithm, the only time `d` was used was as `d - 1`. Because I had mentally skipped over the boilerplate at the top of the function, I spent a decent amount of time trying to figure out why they had subtracted 1 from d until I finally saw the increment above. To this day, I still struggle to decide if the author legitimately did not notice this and just used `d - 1` to try to fix a problem that they didn't know the source of, or if they knew and thought that it was a reasonable way to implement things, and I'm also still conflicted about which would be worse.
  - wombatpm 4y ago
    You can write FORTRAN code in many languages
deleted 4y ago
[deleted]
mikebenfield 4y ago
> They call it "codes" This is not really the core issue here, but yes, I've encountered this many times, and I have to admit that in some professional contexts I use "codes" as a shibboleth to identify people who aren't really in touch with the software engineering world.
- titanomachy 4y ago
  I wonder if this used to be the more common term, and at some point we switched to “code”. Physicists were among the first serious computer users, but their software culture may have bifurcated from ours at some point.
- dguest 4y ago
  I always assumed it came from big international physics experiments where English is most people's second language. People who learn to program in an expat community are going to pick up some jargon that will sound weird to native English speakers in California.
cycomanic 4y ago
You seem to attribute certainl motivations to the PIs based on some misconceptions of the incentives and constrains for doing research. It is true that scientific software engineering is undervalued and this is one of the many issues in modern science. However attributing this to PIs wanting to do everything themselves is completely off the mark. One of the biggest issues in science today is that PIs can't do anything themselves, they, after having been trained as scientist for years, become glorified managers and administrators. The reason why they can't and don't want to hire software engineers to build well designed software is because they don't have any money for it and it is very mismatched with their incentives. If a PI gets a grant for a project, they typically get a constrained budget to achieve the scientific goals of the project. There is usually no money (and it would not be approved) to engineer well designed software, because that doesn't produce scientific outcomes. Now in the situations when there is budget for this, which are the super large institutional research initiatives, like e.g. CERN, they build incredibly well designed software. The SKA for example will have the largest storage cloud in the world. If one wants better designed scientific software one needs to seriously overhaul the scientific funding process and quite significantly increase funding.
sseagull 4y ago
> They call it "codes" They/We have probably been calling it that since before you were alive. I wouldn’t be surprised if it comes from the 60s or 70s. Every group of people has their own terminology for stuff. There is no right or wrong.
- bmitc 4y ago
  I have never heard or read anyone, beside physicists, say codes for software code. It doesn't even make sense. Where have you seen that used?
  - inejge 4y ago
    I have never heard or read anyone, beside physicists, say codes for software code. It doesn't even make sense. Why not? "A code" is a software package which solves some kind of problem computationally. Plural, codes. I'm sure it's been in use almost as long as the programmable computers. Here's[1] a link to an article from 1968 which uses the term in the singular ("The MENE Neutron Transport Code".) [1] https://www.osti.gov/biblio/4819611-mene-neutron-transport-code https://www.osti.gov/biblio/4819611-mene-neutron-transport-c...
    - bmitc 4y ago
      There are plenty of words in English that have the same singular and plural form.
    - mikebenfield 4y ago
      That article appears to be written by a non-native English speaker. I don't think I've ever heard anyone refer to a software package as "a code."
      - inejge 4y ago
        I don't think I've ever heard anyone refer to a software package as "a code." Then you probably haven't been around physicists, or read enough physics-related material. A non-scholarly reference which can't be easily checked online is George Dyson's "Project Orion," where Freeman Dyson (author's father) is quoted as mentioning this or that "code" as significant for the design of various Orion parts. (That should be a native enough English speaker.) The time-frame is late 1950s to early 1960s.
jll29 4y ago
PI here (Research Professor of AI, former industry R&D director, so I have seen both sides). The money will never be the same in academia, and rewards are outside of the control of the PI in most cases (e.g. in Germany, it's a government-level decision and everyone is paid the same), so there are two options: work with average people or find smart & idealistic ones. So... ...please excuse this shameless plug: I'm interested in people who would like to work with me to build research software (both on-premise and AWS cloud) using best practices, in particular using Rust. If you are interesteed in machine learning, natural language processing and information retrieval and would like to work with me as a research software engineer, please do get in touch (leidner at acm dot org). Our new 2 TB RAM dev research group server running Ubuntu 22.04 LTS is waiting for you (CPU/GPU cluster ressources also available) to set up (a VM for) your CI infrastructure. Your software will be used to push forward the frontier of research in machine learning and its applications to language (summarization, question answering) and information retrieval (vertical search engines, learning to rank, unsupervised topic modeling). Edit: academia can be poorly funded and bureaucratic. The former can be fixed by grant applications, the latter must be tolerated until politicians/activists fix it.
- inciampati 4y ago
  The fact that you included a job posting in your comment might be the best evidence in this thread that it's hard to hire RSEs. Good luck of course :)

by bmitc 4y ago

throwawaysleep 4y ago

I recently quit a job as a software engineer on a research project for all the reasons mentioned. They want grunts who don't care about project management or good engineering practices and they will belittle you as not worth the money even when you took a (theoretical, as I am overemployed) massive paycut to go and work for them. I made 1/3 there compared to the jobs I am interviewing for to replace it. These people desperately need software help and have worthwhile projects that need software help. They also need to learn to collaborate over just parsing out obscure goals and being upset when what they get matches their three sentence email and not their vision in their head.

bmitc 4y ago
Yea, it's frustrating because there are indeed some super cool projects just waiting to have interesting software written for them, especially in the more hardware/physical oriented R&D areas. There are some areas that take software seriously, but in general, I have found it really difficult to get scientists and research engineers excited about anything that they don't know about. I don't think its malice. I think its part of their education and institutionalization couple a little with arrogance and somewhat blind view towards areas outside their expertise. I have given presentations before, and I usually only got interesting feedback from non-PI level workers. I always tried to paint a picture that we were doing state of the art technology development, so why shouldn't state of the art software be developed to support that?
- cozzyd 4y ago
  On the flip side of that, many experiments run for decades or longer, and "state of the art" software is often a poor impedance match for that.
  - throwawaysleep 4y ago
    State of the art doesn't necessarily mean React. Just something with unit tests and more than one giant 5000 line file. If the requirement is that it lasts 30 years, you can choose more timeless languages like C, over say Rust.
    - cozzyd 4y ago
      I guess a better word for that might be "not terrible" :). And yes, way too much research code reads like somebody's first coding project.
  - bmitc 4y ago
    I'm not convinced of that. Because what I've seen (several times) on a somewhat small scale perhaps, is someone's code, sometimes written decades ago, never maintained upgraded or properly developed used for years on a system and eventually it gives out, forcing basically a rewrite. And state of the art can mean different things for differently scaled projects.

l0b0 4y ago

I'd encourage you to have a look at a well-known European physics research laboratory's (rhymes with Bern) library and engineering departments (not IT, unless they've started using version control and tests since I worked there). The salary is not as bad as academia, and there's plenty of interesting stuff going on. Most of it is accessible to anyone interested in a way I've not seen outside of academia.

bmitc 4y ago
Certainly cool stuff but out of the question due to location. I've worked for or met with people from similar size (at least in ambition) projects. It's a tough world to break into and especially difficult to bring a different outlook to.
- kyawzazaw 4y ago
  Have you considered US's national labs if you are in the US?
  - myst1 4y ago
    Hard to get into, have to move to one of five places, usually requires security clearances (huge ordeal), extremely clean lifestyle, if your PI is evil expect zero protection, post doc can be considered entry level qualifications, can be asked to work 7 days a week. Ultimately, usually still pays less than entry level software positions offering full time remote work...
- l0b0 4y ago
  I can understand not wanting to move very far, but the location is in many ways great: French alps within driving distance, Jura within walking distance, the lake, old architecture, lots of fields and forests. World-leading public transport. Geneva itself isn't very exciting, to be fair, but you're in the heart of Europe with easy access to all the rest. As for "breaking in," I dunno. The hiring process seems very fair and international (very few locals work there in a technical capacity AFAICT), and there are plenty of people with flexible ideas. Not everyone, for sure, but then there are so many different groups there you couldn't possibly blanket everyone with that statement.

jjtheblunt 4y ago

I think you accidentally revealed the problem when you said "best practices", because asserting such introduces a presumed total order on expertise, and that introduces conflict. thus ends the discussion between a PI and someone telling them they know better?

bmitc 4y ago
It's pretty easy for me to acquiesce all design decisions to their expertise and/or role in addition to being excited by what they do. I'd just like that to be reciprocated.

stackbutterflow 4y ago

See this infamous discussion: https://www.reddit.com/r/MachineLearning/comments/6l2esd/d_why_cant_you_guys_comment_your_fucking_code/ https://www.reddit.com/r/MachineLearning/comments/6l2esd/d_w... Tldr: They view software as just a tool and don't see how they'll benefit from following best practices.

saghm 4y ago
Oh wow, looking at that Python code they linked to brings back memories of my ML course in college. We had skeleton code given to us to fill in for the assignments, and often times it took longer for me to understand what the code they gave was even doing so I'd know what to fill in than it took me to actually write the code. Occasionally there would be random semicolons at the end of the lines, but the author seemed to understand that wasn't necessary in Python since they weren't on most lines, so I guess they were just careless. They also used a very amusing "OO" pattern where they would pass in every input needed to the constructor of objects, then have one method pass all of those instance variables directly into other methods which ignored the fact that they had a `self` parameter and just used the copies of the instance variables (with the same names) passed in as parameters. The biggest confusion I ever had was in one method where they used the typical `n, d = matrix.dimensions` (I can't remember the exact incantation, but it was used in all the code to store the dimensions in `n` and `d`, immediately followed by `d = d + 1`, which I took to assume that the leftmost column of the matrix was just filled with 1s and wasn't needed (which happened in some of the algorithms), but then down in the implementation of the algorithm, the only time `d` was used was as `d - 1`. Because I had mentally skipped over the boilerplate at the top of the function, I spent a decent amount of time trying to figure out why they had subtracted 1 from d until I finally saw the increment above. To this day, I still struggle to decide if the author legitimately did not notice this and just used `d - 1` to try to fix a problem that they didn't know the source of, or if they knew and thought that it was a reasonable way to implement things, and I'm also still conflicted about which would be worse.
- wombatpm 4y ago
  You can write FORTRAN code in many languages

deleted 4y ago

[deleted]

mikebenfield 4y ago

> They call it "codes" This is not really the core issue here, but yes, I've encountered this many times, and I have to admit that in some professional contexts I use "codes" as a shibboleth to identify people who aren't really in touch with the software engineering world.

titanomachy 4y ago
I wonder if this used to be the more common term, and at some point we switched to “code”. Physicists were among the first serious computer users, but their software culture may have bifurcated from ours at some point.
dguest 4y ago
I always assumed it came from big international physics experiments where English is most people's second language. People who learn to program in an expat community are going to pick up some jargon that will sound weird to native English speakers in California.

cycomanic 4y ago

You seem to attribute certainl motivations to the PIs based on some misconceptions of the incentives and constrains for doing research. It is true that scientific software engineering is undervalued and this is one of the many issues in modern science. However attributing this to PIs wanting to do everything themselves is completely off the mark. One of the biggest issues in science today is that PIs can't do anything themselves, they, after having been trained as scientist for years, become glorified managers and administrators. The reason why they can't and don't want to hire software engineers to build well designed software is because they don't have any money for it and it is very mismatched with their incentives. If a PI gets a grant for a project, they typically get a constrained budget to achieve the scientific goals of the project. There is usually no money (and it would not be approved) to engineer well designed software, because that doesn't produce scientific outcomes. Now in the situations when there is budget for this, which are the super large institutional research initiatives, like e.g. CERN, they build incredibly well designed software. The SKA for example will have the largest storage cloud in the world. If one wants better designed scientific software one needs to seriously overhaul the scientific funding process and quite significantly increase funding.

sseagull 4y ago

> They call it "codes" They/We have probably been calling it that since before you were alive. I wouldn’t be surprised if it comes from the 60s or 70s. Every group of people has their own terminology for stuff. There is no right or wrong.

bmitc 4y ago
I have never heard or read anyone, beside physicists, say codes for software code. It doesn't even make sense. Where have you seen that used?
- inejge 4y ago
  I have never heard or read anyone, beside physicists, say codes for software code. It doesn't even make sense. Why not? "A code" is a software package which solves some kind of problem computationally. Plural, codes. I'm sure it's been in use almost as long as the programmable computers. Here's[1] a link to an article from 1968 which uses the term in the singular ("The MENE Neutron Transport Code".) [1] https://www.osti.gov/biblio/4819611-mene-neutron-transport-code https://www.osti.gov/biblio/4819611-mene-neutron-transport-c...
  - bmitc 4y ago
    There are plenty of words in English that have the same singular and plural form.
  - mikebenfield 4y ago
    That article appears to be written by a non-native English speaker. I don't think I've ever heard anyone refer to a software package as "a code."
    - inejge 4y ago
      I don't think I've ever heard anyone refer to a software package as "a code." Then you probably haven't been around physicists, or read enough physics-related material. A non-scholarly reference which can't be easily checked online is George Dyson's "Project Orion," where Freeman Dyson (author's father) is quoted as mentioning this or that "code" as significant for the design of various Orion parts. (That should be a native enough English speaker.) The time-frame is late 1950s to early 1960s.

jll29 4y ago

PI here (Research Professor of AI, former industry R&D director, so I have seen both sides). The money will never be the same in academia, and rewards are outside of the control of the PI in most cases (e.g. in Germany, it's a government-level decision and everyone is paid the same), so there are two options: work with average people or find smart & idealistic ones. So... ...please excuse this shameless plug: I'm interested in people who would like to work with me to build research software (both on-premise and AWS cloud) using best practices, in particular using Rust. If you are interesteed in machine learning, natural language processing and information retrieval and would like to work with me as a research software engineer, please do get in touch (leidner at acm dot org). Our new 2 TB RAM dev research group server running Ubuntu 22.04 LTS is waiting for you (CPU/GPU cluster ressources also available) to set up (a VM for) your CI infrastructure. Your software will be used to push forward the frontier of research in machine learning and its applications to language (summarization, question answering) and information retrieval (vertical search engines, learning to rank, unsupervised topic modeling). Edit: academia can be poorly funded and bureaucratic. The former can be fixed by grant applications, the latter must be tolerated until politicians/activists fix it.

inciampati 4y ago
The fact that you included a job posting in your comment might be the best evidence in this thread that it's hard to hire RSEs. Good luck of course :)