5 ms·

> I’ve never heard of a comparable situation. Yet it is hard to believe scientists would be the ones who only ship flawless software. And easier to imagine tha

by rixed 7y ago

> I’ve never heard of a comparable situation. Yet it is hard to believe scientists would be the ones who only ship flawless software. And easier to imagine that most often, when one realises a previous publication's result was indeed impacted by an error and faces the difficult decision to retract or pretend they haven't noticed, they opt for the other route. I've found scientists in general to have less ego then average, as expected from people trained to care only for the facts, but they still operate in the larger framework of individualistic modern society. It probably takes a scientist and a woman, whose education generally encourage lesser ego to begin with, to admit such an error. Or a software author, but there there is no other alternative to guilt admittance than total ridicule, since we are drowning everyday in a world of errors.

searchableguy 7y ago
> Yet it is hard to believe scientists would be the ones who only ship flawless software. I mean, most of them don't even pin down their dependencies. I don't want to touch a lot of python code in the open written by scientists. Thinking about software reproducibility is the last thing I have noticed on repositories submitted with a paper on arxiv and other places of publishing. Not even a requirements.txt or mentioning what they used.
- dgoldstein0 7y ago
  I've spent years learning and practicing good software engineering practices. Whenever I read about quality issues in scientific code, I wonder: how are they expected to learn these things?
  - searchableguy 7y ago
    I don't think you need to worry about more than a few principles - 1. Reproducibility This is very important for scientists so I expect them to know about package management and containers. You don't need to know everything about it. Virtual environments, docker and using a good package manager like poetry (which is just an abstraction over virtualenv) shouldn't take more than a few weeks. Once you are done with the basics, you can learn more as you build. 2. Git and documentation Add another few weeks. They don't need to write good code. They just need to document it which they should already be good at. Optimization, scalability, distribution and other things can be handled by engineers but without documentation, it makes the work a lot harder.
    - sgt101 7y ago
      Every engineering team that picked up a code base has at some point had a meeting where someone starts ranting about how the code that they are looking after stinks and it would be better to rewrite it, or move to some alternative. This seems to be true if the code is the product of one person working for a few weeks or 20,000 people who work for 10 years. A few years ago I had a conversation where the team insisted to me that the extensive documentation and instructions for a package were worthless because "good code is self documenting" the leader of the pack had shiny eyes and banged the table several times. Be cautious, be kind and look for value in the code that you are using - not just problems. Also, consider: if you can pick up a code base (written by someone who knows the science) would it not be more profitable to repair, document and improve rather than demanding that the scientists learn both your skills and your way of doing things? If you conclude the latter, would you (hand on heart) be able to empirically demonstrate that your way of doing things is the right way?
    - throwaway_pdp09 7y ago
      I disagree with "They don't need to write good code". Reproducability is a check that you got the right answer. As you describe it, it's more like a crappy programmer that relies on the testing team to catch his errors rather than be responsible for his own. > and other things can be handled by engineers Due to lack of funding and low pay I think most of them don't have access to software domain experts (aka seasoned programmers)
  - yummypaint 7y ago
    Physicist here. The lack of any formal training in most programs is a problem. In 2020 it's essentially impossible to do research without being able to write and maintain code, yet there are no real standards of knowledge for the dicipline as a whole. I dont know of any undergrad programs requiring physics majors to take even intro to cs. Most incoming grad students have never used version control, and probably about half of professors have never used it, let alone want to start using it. Larger collaborations tend to be more rigorous with coding practices because they have dedicated software engineers who enforce order and show people the way. Small groups with a few students and a professor are basically the wild west as far as writing software, and vary wildly. In practice most grad students get coding experience by working on projects, which is a great way to learn, but its rare for anyone to look at the code itself afterward if the results are sensible. I dont know of any small group with a formal code review process. I think we could make relatively inexpensive improvements in science by bringing in more dedicated software people who can educate others and help with the software engineering aspects. Its not unusual for a lab with half a dozen groups and 50+ scientists to have no software specialists, and 1 or 2 IT people whose responsibilities are purely infrastructure oriented.
  - dekhn 7y ago
    This is the core question. Can you teach good software engineering, or does it come from years of working with more experienced people/making mistakes and learning? I have met a few "born SREs" who had little or no training but knew exactly what to do in production outage situations. But I haven't met anybody who was a "Great Software Engineer" out of the box. And scientists aren't going to take a lot of time out of their education to get that day-to-day programming experience. I was lucky- I worked in a theory lab, next to a team of good software developers, and worked/learned from them over a period of years. I think what we need are more research engineers, who are people who do software engineering for scientists, so the scientists don't have to become software engineering experts. The scientists also need to learn a bit better to ask for the right tools.
denzil_correa 7y ago
> Yet it is hard to believe scientists would be the ones who only ship flawless software. The job of a scientist is really not to ship software, that's what a team of engineers would do.
- diegolo 7y ago
  > The job of a scientist is really not to ship software, that's what a team of engineers would do. I think that this is the real problem - in academia there is this idea that learning good practices is like a 'dirty' thing that is not required, while instead it would speed up the work and make it more reliable. if you look at chemistry or medicine, there researches have good practices for managing the lab and respect them.
  - 317070 7y ago
    Except that it wouldn't speed things up at all. Academia writes run-once code, which changes spec fourteen times in one week. Their use case is orthogonal to industry. Have you considered that maybe the academics actually know what they are doing?
    - diegolo 7y ago
      Lol I spent 5 years in academia, and I have a PhD in CS - I know what I'm talking about. Specs of code change in academia as in industry, I was able to write unit tests and document my code also in academia. And I know in medicine and chemistry time to publish are much longer - but that is not connected with the fact that they know how to properly use a microscope, clean the lab, and keep an inventory. If you don't write unit tests how do you hedge the possibility of having bugs in your code?
      - neltnerb 7y ago
        Most scientists have no training in computer science, much less engineering, but still need to do it sometimes to build experiments. They've largely taught themselves. You are not the norm. I've taught dozens of grad students enough programming to get the job done and it would have been a total waste of time to make the code that robust. They need experimental results next week, with only one computer ever expected to run the code, not a product demo. The software isn't their research project, it's a nuisance that they have to deal with. Accordingly they neither want to nor have time to do it perfectly. I cannot blame them. That said, there should be a system to encourage actual trained programmers to get involved, including coauthorship and consideration in tenure decisions. The current system is bad, I'm just saying it's not the scientists fault here. This is just literally not their domain of interest or expertise, and I would rather they focus on the thing they're uniquely good at.
nabla9 7y ago
> individualistic modern society. I'm sure you don't really mean that premodern societies were more selfless and humble, but that's how you writing can be interpreted. Family name, fateralism and class based society puts even stronger incentives to skew the the truth than individualism.
- rixed 7y ago
  I happen to believe that a few generations ago people used to care a bit more about whatever group they though they belong to (family, clan, institution, corporation...) and a bit less about standing out with their own individual journey. I do not believe this was any better for the truth in general, as indeed in-group/out-group worldview can be a strong incentive to all sorry of lies. Except in those specific cases where one has to pick between truth that's the best interest of the scientific society and falsehood that's the best interest of the ego.

by rixed 7y ago

searchableguy 7y ago

> Yet it is hard to believe scientists would be the ones who only ship flawless software. I mean, most of them don't even pin down their dependencies. I don't want to touch a lot of python code in the open written by scientists. Thinking about software reproducibility is the last thing I have noticed on repositories submitted with a paper on arxiv and other places of publishing. Not even a requirements.txt or mentioning what they used.

dgoldstein0 7y ago
I've spent years learning and practicing good software engineering practices. Whenever I read about quality issues in scientific code, I wonder: how are they expected to learn these things?
- searchableguy 7y ago
  I don't think you need to worry about more than a few principles - 1. Reproducibility This is very important for scientists so I expect them to know about package management and containers. You don't need to know everything about it. Virtual environments, docker and using a good package manager like poetry (which is just an abstraction over virtualenv) shouldn't take more than a few weeks. Once you are done with the basics, you can learn more as you build. 2. Git and documentation Add another few weeks. They don't need to write good code. They just need to document it which they should already be good at. Optimization, scalability, distribution and other things can be handled by engineers but without documentation, it makes the work a lot harder.
  - sgt101 7y ago
    Every engineering team that picked up a code base has at some point had a meeting where someone starts ranting about how the code that they are looking after stinks and it would be better to rewrite it, or move to some alternative. This seems to be true if the code is the product of one person working for a few weeks or 20,000 people who work for 10 years. A few years ago I had a conversation where the team insisted to me that the extensive documentation and instructions for a package were worthless because "good code is self documenting" the leader of the pack had shiny eyes and banged the table several times. Be cautious, be kind and look for value in the code that you are using - not just problems. Also, consider: if you can pick up a code base (written by someone who knows the science) would it not be more profitable to repair, document and improve rather than demanding that the scientists learn both your skills and your way of doing things? If you conclude the latter, would you (hand on heart) be able to empirically demonstrate that your way of doing things is the right way?
  - throwaway_pdp09 7y ago
    I disagree with "They don't need to write good code". Reproducability is a check that you got the right answer. As you describe it, it's more like a crappy programmer that relies on the testing team to catch his errors rather than be responsible for his own. > and other things can be handled by engineers Due to lack of funding and low pay I think most of them don't have access to software domain experts (aka seasoned programmers)
- yummypaint 7y ago
  Physicist here. The lack of any formal training in most programs is a problem. In 2020 it's essentially impossible to do research without being able to write and maintain code, yet there are no real standards of knowledge for the dicipline as a whole. I dont know of any undergrad programs requiring physics majors to take even intro to cs. Most incoming grad students have never used version control, and probably about half of professors have never used it, let alone want to start using it. Larger collaborations tend to be more rigorous with coding practices because they have dedicated software engineers who enforce order and show people the way. Small groups with a few students and a professor are basically the wild west as far as writing software, and vary wildly. In practice most grad students get coding experience by working on projects, which is a great way to learn, but its rare for anyone to look at the code itself afterward if the results are sensible. I dont know of any small group with a formal code review process. I think we could make relatively inexpensive improvements in science by bringing in more dedicated software people who can educate others and help with the software engineering aspects. Its not unusual for a lab with half a dozen groups and 50+ scientists to have no software specialists, and 1 or 2 IT people whose responsibilities are purely infrastructure oriented.
- dekhn 7y ago
  This is the core question. Can you teach good software engineering, or does it come from years of working with more experienced people/making mistakes and learning? I have met a few "born SREs" who had little or no training but knew exactly what to do in production outage situations. But I haven't met anybody who was a "Great Software Engineer" out of the box. And scientists aren't going to take a lot of time out of their education to get that day-to-day programming experience. I was lucky- I worked in a theory lab, next to a team of good software developers, and worked/learned from them over a period of years. I think what we need are more research engineers, who are people who do software engineering for scientists, so the scientists don't have to become software engineering experts. The scientists also need to learn a bit better to ask for the right tools.

denzil_correa 7y ago

> Yet it is hard to believe scientists would be the ones who only ship flawless software. The job of a scientist is really not to ship software, that's what a team of engineers would do.

diegolo 7y ago
> The job of a scientist is really not to ship software, that's what a team of engineers would do. I think that this is the real problem - in academia there is this idea that learning good practices is like a 'dirty' thing that is not required, while instead it would speed up the work and make it more reliable. if you look at chemistry or medicine, there researches have good practices for managing the lab and respect them.
- 317070 7y ago
  Except that it wouldn't speed things up at all. Academia writes run-once code, which changes spec fourteen times in one week. Their use case is orthogonal to industry. Have you considered that maybe the academics actually know what they are doing?
  - diegolo 7y ago
    Lol I spent 5 years in academia, and I have a PhD in CS - I know what I'm talking about. Specs of code change in academia as in industry, I was able to write unit tests and document my code also in academia. And I know in medicine and chemistry time to publish are much longer - but that is not connected with the fact that they know how to properly use a microscope, clean the lab, and keep an inventory. If you don't write unit tests how do you hedge the possibility of having bugs in your code?
    - neltnerb 7y ago
      Most scientists have no training in computer science, much less engineering, but still need to do it sometimes to build experiments. They've largely taught themselves. You are not the norm. I've taught dozens of grad students enough programming to get the job done and it would have been a total waste of time to make the code that robust. They need experimental results next week, with only one computer ever expected to run the code, not a product demo. The software isn't their research project, it's a nuisance that they have to deal with. Accordingly they neither want to nor have time to do it perfectly. I cannot blame them. That said, there should be a system to encourage actual trained programmers to get involved, including coauthorship and consideration in tenure decisions. The current system is bad, I'm just saying it's not the scientists fault here. This is just literally not their domain of interest or expertise, and I would rather they focus on the thing they're uniquely good at.

nabla9 7y ago

> individualistic modern society. I'm sure you don't really mean that premodern societies were more selfless and humble, but that's how you writing can be interpreted. Family name, fateralism and class based society puts even stronger incentives to skew the the truth than individualism.

rixed 7y ago
I happen to believe that a few generations ago people used to care a bit more about whatever group they though they belong to (family, clan, institution, corporation...) and a bit less about standing out with their own individual journey. I do not believe this was any better for the truth in general, as indeed in-group/out-group worldview can be a strong incentive to all sorry of lies. Except in those specific cases where one has to pick between truth that's the best interest of the scientific society and falsehood that's the best interest of the ego.