4 ms·

These incentives exist but are not a good excuse in my view. I'd recommend reading this blog post about how "incentives" are used to justify all sorts of bad be

These incentives exist but are not a good excuse in my view. I'd recommend reading this blog post about how "incentives" are used to justify all sorts of bad behavior in academia: http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-incentives-its-you/ http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-inc... Right now I'm nearly done a PhD in mechanical engineering. I've worked on several computational projects of varying levels of code quality from good (e.g., NIST's Fire Dynamics Simulator: https://github.com/firemodels/fds https://github.com/firemodels/fds) to bad (won't give unambiguous examples for obvious reasons...). I see clear incentives against uncertainty quantification and high quality model validation in my field. Those topics are treated superficially if at all, because people just want their model to look good, and if they do a rigorous job, that increases the chance that their model looks bad. As far as I'm concerned, if you don't do good quality model validation, you're not doing your job as a scientist. A more specific example: Some codes are kept for a decade or longer if the software is sufficiently complex. One code I worked with was used by multiple graduate students over several PhDs and had very little if anything in terms of documentation. What did exist was far out of date. I spent a fair amount of time trying to understand and document the software, but other people treated me as if I was crazy for wanting to understand the black box I was provided. I don't think I ever actually understood that software fully despite the many hours I spent working on it. And I'm not certain that it provided correct results either. There were almost no tests, and the existing tests were hard to run and were consequently almost never run. The software produced several PhDs, though. The first few PhDs had it fairly easy. I had to pay their technical debt, and I couldn't pay it all back. It made me look bad as far as I'm concerned and did not save time in the long run. If anything, there's a long term incentive toward good software development practices. Ultimately I think the solution would involve having minimum standard for scientific software quality, or else grant funding will be withdrawn. It will anger many people who have so far gotten away with writing low quality software, but beyond that I don't see many downsides.

petschge 8y ago
I agree that not enough benchmarking (comparision between different codes to check that they get compatible results), verification (checking that the code produces results consistent with the underlying model) and validation (checking that the code reproduces nature to a sufficient degree to be useful) it done. I have actually written papers that try do to a bit of that and it was annoyingly hard to publish them. But unlike you and a lot of other posters here on HN I don't think the reason for insufficient validation is that we scientists are too stupid or too lazy. On the contrary, we would love to do more of it. But there is next to no funding to do it.
- btrettel 8y ago
  I appreciate your thoughts. > I have actually written papers that try do to a bit of that and it was annoyingly hard to publish them. I don't know what field you work in, but in my field (fluid dynamics) these papers are not uncommon, though often poorly done. There's a relatively new ASME journal on the subject too: http://verification.asmedigitalcollection.asme.org/journal.aspx http://verification.asmedigitalcollection.asme.org/journal.a... > On the contrary, we would love to do more of it. But there is next to no funding to do it. Perhaps this is the crux of the disagreement then. I don't need extra funding to do what I should be doing in the first place. It can be a big time sink in some cases (e.g., I probably spent 6+ months compiling existing data during my PhD), but often is not. It will probably make your model look worse than you'd like, but if everyone did it then science would progress faster.
  - petschge 8y ago
    Kinetic plasma simulations in space plasma and astrophyscial scenarios. I have seen that journal and I think it is a step in the correct direction. Regarding the funding: you kind DO need that extra funding. because if you don't get funding for that you get out-competed by the other research who skips that step and the next postdoc position or grant goes to that researcher who wrote two more papers instead of testing code that ultimately didn't find any major bugs.
    - btrettel 8y ago
      > you kind DO need that extra funding. because if you don't get funding for that you get out-competed [...] This seems to be a common view, but I don't think it's always true. In my own work, I've found that being careful has led me to write more papers, not less. Particularly given that I've found errors in previous work that I intend to publish or already have published. There's a delay involved (i.e., it takes time to find the problems) but I think I'm coming out ahead in total publications in the long run.
      - petschge 8y ago

petschge 8y ago

I agree that not enough benchmarking (comparision between different codes to check that they get compatible results), verification (checking that the code produces results consistent with the underlying model) and validation (checking that the code reproduces nature to a sufficient degree to be useful) it done. I have actually written papers that try do to a bit of that and it was annoyingly hard to publish them. But unlike you and a lot of other posters here on HN I don't think the reason for insufficient validation is that we scientists are too stupid or too lazy. On the contrary, we would love to do more of it. But there is next to no funding to do it.

btrettel 8y ago
I appreciate your thoughts. > I have actually written papers that try do to a bit of that and it was annoyingly hard to publish them. I don't know what field you work in, but in my field (fluid dynamics) these papers are not uncommon, though often poorly done. There's a relatively new ASME journal on the subject too: http://verification.asmedigitalcollection.asme.org/journal.aspx http://verification.asmedigitalcollection.asme.org/journal.a... > On the contrary, we would love to do more of it. But there is next to no funding to do it. Perhaps this is the crux of the disagreement then. I don't need extra funding to do what I should be doing in the first place. It can be a big time sink in some cases (e.g., I probably spent 6+ months compiling existing data during my PhD), but often is not. It will probably make your model look worse than you'd like, but if everyone did it then science would progress faster.
- petschge 8y ago
  Kinetic plasma simulations in space plasma and astrophyscial scenarios. I have seen that journal and I think it is a step in the correct direction. Regarding the funding: you kind DO need that extra funding. because if you don't get funding for that you get out-competed by the other research who skips that step and the next postdoc position or grant goes to that researcher who wrote two more papers instead of testing code that ultimately didn't find any major bugs.
  - btrettel 8y ago
    > you kind DO need that extra funding. because if you don't get funding for that you get out-competed [...] This seems to be a common view, but I don't think it's always true. In my own work, I've found that being careful has led me to write more papers, not less. Particularly given that I've found errors in previous work that I intend to publish or already have published. There's a delay involved (i.e., it takes time to find the problems) but I think I'm coming out ahead in total publications in the long run.
    - petschge 8y ago