4 ms·

That's a strange comment. It looks like the goal of these software are not at all for "productization" and that even if the author wanted to, there will be no p

by cauch 3y ago

That's a strange comment. It looks like the goal of these software are not at all for "productization" and that even if the author wanted to, there will be no point of porting it or hardening it. It feels like Software Developers are "brainwashed": a good software is a software that is good for what software developers need to do. But a good software is a software that is good for what people who needs it need to do. If the academic people don't need "productization" and still code for productzation, then they are doing a bad job. Inversely, software made by software developers may be really bad in academic sector, as it is the subject of the article: they overengineer when not needed, they complicate things just for portability or future use that will never happen, ... It's a bit like if someone says "these cooks who are preparing omelettes are bad cooks: they are mixing white and yolk together, while real cooks who do meringue always remove the yolk. So the proper way to cook eggs is by removing the yolk". (another aspect that I've noticed is that software developers talk about unit test and code review and are offended if they see scientists not using them properly, but they don't even realise that the goal of these is fulfilled in an arguably more efficient way by other processes that exist in academia. For example, there is sometimes no unit test, but the creator of the algo is also the main user and they will notice directly if a new change has broken something. Or as another example, scientific collaboration often implies several teams working independently and writing their own implementation from scratch, so if team A and team B gets different results from the same inputs, they will investigate and find out the problem in the code)

seadan83 3y ago
Some of your points stand. Though, in the iron triangle of speed, quality, scope, it turns out that quality and speed are linked. Quality is a requirement for speed. So, it seems more akin to making meringue with yolks. Eventually maybe it will work, but if you knew what you are doing and cared, it would be done better faster. The criticism though of losing sight of the goal is valid. That happens.
- cauch 3y ago
  "Quality" for scientists is not the same specificities as "quality" for a software developer. The goal of the software of the scientist is different, so the definition of what is "quality" is different. The article illustrates that: a lot of "software done following the good software developer practice" ends up being of bad quality for the job, and end up wasting a lot of time. Another aspect is the context: the iron triangle is also something built for a specific context. Of course a code that contains very very flexible function will have problems after 5 years of development and usage which will lead to drastic decrease of speed. But scientific code should not be used 5 years later (scientific code is to prove hypothesis, once the article published, you should not use this code, because this code, by construction, contains plenty of hypothesis testing that have been demonstrated not useful). So, the reason "speed" is related to "quality" is different in science.
  - seadan83 3y ago
    > "Quality" for scientists is not the same specificities as "quality" for a software developer. I disagree here. I believe quality is intrinsic to the product. There may be different attributes to the "quality" of the thing that one person may value more than another, but the intrinsic quality is the same. An over-engineered solution is rarely good. Don't use a power-boat to go across a small swimming pool, and don't use a shoddy makeshift raft to across a giant lake. > The goal of the software of the scientist is different, so the definition of what is "quality" is different. I disagree here, but I think I see your point. The general goal for everyone (the goal of software) is to accomplish some task. Software is fundamentally just a tool. Software for the sake of software is bad. This makes me think of an analogy where a person is trying to redo the plumbing on their kitchen sink. Compared to a firm that will do whacky crazy things and leave the situation worse than when they started, and walk away with their bills payed and job half done. Compared to this, certainly a competent novice is better. Though, sometimes it is really important to know about certain O-rings, not only would the "true" professional do the job slightly faster & more methodically than the novice, but they'll know about that O-ring that would only become a problem in the winter. At the same time, sometimes there are no such O-rings, and a simple job is just ultimately a relatively simple job. To that last extent, hiring software engineers can sometimes be scary. When you get very intelligent people and pay them to "solve complex problems," they do tend to build things that are very flexible, FAANG-scalable, very industry latest, AI powered, for the sake of it lasting 5 years; rather than starting simple - Gall's law: “A complex system that works is invariably found to have evolved from a simple system that worked." Or, those engineers may start there because they know those are the "best practices" without yet experiencing why those are best practices, and when to apply those practices and when not to apply those practices. [1] http://principles-wiki.net/principles:gall_s_law http://principles-wiki.net/principles:gall_s_law
pphysch 3y ago
> It's a bit like if someone says "these cooks who are preparing omelettes are bad cooks... No, it's like serving the food on the floor without dishes. It doesn't matter whether the food is fancy or simple. We need not defend the poor practices induced by the corrupt incentives of academia (publish or perish). Ideally, scientific methods and products of research should be highly reproducible (which implies a certain level of quality, in practice).
- cauch 3y ago
  The reason the code of scientist is like it is is not "corrupt incentives of academia", it is just because the goals and the context is different. First, scientific code is exploratory: you start building it without knowing which way will work. For example, you end up with 10 different fit functions, all using different approach (multi-gaussian, kde of control region, home-made shape based on a sum of Gumbel and Gaussian fcts, ...), and during your study, you do your test to conclude which one is the best. Because of that, "building for the future the multi-gaussian fit fct" is just plain stupid: you are spending time for something that have a high probability (>75%) of being totally dropped next week. You also need to build for huge flexibility. For example, having some "god object" allows to pivot very quickly when suddenly you discover a new parameter to add to the equation, rather than to refactorise all the function one by one to add a new parameter of a very specific type to all of them. In this context, the person who codes with strong static typing is not being very smart: in research, their code will change very often and they are just building a tool that will waste their time. Second, the code is the tool, not the product. The scientific conclusion is the product. What needs to be reproducible is the conclusion. You apparently don't understand the goal of reproducibility. Reproducibility is to prove that 2 persons who redo it from scratch obtain the same results, even if they use different tool. By saying that it is important that a scientist B can just blindly run the software of a scientist A, you just screw the reproducibility advantage: if the software has a bug, scientist B will say "I get the same result, so, it is confirmed" while the result is incorrect. At CERN, people from the CMS experiment and people from the Atlas experiment are FORBIDEN to share their code for this reason. Even reading the code can be bad because it can bias the reader. Then, of course, there is a balance and I recommend to share the code with publication, but the reader should be educated enough to know that the code is the last resort to look at if they have a question when trying to reproduce the result. Third, the way scientists work and collaborate creates to some extend "intrinsic" code review and unit testing. Several teams build software in parallel to check the same thing, and will investigate by comparing their result. And scientists are the first user of their software, intensively: they will never build a feature that they will not themselves use in practice later. So, some of the good practices of software dev are just not well designed for these situations. At the end, you are right: serving the food on the floor is bad. The software developer "good practice" are serving the food on the floor because they assumed the dish was where it is not. Finally, one thing to keep in mind is that some software dev people are just full of themselves with big ego. They love to think they are smarter than everyone else. If they are applying good practices themselves, they love feeling superior by thinking the ones not applying them are inferior and that they are so smart. It's a way of rationalising their traditions, and it's then easy to see plenty of ways that confirm this belief. Don't get me wrong, plenty of good practices are really useful, and plenty of scientific code is just badly done. But you just need to have a balance: instead of blindly applying tradition just because it flatter their ego, people should just take a step back and think of why X and Y is better in some context and see how it translates in another context.

by cauch 3y ago

seadan83 3y ago

Some of your points stand. Though, in the iron triangle of speed, quality, scope, it turns out that quality and speed are linked. Quality is a requirement for speed. So, it seems more akin to making meringue with yolks. Eventually maybe it will work, but if you knew what you are doing and cared, it would be done better faster. The criticism though of losing sight of the goal is valid. That happens.

cauch 3y ago
"Quality" for scientists is not the same specificities as "quality" for a software developer. The goal of the software of the scientist is different, so the definition of what is "quality" is different. The article illustrates that: a lot of "software done following the good software developer practice" ends up being of bad quality for the job, and end up wasting a lot of time. Another aspect is the context: the iron triangle is also something built for a specific context. Of course a code that contains very very flexible function will have problems after 5 years of development and usage which will lead to drastic decrease of speed. But scientific code should not be used 5 years later (scientific code is to prove hypothesis, once the article published, you should not use this code, because this code, by construction, contains plenty of hypothesis testing that have been demonstrated not useful). So, the reason "speed" is related to "quality" is different in science.
- seadan83 3y ago
  > "Quality" for scientists is not the same specificities as "quality" for a software developer. I disagree here. I believe quality is intrinsic to the product. There may be different attributes to the "quality" of the thing that one person may value more than another, but the intrinsic quality is the same. An over-engineered solution is rarely good. Don't use a power-boat to go across a small swimming pool, and don't use a shoddy makeshift raft to across a giant lake. > The goal of the software of the scientist is different, so the definition of what is "quality" is different. I disagree here, but I think I see your point. The general goal for everyone (the goal of software) is to accomplish some task. Software is fundamentally just a tool. Software for the sake of software is bad. This makes me think of an analogy where a person is trying to redo the plumbing on their kitchen sink. Compared to a firm that will do whacky crazy things and leave the situation worse than when they started, and walk away with their bills payed and job half done. Compared to this, certainly a competent novice is better. Though, sometimes it is really important to know about certain O-rings, not only would the "true" professional do the job slightly faster & more methodically than the novice, but they'll know about that O-ring that would only become a problem in the winter. At the same time, sometimes there are no such O-rings, and a simple job is just ultimately a relatively simple job. To that last extent, hiring software engineers can sometimes be scary. When you get very intelligent people and pay them to "solve complex problems," they do tend to build things that are very flexible, FAANG-scalable, very industry latest, AI powered, for the sake of it lasting 5 years; rather than starting simple - Gall's law: “A complex system that works is invariably found to have evolved from a simple system that worked." Or, those engineers may start there because they know those are the "best practices" without yet experiencing why those are best practices, and when to apply those practices and when not to apply those practices. [1] http://principles-wiki.net/principles:gall_s_law http://principles-wiki.net/principles:gall_s_law

pphysch 3y ago

> It's a bit like if someone says "these cooks who are preparing omelettes are bad cooks... No, it's like serving the food on the floor without dishes. It doesn't matter whether the food is fancy or simple. We need not defend the poor practices induced by the corrupt incentives of academia (publish or perish). Ideally, scientific methods and products of research should be highly reproducible (which implies a certain level of quality, in practice).

cauch 3y ago
The reason the code of scientist is like it is is not "corrupt incentives of academia", it is just because the goals and the context is different. First, scientific code is exploratory: you start building it without knowing which way will work. For example, you end up with 10 different fit functions, all using different approach (multi-gaussian, kde of control region, home-made shape based on a sum of Gumbel and Gaussian fcts, ...), and during your study, you do your test to conclude which one is the best. Because of that, "building for the future the multi-gaussian fit fct" is just plain stupid: you are spending time for something that have a high probability (>75%) of being totally dropped next week. You also need to build for huge flexibility. For example, having some "god object" allows to pivot very quickly when suddenly you discover a new parameter to add to the equation, rather than to refactorise all the function one by one to add a new parameter of a very specific type to all of them. In this context, the person who codes with strong static typing is not being very smart: in research, their code will change very often and they are just building a tool that will waste their time. Second, the code is the tool, not the product. The scientific conclusion is the product. What needs to be reproducible is the conclusion. You apparently don't understand the goal of reproducibility. Reproducibility is to prove that 2 persons who redo it from scratch obtain the same results, even if they use different tool. By saying that it is important that a scientist B can just blindly run the software of a scientist A, you just screw the reproducibility advantage: if the software has a bug, scientist B will say "I get the same result, so, it is confirmed" while the result is incorrect. At CERN, people from the CMS experiment and people from the Atlas experiment are FORBIDEN to share their code for this reason. Even reading the code can be bad because it can bias the reader. Then, of course, there is a balance and I recommend to share the code with publication, but the reader should be educated enough to know that the code is the last resort to look at if they have a question when trying to reproduce the result. Third, the way scientists work and collaborate creates to some extend "intrinsic" code review and unit testing. Several teams build software in parallel to check the same thing, and will investigate by comparing their result. And scientists are the first user of their software, intensively: they will never build a feature that they will not themselves use in practice later. So, some of the good practices of software dev are just not well designed for these situations. At the end, you are right: serving the food on the floor is bad. The software developer "good practice" are serving the food on the floor because they assumed the dish was where it is not. Finally, one thing to keep in mind is that some software dev people are just full of themselves with big ego. They love to think they are smarter than everyone else. If they are applying good practices themselves, they love feeling superior by thinking the ones not applying them are inferior and that they are so smart. It's a way of rationalising their traditions, and it's then easy to see plenty of ways that confirm this belief. Don't get me wrong, plenty of good practices are really useful, and plenty of scientific code is just badly done. But you just need to have a balance: instead of blindly applying tradition just because it flatter their ego, people should just take a step back and think of why X and Y is better in some context and see how it translates in another context.