5 ms·

No one is saying that code is the science. If I'm given bad information and I act on that information, then problems can occur. Similarly, if the software is

No one is saying that code is the science. If I'm given bad information and I act on that information, then problems can occur. Similarly, if the software is giving the scientist bad information, problems can occur. How many more stories do we have to read about some research getting published in a journal only to have to retract it down the road because they had a bug in the software before we start asking if maybe there needs to be more rigor in the software portion of the research as well? There was a story on HN a while back about a professor who had written software, had come to some conclusions, and even had a Ph.D. student working on research based on that work. Only to find out that a software flaw meant the conclusions weren't useful to anyone and that student ended up wasting years of their life. --- This stuff matters. This isn't a model of reality, it's an exploration of reality. It would be like telling a hiker that terrain doesn't matter. They would, rightfully, disagree with you.

kalenx 6y ago
> How many more stories do we have to read about some research getting published in a journal only to have to retract it down the road because they had a bug in the software before we start asking if maybe there needs to be more rigor in the software We will always hear stories like that, as we will always hear stories about major bugs in stable software releases. Asking a scientist to do better than whole teams of software engineers makes little sense to me. Of course, a bug that was introduced or kept with the counscious intention of fooling the reviewers and the readers is another story.
- RandoHolmes 6y ago
  > Asking a scientist to do better than whole teams of software engineers makes little sense to me. This is not what is being asked, shame on you for the strawman. Your entire post can be summed up with the following sentence: "if we can't be perfect then we may as well not try to be better".
  - kalenx 6y ago
    I was reacting to the part of your post I quoted. The thing is that it has little to do about rigor -- or if I may sin again, it is equivalent to say that software developers lack rigor: sure, some of them do (as some scientists do), but even among the most significant and severe bugs of the history of software, it is seldom the case that we can tell "right, definitely the guy who wrote that lacked rigor and seriousness". Of course this is not a blank forgiveness for every bad scientist out there. Of course we should aim at getting better. But we should make the difference between the ideal science process and the science as performed by a human, prone to errors, misunderstandings and mistakes, and realize that these things will always happen, however many times we call for "more rigor because bugs have consequences".
    - RandoHolmes 6y ago
      All you did was restate the argument that I've already rejected. And stop comparing scientists to software developers, it's a hidden argument by authority, and it isn't needed.
      - kalenx 6y ago
        I don't even understand the point you are making then (apart from me apparently arguing solely with sophisms, which is kind of a prowess). How this "rigor" you are calling for should manifest, then? Put bluntly, my point was that every software has bug, so how "more rigor" would help? What should we do, what should we ask for in _practical_ terms? Also, please do not rephrase this last sentence as "oh so since every software has bugs, then you obviously say that we shouldn't fix bugs, anyway other bugs will remain!".
        RandoHolmes 6y ago
        > Also, please do not rephrase this last sentence as "oh so since every software has bugs, then you obviously say that we shouldn't fix bugs, anyway other bugs will remain!". That's exactly what I'm going to do. Point out that we can demand better even in the face of a lack of perfection. There are two problems here with your stance. 1. The assumption that all bugs are created equal, and 2. The assumption that the truth isn't the overriding concern of science. It's real easy to define the set of bugs that are unacceptable in science. Any bug that would render the results inaccurate is unacceptable. The fact that some jackass web developer wrote a bug that deleted an entire database in no way obviates that responsibility of the scientists.
ufmace 6y ago
I don't entirely disagree, but haven't there also been cases of experimental results being invalidated due to subtle mechanical, electrical, chemical, etc complications with the test equipment, when none of the people involved in the experiment were experts in those fields? I think that, while we could use a bit more training in software engineering best-practices in the science, the thesis is still that science is hard and we need real replication of everything before reaching important conclusions, and over-focusing on one specific type of errors isn't all that helpful.
- RandoHolmes 6y ago
  If they're setting up experiments whose correct results require electrical expertise, then yes, they should either get better training or bring in someone who has it. It's not clear to me why you think I would argue that inaccuracies should be avoided in software but accept that they're ok for electrical systems.

kalenx 6y ago

> How many more stories do we have to read about some research getting published in a journal only to have to retract it down the road because they had a bug in the software before we start asking if maybe there needs to be more rigor in the software We will always hear stories like that, as we will always hear stories about major bugs in stable software releases. Asking a scientist to do better than whole teams of software engineers makes little sense to me. Of course, a bug that was introduced or kept with the counscious intention of fooling the reviewers and the readers is another story.

RandoHolmes 6y ago
> Asking a scientist to do better than whole teams of software engineers makes little sense to me. This is not what is being asked, shame on you for the strawman. Your entire post can be summed up with the following sentence: "if we can't be perfect then we may as well not try to be better".
- kalenx 6y ago
  I was reacting to the part of your post I quoted. The thing is that it has little to do about rigor -- or if I may sin again, it is equivalent to say that software developers lack rigor: sure, some of them do (as some scientists do), but even among the most significant and severe bugs of the history of software, it is seldom the case that we can tell "right, definitely the guy who wrote that lacked rigor and seriousness". Of course this is not a blank forgiveness for every bad scientist out there. Of course we should aim at getting better. But we should make the difference between the ideal science process and the science as performed by a human, prone to errors, misunderstandings and mistakes, and realize that these things will always happen, however many times we call for "more rigor because bugs have consequences".
  - RandoHolmes 6y ago
    All you did was restate the argument that I've already rejected. And stop comparing scientists to software developers, it's a hidden argument by authority, and it isn't needed.
    - kalenx 6y ago
      I don't even understand the point you are making then (apart from me apparently arguing solely with sophisms, which is kind of a prowess). How this "rigor" you are calling for should manifest, then? Put bluntly, my point was that every software has bug, so how "more rigor" would help? What should we do, what should we ask for in _practical_ terms? Also, please do not rephrase this last sentence as "oh so since every software has bugs, then you obviously say that we shouldn't fix bugs, anyway other bugs will remain!".
      - RandoHolmes 6y ago
        > Also, please do not rephrase this last sentence as "oh so since every software has bugs, then you obviously say that we shouldn't fix bugs, anyway other bugs will remain!". That's exactly what I'm going to do. Point out that we can demand better even in the face of a lack of perfection. There are two problems here with your stance. 1. The assumption that all bugs are created equal, and 2. The assumption that the truth isn't the overriding concern of science. It's real easy to define the set of bugs that are unacceptable in science. Any bug that would render the results inaccurate is unacceptable. The fact that some jackass web developer wrote a bug that deleted an entire database in no way obviates that responsibility of the scientists.

ufmace 6y ago

I don't entirely disagree, but haven't there also been cases of experimental results being invalidated due to subtle mechanical, electrical, chemical, etc complications with the test equipment, when none of the people involved in the experiment were experts in those fields? I think that, while we could use a bit more training in software engineering best-practices in the science, the thesis is still that science is hard and we need real replication of everything before reaching important conclusions, and over-focusing on one specific type of errors isn't all that helpful.

RandoHolmes 6y ago
If they're setting up experiments whose correct results require electrical expertise, then yes, they should either get better training or bring in someone who has it. It's not clear to me why you think I would argue that inaccuracies should be avoided in software but accept that they're ok for electrical systems.