4 ms·

In GIS, there's a saying "the map is not the terrain". It seems like HN is in a little SWE bubble, and needs to understand "the code is not the science". In s

In GIS, there's a saying "the map is not the terrain". It seems like HN is in a little SWE bubble, and needs to understand "the code is not the science". In science, code is not an end in-and-of-itself. It is a tool for simulation, data reduction, calculation, etc. It is a way to test scientific ideas. > how do you expect anyone with the right expertise to assess your findings I would expect other experts in the field to write their own implementation of the scientific ideas expressed in a paper. If the idea has any merit, their implementations should produce similar results. Which is exactly what they would do if it were a physical experiment.

yjftsjthsd-h 6y ago
> In GIS, there's a saying "the map is not the terrain". It seems like HN is in a little SWE bubble, and needs to understand "the code is not the science". And if you're a map maker, it's a bit rich to start claiming that the accuracy of your maps is unimportant. If code is "a way to test scientific ideas", then it kinda needs to work if you want meaningful results. Would you run an experiment with thermometers that were accurate to +-30° and reactants from a source known for contamination?
- jnxx 6y ago
  In many parts of scientific research, researchers are, to stay in your metaphor, more travelers using a map, than map makers. Of course, it is a difference whether you make a clinical study on drugs, and use a pocket calculator to compute a mean, or whether you research in numerical analysis, or are presenting a paper in how to use Coq to more efficiently prove the four-color theorem or Fermat's last theorem. In short, much of science is not computer science, and for it, computation is just a tool.
RandoHolmes 6y ago
No one is saying that code is the science. If I'm given bad information and I act on that information, then problems can occur. Similarly, if the software is giving the scientist bad information, problems can occur. How many more stories do we have to read about some research getting published in a journal only to have to retract it down the road because they had a bug in the software before we start asking if maybe there needs to be more rigor in the software portion of the research as well? There was a story on HN a while back about a professor who had written software, had come to some conclusions, and even had a Ph.D. student working on research based on that work. Only to find out that a software flaw meant the conclusions weren't useful to anyone and that student ended up wasting years of their life. --- This stuff matters. This isn't a model of reality, it's an exploration of reality. It would be like telling a hiker that terrain doesn't matter. They would, rightfully, disagree with you.
- kalenx 6y ago
  > How many more stories do we have to read about some research getting published in a journal only to have to retract it down the road because they had a bug in the software before we start asking if maybe there needs to be more rigor in the software We will always hear stories like that, as we will always hear stories about major bugs in stable software releases. Asking a scientist to do better than whole teams of software engineers makes little sense to me. Of course, a bug that was introduced or kept with the counscious intention of fooling the reviewers and the readers is another story.
  - RandoHolmes 6y ago
    > Asking a scientist to do better than whole teams of software engineers makes little sense to me. This is not what is being asked, shame on you for the strawman. Your entire post can be summed up with the following sentence: "if we can't be perfect then we may as well not try to be better".
    - kalenx 6y ago
      I was reacting to the part of your post I quoted. The thing is that it has little to do about rigor -- or if I may sin again, it is equivalent to say that software developers lack rigor: sure, some of them do (as some scientists do), but even among the most significant and severe bugs of the history of software, it is seldom the case that we can tell "right, definitely the guy who wrote that lacked rigor and seriousness". Of course this is not a blank forgiveness for every bad scientist out there. Of course we should aim at getting better. But we should make the difference between the ideal science process and the science as performed by a human, prone to errors, misunderstandings and mistakes, and realize that these things will always happen, however many times we call for "more rigor because bugs have consequences".
      - RandoHolmes 6y ago
        All you did was restate the argument that I've already rejected. And stop comparing scientists to software developers, it's a hidden argument by authority, and it isn't needed.
booleandilemma 6y ago
If you’re saying you produced certain results with code, then the code is indeed the science. Not being able to vouch for the code is like believing a mathematical theorem without seeing the proof.
kazagistar 6y ago
Mathematicians are expected to publish their proofs. Not so that people can do the proof again independently, but so that other mathematicians can find and point out if they have a tangible error in their proof that tangibly invalidates the result. Sure, some people might point out spurious bugs and "design issues" or whatever, boo hoo. But others might actually find flaws in the code that meaningfully affect science itself: true bugs. Sure, they could do this by doing a full replication in a lab and then custom coding everything from scratch. But even then, all you have is two conflicting results, with no good way yet to determine which one is more right or why they disagree. Technically, you can use the scientific progress to eventually find bugs in the scientific process, but why waste so much time when publishing the code will allow for reviews to find bugs so much faster. Its pure benefit to science to not obscure its proofs and rigor.
avasthe 6y ago
You are missing the point. How many actually try to reproduce the results by writing corresponding code themselves? Apparently lot of papers with slightly wrong findings because code errors have passed the peer review (all of us in the SWE bubble know how often bugs occur), at least in less prestigious journals. There is nothing wrong with mandating the code to be supplied with the paper. Because, many time code is somewhere between the experimental setup and proof / result.

yjftsjthsd-h 6y ago

> In GIS, there's a saying "the map is not the terrain". It seems like HN is in a little SWE bubble, and needs to understand "the code is not the science". And if you're a map maker, it's a bit rich to start claiming that the accuracy of your maps is unimportant. If code is "a way to test scientific ideas", then it kinda needs to work if you want meaningful results. Would you run an experiment with thermometers that were accurate to +-30° and reactants from a source known for contamination?

jnxx 6y ago
In many parts of scientific research, researchers are, to stay in your metaphor, more travelers using a map, than map makers. Of course, it is a difference whether you make a clinical study on drugs, and use a pocket calculator to compute a mean, or whether you research in numerical analysis, or are presenting a paper in how to use Coq to more efficiently prove the four-color theorem or Fermat's last theorem. In short, much of science is not computer science, and for it, computation is just a tool.

RandoHolmes 6y ago

No one is saying that code is the science. If I'm given bad information and I act on that information, then problems can occur. Similarly, if the software is giving the scientist bad information, problems can occur. How many more stories do we have to read about some research getting published in a journal only to have to retract it down the road because they had a bug in the software before we start asking if maybe there needs to be more rigor in the software portion of the research as well? There was a story on HN a while back about a professor who had written software, had come to some conclusions, and even had a Ph.D. student working on research based on that work. Only to find out that a software flaw meant the conclusions weren't useful to anyone and that student ended up wasting years of their life. --- This stuff matters. This isn't a model of reality, it's an exploration of reality. It would be like telling a hiker that terrain doesn't matter. They would, rightfully, disagree with you.

kalenx 6y ago
> How many more stories do we have to read about some research getting published in a journal only to have to retract it down the road because they had a bug in the software before we start asking if maybe there needs to be more rigor in the software We will always hear stories like that, as we will always hear stories about major bugs in stable software releases. Asking a scientist to do better than whole teams of software engineers makes little sense to me. Of course, a bug that was introduced or kept with the counscious intention of fooling the reviewers and the readers is another story.
- RandoHolmes 6y ago
  > Asking a scientist to do better than whole teams of software engineers makes little sense to me. This is not what is being asked, shame on you for the strawman. Your entire post can be summed up with the following sentence: "if we can't be perfect then we may as well not try to be better".
  - kalenx 6y ago
    I was reacting to the part of your post I quoted. The thing is that it has little to do about rigor -- or if I may sin again, it is equivalent to say that software developers lack rigor: sure, some of them do (as some scientists do), but even among the most significant and severe bugs of the history of software, it is seldom the case that we can tell "right, definitely the guy who wrote that lacked rigor and seriousness". Of course this is not a blank forgiveness for every bad scientist out there. Of course we should aim at getting better. But we should make the difference between the ideal science process and the science as performed by a human, prone to errors, misunderstandings and mistakes, and realize that these things will always happen, however many times we call for "more rigor because bugs have consequences".
    - RandoHolmes 6y ago
      All you did was restate the argument that I've already rejected. And stop comparing scientists to software developers, it's a hidden argument by authority, and it isn't needed.

booleandilemma 6y ago

If you’re saying you produced certain results with code, then the code is indeed the science. Not being able to vouch for the code is like believing a mathematical theorem without seeing the proof.

kazagistar 6y ago

Mathematicians are expected to publish their proofs. Not so that people can do the proof again independently, but so that other mathematicians can find and point out if they have a tangible error in their proof that tangibly invalidates the result. Sure, some people might point out spurious bugs and "design issues" or whatever, boo hoo. But others might actually find flaws in the code that meaningfully affect science itself: true bugs. Sure, they could do this by doing a full replication in a lab and then custom coding everything from scratch. But even then, all you have is two conflicting results, with no good way yet to determine which one is more right or why they disagree. Technically, you can use the scientific progress to eventually find bugs in the scientific process, but why waste so much time when publishing the code will allow for reviews to find bugs so much faster. Its pure benefit to science to not obscure its proofs and rigor.

avasthe 6y ago

You are missing the point. How many actually try to reproduce the results by writing corresponding code themselves? Apparently lot of papers with slightly wrong findings because code errors have passed the peer review (all of us in the SWE bubble know how often bugs occur), at least in less prestigious journals. There is nothing wrong with mandating the code to be supplied with the paper. Because, many time code is somewhere between the experimental setup and proof / result.