3 ms·

> It looks like you don't understand the definition and the point of reproducibility. If you just click on a button to recompute the results from the same code,

> It looks like you don't understand the definition and the point of reproducibility. If you just click on a button to recompute the results from the same code, this is not "reproducibility". The point of reproducibility is to have other scientists able to confirm your results in a fully independent way. They should be able to do it from scratch, without your code, and reach the same conclusion. I highly disagree with that. In machine learning for instance, it's very important to be able to check authors' claims because otherwise you can't know if they are lying or not, or if they cheated or not ("what do you mean we can't use our test data to improve our training process?") Plus, you can't improve on what others did if they don't let you access their own code and dataset. "We got better results than previous works it seems, but we can't know for sure because maybe that's because our method is better, maybe that's because our code is better, we will never know." My ML approach is more efficient than yours because you wrote it in python on your laptop and I wrote it in C and spent a lot of time to optimize it, plus I have a faster computer. Yeah sure. That's not science.

cauch 4y ago
What you are talking about is important, but it is not what corresponds to the notion of "reproducibility in science". Ideally, what we need is being able to 1) reproduce from scratch, 2) rerun the experiment with the code used (which has a lot of pragmatical advantages too). In fact, I would argue that even the examples you are giving are not "reproducibility". Reproducibility means that if you redo it _from scratch_, you reach the same conclusion. If the conclusion is "this algorithm describe in this paper is 20+-10% better than the algorithm described in this other paper, in those conditions", then, ideally, what you need to do is to read the first paper and follow the explanation to recreate the algorithm, read the second paper and follow the explanation to recreate the algorithm, read the conditions and reproduce them yourself, and then, you check if you get a compatible conclusion. As for improving on top of others, again, the scientific paper explains the algorithm, just apply the recipe in your personal implementation, and you will have the same result. Even better: if you don't, you may have found a bug that exists in their implementation, and you have made a scientific progress that you would not have obtained if you would have just cut-and-pasted their code. But as I've said, the ideal is to have both. Still, the sentence that we reacted on as "not scientific because not compatible with reproducibility" is still valid. It is true that, in practice, the code is not very useful in science (it may be a bit useful, but not as crucial as some may think), and it is true that this does not lead to a reproducibility problem, because the goal of the concept of reproducibility is to guarantee the existence of _independent_ implementations to check if we reach the same conclusion.
- sacado2 4y ago
  Code is little more than a very formal way to describe an algorithm, the difference between a detailed algorithm and a piece of code is very small in many cases. And in the cases where the given algorithm is general and no code is given, reproducibility is at stakes. But ideally you want both: the description of the procedure to rewrite an equivalent program, and the actual code that gave the results detailed in the article. This is happening more and more frequently, and this is a good thing.
  - cauch 4y ago
    > Code is little more than a very formal way to describe an algorithm, the difference between a detailed algorithm and a piece of code is very small in many cases. And in the cases where the given algorithm is general and no code is given, reproducibility is at stakes. If you follow discussions about how to comment code properly, you will see that an important thing to understand to comment code properly is the fact that your comment should not describe the code, but explain the code. In science, the explanation is a very very very important part of what the community wants. The fact that this distinction exists for comments is the proof that "just the code" is not enough. Then, yes, of course, it does not hurt to give the code. I have NEVER said otherwise. But just re-read the discussion: my point is that, in practice and in reality, the EXPLANATION is way more important than the description in scientific research, and, because of that, the scientific community is way more interested in discussing the explanation rather than the description (not that it is 100% useless, the same way that discussing plenty of other things is not 100% useless). > ... reproducibility is at stakes. True. But in this case, trying to solve that by giving the code is the worst way of solving reproducibility. If the EXPLANATION is not well done, than the science is not there. If you can't reproduce because you don't have a good explanation, than, the solution is to fix the explanation, not to give an unexplained code. Describing the code will NOT fix that, but will let bad science being used ("well, I have the code. I don't know exactly why they have done like that, but I can run the code, so let's use it") > But ideally you want both: the description of the procedure to rewrite an equivalent program, and the actual code that gave the results detailed in the article. This is happening more and more frequently, and this is a good thing. No one so far in this discussion disagrees with that. If I was not trying to strongly give the benefit of the doubt, I would be tempted to conclude from this discussion that this disproportionate reaction to just "they are not interested in the code" is from few persons who are totally oblivious of how science works, but are somehow triggered by the idea that the code is not the most important thing in the scientists eyes. It just comforts me in the idea that some computer scientists have no idea of what are the tensions when doing science (some computer scientists, not all) and just refuse even to consider that maybe it is simply slightly different.

cauch 4y ago

What you are talking about is important, but it is not what corresponds to the notion of "reproducibility in science". Ideally, what we need is being able to 1) reproduce from scratch, 2) rerun the experiment with the code used (which has a lot of pragmatical advantages too). In fact, I would argue that even the examples you are giving are not "reproducibility". Reproducibility means that if you redo it _from scratch_, you reach the same conclusion. If the conclusion is "this algorithm describe in this paper is 20+-10% better than the algorithm described in this other paper, in those conditions", then, ideally, what you need to do is to read the first paper and follow the explanation to recreate the algorithm, read the second paper and follow the explanation to recreate the algorithm, read the conditions and reproduce them yourself, and then, you check if you get a compatible conclusion. As for improving on top of others, again, the scientific paper explains the algorithm, just apply the recipe in your personal implementation, and you will have the same result. Even better: if you don't, you may have found a bug that exists in their implementation, and you have made a scientific progress that you would not have obtained if you would have just cut-and-pasted their code. But as I've said, the ideal is to have both. Still, the sentence that we reacted on as "not scientific because not compatible with reproducibility" is still valid. It is true that, in practice, the code is not very useful in science (it may be a bit useful, but not as crucial as some may think), and it is true that this does not lead to a reproducibility problem, because the goal of the concept of reproducibility is to guarantee the existence of _independent_ implementations to check if we reach the same conclusion.

sacado2 4y ago
Code is little more than a very formal way to describe an algorithm, the difference between a detailed algorithm and a piece of code is very small in many cases. And in the cases where the given algorithm is general and no code is given, reproducibility is at stakes. But ideally you want both: the description of the procedure to rewrite an equivalent program, and the actual code that gave the results detailed in the article. This is happening more and more frequently, and this is a good thing.
- cauch 4y ago
  > Code is little more than a very formal way to describe an algorithm, the difference between a detailed algorithm and a piece of code is very small in many cases. And in the cases where the given algorithm is general and no code is given, reproducibility is at stakes. If you follow discussions about how to comment code properly, you will see that an important thing to understand to comment code properly is the fact that your comment should not describe the code, but explain the code. In science, the explanation is a very very very important part of what the community wants. The fact that this distinction exists for comments is the proof that "just the code" is not enough. Then, yes, of course, it does not hurt to give the code. I have NEVER said otherwise. But just re-read the discussion: my point is that, in practice and in reality, the EXPLANATION is way more important than the description in scientific research, and, because of that, the scientific community is way more interested in discussing the explanation rather than the description (not that it is 100% useless, the same way that discussing plenty of other things is not 100% useless). > ... reproducibility is at stakes. True. But in this case, trying to solve that by giving the code is the worst way of solving reproducibility. If the EXPLANATION is not well done, than the science is not there. If you can't reproduce because you don't have a good explanation, than, the solution is to fix the explanation, not to give an unexplained code. Describing the code will NOT fix that, but will let bad science being used ("well, I have the code. I don't know exactly why they have done like that, but I can run the code, so let's use it") > But ideally you want both: the description of the procedure to rewrite an equivalent program, and the actual code that gave the results detailed in the article. This is happening more and more frequently, and this is a good thing. No one so far in this discussion disagrees with that. If I was not trying to strongly give the benefit of the doubt, I would be tempted to conclude from this discussion that this disproportionate reaction to just "they are not interested in the code" is from few persons who are totally oblivious of how science works, but are somehow triggered by the idea that the code is not the most important thing in the scientists eyes. It just comforts me in the idea that some computer scientists have no idea of what are the tensions when doing science (some computer scientists, not all) and just refuse even to consider that maybe it is simply slightly different.