3 ms·
both data and analysis software should be available I think that goes too far. The data alone is sufficient for independent verification, I don't think we shou
by tremon 10y ago
both data and analysis software should be available
I think that goes too far. The data alone is sufficient for independent verification, I don't think we should mandate that scientists publish their tools. However, I would very much support scientists using open tools to begin with. But if the tool is closed, that should not invalidate any results in the paper.
- wolfgke 10y ago> But if the tool is closed, that should not invalidate any results in the paper. As I said in https://news.ycombinator.com/item?id=12100008 https://news.ycombinator.com/item?id=12100008: If it cannot be verified independently as far as possible, this does not mean it is wrong, but also should not be considered as science (since independent verification and the ability to do so as far as possible is central to science). Instead it is just a marketing statement with the author providing evidence for truth. Marketing statements are also not wrong per se (if they were, at least in Germany this would be a criminal offense because of the "Gesetz gegen unlauteren Wettbewerb (UWG)" (which according to the internet dictionary corresponds to "Act Against Unfair Competition" or "fair trade law")), but you strongly have to trust its author to tell you the truth, since you can hardly verify the truth independently.
- elcapitan 10y agoNot sure if there is so much to support - they most likely run some archaic Fortran code that only 5 people in the world understand, compiled for some specific cluster, where it runs multiple days for one dataset, and is only numerically stable in that particular environment.
- wolfgke 10y agoThis is why on https://news.ycombinator.com/item?id=12099720 https://news.ycombinator.com/item?id=12099720 I also suggested to provide a virtual machine image if possible (unluckily providing one is in my opinion not always possible). On the other hand if another team ran the code independently and we really saw that it is numerically unstable in another environment, this would be, in my opinion, a strong case for the importance of providing the source code.
- dalke 10y agoI don't think you understand what elcapitan means by numerically stable. Numeric stability is a function of the environment. If I take an algorithm designed for 128 bit IEEE binary floats and implement it on a system with 64 bit floats, then it may be unstable. That doesn't say anything about the correctness of the original method for the original environment. It only means the method wasn't designed for the new environment. You are used to a world where computers are a commodity, likely based around the Intel architecture, and almost certainly using IEEE floats. What do you do with a computer program written for Anton, a specialized computer for doing molecular dynamics built using specialized ASICs? https://en.wikipedia.org/wiki/Anton_(computer) https://en.wikipedia.org/wiki/Anton_(computer)
- wolfgke 10y agoFirst: IEEE 754 defines exactly (bit per bit) what the result of any floating point operation in each of the five rounding modes as long as no NaN is produced (but in this case only not all bits are defined exactly, but it will still be a NaN). Unluckily C allows some optimizations to be done to code containing floating point code that violates this principle of bit-for-bit reproducability of floating point code. If this causes problems the code can be considered as numerically unstable. In other words: Writing code that uses the IEE 754 defined "gold standard" in its code should be the goal scientific results. The correct way for independent checking, if we really need 128 bit floating point numbers, is to wrap these operations by a software floating point library. > It only means the method wasn't designed for the new environment. If this property (what kind of environment the algorithm depends on) of the algorithm is not documented properly (and an explanation is given why we need this property (say: size of floating points, in particular unusual floating point sizes as 128 bit) for the algorithm to work correctly) any independent reviewer should better not assume these properties to hold. If we indeed find out that this causes problems, this is a strong sign to me that there might be subtile errors in the code. In other words: We should be really careful of the results that the algorithm gave. > What do you do with a computer program written for Anton, a specialized computer for doing molecular dynamics built using specialized ASICs? I gave the answer in the post above: "[U]nluckily providing [a virtual machine] is in my opinion not always possible".