4 ms·

While I certainly share your sentiment that both, data and analysis software, should be available, I do see a conflict of interest here that cannot be easily re

by DasBlob 10y ago

While I certainly share your sentiment that both, data and analysis software, should be available, I do see a conflict of interest here that cannot be easily resolved. On the one hand, we do assert (wrongly, IMO) that competition is a necessary element to drive scientific innovation and consequently require scientific actors to adhere to economic principles, i.e. publish more and better-received papers. On the other hand, allowing researchers to keep their competitive advantage, and having exclusive access to research data certainly is, contradicts the principle of reproducability and unneccessarily restricts further research that could be done based on that data. I don't see a way to easily resolve that in the current scientific framework. One (less than optimal) approach, that is frequently taken in astronomy, is to allow exclusive access to telescope data to the members of the consortium or the principal investigators for a certain amount of time, after which it is released to public. As for the software, it's easier. if the analysis method is sufficiently well documented (and it should be in any publication), people with access to the original data can easily attempt to reproduce the results and either way they'll get a publication out of it. So there is no incentive to keep the analysis software but there is a huge incentive to keep the original data.

wolfgke 10y ago
The solution should rather be that society (and publishers) does not consider papers (or more generally: texts) as research that don't also publish everything possible for independent verification (in this case the source code and raw data). If this is not the case, the text is just a plain marketing statement (in contrast to research) comparable to the marketing statements that are made everyday by lots of companies.
- dalke 10y ago
  The lack of source code does not make everything produced by software "a marketing statement." There's more than one way to verify a conclusion. For example, if I say that RSA-130 = 39685999459597454290161126162883786067576449112810064832555157243 × 45534498646735972188403686897274408864356301263205069600999044599 then you can easily verify it by multiplication - you don't need access to the factorization software I used to determine that result. If I say that the energy difference between two chemical structure according to the Merck Molecular Force Field (MMFF) is X, then you can use any software which implements the MMFF to verify my statement using the raw data - you don't need the source code that I used. In fact, it would be better to use an independent implementation to verify my result than it would be to re-run the same code. And there are many programs which implement the MMFF. If you are used to RDKit, then it will likely be faster to use RDKit to reproduce and verify my numbers than trying to figure out the strange VM I provided, which is based around some home-brew code of mine implemented in MORTRAN.
  - wolfgke 10y ago
    > For example, if I say that RSA-130 = ... × ... then you can easily verify it by multiplication - you don't need access to the factorization software I used to determine that result. I didn't write about independent computation, but about independent verification. In your case you need access to a software for multiplying numbers for an independent check. OK, this is rather trivial code to write. But if it weren't that trivial the author would have to provide it.
    - dalke 10y ago
      I gave the RSA example because it's better known and easier to understand than a molecular modeling example. But I question the meaning of your phrase "independent verification". If the author has to provide the program used for verification, how is that "independent"? At the very least, isn't there a range of dependencies, from "running the same code in an identical environment produced the same answer" to "running a different code in a different environment produced the same answer"? Which verification do you trust more? What level of independence do you need before you can trust the results? Let me give you a few other examples. "We docked 500,000 small molecule structures from CHEMBL against the target using 6,000 GPU days. We took the 1,000 best fits and measured binding affinity. The top 10 structures are shown in table 1. The third is our lead candidate, which reduces the chance of infection in rats from 80% to 10%." Do you need to be able to validate the docking results in order to accept this paper as science? "We used ProteinFitter 4.5 in simulated annealing mode to generate the best protein fold to the given constraints. The resulting structure has a R-factor of 2.3Å against the Hodgkin electron density, which is better than any other published model." Do you need access to ProteinFitter 4.5 to verify this paper? Since all crystallography tools will generate an R-factor for you, why is it critical for the model provider to also supply an environment which contains a tool which does that? Both are examples where it doesn't matter what's in the magic box, or if the magic box is buggy. It could as well be "I had a dream about a snake tying itself into a knot and eating its own tail. I woke up and realized the structure might be a trefoil. It fits all of the known data and makes intuitive sense and hasn't been considered before, so I'm publishing in the hopes that others can explore it more rigorously." Is that science? I say "yes". Now, replace "dream" with "computer program" - is it still science? I still say "yes".
tremon 10y ago
both data and analysis software should be available I think that goes too far. The data alone is sufficient for independent verification, I don't think we should mandate that scientists publish their tools. However, I would very much support scientists using open tools to begin with. But if the tool is closed, that should not invalidate any results in the paper.
- wolfgke 10y ago
  > But if the tool is closed, that should not invalidate any results in the paper. As I said in https://news.ycombinator.com/item?id=12100008 https://news.ycombinator.com/item?id=12100008: If it cannot be verified independently as far as possible, this does not mean it is wrong, but also should not be considered as science (since independent verification and the ability to do so as far as possible is central to science). Instead it is just a marketing statement with the author providing evidence for truth. Marketing statements are also not wrong per se (if they were, at least in Germany this would be a criminal offense because of the "Gesetz gegen unlauteren Wettbewerb (UWG)" (which according to the internet dictionary corresponds to "Act Against Unfair Competition" or "fair trade law")), but you strongly have to trust its author to tell you the truth, since you can hardly verify the truth independently.
- elcapitan 10y ago
  Not sure if there is so much to support - they most likely run some archaic Fortran code that only 5 people in the world understand, compiled for some specific cluster, where it runs multiple days for one dataset, and is only numerically stable in that particular environment.
  - wolfgke 10y ago
    This is why on https://news.ycombinator.com/item?id=12099720 https://news.ycombinator.com/item?id=12099720 I also suggested to provide a virtual machine image if possible (unluckily providing one is in my opinion not always possible). On the other hand if another team ran the code independently and we really saw that it is numerically unstable in another environment, this would be, in my opinion, a strong case for the importance of providing the source code.
ajford 10y ago
I agree on the propriety of data for a limited time. I worked in the field of Radio Astronomy, and it was common to have a two year period where the data was private to the Primary Investigator (and co-investigators/consortium). I think this is a relatively acceptable approach, given how long it can take to process and fact-check data and papers. I'm also for the publishing and tracking of code, as in many fields the data is becoming so vast that it's impossible to process or handle manually, and many pipelines and large scale systems are being built to handle this data. This inevitably means that a simple bug in code (or false/incorrect assumption by the author) can result in many lost hours and years of work. But publishing code can also lead to biases, if there is an unintentional bias in released code or approaches that become popular within a given field. In the field I worked in, there were two very common pipelines used by a huge majority of researchers, and I always wondered if there might be some unintentional selection effect being introduced. Well, it turns out someone else thought so too, so there's now an effort to rewrite one of these tools from the ground up without using any of the original code. It's only something like 60% complete right now, but hopefully soon it will be complete and we'll get to see if results compare, or if we spot any unintentional effects.

by DasBlob 10y ago

wolfgke 10y ago

The solution should rather be that society (and publishers) does not consider papers (or more generally: texts) as research that don't also publish everything possible for independent verification (in this case the source code and raw data). If this is not the case, the text is just a plain marketing statement (in contrast to research) comparable to the marketing statements that are made everyday by lots of companies.

dalke 10y ago
The lack of source code does not make everything produced by software "a marketing statement." There's more than one way to verify a conclusion. For example, if I say that RSA-130 = 39685999459597454290161126162883786067576449112810064832555157243 × 45534498646735972188403686897274408864356301263205069600999044599 then you can easily verify it by multiplication - you don't need access to the factorization software I used to determine that result. If I say that the energy difference between two chemical structure according to the Merck Molecular Force Field (MMFF) is X, then you can use any software which implements the MMFF to verify my statement using the raw data - you don't need the source code that I used. In fact, it would be better to use an independent implementation to verify my result than it would be to re-run the same code. And there are many programs which implement the MMFF. If you are used to RDKit, then it will likely be faster to use RDKit to reproduce and verify my numbers than trying to figure out the strange VM I provided, which is based around some home-brew code of mine implemented in MORTRAN.
- wolfgke 10y ago
  > For example, if I say that RSA-130 = ... × ... then you can easily verify it by multiplication - you don't need access to the factorization software I used to determine that result. I didn't write about independent computation, but about independent verification. In your case you need access to a software for multiplying numbers for an independent check. OK, this is rather trivial code to write. But if it weren't that trivial the author would have to provide it.
  - dalke 10y ago
    I gave the RSA example because it's better known and easier to understand than a molecular modeling example. But I question the meaning of your phrase "independent verification". If the author has to provide the program used for verification, how is that "independent"? At the very least, isn't there a range of dependencies, from "running the same code in an identical environment produced the same answer" to "running a different code in a different environment produced the same answer"? Which verification do you trust more? What level of independence do you need before you can trust the results? Let me give you a few other examples. "We docked 500,000 small molecule structures from CHEMBL against the target using 6,000 GPU days. We took the 1,000 best fits and measured binding affinity. The top 10 structures are shown in table 1. The third is our lead candidate, which reduces the chance of infection in rats from 80% to 10%." Do you need to be able to validate the docking results in order to accept this paper as science? "We used ProteinFitter 4.5 in simulated annealing mode to generate the best protein fold to the given constraints. The resulting structure has a R-factor of 2.3Å against the Hodgkin electron density, which is better than any other published model." Do you need access to ProteinFitter 4.5 to verify this paper? Since all crystallography tools will generate an R-factor for you, why is it critical for the model provider to also supply an environment which contains a tool which does that? Both are examples where it doesn't matter what's in the magic box, or if the magic box is buggy. It could as well be "I had a dream about a snake tying itself into a knot and eating its own tail. I woke up and realized the structure might be a trefoil. It fits all of the known data and makes intuitive sense and hasn't been considered before, so I'm publishing in the hopes that others can explore it more rigorously." Is that science? I say "yes". Now, replace "dream" with "computer program" - is it still science? I still say "yes".

tremon 10y ago

both data and analysis software should be available I think that goes too far. The data alone is sufficient for independent verification, I don't think we should mandate that scientists publish their tools. However, I would very much support scientists using open tools to begin with. But if the tool is closed, that should not invalidate any results in the paper.

wolfgke 10y ago
> But if the tool is closed, that should not invalidate any results in the paper. As I said in https://news.ycombinator.com/item?id=12100008 https://news.ycombinator.com/item?id=12100008: If it cannot be verified independently as far as possible, this does not mean it is wrong, but also should not be considered as science (since independent verification and the ability to do so as far as possible is central to science). Instead it is just a marketing statement with the author providing evidence for truth. Marketing statements are also not wrong per se (if they were, at least in Germany this would be a criminal offense because of the "Gesetz gegen unlauteren Wettbewerb (UWG)" (which according to the internet dictionary corresponds to "Act Against Unfair Competition" or "fair trade law")), but you strongly have to trust its author to tell you the truth, since you can hardly verify the truth independently.
elcapitan 10y ago
Not sure if there is so much to support - they most likely run some archaic Fortran code that only 5 people in the world understand, compiled for some specific cluster, where it runs multiple days for one dataset, and is only numerically stable in that particular environment.
- wolfgke 10y ago
  This is why on https://news.ycombinator.com/item?id=12099720 https://news.ycombinator.com/item?id=12099720 I also suggested to provide a virtual machine image if possible (unluckily providing one is in my opinion not always possible). On the other hand if another team ran the code independently and we really saw that it is numerically unstable in another environment, this would be, in my opinion, a strong case for the importance of providing the source code.

ajford 10y ago

I agree on the propriety of data for a limited time. I worked in the field of Radio Astronomy, and it was common to have a two year period where the data was private to the Primary Investigator (and co-investigators/consortium). I think this is a relatively acceptable approach, given how long it can take to process and fact-check data and papers. I'm also for the publishing and tracking of code, as in many fields the data is becoming so vast that it's impossible to process or handle manually, and many pipelines and large scale systems are being built to handle this data. This inevitably means that a simple bug in code (or false/incorrect assumption by the author) can result in many lost hours and years of work. But publishing code can also lead to biases, if there is an unintentional bias in released code or approaches that become popular within a given field. In the field I worked in, there were two very common pipelines used by a huge majority of researchers, and I always wondered if there might be some unintentional selection effect being introduced. Well, it turns out someone else thought so too, so there's now an effort to rewrite one of these tools from the ground up without using any of the original code. It's only something like 60% complete right now, but hopefully soon it will be complete and we'll get to see if results compare, or if we spot any unintentional effects.