4 ms·

I see a large problem in the fact that (as it seems) the original data and the computer code is not published, too. This is in my opinion a contradiction to the

by wolfgke 10y ago

I see a large problem in the fact that (as it seems) the original data and the computer code is not published, too. This is in my opinion a contradiction to the scientific principle that any result or conclusion has to be able to verified independently as much as possible. Though not publishing data and computer programs (and ideally a virtual machine for independent execution of the program code, too) is unluckily accepted practise in science. If it were otherwise any critic of the result could reapply the computer program to the non-detrended data and see that the conclusion will not change. On the other hand, if the computer program and data is not published he has to trust the conclusions that the researchers came to. So the only way for an independent person to check the result is to find indirect evidence that the conclusion could be wrong. And finding out that a different data set than (wrongly) claimed was used is exactly such indirect evidence (and in my opinion nearly as far as one is able to if one doesn't have access to the used data and computer programs).

DasBlob 10y ago
While I certainly share your sentiment that both, data and analysis software, should be available, I do see a conflict of interest here that cannot be easily resolved. On the one hand, we do assert (wrongly, IMO) that competition is a necessary element to drive scientific innovation and consequently require scientific actors to adhere to economic principles, i.e. publish more and better-received papers. On the other hand, allowing researchers to keep their competitive advantage, and having exclusive access to research data certainly is, contradicts the principle of reproducability and unneccessarily restricts further research that could be done based on that data. I don't see a way to easily resolve that in the current scientific framework. One (less than optimal) approach, that is frequently taken in astronomy, is to allow exclusive access to telescope data to the members of the consortium or the principal investigators for a certain amount of time, after which it is released to public. As for the software, it's easier. if the analysis method is sufficiently well documented (and it should be in any publication), people with access to the original data can easily attempt to reproduce the results and either way they'll get a publication out of it. So there is no incentive to keep the analysis software but there is a huge incentive to keep the original data.
- wolfgke 10y ago
  The solution should rather be that society (and publishers) does not consider papers (or more generally: texts) as research that don't also publish everything possible for independent verification (in this case the source code and raw data). If this is not the case, the text is just a plain marketing statement (in contrast to research) comparable to the marketing statements that are made everyday by lots of companies.
  - dalke 10y ago
    The lack of source code does not make everything produced by software "a marketing statement." There's more than one way to verify a conclusion. For example, if I say that RSA-130 = 39685999459597454290161126162883786067576449112810064832555157243 × 45534498646735972188403686897274408864356301263205069600999044599 then you can easily verify it by multiplication - you don't need access to the factorization software I used to determine that result. If I say that the energy difference between two chemical structure according to the Merck Molecular Force Field (MMFF) is X, then you can use any software which implements the MMFF to verify my statement using the raw data - you don't need the source code that I used. In fact, it would be better to use an independent implementation to verify my result than it would be to re-run the same code. And there are many programs which implement the MMFF. If you are used to RDKit, then it will likely be faster to use RDKit to reproduce and verify my numbers than trying to figure out the strange VM I provided, which is based around some home-brew code of mine implemented in MORTRAN.
    - wolfgke 10y ago
      > For example, if I say that RSA-130 = ... × ... then you can easily verify it by multiplication - you don't need access to the factorization software I used to determine that result. I didn't write about independent computation, but about independent verification. In your case you need access to a software for multiplying numbers for an independent check. OK, this is rather trivial code to write. But if it weren't that trivial the author would have to provide it.
      - 10y ago
jackmott 10y ago
>any result or conclusion has to be able to verified independently as much as possible Well, collecting the data yourself and writing the code yourself would be even more independent.
- wolfgke 10y ago
  > Well, collecting the data yourself and writing the code yourself would be even more independent. I wrote "any conclusion". This does not preclude collecting the data or writing the code yourself (and indeed you should if you doubt, say, the data). But it implies that the code that was used to come to the conclusion must be available.

by wolfgke 10y ago

DasBlob 10y ago

While I certainly share your sentiment that both, data and analysis software, should be available, I do see a conflict of interest here that cannot be easily resolved. On the one hand, we do assert (wrongly, IMO) that competition is a necessary element to drive scientific innovation and consequently require scientific actors to adhere to economic principles, i.e. publish more and better-received papers. On the other hand, allowing researchers to keep their competitive advantage, and having exclusive access to research data certainly is, contradicts the principle of reproducability and unneccessarily restricts further research that could be done based on that data. I don't see a way to easily resolve that in the current scientific framework. One (less than optimal) approach, that is frequently taken in astronomy, is to allow exclusive access to telescope data to the members of the consortium or the principal investigators for a certain amount of time, after which it is released to public. As for the software, it's easier. if the analysis method is sufficiently well documented (and it should be in any publication), people with access to the original data can easily attempt to reproduce the results and either way they'll get a publication out of it. So there is no incentive to keep the analysis software but there is a huge incentive to keep the original data.

wolfgke 10y ago
The solution should rather be that society (and publishers) does not consider papers (or more generally: texts) as research that don't also publish everything possible for independent verification (in this case the source code and raw data). If this is not the case, the text is just a plain marketing statement (in contrast to research) comparable to the marketing statements that are made everyday by lots of companies.
- dalke 10y ago
  The lack of source code does not make everything produced by software "a marketing statement." There's more than one way to verify a conclusion. For example, if I say that RSA-130 = 39685999459597454290161126162883786067576449112810064832555157243 × 45534498646735972188403686897274408864356301263205069600999044599 then you can easily verify it by multiplication - you don't need access to the factorization software I used to determine that result. If I say that the energy difference between two chemical structure according to the Merck Molecular Force Field (MMFF) is X, then you can use any software which implements the MMFF to verify my statement using the raw data - you don't need the source code that I used. In fact, it would be better to use an independent implementation to verify my result than it would be to re-run the same code. And there are many programs which implement the MMFF. If you are used to RDKit, then it will likely be faster to use RDKit to reproduce and verify my numbers than trying to figure out the strange VM I provided, which is based around some home-brew code of mine implemented in MORTRAN.
  - wolfgke 10y ago
    > For example, if I say that RSA-130 = ... × ... then you can easily verify it by multiplication - you don't need access to the factorization software I used to determine that result. I didn't write about independent computation, but about independent verification. In your case you need access to a software for multiplying numbers for an independent check. OK, this is rather trivial code to write. But if it weren't that trivial the author would have to provide it.
    - 10y ago

jackmott 10y ago

>any result or conclusion has to be able to verified independently as much as possible Well, collecting the data yourself and writing the code yourself would be even more independent.

wolfgke 10y ago
> Well, collecting the data yourself and writing the code yourself would be even more independent. I wrote "any conclusion". This does not preclude collecting the data or writing the code yourself (and indeed you should if you doubt, say, the data). But it implies that the code that was used to come to the conclusion must be available.