5 ms·

In astronomy we are still fighting for public data. Verification of results comes from another researcher collecting his own data and running his own analysis.

In astronomy we are still fighting for public data. Verification of results comes from another researcher collecting his own data and running his own analysis. Out of curiosity, do you work for industry or academia?

Blahah 13y ago
That's an extremely inefficient and costly way to validate a result! It would be rendered unnecessary in cases where just looking at the code invalidated the result. I'm in academia (see my profile for details).
- J_Thomas 13y ago
  It's good and useful for independent researchers to collect their own data and analyse it. That's a check on the data collection methods. It's also useful for others to check the analysis of trusted data. If they do it with their own methods and get a different result, then it's time to actually compare the methods. It's a good thing for the original researchers to release their code too. But looking for bugs in the code is no substitute for doing independent analysis and independent data collection. It's an important supplement, but it isn't enough.
  - Blahah 13y ago
    I agree with your points. The saving in the case of released source code is that, if the original analysis is flawed in such a way that there's no need to repeat it because it's wrong, then a large amount of expense and effort can be saved. Of course, if the original analysis holds up given the data and code, independent replication is a necessary thing.
- dalke 13y ago
  Sometimes validation can be done without knowing the actual algorithm or being able to rerun it. For example, if a paper publishes a multiple sequence alignment of 100 sequences, then all you need to do is verify that it's at least as good as what other MSA programs generate. You don't need to be able to rerun the program. Indeed, sometimes you can't. Someone could have manually aligned the sequence, and that process isn't reproducible. You see this as well with fold prediction software, where it can be easy to show that a predicted fold is likely correct (ie, matches experimental observations), and where you don't necessarily care about the method used to get that fold prediction. A genetic algorithm might be very sensitive to the compute environment. For example, the order of float operations generated by two different compiler settings, or by network traffic timings in a distributed system. This can lead to different minima, where the overall effectiveness is the same but the actual configuration is different. The GA search doesn't need to be reproducible; it's only the final effectiveness which is important. For those cases, I don't see the need for access to the underlying software in order to validate the result.
  - Blahah 13y ago
    The problem with the MSA question is that there is no objective measure of goodness. Most people compare them to a subjective 'gold standard', which is a manual alignment. Black box comparisons are not progressive: the only non-subjective advances can come from incorporating the results of sequence evolution experiments into the algorithms, and then the algorithm needs to be trusted, for which it needs to be open to scrutiny. I agree about the genetic algorithm.

Blahah 13y ago

That's an extremely inefficient and costly way to validate a result! It would be rendered unnecessary in cases where just looking at the code invalidated the result. I'm in academia (see my profile for details).

J_Thomas 13y ago
It's good and useful for independent researchers to collect their own data and analyse it. That's a check on the data collection methods. It's also useful for others to check the analysis of trusted data. If they do it with their own methods and get a different result, then it's time to actually compare the methods. It's a good thing for the original researchers to release their code too. But looking for bugs in the code is no substitute for doing independent analysis and independent data collection. It's an important supplement, but it isn't enough.
- Blahah 13y ago
  I agree with your points. The saving in the case of released source code is that, if the original analysis is flawed in such a way that there's no need to repeat it because it's wrong, then a large amount of expense and effort can be saved. Of course, if the original analysis holds up given the data and code, independent replication is a necessary thing.
dalke 13y ago
Sometimes validation can be done without knowing the actual algorithm or being able to rerun it. For example, if a paper publishes a multiple sequence alignment of 100 sequences, then all you need to do is verify that it's at least as good as what other MSA programs generate. You don't need to be able to rerun the program. Indeed, sometimes you can't. Someone could have manually aligned the sequence, and that process isn't reproducible. You see this as well with fold prediction software, where it can be easy to show that a predicted fold is likely correct (ie, matches experimental observations), and where you don't necessarily care about the method used to get that fold prediction. A genetic algorithm might be very sensitive to the compute environment. For example, the order of float operations generated by two different compiler settings, or by network traffic timings in a distributed system. This can lead to different minima, where the overall effectiveness is the same but the actual configuration is different. The GA search doesn't need to be reproducible; it's only the final effectiveness which is important. For those cases, I don't see the need for access to the underlying software in order to validate the result.
- Blahah 13y ago
  The problem with the MSA question is that there is no objective measure of goodness. Most people compare them to a subjective 'gold standard', which is a manual alignment. Black box comparisons are not progressive: the only non-subjective advances can come from incorporating the results of sequence evolution experiments into the algorithms, and then the algorithm needs to be trusted, for which it needs to be open to scrutiny. I agree about the genetic algorithm.