4 ms·
Sometimes validation can be done without knowing the actual algorithm or being able to rerun it. For example, if a paper publishes a multiple sequence alignmen
by dalke 14y ago
Sometimes validation can be done without knowing the actual algorithm or being able to rerun it.
For example, if a paper publishes a multiple sequence alignment of 100 sequences, then all you need to do is verify that it's at least as good as what other MSA programs generate. You don't need to be able to rerun the program.
Indeed, sometimes you can't. Someone could have manually aligned the sequence, and that process isn't reproducible.
You see this as well with fold prediction software, where it can be easy to show that a predicted fold is likely correct (ie, matches experimental observations), and where you don't necessarily care about the method used to get that fold prediction.
A genetic algorithm might be very sensitive to the compute environment. For example, the order of float operations generated by two different compiler settings, or by network traffic timings in a distributed system. This can lead to different minima, where the overall effectiveness is the same but the actual configuration is different. The GA search doesn't need to be reproducible; it's only the final effectiveness which is important.
For those cases, I don't see the need for access to the underlying software in order to validate the result.
- Blahah 14y agoThe problem with the MSA question is that there is no objective measure of goodness. Most people compare them to a subjective 'gold standard', which is a manual alignment. Black box comparisons are not progressive: the only non-subjective advances can come from incorporating the results of sequence evolution experiments into the algorithms, and then the algorithm needs to be trusted, for which it needs to be open to scrutiny. I agree about the genetic algorithm.