4 ms·
No, there was no leakage. We trained on one dataset and evaluated on a completely different one, then did the reverse to show that the model generalized well ir
by nfusi 9y ago
No, there was no leakage. We trained on one dataset and evaluated on a completely different one, then did the reverse to show that the model generalized well irrespective of the training data (Figure 2). The decision of which model to deploy was based on cross-validation over the Avana data. We would have loved to have even more data, but generating data from this kind of experiment is expensive and labor-intensive.
EDIT: we will update the link, thanks. The correct link is https://www.microsoft.com/en-us/research/project/crispr/ https://www.microsoft.com/en-us/research/project/crispr/
- nonbel 9y agoIf you cv on a dataset, then change the features (or hyperparameters) and cv again, picking the best model, then you will will overfit to the cv. This is data leakage, it will lead you to be overly optimistic about your model performance on unseen data. This is well known, and honestly only takes one time working with a real hold out set (no cheating) to learn for life. Eg: https://datascience.stackexchange.com/questions/17288/why-k-fold-cross-validation-cv-overfits-or-why-discrepancy-occurs-between-cv https://datascience.stackexchange.com/questions/17288/why-k-...
- michaelhoffman 9y agoThe final performance evaluation does not use cross-validation, but uses totally held out validation data not used during model selection.
- nonbel 9y agoThanks this is not at all clear from the pre-print. From the final paper it does seem you are right, but the datasets and usage probably could be a bit clearer (eg include a table with that info).