3 ms·

Agreed. Also noted three potential flows: 1. They applied pandas.qcut over the entire returns (i.e. training and validation set) when generating the target per

Agreed. Also noted three potential flows: 1. They applied pandas.qcut over the entire returns (i.e. training and validation set) when generating the target performance labels, which could compromise the validation. 2. The ordering of news is actually important when it comes to forecasting the movements of the market; their training/validation split was done after shuffling the entire dataset, which means the model will have access to information that shouldn't be available to it during training. 3. It makes more sense to use a training-validation-testing setting rather than a training-validation split when reporting the model performance in order to avoid inflated results due to all the hyper-parameter tuning.

rpedela 8y ago
What do you mean by training-validation-testing vs training-validation split? Also why would pandas.qcut compromise the validation?
- deleted 8y ago
  [deleted]
- mlthoughts2018 8y ago
  If the quantile assignments took into account data from both the training set and the test set, then it means the training set quantile label itself contains lookahead information about what the distribution of test set values is. This can lead to artificially increased test set accuracy since by optimizing the model during training, you are partially optimizing based on information you gained by knowing about the test set in advance. Training-validation-testing generally means keeping a portion of the training set held out for evaluating accuracy and convergence after one full iteration of training updates. So you get some information per each training update regarding whether the model had overfitting, or performs unrealistically well on the validation set (which it’s not trained on), or for early stopping criteria. Then, after all that, you move on to checking the final accuracy and metrics on a fully held out test set. This validation approach is common in deep learning because of the many diagnostics you need to get information about during training. Without some information outside the training set itself, it can be hard tonunderstand how the learning rate is affecting you, how likely overfitting is, whether there is a vanishing gradient problem. Waiting until all the way to the end of training to get any feedback on these is sometimes just too inefficient or too risky that you’ll discover an issue only after a huge time sink.

rpedela 8y ago

What do you mean by training-validation-testing vs training-validation split? Also why would pandas.qcut compromise the validation?

deleted 8y ago
[deleted]
mlthoughts2018 8y ago
If the quantile assignments took into account data from both the training set and the test set, then it means the training set quantile label itself contains lookahead information about what the distribution of test set values is. This can lead to artificially increased test set accuracy since by optimizing the model during training, you are partially optimizing based on information you gained by knowing about the test set in advance. Training-validation-testing generally means keeping a portion of the training set held out for evaluating accuracy and convergence after one full iteration of training updates. So you get some information per each training update regarding whether the model had overfitting, or performs unrealistically well on the validation set (which it’s not trained on), or for early stopping criteria. Then, after all that, you move on to checking the final accuracy and metrics on a fully held out test set. This validation approach is common in deep learning because of the many diagnostics you need to get information about during training. Without some information outside the training set itself, it can be hard tonunderstand how the learning rate is affecting you, how likely overfitting is, whether there is a vanishing gradient problem. Waiting until all the way to the end of training to get any feedback on these is sometimes just too inefficient or too risky that you’ll discover an issue only after a huge time sink.