3 ms·

This is not very convincing. It reads like a semester project for an undergrad or a summer intern. In particular, the prediction task used for evaluation is not

by mlthoughts2018 8y ago

This is not very convincing. It reads like a semester project for an undergrad or a summer intern. In particular, the prediction task used for evaluation is not very meaningful, and the intended use case (predicting biotech underperformers) is not very meaningful without predicting by how much they would underperform, i.e. just using the output of this model as yet another regression factor with questionable significance in the end. But in that case, it’s very unlikely a heavy CNN text model would be better suited than simpler methods like a classifier using LSA vectors, or even just extracting the gloVe vectors and doing no convolution. Especially given all the metaparameters they mention needing to tweak. For example, an ablation study to figure out that you need Lecun initialization seems both overkill and requiring way more specialization in deep learning than a typical firm would be looking to have, all just to squeeze out what might be some slight efficacy in one industry group. When I previously worked in quant finance, I used to be very passionate about trying to apply the latest & greatest methods. But over time my feeling is most of it is inapplicable to finance, because it ends up being a lot of work for pretty much no efficacy, like in this post. I found much more value in simpler regression and tree models, and using simple bag of words models with text. The only “advanced” stuff that ever seemed to add significant efficacy was Bayesian hierarchical regression, but only for helping overcome limitations of classical random effects models, not for adding any greater complexity. The post author deserves plenty of credit for skills in deep learning, but the analysis seems unconvincing if it’s supposed to market for using MSFT cloud for deep learning in finance.

hendzen 8y ago
Yep. A lot of quant strategies actually exclude biotech altogether. Simply too volatile. Even with relatively conservative portfolio concentration limits you can still lose millions on a bio name dropping 50% due to a rumor of a failed clinical trial.
RyEgswuCsn 8y ago
Agreed. Also noted three potential flows: 1. They applied pandas.qcut over the entire returns (i.e. training and validation set) when generating the target performance labels, which could compromise the validation. 2. The ordering of news is actually important when it comes to forecasting the movements of the market; their training/validation split was done after shuffling the entire dataset, which means the model will have access to information that shouldn't be available to it during training. 3. It makes more sense to use a training-validation-testing setting rather than a training-validation split when reporting the model performance in order to avoid inflated results due to all the hyper-parameter tuning.
- rpedela 8y ago
  What do you mean by training-validation-testing vs training-validation split? Also why would pandas.qcut compromise the validation?
  - deleted 8y ago
    [deleted]
  - mlthoughts2018 8y ago
    If the quantile assignments took into account data from both the training set and the test set, then it means the training set quantile label itself contains lookahead information about what the distribution of test set values is. This can lead to artificially increased test set accuracy since by optimizing the model during training, you are partially optimizing based on information you gained by knowing about the test set in advance. Training-validation-testing generally means keeping a portion of the training set held out for evaluating accuracy and convergence after one full iteration of training updates. So you get some information per each training update regarding whether the model had overfitting, or performs unrealistically well on the validation set (which it’s not trained on), or for early stopping criteria. Then, after all that, you move on to checking the final accuracy and metrics on a fully held out test set. This validation approach is common in deep learning because of the many diagnostics you need to get information about during training. Without some information outside the training set itself, it can be hard tonunderstand how the learning rate is affecting you, how likely overfitting is, whether there is a vanishing gradient problem. Waiting until all the way to the end of training to get any feedback on these is sometimes just too inefficient or too risky that you’ll discover an issue only after a huge time sink.
gbasin 8y ago
Bingo. Echoes my experience (also in quant finance). ML is just not all that useful in our field. To the extent that some shops are getting mileage from it, I presume it's from combining lots of really alternative datasets. Satellite photos, traffic data, etc?

hendzen 8y ago

Yep. A lot of quant strategies actually exclude biotech altogether. Simply too volatile. Even with relatively conservative portfolio concentration limits you can still lose millions on a bio name dropping 50% due to a rumor of a failed clinical trial.

RyEgswuCsn 8y ago

Agreed. Also noted three potential flows: 1. They applied pandas.qcut over the entire returns (i.e. training and validation set) when generating the target performance labels, which could compromise the validation. 2. The ordering of news is actually important when it comes to forecasting the movements of the market; their training/validation split was done after shuffling the entire dataset, which means the model will have access to information that shouldn't be available to it during training. 3. It makes more sense to use a training-validation-testing setting rather than a training-validation split when reporting the model performance in order to avoid inflated results due to all the hyper-parameter tuning.

rpedela 8y ago
What do you mean by training-validation-testing vs training-validation split? Also why would pandas.qcut compromise the validation?
- deleted 8y ago
  [deleted]
- mlthoughts2018 8y ago
  If the quantile assignments took into account data from both the training set and the test set, then it means the training set quantile label itself contains lookahead information about what the distribution of test set values is. This can lead to artificially increased test set accuracy since by optimizing the model during training, you are partially optimizing based on information you gained by knowing about the test set in advance. Training-validation-testing generally means keeping a portion of the training set held out for evaluating accuracy and convergence after one full iteration of training updates. So you get some information per each training update regarding whether the model had overfitting, or performs unrealistically well on the validation set (which it’s not trained on), or for early stopping criteria. Then, after all that, you move on to checking the final accuracy and metrics on a fully held out test set. This validation approach is common in deep learning because of the many diagnostics you need to get information about during training. Without some information outside the training set itself, it can be hard tonunderstand how the learning rate is affecting you, how likely overfitting is, whether there is a vanishing gradient problem. Waiting until all the way to the end of training to get any feedback on these is sometimes just too inefficient or too risky that you’ll discover an issue only after a huge time sink.

gbasin 8y ago

Bingo. Echoes my experience (also in quant finance). ML is just not all that useful in our field. To the extent that some shops are getting mileage from it, I presume it's from combining lots of really alternative datasets. Satellite photos, traffic data, etc?