3 ms·
Agreed. According to the paper: They trained the data, once, with 3-4 months of data from Feb to May. Then they tested the data, once, with ~6 weeks of data fro
by leeber 12y ago
Agreed. According to the paper: They trained the data, once, with 3-4 months of data from Feb to May. Then they tested the data, once, with ~6 weeks of data from May 6 - June 24.
There was no cross validation involved in assessing the performance of this model. Using one subset for training, and one subset for testing and calling that conclusive is naive.
No interesting conclusions can be taken away from this paper due to flawed methodology and failure to take into account real world variables like bid-ask spread, commissions, how quickly trades can be executed, whether orders will even be filled, etc.
- vasilipupkin 12y agoWell, they did trade it out of sample, so that probably means the model wasn't overfit. But, I agree, they probably used unrealistic execution assumptions
- beejiu 12y agoCross validation too uses training sets and test sets. This sort of time-ordered data will not be independent, so the prequential approach seems to be a more suitable approach to measuring forecast accuracy. (I haven't read the paper.)
- leeber 12y agoCross validation for time series, goes something like this: http://robjhyndman.com/hyndsight/crossvalidation/ http://robjhyndman.com/hyndsight/crossvalidation/ (near bottom of the article) Anyways, as somebody who has spent GIANT amounts of time experimenting with machine learning using market data (mainly stocks), I can give you TONS algorithms I've created that would show similar, even much higher, gains when you only test on 6 weeks of data. Been there done that. For example 6 weeks you make a 100% return, then in the next 3 weeks you suffer a 50% loss. Reality sets in...