2 ms·

Cross validation doesn't change the fact that you're trying to predict a non-stationary distribution. Machine learning techniques generally make an assumption t

Cross validation doesn't change the fact that you're trying to predict a non-stationary distribution. Machine learning techniques generally make an assumption that your samples are being drawn from at least a close enough distribution to any future data that may enter the system. With problems like text classification in NLP you can generally make a safe assumption that, though language does change, it changes slowly enough that you don't have to worry. In other cases a model may simply need to be retrained after a certain period of time. Even in text classification real world systems will often incorporate an unsupervised novelty detector as well to indicate when the models may need to be retrained. Additionally normal cross validation (random or stratified sampling) does not work on time series data since you are in effect cheating by training on events in the future, which your model will not be able to predict. In order to test your model on time series data you would need to hold out a significant chunk of data at the end of the time series. If you're interested in the general case of predicting the stock market you might enjoy reading up on the Efficient Market Hypothesis[0] which deals quite well with just that question. http://en.wikipedia.org/wiki/Efficient-market_hypothesis http://en.wikipedia.org/wiki/Efficient-market_hypothesis

fchollet 14y ago
Indeed. One reason why ML is quite powerless when looking at the stockmarket (from the point of view of past stock values) is that, even if you assume there is a latent model to be learned, it would seem reasonable that this model evolves as fast as the economic landscape and that is to say, quite fast. To such an extent that the valid data points available (the past few years of price variations) would not suffice to train even a very low-dimensional latent model.

fchollet 14y ago

Indeed. One reason why ML is quite powerless when looking at the stockmarket (from the point of view of past stock values) is that, even if you assume there is a latent model to be learned, it would seem reasonable that this model evolves as fast as the economic landscape and that is to say, quite fast. To such an extent that the valid data points available (the past few years of price variations) would not suffice to train even a very low-dimensional latent model.