4 ms·
If you are going to use machine learning on the data, though, make sure you know what you are doing unless you are just using someone else's complete package. I
by tzs 10y ago
If you are going to use machine learning on the data, though, make sure you know what you are doing unless you are just using someone else's complete package. It's really easy to screw up machine learning.
I recall an example given in a class I took. (I may be misremembering the details, though).
Some people were trying to apply machine learning to currency trading. They had a bunch of data. They normalized the data (a common step in machine learning), and divided it into training and test and validation sets, and trained their model. Everything looked great, and they were getting excellent results on the test set.
When they went live with real money instead of making the nice profit predicted, they lost a lot of money.
Their mistake? They normalized the whole data set up front. What they should have done is split it into the training, test, and validation sets, and normalized each of those individually. Normalizing before splitting compromises the independence of the sets, biasing the learning.
(I must admit I never did quite understand this. Normalizing is optional. As far as I understand if one does an arbitrary transformation on one's data as a whole that should not actually make learning go bad, at least as long as the same transformation is applied to all the input when you go live. So when they did a normalizing step on the whole data set, why wasn't that just like doing any other arbitrary transform? There is serious dark magic here...)
- matheweis 10y agoI've read about a similar scenario (possibly the same one?), and you got the important bits right. You cannot normalize the entire dataset at once; the normalization contains some parity effectively allowing the algorithm to cheat and see the future... but in real life we can't see the future. The simple example that eliminates some of the "black magic" is that normalizing against the entire set lets the algorithm know what the highest and lowest points across the entire data set are - and knowing the lowest and highest lets the algorithm buy low/sell high for all of the known data.
- mrkgnao 10y agoWithout a minute understanding of what's involved, I'd guess that normalizing is a non-local operation, in the sense that the effect of normalizing some x depends on the other elements of your dataset, so that normalization includes the "information" of the rest of the data implicitly. In real life, that's sort of like having access to the answers when you're given the questions. Being trained in that environment won't do much good.
- matheweis 10y agoI think you've got it. The part about normalization across the entire dataset implicitly including information is by definition true (it contains, at a minimum the true min/max). Can't speak to the rest, though, as I'm still learning myself... :)