3 ms·
Ask HN: 0.000068 MSE for TensorFlow Model?
Question, just for comparison purposes, is anyone getting an MSE like this in any of their models for validation metrics?
Mine is doing some predictions, and over the past few weeks of iterations I've gone from 0.2 down to this, and trying to push it lower.
I know this all depends on context and the number means nothing by itself, but would be curious to know if anyone has a really low number, much lower than this.
- neximo64 3y agoThis is almost a meaningless, the MSE depends on the model structure and can't be compared arbitrarily. If you had a single perceptron/single layer model, its a fantastic MSE.
- deleted 3y ago[deleted]
- jzombie 3y agoI thought MSE was tied to the predictions it makes and not the structure itself (i.e. if you get the same value out for the same input, using different structures, you should at least get the same MSE).
- verdverm 3y agowithout more details, this sounds like overfitting or training data leakage
- jzombie 3y agoI can assure the data is not shuffled and is ordered in chronological order (time series model). However at present I've only got training/validation splits and not an additional testing.
- verdverm 3y agoOne thing you might try is creating some brownian noise and train on that. If you get really low errors it would be sus
- jzombie 3y agoThanks for the feedback. Will try that. Been mostly using Keras Tuner and Tensorboard to slowly get to the point I'm at and have had some wild results in the process.
- verdverm 3y agoThe best test will be to run it against the real market, but without actual money. With results like that, I would not wager my dollars
- jzombie 3y agoYes, actually I was planning on training a separate model to determine buy/sell signals, using this prediction model as possibly part of a transfer learning process, and do paper trading w/ it to start w/.
- verdverm 3y agoFun story, my first market model (2003) was a simple feed-forward network trained using an evolutionary algorithm. This led me to LSTM and backprop, which required calculus, and pushed me to go back to college. I've been thinking about building something again, based in P/E ratio and some portfolio mgmt strats I used for a college paper. Got the best returns for the semester but only a C grade because the prof didn't like my mechanical approach. I'm all about removing the fear and greed from the decision making.
- jzombie 3y agoI started w/ an LSTM and wound up w/ a mixture of transformers, CNNs and skip connections.
- verdverm 3y agoI think my next attempt will be a handful of hand crafted rules on daily data. I'm definitely never going to try day trading scale strategies or try to pick good points intraday to buy. It should be resilient to the price difference over a day or two