4 ms·
>Back to the computer science: 470,000+ features sounds nice at first, but is a recipe for overfitting when we only have 700 samples at our disposal. Proceeds
by chewxy 7y ago
>Back to the computer science: 470,000+ features sounds nice at first, but is a recipe for overfitting when we only have 700 samples at our disposal.
Proceeds to use (1024^2 * 2 + 1024) parameters in the neural network.
- starchild_3001 7y agoYeah, that's what I thought too. Interesting work nonetheless!
- jjoonathan 7y agoJust use dropout, dropout prevents overfitting! /s /s for this post, I mean. I have had this very suggestion made to me non-sarcastically under similar circumstances.
- deepnotderp 7y agoI know this is a joke, but the theory of generalization in NNs is rapidly advancing and it's not quite that simplistic: https://arxiv.org/abs/2003.02139 https://arxiv.org/abs/2003.02139
- chewxy 7y agoYa. And the choice of optimizer (in this case adam) also imposes upon it some regularization scheme. I just thought I'd highlight a bit of funniness.
- profunctor 7y agoHow does Adam provide regularisation? I’d never heard of this before and I don’t recall it from when I read the paper.
- Buraksr 7y agoPart of the issue is that many of the neurons in a neural network end up contributing little to the performance. This is why the practice of pruning, the removal of neurons that don't contribute much, exists [1]. Neurons can end up having so little gradient that they end up stuck (Big problem with RELU), or can end up being insignificant due to the subsequent layers weights of that neuron. As the author is using RELU, he will have a decent number of neurons 'die'. So some 'over-provisioning' is not a bad idea in theory. Also, if my keras isn't so rusty, I think the author is using less parameters than you are stating. Still a more reasonable dropout rate and maybe some regularization/batch normalization might help, but I would say not over fitting on only 700 samples is a hard task, even with a network much smaller than that. I would be pretty shocked if this neural net wasn't over fit [1]-https://towardsdatascience.com/pruning-deep-neural-network-56cae1ec5505 https://towardsdatascience.com/pruning-deep-neural-network-5...
- ruborcalor 7y agowhat a meme haha :) after many similar comments I plan on implementing a simpler model and seeing how it compares
- chewxy 7y agoI forked your colab. There was abit of code it couldn't run. `Gxxx_matrix.csv` not found.