Part of the issue is that many of the neurons in a neural network end up contributing little to the performance. This is why the practice of pruning, the remova

Part of the issue is that many of the neurons in a neural network end up contributing little to the performance. This is why the practice of pruning, the removal of neurons that don't contribute much, exists [1]. Neurons can end up having so little gradient that they end up stuck (Big problem with RELU), or can end up being insignificant due to the subsequent layers weights of that neuron. As the author is using RELU, he will have a decent number of neurons 'die'. So some 'over-provisioning' is not a bad idea in theory. Also, if my keras isn't so rusty, I think the author is using less parameters than you are stating. Still a more reasonable dropout rate and maybe some regularization/batch normalization might help, but I would say not over fitting on only 700 samples is a hard task, even with a network much smaller than that. I would be pretty shocked if this neural net wasn't over fit [1]-https://towardsdatascience.com/pruning-deep-neural-network-56cae1ec5505 https://towardsdatascience.com/pruning-deep-neural-network-5...