It's easy to overfit models with huge numbers of parameters when there isn't much data, which produces a model that's possibly amusing but not any more useful t

It's easy to overfit models with huge numbers of parameters when there isn't much data, which produces a model that's possibly amusing but not any more useful than an overcomplicated key value store that memorises the training data. I'm not familiar with RNN / LSTN models, or how torch-rnn encodes features, so it's pretty opaque looking at the writeup and results to gauge if there's a bunch of overfitting going on here. Is 2 megs of javascript sufficient to train a model of this complexity? Or would 20kb suffice? Or is 2GB really the minimum needed? I have no idea, but it doesn't really seem like this article knows either. I believe in the olden days where computational power was limited, statisticians would hand craft simple statistical models with low numbers of parameters, and figure out how much reasonably distributed data they'd need to collect and encode to the constrain the parameters in their model (i.e. so the resulting model fit wasn't complete nonsense and had some predictive accuracy or explanatory power). These days with much more computational power, it is relatively easy to fit highly complex models with huge numbers of parameters, but arguably not much easier (or perhaps harder?) to assess if the model fit / model fitting procedure is complete nonsense or not, and if the resulting model is any good. I guess k-fold cross validation and so on can help, it doesn't read like that was explicitly done here. By default the torch-rnn docco claims to use 0.1th of the data for test and validation sets. This sounds... small? The writeup gives results in terms of "Loss" for a number of configurations, but doesn't define what the loss is (which subset of the data was used to compute it) or really explain what the experimental protocol was for training, validation, parameter tuning, etc. There is a risk that the author has observed the loss measured on the test or validation set for each combination of hyperparameters (network topology, dropout, etc) and then used this information to manually implement hyperparameter optimisation (to choose network topology, dropout parameter, to minimuse loss on the test / validation set). this would result in there being no "fresh" unobserved data left to give an independent validation of if the resulting model fitting strategy (including the manually-executed hyperparameter tuning algorithm) has any predictive accuracy as all.