To summarize the article: using backpropagation to train a small LSTM to accept strings of the form (a^n b^n), regularization with both L1 and L2 losses result

To summarize the article: using backpropagation to train a small LSTM to accept strings of the form (a^n b^n), regularization with both L1 and L2 losses result in loss spaces where the loss-optimal solution is not the generally-correct one. The resulting networks fail out-of-sample. The LSTM architecture does permit a correct, 'general' solution to exist, and the authors show that the general solution is an optimum when using a minimum-description-length error function. The MDL error function is effectively entropy(weights) + relative entropy(training data | model). The researcher's problem is that entropy(weights) is not well-defined. To prevent the network from "smuggling" information through highly precise but otherwise random-looking weights, they define an entropy (coding length) that rewards simple rational fractions (1/2, 3/4, 7/11) and penalizes complex ones (715/937). With this loss function, their hand-crafted optimal solution also lies at a loss-optimum. Unfortunately, the resulting loss space is non-differentiable, making it useless for any gradient-descent-like training. To muse about this, the authors' problem of weight entropy here is similar to the problem faced by autoencoders, whereby they want to have a minimally-structured latent space to avoid the same information-smuggling problem. Perhaps we could evolve a differentiable version of the model-entropy loss function by treating the model weights as random variables, drawn from a distribution of learned mean and variance with the same 'reparameterization trick' that makes variational autoencoders work. The model entropy loss term is then the same as the latent-space regularization term in VAE training.