The RNN in question is over-parameterized, so most loss functions wouldn't be expected to coincide with MDL (unless maybe you added in heavier regularization th

The RNN in question is over-parameterized, so most loss functions wouldn't be expected to coincide with MDL (unless maybe you added in heavier regularization than just L1/L2). There is no incentive towards simplicity. But it is also a single-task network, and it is increasingly rare to train any of those. A multi-task network must share its weights across all the tasks, and so the more tasks it does for a particular parameterization, the more incentive it has towards simplicity in order to do the tasks at all. So I wonder if scaling up task diversity would increasingly approximate MDL and presumably help it learn the true algorithmic cores?