Short answer is surely an information theoretic one: if a neural net acquires knowledge, it needs to put that knowledge somewhere, and the state of the net is t

Short answer is surely an information theoretic one: if a neural net acquires knowledge, it needs to put that knowledge somewhere, and the state of the net is the only place it can be going. Sure, training the net forces it to store that knowledge as efficiently/lossily as it can get away with per the training criteria, but the fundamental information has to go somewhere. Doesn't it follow that if you run a DNN with a smaller memory footprint it will be able to learn less stuff (or its learning will be more sloppy/lossy)? It sounds like some of this piece argues that implementation choices like SIMD instructions are forcing particular pieces of DNN state to be constrained to certain sizes for efficiency; so I guess the argument there is that there is wasted precision in a lot of implemented state storage that isn't meaningfully useful for the network to capture its learning; that's reasonable - the bit resolution for a network weight should be driven by the mathematical precision of the model, not the practical constraints of hardware. But if you ran those numbers it's also perfectly possible (I guess) that You would find your DNN wants 128bit weights and on current hardware is actually limited by the precision of the weights and states we permit them to store - and that given more memory depth they would be able to learn more. This seems to start from the premise that we're giving DNNs more memory than they need. Is that actually a given?