That's a great way of saying it concisely. And to continue that generalization - one of the consequences of information theory is that there exist trivially inc

That's a great way of saying it concisely. And to continue that generalization - one of the consequences of information theory is that there exist trivially incompressible strings. This plays nicely with the recent (2018) Lei-Luo-Yau-Gu result that there exist manifolds which cannot be learned. As I harp on every chance I can get, I have a pet hypothesis that there's a very deep corollary here waiting to be proven rigorously. Namely, that we can show there exist adversarial inputs that exploit neural networks because they're incompressible. Furthermore, that these inputs are information theoretically guaranteed to exploit the neural network (even if there are practical complexity theoretic workarounds).