I love how I've seen Ilya talk about it. If we could find the shortest program to reproduce the training set, that would be optimal compression. But it's an int

I love how I've seen Ilya talk about it. If we could find the shortest program to reproduce the training set, that would be optimal compression. But it's an intractable problem, we can't even come close, there's just no way to come at it. But with deep learning we can instead find a circuit that approaches reproducing the training set. This is lossy compression. There's nothing "just glorified" about it though; the result is astounding. A more appropriate takeaway might be that sufficient compression is mind-bendingly more powerful than intuition might otherwise guess. Does calling it "just glorified" guide any intuition that in order to compress amazon reviews a neural net is gonna have weights that correspond to sentiment? Does it tell you that such compression also ends up producing an artifact that can be put to work in a generative framework? And that it'll be a very useful generative framework because such compression required weights that correspond to all sorts of useful ideas, a compression of something more meaningful than just text? Calling it "just glorified X" is clickbait. It's compression alright, and it's either 1) also whole lot more, or 2) compression is a whole lot more wild and impressive than you thought, or both.