4 ms·
Always thought of the opposite: DL model as a compression method. Like taking a initial frame of a movie and training a DL to reproduce the following frames. A
by woliveirajr 6y ago
Always thought of the opposite: DL model as a compression method.
Like taking a initial frame of a movie and training a DL to reproduce the following frames. And then transmitting the frame + parameters to reproduce the movie.
- wenc 6y agoThat’s what NVIDIA Maxine does. https://www.dpreview.com/news/5756257699/nvidia-research-develops-a-neural-network-to-replace-traditional-video-compression https://www.dpreview.com/news/5756257699/nvidia-research-dev...
- fantod 6y agoMaybe this could be the beginning of a plausible explanation for why knowledge distillation works.
- astrange 6y agoDeep learning for video compression seems like it'll have a natural advantage just because video codecs have never been particularly large (maybe 100 KBs?) but everyone expects an ML model to be enormous. That's a lot of space to store common data in.
- abyesilyurt 6y agoIt is not video, but I have worked on image compression using neural networks on my master thesis. It was an improvement over an existing method, so there is quite a bit of research going on in this area. Briefly, the idea was to use an auto-encoder to transform the image, then quantize and encode the transform coefficients. So, you can actually “teach” the network to be resilient against quantization operation. Very similar to what the author describes.
- unemphysbro 6y agoYeah, I've thought something along those lines, maybe applied to a layer(s)/architectural element-wise basis. I'm sure someone's done it because it; just seems like another optimization problem.
- chestervonwinch 6y agoAny dimensionality reduction method (DL-based or otherwise) can be viewed as a form of compression. But, the end goal in data dimensionality reduction is usually quite different than with data compression. Either way, this is much different than model compression/distillation, which is compression of the parametrized functional mapping itself. As a silly example, imagine you fit a 100-degree polynomial to noisy linear data using some proper regularization. You would find that can distill/compress your 100-degree polynomial model into a 1-degree model with comparable accuracy.