3 ms·

Very interesting. This makes me wonder if a similar technique can be used for compression?

by bcheung 10y ago

adriancooney 10y ago
Middle-out compression.
aab0 10y ago
Absolutely. The point of an autoencoder is dimensionality reduction: boil a big set of data down to a few hundred or thousand numbers in a vector which summarizes it. You could treat it either as lossy compression and store just the encoding, or you can treat it as a hybrid format in which the autoencoder lossy encoding is then corrected to lossless by additional bits in the stream. In practice, even the hyper-efficient compression algorithms used in something like zpaq tend to use only very small shallow predictive neural networks because no one wants to wait days for their data to be compressed or ship around big neural nets as part of their archives, so it's more of an information-theoretic curiosity. Few enough people will even use 'xz'.
- vintermann 10y ago
  You could, but I don't think this would be competitive on compression ratio at all, even allowing for an order of magnitude more time.
  - sp332 10y ago
    The technique is very effective, after all a variant of paq holds a compression record. http://prize.hutter1.net/ http://prize.hutter1.net/ You can try paq yourself in PeaZip. http://filehippo.com/download_peazip_64/ http://filehippo.com/download_peazip_64/
    - argonaut 10y ago
      PAQ does not use autoencoders, that's the difference. Neural network models for compression are extremely slow and can use a crapton of memory (one top benchmark result uses 32GB of RAM): https://www.quora.com/What-is-the-potential-of-neural-networks-in-data-compression https://www.quora.com/What-is-the-potential-of-neural-networ...
    - vintermann 10y ago
      Last I checked, PAQ only uses a shallow (two layer) neural network as a last step to weight the predictions from the multiple handmade next-bit prediction models it contains.
      - sp332 10y ago
        So, you think the additional accuracy of the larger neural network isn't enough to overcome the storage size of the network itself?
astrange 10y ago
This is too powerful to use for product work, I think. Professionals like to use something they can understand, and when making a BD or streaming source they know what a compression artifact looks like and which frames' bitrates to tweak to hide it. They pretty much sit there all day and just do that. Maybe live streaming?
csn 10y ago
I was intrigued by this a while back. I think of training a NN as generating a function, an equation, from a training set which, given a specific input, outputs a prediction. If you can come up with an equation-input pair that, when executed, a) accurately enough approximates some data, and b) requires less space than the original file, you have achieved (most likely lossy) compression.

Very interesting. This makes me wonder if a similar technique can be used for compression?

by bcheung 10y ago

Very interesting. This makes me wonder if a similar technique can be used for compression?

adriancooney 10y ago

Middle-out compression.

aab0 10y ago

Absolutely. The point of an autoencoder is dimensionality reduction: boil a big set of data down to a few hundred or thousand numbers in a vector which summarizes it. You could treat it either as lossy compression and store just the encoding, or you can treat it as a hybrid format in which the autoencoder lossy encoding is then corrected to lossless by additional bits in the stream. In practice, even the hyper-efficient compression algorithms used in something like zpaq tend to use only very small shallow predictive neural networks because no one wants to wait days for their data to be compressed or ship around big neural nets as part of their archives, so it's more of an information-theoretic curiosity. Few enough people will even use 'xz'.

vintermann 10y ago
You could, but I don't think this would be competitive on compression ratio at all, even allowing for an order of magnitude more time.
- sp332 10y ago
  The technique is very effective, after all a variant of paq holds a compression record. http://prize.hutter1.net/ http://prize.hutter1.net/ You can try paq yourself in PeaZip. http://filehippo.com/download_peazip_64/ http://filehippo.com/download_peazip_64/
  - argonaut 10y ago
    PAQ does not use autoencoders, that's the difference. Neural network models for compression are extremely slow and can use a crapton of memory (one top benchmark result uses 32GB of RAM): https://www.quora.com/What-is-the-potential-of-neural-networks-in-data-compression https://www.quora.com/What-is-the-potential-of-neural-networ...
  - vintermann 10y ago
    Last I checked, PAQ only uses a shallow (two layer) neural network as a last step to weight the predictions from the multiple handmade next-bit prediction models it contains.
    - sp332 10y ago
      So, you think the additional accuracy of the larger neural network isn't enough to overcome the storage size of the network itself?

astrange 10y ago

This is too powerful to use for product work, I think. Professionals like to use something they can understand, and when making a BD or streaming source they know what a compression artifact looks like and which frames' bitrates to tweak to hide it. They pretty much sit there all day and just do that. Maybe live streaming?

csn 10y ago

I was intrigued by this a while back. I think of training a NN as generating a function, an equation, from a training set which, given a specific input, outputs a prediction. If you can come up with an equation-input pair that, when executed, a) accurately enough approximates some data, and b) requires less space than the original file, you have achieved (most likely lossy) compression.