3 ms·

>Maybe neural nets will become the data compressors of tomorrow? Long a topic of research, with many interesting ramifications: https://ai.googleblog.com/2016/

>Maybe neural nets will become the data compressors of tomorrow? Long a topic of research, with many interesting ramifications: https://ai.googleblog.com/2016/09/image-compression-with-neural-networks.html https://ai.googleblog.com/2016/09/image-compression-with-neu... In analogy to standard compression techniques, you can think of the neural net as a very large "dictionary". For text compression, there is of course the famous Hutter Prize, launched in 2006: https://en.wikipedia.org/wiki/Hutter_Prize https://en.wikipedia.org/wiki/Hutter_Prize ("Prediction is the golden key that opens all locks". Compressing each byte of wikipedia text is equivalent to predicting it-- to compactly represent its knowledge is to understand it.)

vintermann 5y ago
1. If the first byte is 0, insert the text of Wikipedia, 2. If it isn't, ignore it, and all further bytes are interpreted literally. To avoid these sort of "joke" decompressors, they evaluate compressors on (size of compressed data) + (size of decompressor) in the compression competitions last time I checked. That means we won't get a winner based on GPT3 anytime soon. 350+ GB of weights is a lot to overcome :) Though of course, given enough data to compress, it might well be that full-on neural language models are still worth it.
- deleted 5y ago
  [deleted]
- visarga 5y ago
  > That means we won't get a winner based on GPT3 anytime soon. 350+ GB of weights is a lot to overcome :) You're right, and in-there lays the crucial difference between compression and language modelling. Models are concerned with having a good representation for both past and future distributions, while compressors only care about the past. Models support many tasks while compressors are just for input replication.
  - vintermann 5y ago
    No, I don't quite agree. All compressors have a model, at least the general purpose ones. If you're writing self-extracting compression for your 8-bit demo, maybe you only care about the data you have right in front of you... But for anything general purpose, from DEFLATE to paq, there's an implicit, fixed model of future data in there (even if we sometimes prefer to see it as an "adaptive" model). A good illustration of this is that you can quite easily use a general purpose compression algorithm for classification, and get surprisingly good performance.
- injidup 5y ago
  I don't think it is unreasonable that both sender and receiver have the 350GB NN at each end. If for example we are talking video conferencing and compression of video data then only a small amount of data need to be transmitted real time and at each end a high fidelity image can be reconstructed.
  - vintermann 5y ago
    Maybe one day that will be practical. But right now, you can't even fit that into a top-tier GPU you know...

vintermann 5y ago

1. If the first byte is 0, insert the text of Wikipedia, 2. If it isn't, ignore it, and all further bytes are interpreted literally. To avoid these sort of "joke" decompressors, they evaluate compressors on (size of compressed data) + (size of decompressor) in the compression competitions last time I checked. That means we won't get a winner based on GPT3 anytime soon. 350+ GB of weights is a lot to overcome :) Though of course, given enough data to compress, it might well be that full-on neural language models are still worth it.

deleted 5y ago
[deleted]
visarga 5y ago
> That means we won't get a winner based on GPT3 anytime soon. 350+ GB of weights is a lot to overcome :) You're right, and in-there lays the crucial difference between compression and language modelling. Models are concerned with having a good representation for both past and future distributions, while compressors only care about the past. Models support many tasks while compressors are just for input replication.
- vintermann 5y ago
  No, I don't quite agree. All compressors have a model, at least the general purpose ones. If you're writing self-extracting compression for your 8-bit demo, maybe you only care about the data you have right in front of you... But for anything general purpose, from DEFLATE to paq, there's an implicit, fixed model of future data in there (even if we sometimes prefer to see it as an "adaptive" model). A good illustration of this is that you can quite easily use a general purpose compression algorithm for classification, and get surprisingly good performance.
injidup 5y ago
I don't think it is unreasonable that both sender and receiver have the 350GB NN at each end. If for example we are talking video conferencing and compression of video data then only a small amount of data need to be transmitted real time and at each end a high fidelity image can be reconstructed.
- vintermann 5y ago
  Maybe one day that will be practical. But right now, you can't even fit that into a top-tier GPU you know...