4 ms·

I find the idea of AI-based image compression rather fascinating. In theory it seems like a great fit for AI; "perceived quality" is exactly the sort of fuzzy

I find the idea of AI-based image compression rather fascinating. In theory it seems like a great fit for AI; "perceived quality" is exactly the sort of fuzzy metric AI tends to be good at optimizing for, and a compression algorithm possessing an understanding of high-level visual features like shapes, lighting, etc seems like it would have a lot of potential. Taken to the extreme, a sufficiently advanced AI compression system could theoretically end up storing representations of high-level features like faces in the compressed data; which could lead to some really bizarre looking compression artifacts when the system breaks down.

derefr 7y ago
> Taken to the extreme, a sufficiently advanced AI compression system could theoretically end up storing representations of high-level features like faces in the compressed data Or, you could have a compression system created specifically for photographic images (as opposed to a “general compressor” like LZ77), where these high-level features are part of the (de)compressor itself. Now, lossy domain-specific compressors are pretty common; one good example is audio “voice codecs” for telecom, in which every snippet of audio is encoded as a phoneme-like symbol, such that noise comes out the other end as a gobbledygook that sounds a lot like words. But much more interesting to me is a lossless domain-specific compressor, which I’m not aware of any examples of. (Please tell me if about some if you know them!) In such a compressor, the input would be “rebased” against the model, such that the output looks like an instruction stream ala “start with [this synthesized template with these parameters]; and then add point-modifications {A, B, C, ...} to get from there back to the original input.” So inputs that were more like something it “understood” would compress well (i.e. just be described by a template) while inputs that were less-well-understood wouldn’t get much from the procedural-generation phase, and would end up being specified entirely as point-corrections, and so not really compress at all.
- imurray 7y ago
  A relevant NeurIPS 2019 paper for lossless compression (for e.g., medical imaging): https://papers.nips.cc/paper/9383-integer-discrete-flows-and-lossless-compression https://papers.nips.cc/paper/9383-integer-discrete-flows-and...
- jfkebwjsbx 7y ago
  FLAC for audio is one
  - gugagore 7y ago
    That was my first thought too. To see how it leverages domain-specific properties, see: https://xiph.org/flac/format.html#interchannel https://xiph.org/flac/format.html#interchannel https://xiph.org/flac/format.html#prediction https://xiph.org/flac/format.html#prediction
- sdenton4 7y ago
  It's pretty easy to build a lossless learned encoder. Encode the image with your model, then apply a lossless encoding to the error. You can also include a measure of the entropy of the error in your model's loss function. The model should make the errors small, and including the entropy measure of the error in the loss function means that the model produces "cheaper to correct" errors. A nice keyword on this front is "bits back argument".
  - barrkel 7y ago
    That's a nice theory, but what if your ML encoder uses textures to encode perceptual equivalence efficiently that's actually got very high errors measured by pixel? It seems to me the best ML-driven encoders would work by inferring higher abstraction details from the image in ways which are hard to get back to lossless - that there's an intrinsic tension there.
    - fwip 7y ago
      Then choose a different encoder, or add another layer to shift/correct the texture.
- woeirua 7y ago
  What you're basically describing is compressive sensing. The basic idea is that if you can find a domain in which your data is maximally sparse, then you can transmit the same information with many fewer bits. Unless you're extremely lucky with your basis functions, it is not strictly speaking a lossless compression algorithm.
- leijurv 7y ago
  Yes, exactly! One of the winning entries at http://prize.hutter1.net/ http://prize.hutter1.net/ (challenge to compress English Wikipedia) is/was a LSTM that predicted the next character then used arithmetic coding on the final bitstream (correct bit / incorrect bit). Specifically, see https://arxiv.org/abs/1609.07959 https://arxiv.org/abs/1609.07959
- 0-_-0 7y ago
  > But much more interesting to me is a lossless domain-specific compressor, which I’m not aware of any examples of. (Please tell me if about some if you know them!) A bit academic, but if you check out this paper from 2015 https://arxiv.org/pdf/1502.04623.pdf https://arxiv.org/pdf/1502.04623.pdf on a neural network trained on binarised MNIST (28x28 pixel images), Table 2 quotes entropy figures of 116.815 after converting to bits, which would be 28*28/116.815=6.71x compression (if used as a compressor)
dabei 7y ago
The tricky part is how to measure the perceived quality. With video there is VMAF but seems there has been no development on the image quality assessment.
andbberger 7y ago
In theory it is a great fit for ML. But not for the reasons you're suggesting. The success of a compression scheme ultimately comes down to its ability to estimate p(x), for some sample x. And estimating p(x) is precisely the problem the field of ML addresses.
taneq 7y ago
> Taken to the extreme, a sufficiently advanced AI compression system could theoretically end up storing representations of high-level features like faces in the compressed data I think that's the major risk for straightforward "AI" compression. You'll end up with some high level description of the scene and the decompressed image will then be an "artist's impression" which contains details that may not have been in the original image. You'll take a photo of a cheque, compress it, decompress it, and you'll get a picture of a cheque, but it might not be the same cheque.
- deleted 7y ago
  [deleted]
bllguo 7y ago
the idea of learning as compression is actually an old one. Take a look at https://en.wikipedia.org/wiki/Minimum_description_length https://en.wikipedia.org/wiki/Minimum_description_length

derefr 7y ago

> Taken to the extreme, a sufficiently advanced AI compression system could theoretically end up storing representations of high-level features like faces in the compressed data Or, you could have a compression system created specifically for photographic images (as opposed to a “general compressor” like LZ77), where these high-level features are part of the (de)compressor itself. Now, lossy domain-specific compressors are pretty common; one good example is audio “voice codecs” for telecom, in which every snippet of audio is encoded as a phoneme-like symbol, such that noise comes out the other end as a gobbledygook that sounds a lot like words. But much more interesting to me is a lossless domain-specific compressor, which I’m not aware of any examples of. (Please tell me if about some if you know them!) In such a compressor, the input would be “rebased” against the model, such that the output looks like an instruction stream ala “start with [this synthesized template with these parameters]; and then add point-modifications {A, B, C, ...} to get from there back to the original input.” So inputs that were more like something it “understood” would compress well (i.e. just be described by a template) while inputs that were less-well-understood wouldn’t get much from the procedural-generation phase, and would end up being specified entirely as point-corrections, and so not really compress at all.

imurray 7y ago
A relevant NeurIPS 2019 paper for lossless compression (for e.g., medical imaging): https://papers.nips.cc/paper/9383-integer-discrete-flows-and-lossless-compression https://papers.nips.cc/paper/9383-integer-discrete-flows-and...
jfkebwjsbx 7y ago
FLAC for audio is one
- gugagore 7y ago
  That was my first thought too. To see how it leverages domain-specific properties, see: https://xiph.org/flac/format.html#interchannel https://xiph.org/flac/format.html#interchannel https://xiph.org/flac/format.html#prediction https://xiph.org/flac/format.html#prediction
sdenton4 7y ago
It's pretty easy to build a lossless learned encoder. Encode the image with your model, then apply a lossless encoding to the error. You can also include a measure of the entropy of the error in your model's loss function. The model should make the errors small, and including the entropy measure of the error in the loss function means that the model produces "cheaper to correct" errors. A nice keyword on this front is "bits back argument".
- barrkel 7y ago
  That's a nice theory, but what if your ML encoder uses textures to encode perceptual equivalence efficiently that's actually got very high errors measured by pixel? It seems to me the best ML-driven encoders would work by inferring higher abstraction details from the image in ways which are hard to get back to lossless - that there's an intrinsic tension there.
  - fwip 7y ago
    Then choose a different encoder, or add another layer to shift/correct the texture.
woeirua 7y ago
What you're basically describing is compressive sensing. The basic idea is that if you can find a domain in which your data is maximally sparse, then you can transmit the same information with many fewer bits. Unless you're extremely lucky with your basis functions, it is not strictly speaking a lossless compression algorithm.
leijurv 7y ago
Yes, exactly! One of the winning entries at http://prize.hutter1.net/ http://prize.hutter1.net/ (challenge to compress English Wikipedia) is/was a LSTM that predicted the next character then used arithmetic coding on the final bitstream (correct bit / incorrect bit). Specifically, see https://arxiv.org/abs/1609.07959 https://arxiv.org/abs/1609.07959
0-_-0 7y ago
> But much more interesting to me is a lossless domain-specific compressor, which I’m not aware of any examples of. (Please tell me if about some if you know them!) A bit academic, but if you check out this paper from 2015 https://arxiv.org/pdf/1502.04623.pdf https://arxiv.org/pdf/1502.04623.pdf on a neural network trained on binarised MNIST (28x28 pixel images), Table 2 quotes entropy figures of 116.815 after converting to bits, which would be 28*28/116.815=6.71x compression (if used as a compressor)

dabei 7y ago

The tricky part is how to measure the perceived quality. With video there is VMAF but seems there has been no development on the image quality assessment.

andbberger 7y ago

In theory it is a great fit for ML. But not for the reasons you're suggesting. The success of a compression scheme ultimately comes down to its ability to estimate p(x), for some sample x. And estimating p(x) is precisely the problem the field of ML addresses.

taneq 7y ago

> Taken to the extreme, a sufficiently advanced AI compression system could theoretically end up storing representations of high-level features like faces in the compressed data I think that's the major risk for straightforward "AI" compression. You'll end up with some high level description of the scene and the decompressed image will then be an "artist's impression" which contains details that may not have been in the original image. You'll take a photo of a cheque, compress it, decompress it, and you'll get a picture of a cheque, but it might not be the same cheque.

deleted 7y ago
[deleted]

bllguo 7y ago

the idea of learning as compression is actually an old one. Take a look at https://en.wikipedia.org/wiki/Minimum_description_length https://en.wikipedia.org/wiki/Minimum_description_length