7 ms·

Hi, author here! This is the demo of a research project demonstrating the potential of using GANs for compression. For this to be used in practice, more researc

Hi, author here! This is the demo of a research project demonstrating the potential of using GANs for compression. For this to be used in practice, more research is required to make the model smaller/faster, or to build a deticated chip! Currently, this runs at approximately 0.7 megapixel/sec on a GPU with unoptimized TensorFlow. Happy to answer any questions.

ksec 6y ago
>Currently, this runs at approximately 0.7 megapixel/sec on a GPU with unoptimized TensorFlow. Forgive me, is that in encoding or decoding? And size of library required to decode it? ( Edit: Looks like answered below ) It was only an hour ago I was looking at some pictures encoded with VVC / H.266 reference encoder, it was better than BPG / HEVC, but this HiFiC still beats it by quite a bit. This whole thing blows up my mind on limitation I thought we had on image compression. May be I should take back my words on JPEG XL is the future. This seems revolutionary.
- gtoderici 6y ago
  (coauthor here) The 0.7 megapixels/sec is PNG decode (to get input)+encoding+decoding+PNG encoding (to get output we can visualize in a browser) speed. Thanks for your kind comment!
- JyrkiAlakuijala 6y ago
  JPEG XL and HiFiC are developed in the same team. JPEG XL is just positioned for slightly less compute than HiFiC.
petermcneeley 6y ago
These are truly amazing results. Looking closely at the results vs original it would appear that much of the details are very different at almost a noise level. Is the perceptual evaluation allowing for these similar but completely different noise details?
- gtoderici 6y ago
  (coauthor here) We used an adversarial loss in addition to a perceptual loss and MSE. None of these work super-well when the others are not used. The adversarial loss "learns" what is a compressed image and tries to make the decoder go away from such outputs. The perceptual (LPIPS) is not so sensitive to pure noise and allows for it, but is sensitive to texture details. MSE tries to get the rough shape right. We also asked users in a study to tell us which images they preferred when having access to the original. Most prefer the added details even if they're not exactly right.
  - nbardy 6y ago
    > The adversarial loss "learns" what is a compressed image and tries to make the decoder go away from such outputs. Could you expand on this point
    - atorodius 6y ago
      The idea is that any distortion loss imposes specific artifacts. For example, MSE tends to blur outputs, CNN-feature-based losses (VGG, LPIPS) tend to produce gridding patterns or also blur. Now, when the discriminator network sees these artifacts, those artifacts very obviously distinguish the reconstructions from the input, and thus the gradients from the discriminator guide the optimization away from these artifacts. Let me know if this helps!
- fiddlerwoaroof 6y ago
  Yeah, it seems like the HiFiC version has the same large-scale structure, but the details are sort of "regenerated": e.g. it's as if the image format says "there's some hair here", and the decoder just paints in hair.
  - gtoderici 6y ago
    That was exactly the goal of the project! Basically if the size doesn't allow to have detail, we need to "hallucinate it". This of course is not necessary if there's enough bandwidth available for transmission or enough storage. On the other hand, in our paper we show that some generated detail can help even at higher bitrates.
UnbugMe 6y ago
The demo images used on hific.github.io appear to be part of the datasets used to train the system. In another comment you say the trained model is 726MB. The combined size of the training datasets appear to be about 8GB zipped. Is the currently trained model usable on images that are not part of the training datasets? with output of similar quality and size?
- atorodius 6y ago
  Hi- no the images on the demo page are not part of the training set, they are only used to evaluate the method. Arbitrary images of natural scenes will look similar at these bitrates. We‘ll release trained models and a colab soon!
  - UnbugMe 6y ago
    Thank you and my apologies. I should have read the pdf more carefully where the distinction between the training data and evaluation datasets is described Our training set consists of a large set of high-resolution images collected from theInternet, which we downscale to a random size between 500 and 1000 pixels, and then randomlycrop to256×256. We evaluate on three diverse benchmark datasets collected independently ofour training set to demonstrate that our method generalizes beyond the training distribution: thewidely usedKodak[22] dataset (24 images), as well as theCLIC2020[43] testset (428 images), andDIV2K[2] validation set (100 images).
cs702 6y ago
Great work. Thank you for making it so accessible online and sharing it on HN! The first thought I had, after looking at some of the examples, is that it should be fairly trivial to train or finetune the compression network to hallucinate missing detail with "BPG-style" or "JPG-style" smoothing artifacts by feeding compressed images as "real" samples to the adversarial network during training/finetuning. I wonder if doing that would make it possible to achieve even greater compression with loss of detail that is indistinguishable from traditional compression methods and therefore still acceptable to the human eye.
- gtoderici 6y ago
  What you suggest has already been done: train a neural network with the output of BPG or JPEG, and ask to reconstruct the input with just the decompressed pixels being available. It definitely is a valid approach but the limitation is that if the network needs some texture-specific information that cannot be extracted from the decoded pixels, it can't really do much. There were approaches where such information was also sent on the side, which yielded better results, of course. The field is wide-open and each approach has its own challenges (e.g., you may need to train one network for quantization level for example if you're going to do restoration).
keyi 6y ago
Have you compared the result with latest image compression formats, such as AVIF[0]? [0]: https://aomediacodec.github.io/av1-avif/ https://aomediacodec.github.io/av1-avif/
- gtoderici 6y ago
  We haven't specifically compared to AVIF, which as far as we know is still under development. We'd be happy to compare, but it's unlikely that we'd learn much out of it. As far as we know, AVIF is better by <100% than HEVC, but we're comparing against HEVC at 300% of the bitrate. Of course, we'd be happy to add any additional images from other codecs if they're available.
  - bscphil 6y ago
    I would add JPEG-XL in addition if you're looking for suggestions for other codecs to compare to. It's very competitive with AV1 and beats it, in my opinion, at higher bitrates. Admittedly, you're not likely to learn much from this that is useful for your research, but most of the interest from people clicking on this is probably wanting to see the latest developments in image compression.
magicalhippo 6y ago
Looks really interesting, thanks for sharing! One silly question, the red door example image... why is the red so saturated in all the compressed versions vs the original? edit: Ah seems to be some kind of display issue in Firefox. When saving the images and comparing the saturation level is roughly equal.
- atorodius 6y ago
  Interesting, I did not notice this before. This is likely due to the original having some color profile attached. Not sure why this only renders differently in Firefox.
belval 6y ago
Hi, could you give a general comparison vs CLIC workshop[1] submissions at CVPR? It seems to be the same end goal and CLIC was even sponsored by Google so I am a little puzzled as to why I didn't see your work there. EDIT: You mention CLIC2020 in your paper so I'm a bit more confused. Another question would be,what are your thoughts on the future of learned image compression and ideas for future work? [1] https://www.compression.cc/ https://www.compression.cc/

ksec 6y ago

>Currently, this runs at approximately 0.7 megapixel/sec on a GPU with unoptimized TensorFlow. Forgive me, is that in encoding or decoding? And size of library required to decode it? ( Edit: Looks like answered below ) It was only an hour ago I was looking at some pictures encoded with VVC / H.266 reference encoder, it was better than BPG / HEVC, but this HiFiC still beats it by quite a bit. This whole thing blows up my mind on limitation I thought we had on image compression. May be I should take back my words on JPEG XL is the future. This seems revolutionary.

gtoderici 6y ago
(coauthor here) The 0.7 megapixels/sec is PNG decode (to get input)+encoding+decoding+PNG encoding (to get output we can visualize in a browser) speed. Thanks for your kind comment!
JyrkiAlakuijala 6y ago
JPEG XL and HiFiC are developed in the same team. JPEG XL is just positioned for slightly less compute than HiFiC.

petermcneeley 6y ago

These are truly amazing results. Looking closely at the results vs original it would appear that much of the details are very different at almost a noise level. Is the perceptual evaluation allowing for these similar but completely different noise details?

gtoderici 6y ago
(coauthor here) We used an adversarial loss in addition to a perceptual loss and MSE. None of these work super-well when the others are not used. The adversarial loss "learns" what is a compressed image and tries to make the decoder go away from such outputs. The perceptual (LPIPS) is not so sensitive to pure noise and allows for it, but is sensitive to texture details. MSE tries to get the rough shape right. We also asked users in a study to tell us which images they preferred when having access to the original. Most prefer the added details even if they're not exactly right.
- nbardy 6y ago
  > The adversarial loss "learns" what is a compressed image and tries to make the decoder go away from such outputs. Could you expand on this point
  - atorodius 6y ago
    The idea is that any distortion loss imposes specific artifacts. For example, MSE tends to blur outputs, CNN-feature-based losses (VGG, LPIPS) tend to produce gridding patterns or also blur. Now, when the discriminator network sees these artifacts, those artifacts very obviously distinguish the reconstructions from the input, and thus the gradients from the discriminator guide the optimization away from these artifacts. Let me know if this helps!
fiddlerwoaroof 6y ago
Yeah, it seems like the HiFiC version has the same large-scale structure, but the details are sort of "regenerated": e.g. it's as if the image format says "there's some hair here", and the decoder just paints in hair.
- gtoderici 6y ago
  That was exactly the goal of the project! Basically if the size doesn't allow to have detail, we need to "hallucinate it". This of course is not necessary if there's enough bandwidth available for transmission or enough storage. On the other hand, in our paper we show that some generated detail can help even at higher bitrates.

UnbugMe 6y ago

The demo images used on hific.github.io appear to be part of the datasets used to train the system. In another comment you say the trained model is 726MB. The combined size of the training datasets appear to be about 8GB zipped. Is the currently trained model usable on images that are not part of the training datasets? with output of similar quality and size?

atorodius 6y ago
Hi- no the images on the demo page are not part of the training set, they are only used to evaluate the method. Arbitrary images of natural scenes will look similar at these bitrates. We‘ll release trained models and a colab soon!
- UnbugMe 6y ago
  Thank you and my apologies. I should have read the pdf more carefully where the distinction between the training data and evaluation datasets is described Our training set consists of a large set of high-resolution images collected from theInternet, which we downscale to a random size between 500 and 1000 pixels, and then randomlycrop to256×256. We evaluate on three diverse benchmark datasets collected independently ofour training set to demonstrate that our method generalizes beyond the training distribution: thewidely usedKodak[22] dataset (24 images), as well as theCLIC2020[43] testset (428 images), andDIV2K[2] validation set (100 images).

cs702 6y ago

Great work. Thank you for making it so accessible online and sharing it on HN! The first thought I had, after looking at some of the examples, is that it should be fairly trivial to train or finetune the compression network to hallucinate missing detail with "BPG-style" or "JPG-style" smoothing artifacts by feeding compressed images as "real" samples to the adversarial network during training/finetuning. I wonder if doing that would make it possible to achieve even greater compression with loss of detail that is indistinguishable from traditional compression methods and therefore still acceptable to the human eye.

gtoderici 6y ago
What you suggest has already been done: train a neural network with the output of BPG or JPEG, and ask to reconstruct the input with just the decompressed pixels being available. It definitely is a valid approach but the limitation is that if the network needs some texture-specific information that cannot be extracted from the decoded pixels, it can't really do much. There were approaches where such information was also sent on the side, which yielded better results, of course. The field is wide-open and each approach has its own challenges (e.g., you may need to train one network for quantization level for example if you're going to do restoration).

keyi 6y ago

Have you compared the result with latest image compression formats, such as AVIF[0]? [0]: https://aomediacodec.github.io/av1-avif/ https://aomediacodec.github.io/av1-avif/

gtoderici 6y ago
We haven't specifically compared to AVIF, which as far as we know is still under development. We'd be happy to compare, but it's unlikely that we'd learn much out of it. As far as we know, AVIF is better by <100% than HEVC, but we're comparing against HEVC at 300% of the bitrate. Of course, we'd be happy to add any additional images from other codecs if they're available.
- bscphil 6y ago
  I would add JPEG-XL in addition if you're looking for suggestions for other codecs to compare to. It's very competitive with AV1 and beats it, in my opinion, at higher bitrates. Admittedly, you're not likely to learn much from this that is useful for your research, but most of the interest from people clicking on this is probably wanting to see the latest developments in image compression.

magicalhippo 6y ago

Looks really interesting, thanks for sharing! One silly question, the red door example image... why is the red so saturated in all the compressed versions vs the original? edit: Ah seems to be some kind of display issue in Firefox. When saving the images and comparing the saturation level is roughly equal.

atorodius 6y ago
Interesting, I did not notice this before. This is likely due to the original having some color profile attached. Not sure why this only renders differently in Firefox.

belval 6y ago

Hi, could you give a general comparison vs CLIC workshop[1] submissions at CVPR? It seems to be the same end goal and CLIC was even sponsored by Google so I am a little puzzled as to why I didn't see your work there. EDIT: You mention CLIC2020 in your paper so I'm a bit more confused. Another question would be,what are your thoughts on the future of learned image compression and ideas for future work? [1] https://www.compression.cc/ https://www.compression.cc/