17 ms·

Show HN: HiFiC – High-Fidelity Generative Image Compression – demo and paper

281 points by atorodius 6y ago

deweller 6y ago
This is really impressive. But it raises some questions for me. What size library would be required to decode these type of images? And would the decoding library be updated on a regular basis? Would the image change when decoding library is updated? Would images be tagged with a version of the library when encoded (HiFiC v1.0.2?)
- ache7 6y ago
  And I also have questions about decompression speed and memory requirements.
- atorodius 6y ago
  Thanks for the kind words! > What size library would be required to decode these type of images? The model is 726MB. Keep in mind that this is a research project - further research needs to be done on how to get faster now that we know that this kind of results are possible! > And would the decoding library be updated on a regular basis? Only if you want even better reconstructions! > Would the image change when decoding library is updated? Would images be tagged with a version of the library when encoded (HiFiC v1.0.2?) Yes, some header would have to be added.
  - pornel 6y ago
    I'm very curious how such thing could be standardized as an image format. With classic image formats there's an expectation one can write a spec and an independent implementation from scratch. "Take my X-MB large pre-trained model" is unprecedented. Would it still be competitive with H.265 if the model was 10MB or 50MB in size? 0.7GB may be difficult for widespread adoption.
    - gtoderici 6y ago
      Independently of this work, we have models which are competitive with HEVC while being significantly smaller (this is from previous work). They will not look nearly as good as what you see in the website demo, but they're still better. I don't have any such model handy but perhaps it's 10x-20x smaller. We don't claim that this (or even the previous work) is the way to go forward for images, but we hope to incentivize more researchers to look in this direction so that we can figure out how to deploy these kinds of methods since the results they produce are very compelling.
mmastrac 6y ago
In addition to being an amazing result, props for including a variety different sets of skin tones in the hero sample box. It's extremely good to see that there was thought put into reducing any potential bias for this compression algorithm which has unfortunately been an issue in the history of image capture [1] [1] https://www.nytimes.com/2019/04/25/lens/sarah-lewis-racial-bias-photography.html https://www.nytimes.com/2019/04/25/lens/sarah-lewis-racial-b...
atorodius 6y ago
Hi, author here! This is the demo of a research project demonstrating the potential of using GANs for compression. For this to be used in practice, more research is required to make the model smaller/faster, or to build a deticated chip! Currently, this runs at approximately 0.7 megapixel/sec on a GPU with unoptimized TensorFlow. Happy to answer any questions.
- ksec 6y ago
  >Currently, this runs at approximately 0.7 megapixel/sec on a GPU with unoptimized TensorFlow. Forgive me, is that in encoding or decoding? And size of library required to decode it? ( Edit: Looks like answered below ) It was only an hour ago I was looking at some pictures encoded with VVC / H.266 reference encoder, it was better than BPG / HEVC, but this HiFiC still beats it by quite a bit. This whole thing blows up my mind on limitation I thought we had on image compression. May be I should take back my words on JPEG XL is the future. This seems revolutionary.
  - gtoderici 6y ago
    (coauthor here) The 0.7 megapixels/sec is PNG decode (to get input)+encoding+decoding+PNG encoding (to get output we can visualize in a browser) speed. Thanks for your kind comment!
  - JyrkiAlakuijala 6y ago
    JPEG XL and HiFiC are developed in the same team. JPEG XL is just positioned for slightly less compute than HiFiC.
- petermcneeley 6y ago
  These are truly amazing results. Looking closely at the results vs original it would appear that much of the details are very different at almost a noise level. Is the perceptual evaluation allowing for these similar but completely different noise details?
  - gtoderici 6y ago
ricardobeat 6y ago
This is amazing. Not only it completely blows JPEG and BPG out of the water in efficiency, but it makes some images look better than the original! It seems to reduce local contrast which works well for skin (photos 1, 2 and 5), and what it does for repeating patterns is quite pleasant to the eyes (1 and 4). The only downside I see is the introduction of artifacts like you can see on the right-side face of the clock tower.
- atorodius 6y ago
  Thanks for the kind words! We hope that the algorithm we presented is a step in the right direction, and we acknowledge that there's more work to be done! Just like with any algorithm there are ways in which it can be improved. Please check out the supplementary materials [1], to find some examples that can definitely be improved: small faces and small text. Overall, as you say, the algorithm does a great job though! [1] https://storage.googleapis.com/hific/data/appendixb/appendixb.pdf https://storage.googleapis.com/hific/data/appendixb/appendix...
- pbowyer 6y ago
  > it makes some images look better than the original! It seems to reduce local contrast which works well for skin (photos 1, 2 and 5) It's better than the lossy JPEGs but I strongly disagree it's better than the original - or even close. Examining image 1 closely, the loss of contrast and the regularising of the texture of the skin make the HiFIC image feel fake when I look at it. I downloaded and opened the images at random on my computer to get rid of the bias of "the right one is the original" and it was the original I preferred each time. Regardless, it's impressive work and I look forward to future developments!
  - ricardobeat 6y ago
    That effect is very close to what a photographer would aim for with manual editing. It is not about fidelity but that blemish-free look which you'll find on any magazine. The authors also mentioned that in their research most users preferred the compressed one!
crazygringo 6y ago
I've been waiting for something to implement this concept for so long, and I'm so happy to finally get a chance to explore how it works in practice! It's really fascinating to zoom comparing the compressed version with the original -- the "general idea" is all there and the quality is roughly the same, but strands of hair move to different positions, skin pores are in a completely different pattern, etc. -- it's all regenerated to look "right", but at the lowest level the texture isn't "real". Which is so fascinatingly different from JPEG or other lossy schemes -- the compression artifacts are obviously exactly that, you'd never confuse them for real detail. While here, you could never guess what is real detail vs. what is reconstructed detail. My hunch is that this won't actually take off at first for images, because images on the internet don't use up that much of our bandwidth anyways, and the performance and model storage requirements here are huge. BUT -- if the same idea can be applied to 1080p video with a temporal dimension, so that even if reconstructed textures are "false" they persist across frames, then suddenly the economics here would start to make sense: dedicated chips for encoding and decoding, with a built-in 10 GB model in ROM, trained on a decent proportion of every movie and TV show ever made... imagine if 2-hour movies were 200 MB instead of 2 GB? (And once we have chips for video, they could be trivially re-used for still images, much like Apple reuses h.265 for HEIC.)
- dividuum 6y ago
  Regarding your last paragraph: I wouldn't be surprised if existing "upscale" chips already work similar to what you suggested: They convert lower res content to (for example) 4K. See https://www.sony.com/electronics/4k-resolution-4k-upscaling-tvs https://www.sony.com/electronics/4k-resolution-4k-upscaling-... for an example.
- swiley 6y ago
  I think it will be extremely popular. Most people are concerned about how computers look. Wether or not the behavior is correct usually only bothers people who spend time working on computers. I’m not sure if it’s because the first group is just used to life being unpredictable and don’t realize things could be better, or if they really don’t care.
- ksec 6y ago
0-_-0 6y ago
I'm very impressed, I was waiting for an image codec that combines something like VGG loss + GANs! (Another thing that I'm waiting for is a neural JPEG decoder with a GAN, which would be backwards compatible with all the pictures already out there!) Now we need to get some massive standardisation process going to make this more practical and perfect it, just like it was done for JPEG in the old days! (And then do it for video and audio too!) What happens if you compress a noisy image? Does the compression denoise the image?
- gtoderici 6y ago
  On the standardization issue: the advantage of such a method that we presented is that as long as there exists a standard for model specification, we can encode every image with an arbitrary computational graph that can be linked from the container. Imagine being able to have domain specific models - say we could have a high accuracy/precision model for medical images (super-close to lossless), and one for low bandwidth applications where detail generation is paramount. Also imagine having a program written today (assuming the standard is out), and it being able to decode images created with a model invented 10 years from today doing things that were not even thought possible when the program was originally written. This should be possible because most of the low level building blocks (like convolution and other mathematical operations) is how we define new models! On noise: I'll let my coauthors find some links to noisy images to see what happens when you process those.
  - 0-_-0 6y ago
    Absolutely! Being able to improve a decoder for an existing encoder (and vica versa) is a great advantage!
- atorodius 6y ago
  Thanks! Noise is actually preserved really well, and is one of the strengths of using a GAN. Check out this visual example from the "All Evaluation Images" link: https://storage.googleapis.com/hific/clic2020/visualize.html?perPage=10&imgs=8c3f64635c07801bc2c9def490a27168 https://storage.googleapis.com/hific/clic2020/visualize.html...
jasonjayr 6y ago
This is impressive; but isn't this also susceptible to the type of bug that Xerox copiers got hit with many years ago? https://www.theregister.com/2013/08/06/xerox_copier_flaw_means_dodgy_numbers_and_dangerous_designs/ https://www.theregister.com/2013/08/06/xerox_copier_flaw_mea...
- atorodius 6y ago
  Small text is indeed one of the fail cases, and more research needs to be done here (see also my other comment [1]). We mention this issue in the supplementary materials [2], and you can check out an example here at [3] [1] https://news.ycombinator.com/item?id=23654161 https://news.ycombinator.com/item?id=23654161 [2] https://storage.googleapis.com/hific/data/appendixb/appendixb.pdf https://storage.googleapis.com/hific/data/appendixb/appendix... [3] https://storage.googleapis.com/hific/userstudy/visualize.html?perPage=1&page=5 https://storage.googleapis.com/hific/userstudy/visualize.htm...
  - mmastrac 6y ago
    This might be a good opportunity to lead to research on a "good faith watermark" for GAN-compressed images that may include hallucinated details.
    - gtoderici 6y ago
      One of the things we discussed to address this is to have the ability to: a) turn off detail hallucination completely given the same bitstream; and b) store the median/maximum absolute error across the image (b) should allow the user to determine whether the image is suitable for their use-case.
      - AbrahamParangi 6y ago
        You could use an OCR encoder and cat that to the feature vector. Great work by the way!
stereo 6y ago
The obvious question when looking at the comparison is, what kind of jpeg did they pick, and was it fair? It turns out that mozjpeg, which is pretty much state of the art, can squeeze out an extra 5% out of shades_jpg_4x_0.807.jpg. Your comparison would be even more impressive if you produced the best jpegs you can at those sizes.
- atorodius 6y ago
  Thank you for your suggestion. We use libjpeg for the Demo. Keep in mind that we compare against JPEG files which are at around 100%-400% of the size of the proposed method, so 5% would not really add much as far as we can tell.
lokl 6y ago
Has this been extended to video (beyond compressing each frame separately)?
- atorodius 6y ago
  This is the first time that GANs have been used to obtain high-fidelity reconstructions at useful bitrates. We haven't tried extending it to video.
roywiggins 6y ago
Finally someone's instantiated something like Vernor Vinge's "evocation" idea, quoted here: https://www.danvk.org/wp/2014-02-10/vernor-vinge-on-video-calls/index.html https://www.danvk.org/wp/2014-02-10/vernor-vinge-on-video-ca...
nullc 6y ago
Generally lossy compression methods have a 'knee' below which the percieved quality rapidly drops off. The default jpeg examples here are well below that knee. Usually comparing below the knee isn't very useful except to better help understand the sort of artefacts that one might want to also look for in higher rate images. It would be interesting to see some examples from a lower rate HiFiC specifically for the purpose of bringing out some artefacts to help inform comparisons at higher rates.
- ebg13 6y ago
  > The default jpeg examples here are well below that knee. The examples are chosen as multiples of their file size 1x,2x,...
  - nullc 6y ago
    I used the word 'default' for a reason! :)
    - atorodius 6y ago
      The HiFiC model we show is already the low model :) We show JPEG and BPG at the same rate to visualize how low the bitrate of our model actually is. And for JPEG and BPG we show 1x, 2x, and so on to visualize how many more bits the previous methods need to look similar visually.
      - nullc 6y ago
        Sure, but it isn't low rate enough to produce the level of gross artefacts needed to train the viewer to recognize faults in the images. E.g. after looking at those jpeg images you're able to flip to much higher rate jpegs and notice blocking artefacts that a completely unprepared subject would fail to notice. In my work on lossy codecs I found that having trained listeners was critical to detecting artifacts that would pass on quick inspection but would become annoying after prolonged exposure. From a marketing fairness point since the only codec you evaluate 'below knee' is jpeg, it risks exaggerating the difference. It would be just about as accurate to say that jpeg can't achieve those low rates-- since no one will use jpeg at a point where its falling apart. This work is impressive enough that it doesn't need any (accidental) exaggeration. I think it's best to either have all comparison points above the point where the codecs fall apart, or have a below-knee example for all so that it's clear how they fail and where they fail... rather than asking users to compare the gigantic difference between a failed and non-failing output.
fizixer 6y ago
Fantastic. When can we install the 1TiB video player/codec/plugin, that'll allow us to stream 4K movies on a 1Mbps connection?
- atorodius 6y ago
  Hopefully soon! However, in addition to research needed to make this work for video, it also needs research to make it smaller and/or to put it in silicon (see my other comments). The network shown is "only" 726MB BTW :)
isoprophlex 6y ago
Super exciting! Fantastic work and cool demo! I hope this can be 'stabilized' to allow video compression. If you run eg. style transfer networks on video, the artifacts in the outputted frames arent stable, they jump around from frame to frame. (Not sure if you can decode this at 24 fps on normal hardware, but still...)
- atorodius 6y ago
  Thanks! And agreed - doing is in real time for high-resolution videos still needs quite some research though.
  - mikepurvis 6y ago
    Shipping the network in device ROMs seems like a pretty far-off thing at this point, but I wonder if there could be something in the nearer term around time-shifting bandwidth usage, eg: - Media appliance under your TV downloads a 5GB updated network during the night so that you can use smaller-encoded streams of arbitrary content in the evening. - Spotify maintains a network on your phone that is updated while you're at home so that you can stream audio on the road with minimal data usage. - Your car has a network in it that is periodically updated over wifi and allows you to receive streetview images over a manufacturer-supplied M2M connection.
ebg13 6y ago
A number of comments here mention the regularity of the noise in the Lo variant as producing a slightly artificial appearance (zoom on the woman's forehead in picture 1). Would it improve anything to look for swathes where the dominant texture is noticeably uniform noise and mix it up a bit?
- gtoderici 6y ago
  Not sure how that would work out in practice. We kind of rushed with the experiments in any case - maybe we could let the methods train for longer. In the end the adversary might learn any regular patters that appear, therefore forcing the generator to come up with something that cannot be detected easily. (We trained everything to 1M steps. Perhaps letting it train to 2M would solve it.)
mchusma 6y ago
If you are into this you may also like to see a similar model facebook used for foveated rendering (Nov 2019): https://ai.facebook.com/blog/deepfovea-using-deep-learning-for-foveated-reconstruction-in-ar-vr/ https://ai.facebook.com/blog/deepfovea-using-deep-learning-f...
microcolonel 6y ago
Hmm, it's too bad that none of the methods (including the novel one) accurately represent her pupils. In the original image, you can tell which direction she's looking, and see how dilated her pupils are. What I feel could be a high priority is machine learning for rate distortion. It would be really good to have higher rate on the pupils.
- atorodius 6y ago
  FWIW, the network uses the hyperprior probability estimation network (citation [6] in the paper), which already adapts the rate depending on the image region.
littlestymaar 6y ago
Amazing work! It's something we knew would come at some point, but I didn't expect it to be that good so early! Since some of the authors are in this thread answering questions, I have one: I wonder if such project requires Google-scale infrastructure or if it's something that can easily be replicated. For instance, how big is the training set? And how much compute was necessary to train the model?
- atorodius 6y ago
  Hi! It depends on how far you want to go. For this project, we did a lot of exploration, because we had Google-scale infrastructure. Replicating all the exploration will need a lot of GPUs (in the 100s), replicating all the experiments that actually went into the paper maybe a few dozens. Training something similar as what's in the demo with the code we will release takes 1 V100 :) We can't release the internal training set, but expect a dataset of a few hundred thousand images (e.g., openimages) to be sufficient, maybe even less (AFAIK this has not been explored in a controlled setting).
  - littlestymaar 6y ago
    Thanks a lot!

281 points by atorodius 6y ago

This is really impressive. But it raises some questions for me. What size library would be required to decode these type of images? And would the decoding library be updated on a regular basis? Would the image change when decoding library is updated? Would images be tagged with a version of the library when encoded (HiFiC v1.0.2?)

ache7 6y ago
And I also have questions about decompression speed and memory requirements.
atorodius 6y ago
Thanks for the kind words! > What size library would be required to decode these type of images? The model is 726MB. Keep in mind that this is a research project - further research needs to be done on how to get faster now that we know that this kind of results are possible! > And would the decoding library be updated on a regular basis? Only if you want even better reconstructions! > Would the image change when decoding library is updated? Would images be tagged with a version of the library when encoded (HiFiC v1.0.2?) Yes, some header would have to be added.
- pornel 6y ago
  I'm very curious how such thing could be standardized as an image format. With classic image formats there's an expectation one can write a spec and an independent implementation from scratch. "Take my X-MB large pre-trained model" is unprecedented. Would it still be competitive with H.265 if the model was 10MB or 50MB in size? 0.7GB may be difficult for widespread adoption.
  - gtoderici 6y ago
    Independently of this work, we have models which are competitive with HEVC while being significantly smaller (this is from previous work). They will not look nearly as good as what you see in the website demo, but they're still better. I don't have any such model handy but perhaps it's 10x-20x smaller. We don't claim that this (or even the previous work) is the way to go forward for images, but we hope to incentivize more researchers to look in this direction so that we can figure out how to deploy these kinds of methods since the results they produce are very compelling.

mmastrac 6y ago

In addition to being an amazing result, props for including a variety different sets of skin tones in the hero sample box. It's extremely good to see that there was thought put into reducing any potential bias for this compression algorithm which has unfortunately been an issue in the history of image capture [1] [1] https://www.nytimes.com/2019/04/25/lens/sarah-lewis-racial-bias-photography.html https://www.nytimes.com/2019/04/25/lens/sarah-lewis-racial-b...

atorodius 6y ago

Hi, author here! This is the demo of a research project demonstrating the potential of using GANs for compression. For this to be used in practice, more research is required to make the model smaller/faster, or to build a deticated chip! Currently, this runs at approximately 0.7 megapixel/sec on a GPU with unoptimized TensorFlow. Happy to answer any questions.

ksec 6y ago
>Currently, this runs at approximately 0.7 megapixel/sec on a GPU with unoptimized TensorFlow. Forgive me, is that in encoding or decoding? And size of library required to decode it? ( Edit: Looks like answered below ) It was only an hour ago I was looking at some pictures encoded with VVC / H.266 reference encoder, it was better than BPG / HEVC, but this HiFiC still beats it by quite a bit. This whole thing blows up my mind on limitation I thought we had on image compression. May be I should take back my words on JPEG XL is the future. This seems revolutionary.
- gtoderici 6y ago
  (coauthor here) The 0.7 megapixels/sec is PNG decode (to get input)+encoding+decoding+PNG encoding (to get output we can visualize in a browser) speed. Thanks for your kind comment!
- JyrkiAlakuijala 6y ago
  JPEG XL and HiFiC are developed in the same team. JPEG XL is just positioned for slightly less compute than HiFiC.
petermcneeley 6y ago
These are truly amazing results. Looking closely at the results vs original it would appear that much of the details are very different at almost a noise level. Is the perceptual evaluation allowing for these similar but completely different noise details?
- gtoderici 6y ago

ricardobeat 6y ago

This is amazing. Not only it completely blows JPEG and BPG out of the water in efficiency, but it makes some images look better than the original! It seems to reduce local contrast which works well for skin (photos 1, 2 and 5), and what it does for repeating patterns is quite pleasant to the eyes (1 and 4). The only downside I see is the introduction of artifacts like you can see on the right-side face of the clock tower.

atorodius 6y ago
Thanks for the kind words! We hope that the algorithm we presented is a step in the right direction, and we acknowledge that there's more work to be done! Just like with any algorithm there are ways in which it can be improved. Please check out the supplementary materials [1], to find some examples that can definitely be improved: small faces and small text. Overall, as you say, the algorithm does a great job though! [1] https://storage.googleapis.com/hific/data/appendixb/appendixb.pdf https://storage.googleapis.com/hific/data/appendixb/appendix...
pbowyer 6y ago
> it makes some images look better than the original! It seems to reduce local contrast which works well for skin (photos 1, 2 and 5) It's better than the lossy JPEGs but I strongly disagree it's better than the original - or even close. Examining image 1 closely, the loss of contrast and the regularising of the texture of the skin make the HiFIC image feel fake when I look at it. I downloaded and opened the images at random on my computer to get rid of the bias of "the right one is the original" and it was the original I preferred each time. Regardless, it's impressive work and I look forward to future developments!
- ricardobeat 6y ago
  That effect is very close to what a photographer would aim for with manual editing. It is not about fidelity but that blemish-free look which you'll find on any magazine. The authors also mentioned that in their research most users preferred the compressed one!

crazygringo 6y ago

I've been waiting for something to implement this concept for so long, and I'm so happy to finally get a chance to explore how it works in practice! It's really fascinating to zoom comparing the compressed version with the original -- the "general idea" is all there and the quality is roughly the same, but strands of hair move to different positions, skin pores are in a completely different pattern, etc. -- it's all regenerated to look "right", but at the lowest level the texture isn't "real". Which is so fascinatingly different from JPEG or other lossy schemes -- the compression artifacts are obviously exactly that, you'd never confuse them for real detail. While here, you could never guess what is real detail vs. what is reconstructed detail. My hunch is that this won't actually take off at first for images, because images on the internet don't use up that much of our bandwidth anyways, and the performance and model storage requirements here are huge. BUT -- if the same idea can be applied to 1080p video with a temporal dimension, so that even if reconstructed textures are "false" they persist across frames, then suddenly the economics here would start to make sense: dedicated chips for encoding and decoding, with a built-in 10 GB model in ROM, trained on a decent proportion of every movie and TV show ever made... imagine if 2-hour movies were 200 MB instead of 2 GB? (And once we have chips for video, they could be trivially re-used for still images, much like Apple reuses h.265 for HEIC.)

dividuum 6y ago
Regarding your last paragraph: I wouldn't be surprised if existing "upscale" chips already work similar to what you suggested: They convert lower res content to (for example) 4K. See https://www.sony.com/electronics/4k-resolution-4k-upscaling-tvs https://www.sony.com/electronics/4k-resolution-4k-upscaling-... for an example.
swiley 6y ago
I think it will be extremely popular. Most people are concerned about how computers look. Wether or not the behavior is correct usually only bothers people who spend time working on computers. I’m not sure if it’s because the first group is just used to life being unpredictable and don’t realize things could be better, or if they really don’t care.
ksec 6y ago

0-_-0 6y ago

I'm very impressed, I was waiting for an image codec that combines something like VGG loss + GANs! (Another thing that I'm waiting for is a neural JPEG decoder with a GAN, which would be backwards compatible with all the pictures already out there!) Now we need to get some massive standardisation process going to make this more practical and perfect it, just like it was done for JPEG in the old days! (And then do it for video and audio too!) What happens if you compress a noisy image? Does the compression denoise the image?

gtoderici 6y ago
On the standardization issue: the advantage of such a method that we presented is that as long as there exists a standard for model specification, we can encode every image with an arbitrary computational graph that can be linked from the container. Imagine being able to have domain specific models - say we could have a high accuracy/precision model for medical images (super-close to lossless), and one for low bandwidth applications where detail generation is paramount. Also imagine having a program written today (assuming the standard is out), and it being able to decode images created with a model invented 10 years from today doing things that were not even thought possible when the program was originally written. This should be possible because most of the low level building blocks (like convolution and other mathematical operations) is how we define new models! On noise: I'll let my coauthors find some links to noisy images to see what happens when you process those.
- 0-_-0 6y ago
  Absolutely! Being able to improve a decoder for an existing encoder (and vica versa) is a great advantage!
atorodius 6y ago
Thanks! Noise is actually preserved really well, and is one of the strengths of using a GAN. Check out this visual example from the "All Evaluation Images" link: https://storage.googleapis.com/hific/clic2020/visualize.html?perPage=10&imgs=8c3f64635c07801bc2c9def490a27168 https://storage.googleapis.com/hific/clic2020/visualize.html...

jasonjayr 6y ago

This is impressive; but isn't this also susceptible to the type of bug that Xerox copiers got hit with many years ago? https://www.theregister.com/2013/08/06/xerox_copier_flaw_means_dodgy_numbers_and_dangerous_designs/ https://www.theregister.com/2013/08/06/xerox_copier_flaw_mea...

atorodius 6y ago
Small text is indeed one of the fail cases, and more research needs to be done here (see also my other comment [1]). We mention this issue in the supplementary materials [2], and you can check out an example here at [3] [1] https://news.ycombinator.com/item?id=23654161 https://news.ycombinator.com/item?id=23654161 [2] https://storage.googleapis.com/hific/data/appendixb/appendixb.pdf https://storage.googleapis.com/hific/data/appendixb/appendix... [3] https://storage.googleapis.com/hific/userstudy/visualize.html?perPage=1&page=5 https://storage.googleapis.com/hific/userstudy/visualize.htm...
- mmastrac 6y ago
  This might be a good opportunity to lead to research on a "good faith watermark" for GAN-compressed images that may include hallucinated details.
  - gtoderici 6y ago
    One of the things we discussed to address this is to have the ability to: a) turn off detail hallucination completely given the same bitstream; and b) store the median/maximum absolute error across the image (b) should allow the user to determine whether the image is suitable for their use-case.
    - AbrahamParangi 6y ago
      You could use an OCR encoder and cat that to the feature vector. Great work by the way!

stereo 6y ago

The obvious question when looking at the comparison is, what kind of jpeg did they pick, and was it fair? It turns out that mozjpeg, which is pretty much state of the art, can squeeze out an extra 5% out of shades_jpg_4x_0.807.jpg. Your comparison would be even more impressive if you produced the best jpegs you can at those sizes.

atorodius 6y ago
Thank you for your suggestion. We use libjpeg for the Demo. Keep in mind that we compare against JPEG files which are at around 100%-400% of the size of the proposed method, so 5% would not really add much as far as we can tell.

lokl 6y ago

Has this been extended to video (beyond compressing each frame separately)?

atorodius 6y ago
This is the first time that GANs have been used to obtain high-fidelity reconstructions at useful bitrates. We haven't tried extending it to video.

roywiggins 6y ago

Finally someone's instantiated something like Vernor Vinge's "evocation" idea, quoted here: https://www.danvk.org/wp/2014-02-10/vernor-vinge-on-video-calls/index.html https://www.danvk.org/wp/2014-02-10/vernor-vinge-on-video-ca...

nullc 6y ago

Generally lossy compression methods have a 'knee' below which the percieved quality rapidly drops off. The default jpeg examples here are well below that knee. Usually comparing below the knee isn't very useful except to better help understand the sort of artefacts that one might want to also look for in higher rate images. It would be interesting to see some examples from a lower rate HiFiC specifically for the purpose of bringing out some artefacts to help inform comparisons at higher rates.

ebg13 6y ago
> The default jpeg examples here are well below that knee. The examples are chosen as multiples of their file size 1x,2x,...
- nullc 6y ago
  I used the word 'default' for a reason! :)
  - atorodius 6y ago
    The HiFiC model we show is already the low model :) We show JPEG and BPG at the same rate to visualize how low the bitrate of our model actually is. And for JPEG and BPG we show 1x, 2x, and so on to visualize how many more bits the previous methods need to look similar visually.
    - nullc 6y ago
      Sure, but it isn't low rate enough to produce the level of gross artefacts needed to train the viewer to recognize faults in the images. E.g. after looking at those jpeg images you're able to flip to much higher rate jpegs and notice blocking artefacts that a completely unprepared subject would fail to notice. In my work on lossy codecs I found that having trained listeners was critical to detecting artifacts that would pass on quick inspection but would become annoying after prolonged exposure. From a marketing fairness point since the only codec you evaluate 'below knee' is jpeg, it risks exaggerating the difference. It would be just about as accurate to say that jpeg can't achieve those low rates-- since no one will use jpeg at a point where its falling apart. This work is impressive enough that it doesn't need any (accidental) exaggeration. I think it's best to either have all comparison points above the point where the codecs fall apart, or have a below-knee example for all so that it's clear how they fail and where they fail... rather than asking users to compare the gigantic difference between a failed and non-failing output.

fizixer 6y ago

Fantastic. When can we install the 1TiB video player/codec/plugin, that'll allow us to stream 4K movies on a 1Mbps connection?

atorodius 6y ago
Hopefully soon! However, in addition to research needed to make this work for video, it also needs research to make it smaller and/or to put it in silicon (see my other comments). The network shown is "only" 726MB BTW :)

isoprophlex 6y ago

Super exciting! Fantastic work and cool demo! I hope this can be 'stabilized' to allow video compression. If you run eg. style transfer networks on video, the artifacts in the outputted frames arent stable, they jump around from frame to frame. (Not sure if you can decode this at 24 fps on normal hardware, but still...)

atorodius 6y ago
Thanks! And agreed - doing is in real time for high-resolution videos still needs quite some research though.
- mikepurvis 6y ago
  Shipping the network in device ROMs seems like a pretty far-off thing at this point, but I wonder if there could be something in the nearer term around time-shifting bandwidth usage, eg: - Media appliance under your TV downloads a 5GB updated network during the night so that you can use smaller-encoded streams of arbitrary content in the evening. - Spotify maintains a network on your phone that is updated while you're at home so that you can stream audio on the road with minimal data usage. - Your car has a network in it that is periodically updated over wifi and allows you to receive streetview images over a manufacturer-supplied M2M connection.

ebg13 6y ago

A number of comments here mention the regularity of the noise in the Lo variant as producing a slightly artificial appearance (zoom on the woman's forehead in picture 1). Would it improve anything to look for swathes where the dominant texture is noticeably uniform noise and mix it up a bit?

gtoderici 6y ago
Not sure how that would work out in practice. We kind of rushed with the experiments in any case - maybe we could let the methods train for longer. In the end the adversary might learn any regular patters that appear, therefore forcing the generator to come up with something that cannot be detected easily. (We trained everything to 1M steps. Perhaps letting it train to 2M would solve it.)

mchusma 6y ago

If you are into this you may also like to see a similar model facebook used for foveated rendering (Nov 2019): https://ai.facebook.com/blog/deepfovea-using-deep-learning-for-foveated-reconstruction-in-ar-vr/ https://ai.facebook.com/blog/deepfovea-using-deep-learning-f...

microcolonel 6y ago

Hmm, it's too bad that none of the methods (including the novel one) accurately represent her pupils. In the original image, you can tell which direction she's looking, and see how dilated her pupils are. What I feel could be a high priority is machine learning for rate distortion. It would be really good to have higher rate on the pupils.

atorodius 6y ago
FWIW, the network uses the hyperprior probability estimation network (citation [6] in the paper), which already adapts the rate depending on the image region.

littlestymaar 6y ago

Amazing work! It's something we knew would come at some point, but I didn't expect it to be that good so early! Since some of the authors are in this thread answering questions, I have one: I wonder if such project requires Google-scale infrastructure or if it's something that can easily be replicated. For instance, how big is the training set? And how much compute was necessary to train the model?

atorodius 6y ago
Hi! It depends on how far you want to go. For this project, we did a lot of exploration, because we had Google-scale infrastructure. Replicating all the exploration will need a lot of GPUs (in the 100s), replicating all the experiments that actually went into the paper maybe a few dozens. Training something similar as what's in the demo with the code we will release takes 1 V100 :) We can't release the internal training set, but expect a dataset of a few hundred thousand images (e.g., openimages) to be sufficient, maybe even less (AFAIK this has not been explored in a controlled setting).
- littlestymaar 6y ago
  Thanks a lot!