4 ms·

These are truly amazing results. Looking closely at the results vs original it would appear that much of the details are very different at almost a noise level.

These are truly amazing results. Looking closely at the results vs original it would appear that much of the details are very different at almost a noise level. Is the perceptual evaluation allowing for these similar but completely different noise details?

gtoderici 6y ago
(coauthor here) We used an adversarial loss in addition to a perceptual loss and MSE. None of these work super-well when the others are not used. The adversarial loss "learns" what is a compressed image and tries to make the decoder go away from such outputs. The perceptual (LPIPS) is not so sensitive to pure noise and allows for it, but is sensitive to texture details. MSE tries to get the rough shape right. We also asked users in a study to tell us which images they preferred when having access to the original. Most prefer the added details even if they're not exactly right.
- nbardy 6y ago
  > The adversarial loss "learns" what is a compressed image and tries to make the decoder go away from such outputs. Could you expand on this point
  - atorodius 6y ago
    The idea is that any distortion loss imposes specific artifacts. For example, MSE tends to blur outputs, CNN-feature-based losses (VGG, LPIPS) tend to produce gridding patterns or also blur. Now, when the discriminator network sees these artifacts, those artifacts very obviously distinguish the reconstructions from the input, and thus the gradients from the discriminator guide the optimization away from these artifacts. Let me know if this helps!
fiddlerwoaroof 6y ago
Yeah, it seems like the HiFiC version has the same large-scale structure, but the details are sort of "regenerated": e.g. it's as if the image format says "there's some hair here", and the decoder just paints in hair.
- gtoderici 6y ago
  That was exactly the goal of the project! Basically if the size doesn't allow to have detail, we need to "hallucinate it". This of course is not necessary if there's enough bandwidth available for transmission or enough storage. On the other hand, in our paper we show that some generated detail can help even at higher bitrates.

gtoderici 6y ago

(coauthor here) We used an adversarial loss in addition to a perceptual loss and MSE. None of these work super-well when the others are not used. The adversarial loss "learns" what is a compressed image and tries to make the decoder go away from such outputs. The perceptual (LPIPS) is not so sensitive to pure noise and allows for it, but is sensitive to texture details. MSE tries to get the rough shape right. We also asked users in a study to tell us which images they preferred when having access to the original. Most prefer the added details even if they're not exactly right.

nbardy 6y ago
> The adversarial loss "learns" what is a compressed image and tries to make the decoder go away from such outputs. Could you expand on this point
- atorodius 6y ago
  The idea is that any distortion loss imposes specific artifacts. For example, MSE tends to blur outputs, CNN-feature-based losses (VGG, LPIPS) tend to produce gridding patterns or also blur. Now, when the discriminator network sees these artifacts, those artifacts very obviously distinguish the reconstructions from the input, and thus the gradients from the discriminator guide the optimization away from these artifacts. Let me know if this helps!

fiddlerwoaroof 6y ago

Yeah, it seems like the HiFiC version has the same large-scale structure, but the details are sort of "regenerated": e.g. it's as if the image format says "there's some hair here", and the decoder just paints in hair.

gtoderici 6y ago
That was exactly the goal of the project! Basically if the size doesn't allow to have detail, we need to "hallucinate it". This of course is not necessary if there's enough bandwidth available for transmission or enough storage. On the other hand, in our paper we show that some generated detail can help even at higher bitrates.