4 ms·
> A pipedream would be an entirely differentiable image compression pipeline where all the steps can be fine tuned together to optimize a particular image with
by londons_explore 6y ago
> A pipedream would be an entirely differentiable image compression pipeline where all the steps can be fine tuned together to optimize a particular image with respect to any differentiable loss function.
Neural Image Compression? https://arxiv.org/abs/1908.08988 https://arxiv.org/abs/1908.08988
- amelius 6y agoI fear that you might end up with hallucinations in your images ...
- 0-_-0 6y agoIsn't that a feature?
- IanCal 6y agoSimilar to imagining Ryan Gosling is in your background https://petapixel.com/2020/08/17/gigapixel-ai-accidentally-added-ryan-goslings-face-to-this-photo/ https://petapixel.com/2020/08/17/gigapixel-ai-accidentally-a...
- alanbernstein 6y agoRyan Gosling? That's quite a stretch.
- MiroF 6y agoIt's not trained to generate faces, so I don't think so.
- tonic_section 6y agoUnfortunately you wouldn't have any guarantees on the output of any particular image though, just some reassurances about the expected behaviour over the training set.
- MiroF 6y ago> any guarantees on the output of any particular image though You don't have any guarantees with this non-convex optimization. I think most of these methods would work OK on out-of-domain data.
- tonic_section 6y agoIn terms of the decoded image, yes - it's very unlikely you would get something substantially different from the original image. But in terms of the bitrate it's not hard to find examples where the compressed bitrate can be several standard deviations above the average bitrate on the training set - see e.g. the last example here: https://github.com/Justin-Tan/high-fidelity-generative-compression https://github.com/Justin-Tan/high-fidelity-generative-compr... (Lossy) neural compression methods may also synthesize small portions of an image to avoid compression artefacts associated with standard image codecs, so should definitely not be used in sensitive applications where small details can make a big difference such as security imaging, guarantees or none.
- MiroF 6y agoThose are very cool examples and of considerably higher quality+bitrate than when I last tuned into this field half a year ago. Unrelated, but I actually recognize your name from Github - I guess deep image compression is a pretty small space.
- underanalyzer 6y agoYea neural image compression looks pretty neat! The reason I bring up jpeg is because it's so well established and if you come up with a more optimized jpeg (which I'm not really convinced is possible, again a pipedream) you don't have to force people to transition to a new image format. In the end methods like neural style or whatever comes after are probably the better pick but there is a transition period.
- MiroF 6y ago> more optimized jpeg The thing about compression is that there is no single "more optimized" knob - there's a bunch of different tradeoffs. Want a compression algo that can compress existing images to smaller sizes than JPEG? You can already do that with neural image compression. Want a compression algo that can decode that compressed image in 0.01 seconds? You need JPEG.
- underanalyzer 6y agoSorry I could have been more specific. By more optimized jpeg I meant better perceptual quality (again subjective) within the confines of what a jpeg decoder could understand. To co opt your knobs analogy I imagine each of the steps of a complex image compression pipeline comes with its own knobs each with its own tradeoffs. The dream here would be to tune all those knobs at the same time to optimize some sense of quality in a particular image. Of course huge disclaimer I’m not an image or signal processing expert. It’s also very possible that these “knobs” have been tuned well enough so that even if we optimized them for a specific image the quality difference would not be noticeable.
- MiroF 6y agoThe problem here is that the JPEG decoder is not differentiable.
- MiroF 6y ago> entirely differentiable image compression pipeline Depending on what is meant by entirely differentiable, this might be impossible without relaxation. ie. you can't differentiate through the quantization step
- tonic_section 6y agoThere are a couple of solutions which work empirically - as you mentioned, one solution is a dithering-like differentiable relaxation where uniform noise is added, which simulates quantization, or just to ignore the quantization operation when taking gradients, essentially treating it as an identity operation in the backward pass.
- MiroF 6y agoBut how do you optimize the lossless encoding of the quantized latent space? ie. how do you tell the encoder to produce something that can be well encoded, given that the encoding is a bunch of discrete steps.
- tonic_section 6y agoUsually the lossless encoding is offloaded to a standard entropy coder, e.g. arithmetic, ANS, etc. because these approach the theoretical minimum rate given by the source entropy pretty closely, so there wouldn't be a point building a fancy differentiable replacement.
- MiroF 6y agoThat makes sense, I don't think I stated my question very clearly: how do you control/optimize the entropy of the latent space? ie. what stops the network from laundering all of the information for reconstructing the image through a super high entropy latent space that is hard to code but allows it to reconstruct perfectly e: I guess I should just get up to date by reading some papers
- 6y ago