4 ms·
This is pretty neat. But is it just me or does the dog picture look better in JPEG? When zoomed in, the JPEG artifacts are quite apparent and the RNN produces
by emcq 10y ago
This is pretty neat. But is it just me or does the dog picture look better in JPEG?
When zoomed in, the JPEG artifacts are quite apparent and the RNN produces a much smoother image. However, to my eye when zoomed out the high frequency "noise", particularly in the snout area, looks better in JPEG. The RNN produces a somewhat blurrier image that reminds me of the soft focus effect.
- jcl 10y agoIt looks like they are using a "multi-scale structural similarity" (MS-SIM) metric as a proxy for how well a human will think a compressed image reproduces the original image. Both the JPEG and the RNN images are compressed to the same MS-SIM metric, which in this case makes the RNN image take 25% fewer bytes. Part of the reason JPEG looks better in this case is that they have chosen to feature a detail- and noise-rich portion of the image, which favors JPEG... High JPEG compression has a sharpening effect around edges, including "ringing" which hides well in noise. However, if you view the full-sized comparison of the dog images, you can see that JPEG has essentially no detail in the darker portions of the image. Presumably both the MS-SIM metric and humans would judge RNN better for these portions.
- highd 10y agoThanks for describing MS-SIM - I've always been concerned about using L2 or similar metrics for image similarity. Is there any work on learning an image similarity metric? I would think if you Mechanical Turk'd a bunch of A/B images (with or without perturbations of various sorts or even totally different images) and asked the user to score the similarity you should be able to learn a pretty interesting scoring.
- wyager 10y ago>High JPEG compression has a sharpening effect around edges, including "ringing" which hides well in noise. Ringing is the opposite of a sharpening effect, no? JPEG essentially acts as a low-pass filter over the DCT tiles of the image, so you get ringing because you eliminate the high-frequency components that make the edge sharp and you're left with a wavelet that extends past the sharp edge and isn't cancelled out the way it used to be.
- TD-Linux 10y agoNope, ringing is the effect of a high-pass (or band-pass) filter - it's the opposite of smoothing. JPEG looses information by quantization, not low-passing. It is true that small high frequency coefficients will get rounded to 0 during quantization, and this does save quite a few bits. But the ringing is just from being imprecise about each DCT coefficient, coding a value that is either too large or too small.
- gugagore 10y agoI don't think it makes sense to say that ringing is (necessarily) an effect of high-pass, low-pass, or band-pass filters. Any second-order system (that is undamped) could exhibit ringing. If I concatenate a low-pass filter with one of these ringing filters, I could get a ringing and low-pass filter
- 0x09 10y agoRemember that a brickwall band pass filter (low, high, or anywhere else) is equivalent to convolution with this https://www.dsprelated.com/josimages_new/mdft/img1768.png https://www.dsprelated.com/josimages_new/mdft/img1768.png in the spatial domain. Any time quantization is high enough for coefficients in a given band to become zero while the surrounding ones remain nonzero it will introduce at least some level of ringing. wyager accurately described why in terms of the DCT though -- cosines naturally "ring" and the only way a non-sinusoid can be represented is through the combination of many other terms, so any time a coefficient contributing to this shape is eliminated, the structure "decays" so to speak and the underlying sinusoids become part of the reconstruction.
- nomel 10y agoThe NN images, for sure, have a problem reproducing sharp edges/have less high frequency content. Just look at the grass. It appears to be a severely flawed metric.
- nomel 10y agoBy severely flawed metric, I meant, > Presumably both the MS-SIM metric and humans would judge RNN better for these portions. doesn't remotely hold for detailed/high contrast. If I can see a huge difference, and vastly prefer one over the other, the metric is not useful by its definition. Please don't get emotional about figures of merit. I'm guessing a genuinely useful figure of merit would have put both file sizes a bit closer together, and the NN would have shined, especially since it works so well in low contrast areas.
- deleted 10y ago[deleted]
- derefr 10y agoOne thing I've seen suggested for doing algorithms like seam carving, is creating a small embedded "noise texture sample library" that can be mixed and matched to synthesize "more of" a surrounding texture, rather than simply cloning that texture. You can, of course, recognize (and remove) textures using the convolution nodes of a convolutional NN, encode the texturing parameters as part of the output of the network, and then reverse the process with a deconvolutional NN, with the textures reappearing at the end. The deconvolution nodes are in effect equivalent to that "noise texture sample library." But this is, in essence, two algorithms: one that compresses textures, and then another that compresses "everything else" once you've subtracted the texture. Thinking about the second algorithm, I wonder whether you need to go to the trouble of an ML approach at all: a non-textured image would effectively be a smooth color gradient, perfectly amenable to wavelet compression.