18 ms·
The dangers behind image resizing (2021)
- hgomersall 4y agoThe bigger problem is that the pixel domain is not a very good domain to be operating in. How many hours and of training and thousands of images are used to essentially learn about Gabor filters.
- thr0wnawaytod4y 4y agoCame here for a new ImageTragick but got actual resizing problems
- account42 4y ago> The definition of scaling function is mathematical and should never be a function of the library being used. Horseshit. Image resizing or any other kind of resampling is essentially always about filling in missing information. The is no mathematical model that will tell you for certain what the missing information is.
- deleted 4y ago[deleted]
- planede 4y agoArguably downscaling does not fill in missing information, it only throws away information. Still, implementations vary a lot here. There might not be a consensus of a unique correct way to do downscaling, but there are certain things that you certainly don't want to do. Like doing naive linear arithmetic on sRGB color values.
- HPsquared 4y agoInterpolation is still filling in missing information, it's just possible to get a pretty good estimate.
- willis936 4y agoThis is wrong. Interpolation below Nyquist (downsampling) results in a subset of the original Information (capital I information theory information).
- astrange 4y agoImages aren't bandlimited so the conditions don't apply for that. That's why a vector image rendered at 128x128 can look better/sharper than one rendered at 256x256 and scaled down.
- willis936 4y agoThey are band-limited. That's why you get aliasing when taking unfiltered photos above Nyquist without AA filters. In your example the lower res image would be using most of its bandwidth while the higher res image would be using almost none of its bandwidth. Images are 2D discrete signals. Everything you know about 1D DSP applies to them.
- orlp 4y agoOne interesting complication for a lot of photos is that the bandwidth of the green channel is twice as high as the red and blue channels due to the Bayer filter mosaic.
- astrange 4y agoIf some of the edges are infinitely sharp, and you know which ones they are by looking at them, as in my example, then it's using more than all its bandwidth at any resolution.
- willis936 4y agoThat's true in the 1D case as well. That requires upsampling with information generation before downsampling. Using priori to guess missing information is a task that will never be finished and is interesting. It isn't necessary for a satisfactory downsampling result.
- Gordonjcp 4y agoAha, no! Downscaling *into a discrete space by an arbitrary amount* is absolutely filling in missing information. Take the naive case where you downscale a line of four pixels to two pixels - you can simply discard two of them so you go from `0,1,2,3` to `0,2`. It looks okay. But what happens if you want to scale four pixels to three? You could simply throw one away but then things will look wobbly and lumpy. So you need to take your four pixels, and fill in a missing value that lands slap bang between 1 and 2. Worse, you actually need to treat 0 and 3 as missing values too because they will be somewhat affected by spreading them into the middle pixel. So yes, downscaling does have to compute missing values even in your naive linear interpolation!
- im3w1l 4y agoFor downscaling, area averaging is simple and makes a lot of intuitive sense and gives good results. To me it's basically the definition of downscaling. Like yeah, you can try to get clever and preserve the artistic intent or something with something like seamcarving but then I wouldn't call it downscaling anymore.
- meindnoch 4y ago>Take the naive case where you downscale a line of four pixels to two pixels - you can simply discard two of them so you go from `0,1,2,3` to `0,2`. It looks okay. This is already wrong, unless the pixels are band-limited to Nyquist/4. Trivial example where this is not true: 1 0 1 0 If such a signal is decimated by 2 you get 1 1 Which is not correct.
- deleted 4y ago[deleted]
- actionfromafar 4y agoThe article talks about downsampling, not upsampling, just so we are clear about that. And besides, a ranty blog post pointing out pitfall can still be useful for someone else coming from the same naïve (in a good/neutral way) place as the author.
- mytailorisrich 4y agoNot at all. He is correct that those functions are defined mathematically and that the results should therefore be the same using any libraries which claim to implement them. An example used in the article: https://en.wikipedia.org/wiki/Lanczos_resampling https://en.wikipedia.org/wiki/Lanczos_resampling
- planede 4y agoProblems with image resizing is a much deeper rabbit hole than this. Some important talking points: 1. The form of interpolation (this article). 2. The colorspace used for doing the arithmetic for interpolation. You most likely want a linear colorspace here. 3. Clipping. Resizing is typically done in two phases, once resizing in x then in y direction, not necessarily in this order. If the kernel used has values outside of the range [0, 1] (like Lanczos) and for intermediate results you only capture the range [0,1], then you might get clipping in the intermediate image, which can cause artifacts. 4. Quantization and dithering. 5. If you have an alpha channel, using pre-multiplied alpha for interpolation arithmetic. I'm not trying to be exhaustive here. ImageWorsener's page has a nice reading list[1]. [1] https://entropymine.com/imageworsener/ https://entropymine.com/imageworsener/
- peepee1982 4y agoWouldn't the clipping be solved by using floating point numbers during the filtering process?
- planede 4y agoIt would. It would also not accumulate quantization errors from an intermediate result. Having said that there are precedents for having the intermediate image pixels in integral values. Here is imageworsener's article about this[1] [1] https://entropymine.com/imageworsener/clamp-int/ https://entropymine.com/imageworsener/clamp-int/
- peepee1982 4y agoI love sites like these. Had never heard of Image Worsener before. Thanks!
- actionfromafar 4y agoWow, points 2, 3 and 5 wouldn't have occured to me even if I tried. Thanks. I now have a mental note to look stuff up if my resizing ever gives results I'm not happy with. :)
- deleted 4y ago[deleted]
- WithinReason 4y agotorch.nn.functional.interpolate has an "antialias" switch that's off by default
- qwertyforce 4y agoIt seems it was introduced after 1.9.0 https://pytorch.org/docs/1.9.0/generated/torch.nn.functional.interpolate.html https://pytorch.org/docs/1.9.0/generated/torch.nn.functional...
- WithinReason 4y agoYou're right, looks like it was added with 1.11 on March 10, 2022. Seems like an important feature to miss so long!
- singularity2001 4y agofunny that they use tf and pytorch in this context without even mentioning their fantastic upsampling capabilities
- mythz 4y agoWhat are some good image upscaler libraries that exist? I'm assuming the high quality ones would need to use some AI model to fill in missing detail.
- soderfoo 4y agoWaifu2x - I've used the library to upscale both old photos and videos with enough success to be pleased with the results. https://github.com/nagadomi/waifu2x https://github.com/nagadomi/waifu2x
- brucethemoose2 4y agoDepends on your needs! Zimg is a gold standard to me, but yeah, you can get better output depending on the nature of your content and hardware. I think ESRGAN is state-of-the-art above 2x scales, with the right community model from upscale.wiki, but it is slow and artifacty. And pixel art, for instance, may look better upscaled with xBRZ.
- biscuits1 4y agoThis article throws a red flag on proving negative(s). This is impossible with maths. The void is filled by human subjectivity. In a graphical sense, "visual taste."
- version_five 4y agoI'd argue that if your ML model is sensitive to the anti-aliasing filter used in image resizing, you've got bigger problems than that. Unless it's actually making a visible change that spoils whatever it is the model supposed to be looking for. To use the standard cat / dog example, filter choice or resampling choice is not going to change what you've got a picture of, and if your model is classifying based in features that change with resampling, it's not trustworthy. If one is concerned about this, one could intentionally vary the resampling or deliberately add different blurring filters during training to make the model robust to these variations
- hprotagonist 4y ago> I'd argue that if your ML model is sensitive to the anti-aliasing filter used in image resizing, you've got bigger problems than that. I’ve seen it cause trouble in every model architecture i’ve tried.
- version_five 4y agoWhat kinds of model architectures? I'm curious to play with it myself
- hprotagonist 4y agomost object detection models will show variability in bounding box confidences and coordinates. it’s not a huge instability, but you can absolutely see performance changes.
- deleted 4y ago[deleted]
- derefr 4y agoYou say that “if your model is classifying based in features that change with resampling, it’s not trustworthy.” I say that choice of resampling algorithm is what determines whether a model can learn the rule “zebras can be recognized by their uniform-width stripes” or not; as a bad resample will result in non-uniform-width stripes (or, at sufficiently small scales, loss of stripes!)
- ricardobeat 4y agoWas hoping to see libvips in the comparison, which is widely used. I wonder why it's not adopted by any of these frameworks?
- brucethemoose2 4y agoFor those going down this rabbit hole, perceptual downscaling is state of the art, and the closest thing we have to a Python implementation is here (with a citation of the original paper): https://github.com/WolframRhodium/muvsfunc/blob/master/muvsfunc.py#L3671 https://github.com/WolframRhodium/muvsfunc/blob/master/muvsf... Other supposedly better CUDA/ML filters give me strange results.
- thrdbndndn 4y agoThere are so many gems in VapourSynth scene. I really wish there are some better general-purpose imaging libraries that steadily implement/copy these useful filters, so that more people can use them out of the box. Most of languages I've involved are surprisingly lacking in this regard despite their huge potential use cases. Like, in case of Python, Pillow is fine but it has nothing fancy. You can't even fine-tune parameters of bicubic, let alone billions of new algorithms from video communities. OpenCV or ML tools like to re-invent the wheels themselves, but often only the most basic ones (and badly as noted in this article).
- brucethemoose2 4y agoVapourSynth is great for ML stuff actually, as it can ingest/output numpy arrays or PNGs, and work with native FP32. A big sticking point is variable resolution, which it technically supports but doesn't really like without some workarounds. But yeah I agree, its kinda tragic that the ML community is stuck with the simpler stuff.
- anotheryou 4y agoHm, any examples of that? I found https://dl.acm.org/doi/10.1145/2766891 https://dl.acm.org/doi/10.1145/2766891 but I don't like the comparisons. Any designer will tell you, after down-scaling you do a minimal sharpening pass. The "perceptual downscaling" looks slightly over-sharpened to me. I'd love to compare something I sharpened in photoshop with these results.
- brucethemoose2 4y ago
- fIREpOK 4y agoI favored cropping even back in 2021
- jcynix 4y agoNow that's an interesting topic for photographers who like to experiment with anamorphic lenses for panoramas. An anamorphic lens (optically) "squeezes" the image onto the sensor, and afterwards the digital image has to be "desqueezed" (i.e. upscaled in one axis) to give you the "final" image. Which in turn is downscaled to be viewed on either a monitor or a printout. But the resulting images I've seen until now nevertheless look good. I think that's because in natural images you have not that many pixel-level details. And we mostly see downscaled images on the web or in youtube videos most of the time ...
- cynicalsecurity 4y agoFinally someone said it.
- thrdbndndn 4y agoI'm shocked. I don't even know this is a thing. By that I mean, I know what bilinear/bicubic/lanczos resizing algorithms are, and I know they should at least have acceptable results (compared to NN). But I don't know famous libraries (especially OpenCV which is a computer vision library!) could have such poor results. Also a side note, IIRC bilinear and bicubic have constants in the equation. So technically when you're comparing different implementations you need to make sure this input (parameters) is the same. But this shouldn't excuse the extreme poor results in some.
- NohatCoder 4y agoAt least bilinear and bicubic have a widely agreed upon specific definition. The poor results are the result of that definition. They work reasonably for upscaling, but downscaling more than a trivial amount causes them to weigh a few input pixels highly and outright ignore most of the rest.
- leni536 4y ago> bicubic have a widely agreed upon specific definition Not so fast: https://entropymine.com/imageworsener/bicubic/ https://entropymine.com/imageworsener/bicubic/
- NohatCoder 4y agoFair. To be clear the issue remains no matter the choice of these parameters.
- pacaro 4y agoI've seen more than one team find that reimplementing an OpenCV capability that they use gain them both in quality and performance. This isn't necessarily a criticism of OpenCV, often the OpenCV implementation is, of necessity, quite general, and a specific use-case can engage optimizations not available in the general case
- JackFr 4y agoHmmm. With respect to feeding an ML system, are visual glitches and artifacts important? Wouldn't the most important thing to use a transformation which preserves as much information as possible and captures relevant structure? If the intermediate picture doesn't look great, who cares if the result is good. Ooops. Just thought about generative systems. Nevermind.
- brucethemoose2 4y agoJust speaking from experience, GAN upscalers pick up artifacts in the training dataset like a bloodhound. You can use this to your advantage by purposely introducing them into the lowres inputs so they will be removed.
- IYasha 4y agoSo, what are the dangers? (what's the point of the article?) That you'll get different model with same originals processed by different algorithms? The comparison of resizing algorithms is not something new, importance of adequate input data is obvious, difference in image processing algorithms availability is also understandable. Clickbaity.
- TechBro8615 4y agoIf you read to the end, they link to a library they made for solving the problem by wrapping Pillow C functions to be callable in C++
- azubinski 4y agoA friend of mine decided to take up image resizing on the third lane of a six-lane highway. And he was hit by a truck. So it's true about the danger of image resizing.
- IYasha 4y agoplot twist: a Tesla truck, with autopilot using bad image resizing algorithms )
- godshatter 4y agoIf their worry is the differences between algorithms in libraries in different execution environments, shouldn't they either find a library they like that can be called from all such environments or if they can't find one or there is no single library that can be used in all environments then shouldn't they just write their own using their favorite algorithm? Why make all libraries do this the same way? Which one is undeniably correct?
- TechBro8615 4y agoThat's basically what they did, which they mention in the last paragraph of the article. They released a wrapper library [0] for Pillow so that it can be called from C++: > Since we noticed that the most correct behavior is given by the Pillow resize and we are interested in deploying our applications in C++, it could be useful to use it in C++. The Pillow image processing algorithms are almost all written in C, but they cannot be directly used because they are designed to be part of the Python wrapper. We, therefore, released a porting of the resize method in a new standalone library that works on cv::Mat so it would be compatible with all OpenCV algorithms. You can find the library here: pillow-resize. [0] https://github.com/zurutech/pillow-resize https://github.com/zurutech/pillow-resize
- est 4y agoIs there any hacks/study to maximize the downsampling errors? E.g. looks totally different on original vs 224x224 pictures
- version_five 4y agoThere is a "resizing attack" that's been published that does what you're suggesting https://embracethered.com/blog/posts/2020/husky-ai-image-rescaling-attacks/ https://embracethered.com/blog/posts/2020/husky-ai-image-res...
- dark-star 4y agodownscaling images introduces artifacts and throws away information! news at 5!
- erulabs 4y agoImage resizing is one of those things that most companies seem to build in-house over and over. There are several hosted services, but obviously sending your users photos to a 3rd party is pretty weak. For those of us looking for a middle-ground: I've had great success with imgproxy (https://github.com/imgproxy/imgproxy https://github.com/imgproxy/imgproxy) which wraps libvips and well is maintained.
- intrasight 4y agoI was sort of expecting them to describe this danger to resizing: one can feed a piece of an image into one of these new massive ML models and get back the full image - with things that you didn't want to share. Like cropping out my ex. IS ML sort of like a universal hologram in that respect?
- AtNightWeCode 4y agoThought this article was going to be about DDOS...
- pallas_athena 4y agoIf you upscale (with interpolation) some sensitive image (think security camera), could that be dismissed in court as it "creates" new information that wasn't there in the original image?