10 ms·
Image Super-Resolution via Iterative Refinement
- tech_timc 5y agoCan you provide some image difference results between the reference images and the resulting super-resolution output? That would help to visualize what sort of structure, if any, is introduced by the super-resolution process.
- xscott 5y agoI'm curious to know if the various blemishes (acne scars, moles, etc...) are there in the true images. Also, the braids in one of the pictures don't look as simple/contiguous as I think real braids would be. Still, it's very cool how it fills in realistic looking details.
- jszymborski 5y agoIf you scroll down, under the "Super-Resolution Results" header, there's a comparison with a "Reference" column.
- tech_timc 5y agoThanks, what I am asking for is a visualization on the pixel level which shows the difference between the reference image, and the associated super-resolution image, as is often used to highlight the minimal differences in adversarial examples that confuse image classifiers. For example, see: https://christophm.github.io/interpretable-ml-book/adversarial.html https://christophm.github.io/interpretable-ml-book/adversari... https://openai.com/blog/adversarial-example-research/ https://openai.com/blog/adversarial-example-research/
- ssivark 5y agoThe "leopard spots" example is particularly interesting in how the super-resolution just hallucinates seemingly similar textures which can be completely different from the actual texture in the reference patch. (Such artifacts have been specifically pointed out in the context of analogous deep learning approaches applied to medical images).
- pfortuny 5y agoIt is not structure per se (although in the case of the leopard there is structure). It is an illicit inference. Given a diffusion process (as blur) there are an infinite number of initial states which converge to a specific final state. This is the nature of diffusion (think of it as: as long as the total initial energy is the same, the final state of a diffusion process is “homogeneous density of energy”). So inferring an “initial state” is totally invalid. Edit: look at the 64 to 256 to 1024 example in the “Unconditional…” section and take a look at the artifact on the bottom left (to the viewer) teeth and lip. If that is not an artifact… Same on the top-right teeth. Also: how does the algorithm know it is facial hair and not just makeup? It might be both but it generates facial hair.
- floxy 5y agoTwo Minute Papers did a ̶r̶e̶v̶i̶e̶w̶ ̶o̶f̶ ̶t̶h̶i̶s̶ ̶w̶o̶r̶k̶ thing: https://www.youtube.com/watch?v=WCAF3PNEc_c https://www.youtube.com/watch?v=WCAF3PNEc_c
- krona 5y agoThat's not a review. The guy just described the title of the paper without even reading the abstract.
- tehsauce 5y agohaha it’s a shame the level of effort in his paper reviews has really gone downhill, this one was especially so.
- jcims 5y agoEarly TMP subscriber here. Recently subbed to bycloud as well, he does a pretty good job of just getting down to business when reviewing papers: https://youtube.com/c/bycloudAI https://youtube.com/c/bycloudAI
- tehsauce 5y agothanks for the share! My personal taste is more like Yannic Kilcher videos I think.
- manpagereader 5y agoWhere's the code?
- floxy 5y agohttps://github.com/Janspiry/Image-Super-Resolution-via-Iterative-Refinement https://github.com/Janspiry/Image-Super-Resolution-via-Itera...
- leptoniscool 5y agoThis is an "unofficial implementation".
- gdevenyi 5y agoPage doesn't render properly in chrome mobile.
- mrtksn 5y agoWhile brilliant, I can’t stop myself from reminding the instance where another tool put Ray Gosling’s face on an image [0]. It’s fascinating, I especially enjoy when AI screws up because it breaks the illusion and gives us a clue to reason how this have happened. [0] https://news.ycombinator.com/item?id=24196650 https://news.ycombinator.com/item?id=24196650
- atdrummond 5y agoRay or Ryan? I’ve never heard of the former.
- mdp2021 5y agoWith some probability both, potentially: > CelebFaces Attributes (CelebA) Dataset [...] 10,177 unique identities, but names of identities are not given https://www.kaggle.com/jessicali9530/celeba-dataset https://www.kaggle.com/jessicali9530/celeba-dataset http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html
- mrtksn 5y ago> I’ve never heard of the former. That should probably give you a clue but you can always check the source and find out if it's Ray or Ryan Gosling.
- boibombeiro 5y agoThose old face super resolution cnn were usually trained with the celebA dataset. And because celebs have above average attractiveness, so did the generated images. Also the dataset were primarly white. Creating the meme image white Obama.
- mdp2021 5y ago> Also the dataset were primarly But offers «40 binary attribute annotations per image», which the (virtual) algorithm may use for choosing a path.
- trhway 5y agoso we can go basically to the old times of speech based tales just with computer AI instead of your own imagination - i mean the modern version would be that Netflix sends to your computer "The superhero hits the villain", and the AI on you computer makes it into a glorious 8K sequence of frames of the glorious battle. Depending on the model you leased of course, premium custom or regular one, etc. And there is a startup here to turn book pages you would read otherwise immediately into a movie.
- social_quotient 5y agoIt’s an interesting thought. Similar to the ideas of sending the audio on a zoom call and some deep fake model of your face and then the viewer sees a high res face talking but doesn’t need to get a full video stream.
- designium 5y agoEnhance 34 to 36.
- 6gvONxR4sf7o 5y agoSuper-resolution has been in the news over the last few days after it came up in the Rittenhouse trial. The defense lawyers said something along the lines of disallowing the ipad pinch-and-zoom as a way to present evidence because (they claimed) that it does super-resolution, which would potentially introduce fake evidence.
- djsbs 5y agoThats not what they claimed. They pointed out that the onus is on the prosecution to their get evidence validated. If they want to zoom in they have to get an expert to testify that iOS doesnt add any pixles. Its pretty basic trial procedure, the prosecution knows the rules and the judge gave them time to find an expert.
- 6gvONxR4sf7o 5y agoI read “add any pixels” in the context of zooming in as super resolution.
- djsbs 5y agoIs interpolation “super resolution”. Id argue not, but it doesn't matter. If the software adds pixels to the image than it has made a material change to the picture. Defense has a right to call the prosecution to prove that the zoom effect does not alter the picture. And its not a moot point. Years ago Xerox started doing OCR (?) on photocopies to same on storage before photocopying (by storing the image as text). Everyone assumed it was a materially identical copy until people noticed it was making typographical and lexical changes to legal contracts in materially meaningful ways. [1] [1] https://www.cbc.ca/news/science/xerox-copiers-might-alter-numbers-text-in-documents-1.1303751 https://www.cbc.ca/news/science/xerox-copiers-might-alter-nu...
- kevin_thibedeau 5y agoIt would have to be an interpolation algorithm that doesn't add or remove signal information. Everybody uses approximations for efficient hardware utilization. I wouldn't consider that admissible.
- endisneigh 5y agoI'd love how many times you can regressively run this process until the first generation source no longer looks (subjectively) like the nth generation refined image. e.g. [woah] -> [wooah] displacement: 1 [wooah] -> [woowah] displacement: 3 [woowah] -> [oowaha] displacement: 6
- tpoacher 5y agoWhat an extremely mobile-unfriendly website. I'll try to remember to read this on the pc later ...
- icf80 5y agois this real? something doesn't feel right
- RobinL 5y agoHas anyone applied this to video (i.e. apply frame by frame)? A quick Google didn't turn anything up...
- blueboo 5y agoThere’s no guarantee the detail will be stable frame-to-frame. You’d almost surely get textures stuttering and winking in and out of existence.
- Mikhail_K 5y agoTLDR: "machine learning", that is, some neural network again. That has nothing to do with science.
- holoduke 5y agoI remenber movies from the past where a police Investigation team magically upscaled a blurred picture of a numberplate, person etc. Always smiled and thought this is such bullshit. But it might actually become reality. Number plates still hard though
- _Microft 5y agoThere are a lot of amazing things you can do when you have extra constraints for the blurred information, like which font was used for a blurred text or knowing that it was a valid barcode that got blurred. In general "enhancing" images just does not work though. It will fake it in a way that looks believable to us but it cannot recover information that is not even there.
- occamrazor 5y agoBut sometimes there is enough information, just in a format that isn’t easily useable. For example a low-res video of a car driving away: each frane is too blurry to recover the plate number, but all frames together have enough information. I would expect DL models to perform wel on this task, but I could not find many results.
- _Microft 5y agoYou are correct, of course. In that case, the information is actually there albeit spread out temporally over several frames. I should have said that I was talking about single photographs.
- mdp2021 5y agoDo you know about that technology that works like a "spying microphone", reconstructing the environmental sound in a (ordinary quality, ordinary rate) video through the minuscule vibrations in the objects computed from the frames? Found: https://news.mit.edu/2014/algorithm-recovers-speech-from-vibrations-0804 https://news.mit.edu/2014/algorithm-recovers-speech-from-vib... > ... they were able to recover intelligible speech from the vibrations of a potato-chip bag photographed from 15 feet away through soundproof glass I see that (as expected) it has been discussed on HN: https://news.ycombinator.com/item?id=8131785 https://news.ycombinator.com/item?id=8131785
- okasaki 5y agoI've never seen this mentioned, but when I tried these sorts of methods, they mostly only worked on images that started as high quality images and were deliberately down-scaled. If the image original was low quality, or had artifacts, then the results were useless.
- magicalhippo 5y agoThis reminded me of the recent "VFX artists react" video by Corridor Crew[1], they talk about how the artist(s) added a ton of details to a scene, just for almost all of it to be blurred out and hidden by subsequent passes. The point made was that even though the end result is blurred out (out of focus etc), the brain can tell if it started out looking appropriate or not. So to make a good shot, detail is required. In that case it's our brain doing the up-scaling of sorts, and the issue seems similar. Down-scaling (blurring) a high-quality image and a low-quality one will result it two images that have critical differences. When those again are up-scaled they will lead to very different outcomes. Similarly if noise is introduced in the down-scaled (blurred) image, that is in essence the same as changing the image it was down-scaled from, thus up-scaling would again lead to something different. [1]: https://youtu.be/-CjNwVyojPA?t=1034 https://youtu.be/-CjNwVyojPA?t=1034
- Syzygies 5y agoTrue, but most images on the web have been deliberately downscaled.
- rsfern 5y agoThere’s a lot of interest in superresolution imaging in quantitative microscopy, where this is definitely a huge limitation
- mdp2021 5y agoI am afraid these methods ("learn the typical detail and "re"-apply it") on the said context ("obtain telling detail where we miss it") would be "dreaming" the detail. There methods are for "beauty", not to "induce real facts": what is "probable" is not what is "real". -- Edit: I have an example for you. Check https://github.com/Janspiry/Image-Super-Resolution-via-Iterative-Refinement/blob/master/misc/sr_64_512_0_hr.png https://github.com/Janspiry/Image-Super-Resolution-via-Itera... and notice the black spot under the iris of the right eye, on the left: does the actual person really have it? It is not in the blurred source image, https://github.com/Janspiry/Image-Super-Resolution-via-Iterative-Refinement/blob/master/misc/sr_64_512_0_inf.png https://github.com/Janspiry/Image-Super-Resolution-via-Itera... . Now think of an "empiricist" perceptive instrument returning that kind of "noise"... -- I understand you work on the topic (ANN for assessment), so you surely meant something different that what contextually appears.
- IshKebab 5y agoFunny how everyone was mocking that judge for thinking that zooming might add data, yet here is an example showing that zooming reduces the age of someone's face and removes the aggression from a tiger's expression. You might say "well of course we all know that this AI based super-resolution uses too much computing power to do in real time on an iPad, where they can only do sinc interpolation" but how long will that be true for? It's not really unreasonable for a judge not to know about technical details like that. I was pretty shocked at how all of the comments on the Ars article about it were so misguided.