12 ms·
Super-resolution from a single image
- jjcm 14y agoAre there any decent photoshop plugins that will achieve a similar result to this? Would love to have this in my workflow.
- CubicleNinjas 14y agoWhile not a PS plugin, Perfect Resize 7 Pro does shocking things. Our design team used it to resize images to the vertical size of a three story building and they still looked really good. http://www.ononesoftware.com/products/perfect-resize/ http://www.ononesoftware.com/products/perfect-resize/
- rorrr 14y agoS-spline Genuine fractals (now called Perfect Resize)
- kawera 14y agoPhotoAcute does super-resolution very well: http://www.photoacute.com/ http://www.photoacute.com/
- rhplus 14y agoI just noticed that this is from a 2009 paper. Has this method made its way into any products by now?
- teuobk 14y agoIf you have Matlab, you might try the tools here: http://www.ece.lsu.edu/ipl/Software.html http://www.ece.lsu.edu/ipl/Software.html
- jandrewrogers 14y agoTechniques like this have been used in some high-end sensor applications since (at least) the 1990s. It was an option for increasing discrimination resolution via DSP that could not practically be accomplished via higher resolution sensors due to things like space constraints given the technology of the time. It is probably less useful today due to ongoing miniaturization of imaging sensors.
- linhat 14y ago> It is probably less useful today due to ongoing miniaturization of imaging sensors. Not really. Dig into concepts called sparsity and compressed sensing, which are quite similar algorithms (not really the algorithms, but the underlying ideas). I can't find the authors' original website, only this old wired article from 2010: http://www.wired.com/magazine/2010/02/ff_algorithm/2/ http://www.wired.com/magazine/2010/02/ff_algorithm/2/ So especially with the ongoing miniaturization of imaging sensors, which will allow you to take even bigger and bigger pictures, there are issues these algorithms could solve, unless SD cards get really cheap really fast I guess.
- jandrewrogers 14y agoI meant less useful for the purposes which such algorithms were originally designed for two decades ago. They original use cases were discrimination, not making prettier pictures. I am quite familiar with compressive sampling.
- ansgri 14y agoThanks for the link. I'm now in the process of choosing the direction of my postgraduate studies, and these things quasifractal-related seem rather interesting.
- kemiller 14y agoSo this means that CSI's "enhance" is actually more possible than reputed.
- jhdevos 14y agoNot directly (but of course you were not entirely serious). Look at, for instance, the letter chart. The letters at the bottom are turned into very sharp forms, but not all look like letters, and those that do might not be turned into the correct ones... For instance, if they used this to "enhance" a picture with a piece of paper in someone's pocket, these letters would be enhanced based on other forms somewhere else in the picture -- so some vague letter shapes might become 'Burger King' just because the photo also contains a Burger King logo somewhere :)
- ynniv 14y agoThis doesn't actually add information, it just creates a magnified image that looks nicer than bicubic.
- hamburglar 14y agoYeah, and in fact the fact that it looks crisper can be misleading. You can't rely on a detail that a method like this infers from the low resolution image. Look at the green tiles on the kitchen floor in the 6th example. The "Our SR Result" image looks fantastic, but it's obvious that it's false, because the tiles end up misshapen and unevenly spaced. You really wouldn't want to convict someone of murder based on this type of enhancement.
- jessriedel 14y agoActually, it does add information. It takes information from elsewhere in the picture and uses it to interpolate/guess what high-resolution image produced the low-resolution pixels.
- daenz 14y ago"Adding information" in this context means providing accurate data that was not previously there. The guesses that the algorithm makes are approximations, but there are many original signals that could produce those same approximations, so the information is not accurate.
- sp332 14y agoThe server is already slow, so if it stops working I think this coral cache version should still load. http://www.wisdom.weizmann.ac.il.nyud.net/~vision/SingleImageSR.html http://www.wisdom.weizmann.ac.il.nyud.net/~vision/SingleImag...
- jawns 14y agoI'd be curious to know at what magnification this breaks down, if any. For instance, most of the images have been enlarged by about 3x or 4x. What happens when you blow them up to, say, 20x? It seems like this algorithm does some sort of vectorization -- because look at how crisp the lines are -- which would suggest to me that it scale well. But maybe I'm wrong.
- eridius 14y agoit could just be emphasizing edges. The very first picture of the baby had a rather overemphasized edge around his face, and yet it doesn't look like a vector.
- petekp 14y agoThis should be integrated into web browsers. Seems like it would particularly benefit people with high-density displays.
- CodeCube 14y agoMaybe Apple can take this algorithm and use it as the scaling algorithm for low res images on retina displays.
- HardyLeung 14y agoThis algorithm is likely very slow. In fact, the authors didn't mention runtime at all.
- ynniv 14y agoYes, there are other high performance candidates that work well with line art. http://en.wikipedia.org/wiki/Hq3x http://en.wikipedia.org/wiki/Hq3x
- rsiqueira 14y agoDepixelizing Pixel Art may work even better for some types of images: http://research.microsoft.com/en-us/um/people/kopf/pixelart/ http://research.microsoft.com/en-us/um/people/kopf/pixelart/ "The algorithm extracts a smooth, resolution-independent vector representation from the image, which enables magnifying the results by an arbitrary amount without image degradation."
- Evbn 14y agoThis is great work. The one thing that bugs me is that the algorithm is rather timid about corners (even though we its major contribution is in aggressively committing to hard edges) and results are rather blobby instead of forming sharp outlines.
- scriptproof 14y agoI tried HQX and the results are impressive! I have abandonned the Python tool because the installation is tedious but this one is very straightforward.
- AshleysBrain 14y agoIt doesn't have to be slow - it may be possible to write a shader that does it on a GPU.
- wahnfrieden 14y agoAre there any implementations of this available?
- guard-of-terra 14y agoThe problem with these: Every half a year we hear about new revolutionary image transformation. Today it's super-resolution, last year it was low-res PC games graphics interpolation, year before that - filter that could resize images by one direction (e.g. width) while preserving proportions and throwing away less interesting parts. The image will become 2x narrower and you'll hardly notice. Sadly, this is usually first and last time we see the technology in question. They do not seem to produce any impact that could increase our quality of life. They just sit on some dusty shelves somewhere.
- archivator 14y agoThat's hardly the case. The low-res texture stuff is widely used in emulators (both mobile and on PCs) - after all, that's exactly where it was meant to be used. The seam carving (the "least interesting line removal" thing) is now a major feature of Photoshop (called content-aware scale) and has become really well integrated (I hear there are attempts to do in video as well). The fact that you don't see this tech in your line of work doesn't mean it's not out there.
- guard-of-terra 14y agoGlad to hear seam carving took off. It looked like the most amazing thing when it was introduced.
- deleted 14y ago[deleted]
- mintplant 14y agoDo you have sources for each those, especially the uni-directional resizing? They sound quite interesting.
- avidal 14y agoThe technique is called Seam Carving: https://en.wikipedia.org/wiki/Seam_carving https://en.wikipedia.org/wiki/Seam_carving It effectively eliminates the least interesting line when performing a resize, allowing you, for example, to resize a skyline by eliminating the uninteresting smaller buildings and gaps.
- jhuckestein 14y agoSome of these images are in uncanny-valley territory for me. The bicubic images just look resized to me, but super-resolution images look real but "wrong". Given how much more "real" they still look than many computer graphics (especially realtime), I guess we won't leave uncanny valley behind us anytime soon.
- madmax108 14y agothank you for saying the same thing I was thinking. It's pretty neat when you consider the title of the paper, but the actual results just seem wrong. Not really sure why!
- stevewilhelm 14y agoJust a comment to the authors, my I suggest that the example magnified photos default to displaying your SR algorithm instead of NN interpolation.
- daten 14y agoI personally liked how the examples seamed to start with worst-cast and progress toward their new possibly "best case" method. What do you think is gained by starting with the new result?
- stevewilhelm 14y agoSince the paper is about the new method, one assumes the examples show the new method's results. The first impression, is that the new method isn't very effective.
- kyberias 14y agoNo, don't do that. The current order is fine. I want to see the algorithm's effect when compared to the old algorithm.
- JTxt 14y agoI was skeptical of the bottom line of the eye chart image. characters are 3x3 pixels. The o should translate to a dot. there is no hole. Their explanation for it: "The lines of letters have been recovered quite well due to the existence of cross-scale patch recurrence in those image areas. However, the small digits on the left margin of the image could not be recovered, since their patches recurrence occurs only within the same (input) scale. Thus their resulting unified SR constraints reduce to the “classical” SR constraints (imposed on multiple patches within the input image). The resulting resolution of the digits is better than the bicubic interpolation, but suffers from the inherent limits of classical SR [3, 14]." So it's guessing based on the larger characters. Neat. Here's their last line: DKN?MULJSPAVMRA?C?OVZO Here's the actual: DKNTWULJSPXVMRAHCFOYZG (from what I can read.) ( http://1.bp.blogspot.com/-VgvutrSWaFk/T4VI-2tDH2I/AAAAAAAAAXY/tx60XCDeAxI/s1600/reading+glasses+eye+chart.jpg http://1.bp.blogspot.com/-VgvutrSWaFk/T4VI-2tDH2I/AAAAAAAAAX... ) This could probably help ocr. Other than this, I think Genuine Fractals and BenVista PhotoZoom give similar results.
- daenz 14y agoSorry, could you clarify what you mean about the eye chart image? I'm amazed at the last line of the chart too. What are you suggesting they did to get that clarity?
- mbrubeck 14y agoPart of their algorithm finds areas of the image that are close matches but have different sizes, and uses the larger areas to create high-resolution replacements for the matching smaller areas. Of course this produces especially good-looking output for images like the eye chart (and also the final image on the page) that have many similar elements repeated at different scales. Essentially it is looking at the large letters (or pieces of them) to guess how the small letters should look. And as JTxt notes above, sometimes it chooses the wrong large letter. Even on the third-to-last line it couldn't get one of the letters correct. The output looks like "HKO" while the actual chart has "HKG" -- but since there is no larger "G" for the algorithm to use as an example, it ended up with a different but similar shape. This could probably be improved by the other SR techniques they mention that use libraries of sample images.
- beggi 14y agohttp://en.wikipedia.org/wiki/Super-resolution http://en.wikipedia.org/wiki/Super-resolution Implementation in Python: http://mentat.za.net/supreme/ http://mentat.za.net/supreme/
- iwalton3 14y agoHow do I use the Python Implementation?
- regularfry 14y agoLooks like http://mentat.za.net/supreme/doc/api/supreme.resolve.html#solve http://mentat.za.net/supreme/doc/api/supreme.resolve.html#so... might help.
- Sapemeg 14y agoi wonder whether this kind of method can be as fast as the older ones?
- oulipo 14y agoIt is the principle of NL-means denoising: since an image is redundant, you use informations about shapes that are similar and are present at different scales in the image to guess the shape of the zoomed version of the smallest ones by using the (more thorough) content of the bigger ones
- mistercow 14y agoThe subjective effect reminds me strongly of 2xSaI and hqnx upscaling filters, although of course those only work well for pixel art.
- deleted 14y ago[deleted]
- ck2 14y agoIt looks like contrasting lines are almost vectorized then expanded?
- eridius 14y agoIs there any information on how computationally expensive this is?
- ansgri 14y agoAs I understand, the complexity is not much worse than O(A * K), where A is upscaled image area and K is approximate KNN lookup time in a dataset of all tiles for all scales. The question is, how fast is KNN on such high-dimensional data.
- jostmey 14y agoI am impressed. Still, I suspect all the images shown on the webpage closely follow the frequency spectrum of natural images. Different methods will do better under different frequency spectrums.
- chmike 14y agoApplying this on satellite images with much more data available could yield interesting results. Anyone interested to test this method with google satellite images ?
- duncan 14y agohttp://www.youtube.com/watch?v=Vxq9yj2pVWk http://www.youtube.com/watch?v=Vxq9yj2pVWk
- alister 14y agoDear Mr. Glasner, Bagon, and Irani (authors of the super-resolution paper): Please apply this technique to resolve a historically important question about the assassination of President John F. Kennedy by identifying the license plate number of the car immediately behind the bus: http://i754.photobucket.com/albums/xx183/twsxpress/Forum2/Yellow_Transit_Van.png http://i754.photobucket.com/albums/xx183/twsxpress/Forum2/Ye... http://i754.photobucket.com/albums/xx183/twsxpress/Forum2/murry1240.jpg http://i754.photobucket.com/albums/xx183/twsxpress/Forum2/mu... Better resolution photos should be available since the ones I've seen printed in books are better than these. And there are plenty of photos of Texas license plates from 1963 with which to seed your algorithm. The reason this is interesting is because Roger Craig, a Deputy Sheriff in Dallas, after he witnessed the assassination said that he saw Lee Harvey Oswald run from the Texas School Book Depository and get into Nash Rambler station wagon driven by a another man. These photos show a Nash Rambler station wagon that was indeed passing the Depository just after the assassination. If motor vehicle records still exist for 1963, then its owner can be identified. The key point is that the existence of a person picking up Oswald after the assassination would strongly indicate the possibility that he was not acting alone. I'll readily grant that this isn't likely to settle the issue, but I think it still would be amazing if the super-resolution approach was able to generate new information about a topic that no one expects to ever see new information about.
- jacobolus 14y agoThat’s not how this technique works. There’s no magic here, just some algorithms. It doesn’t use time and space travel to reach back into the original scene and pull data that wasn’t captured by the camera. Instead, it guesses at what the blown up image should look like based on an examination of other features already in the image, with some expectation that shapes and textures will be similar in different parts of the image and at different scales – will have a sort of fractal nature – and by aiming to make shapes with edges instead of fuzzy boundaries. Using these algorithms can’t reveal the precise content of a license plate that was too small in the original image to see the letters of.
- alister 14y ago
- rburhum 14y agoIs there a BSD/LGPL licensed c/c++/ implementation? I'd love to try it out in some public domain satellite imagery to see the results
- ANH 14y agoThe first application that comes to my mind is improving the quality of photos in real estate listings. Half the time it looks like they were taken by a camera phone circa 1995 by someone who was skipping from room to room.
- gwern 14y agoInteresting how these super-resolution approaches are slowly inching onto AI/machine learning territory.
- t3kcit 14y agoIn the context of computer vision, this paper is pretty "old". Since then people have more focused on denoising, though.
- cypherpnks 14y agoI claim bullshit. The theoretically optimal algorithm is, and has been known to be for 40+ years, sinc interpolation. Gimp does this (Lanczos approximation). Comparing to bicubic leaves aliasing relics and looks bad. I'd be very interested in seeing the algorithm compared to something that's not a pure known strawman. (In CS terms, this is akin to comparing your algorithm to something using a bubble sort, and ignoring the invention of n log n sorting algorithms)
- pohl 14y agoI don't know anything about this area of specialty, but I thought it important to point out that Huffman compression was also proven to be theoretically optimal, and then along came arithmetic compression. I only mean to say that you should be very precise about what the proof of optimality actually showed. And, with that, I'm bracing myself to get schooled. Edit: This reminds me of the Iterated Fractal Systems proprietary lossy image compression algorithm they tried to commercialize in the 90s. It was able to decompress to a larger scale image that introduced synthetic detail that was often convincing to the eye. Notice how this article talks about different scales.
- cypherpnks 14y agoThat wasn't my point. You can do better than theoretically optimal -- indeed, their results, in many cases, do look better than sinc. Theoretically optimal makes assumptions that the source image is, in some ways, random, in a way that the real world doesn't conform to. If you know there are hard edges, you can do better. The point is that they should be comparing to sinc. They're comparing to a known-stupid algorithm to make themselves look better than they are.
- apu 14y agoI'm a little late to the party, but super-resolution is a topic of interest in computer vision, and this approach is one of many that have been recently proposed. At a high level, I should say that the field is still searching for the right approach, and there is by no means a consensus that this method (or more generally, this philosophy) is the right one. As a general rule of thumb, the performance shown in example figures of a computer vision paper should be taken as best cases (and in some papers, as outliers), and even the "failures" shown are often not the worst or most typical. Similarly, quantitative results are generally as optimistic as possible, arrived at through "graduate student descent" of the parameter space. So I think we are still quite far from a super-resolution method that is truly practical OR effective. Some specific points about this paper: - This research group has a long history of exploring methods based on exploiting self-similarity in images for various tasks (super-resolution, denoising, segmentation, etc.), and although they have shown remarkable progress on this front, it is generally agreed that using more than just a single image would improve results drastically for just about any task. - The use of more-than-exhaustive self-similarity search is EXTREMELY expensive, however, and that's why runtime is often not mentioned in these papers. It's not uncommon for processing times to be on the order of DAYS, for a single tiny VGA image. (I don't remember whether they quote a number in this paper or not, but it's certainly not faster than a few hours per image, unless you start using some hacks.) - As others have commented, this method is not actually extracting new information from the image, but rather "hallucinating" information based on similar parts of the image. There is certainly reasonable justification for its use as a prior, but it's not clear whether it's optimal, or even close to it. - My gut feeling is that super resolution methods will find much more application in compression-related areas rather than CSI-esque forensics. For example, JPEG compression is roughly 30%. But downsample an image by 2x in each direction and your compression ratio is already 25%. So if you can downsample images a few times and then use super-resolution to "hallucinate" a high-resolution version, that's probably good enough for the vast majority of common images (e.g., facebook photos), where users aren't very sensitive to quantitative "correctness". And of course with mobile adoption happening much faster than mobile bandwidth is increasing, this becomes an ideal application domain.
- tripzilch 14y ago> "graduate student descent" of the parameter space that is a wonderful phrase :) going to keep it in mind :)