9 ms·
Using Waifu2x to Upscale Japanese Prints
- deleted 11y ago[deleted]
- xatnys 11y agoInteresting! The effect looks quite similar to warpsharp (http://avisynth.nl/index.php/WarpSharp http://avisynth.nl/index.php/WarpSharp), a sharpening filter that seemed to have some vanity among anime encoders back when video sources were not as crisp as they are today. There's quite a lot of detail loss in Resig's ukiyo-e example, but I imagine for most people the most striking part of it will be how much smoother the result appears.
- mahouse 11y agoHe forgot to comment on how the filter destroyed the letters.
- jeresig 11y agoI'm not sure I'd go so far as to say "destroyed". Compare the text in this cartouche: https://imgur.com/7fGJg4s,iWf4pXG https://imgur.com/7fGJg4s,iWf4pXG At worst it seems comparable to the previous result. At least to my eyes.
- mahouse 11y agoFor example, 4th character, two lines were converted into a stain.
- jeresig 11y agoGreat point. FWIW, I've updated the post to include some of the cartouches, along with a cartouche at the "low noise reduction" level. The two lines in the fourth character appear to still be relatively distinct in this case.
- mahouse 11y agoThat being said, I don't want by any means to disrespect the work, because it's clearly impressive.
- ComputerGuru 11y agoThe "cleanliness" of the resulting images is undeniable, but once you get past the sheer awe at how crisp and clear the upscaled image is, you'll immediately notice the loss of detail. It completely does away with any and all texturing, which is especially noticeable in the last image ([1] vs [2]) - look at the scales and patterned lines on the snake (?) around his neck and the white strands in his hair, and of course, the letters have been turned into (unrecognizable?) squiggles. Still, in terms of pure shock and awe - they're jaw-droppingly nice for upscaled versions, to the point where if you didn't have the original, it wouldn't occur to you that this wasn't it. 1: http://ukiyo-e.org/image/mfa/sc165440 http://ukiyo-e.org/image/mfa/sc165440 2: http://i.imgur.com/541uG5t.png http://i.imgur.com/541uG5t.png
- cat9 11y agoMaybe? I'm definitely biased in that I have substantial computer vision & image processing experience, but the output looks riddled with obvious filter and vectorization artifacts to me. Re: Image 2, http://i.imgur.com/541uG5t.png http://i.imgur.com/541uG5t.png
- jeresig 11y agoYeah - the background looking like crumpled-up wrapping paper is definitely not ideal. I suspect that it's having trouble with mostly-uniform areas of color that have slight variations. It appears to be extrapolating and creating these larger effects.
- sthatipamala 11y agoHave you tried running the original through a high pass filter (to get the textures) and applying it over the vectorized version? It might work for the background texture, though it would probably suck for the text.
- jacobolus 11y agoIt would preserve all the JPEG artifacts, which are the main reason the naively upscaled version looks so crappy in the first place.
- yellowapple 11y agoThis looks like it could be applied to a real-life "ENHANCE" button. By training similar algorithms with photographs instead of anime prints, would this be a feasible means of approximating detail from enlarged photographs CSI-style (not quite to the extreme one sees on TV, but perhaps enough for a police sketch or something)?
- deleted 11y ago[deleted]
- kibwen 11y agoSomething to keep in mind is that when upscaling, you are actually inventing (fabricating) detail. Tools like the one presented here are content to invent detail that looks pleasing to the eye, but if you tried to do something like this for photographs you wouldn't get anything that would hold up as evidence. You also wouldn't want to use this to guide a police sketch, because the "enhanced" image actually contains false information compared to the original.
- woodman 11y agoThis upscaling can be considered a form of lossy compression. The pixelated images, while ugly, contain more information - the process cannot be reversed due to this loss of information. > You also wouldn't want to use this to guide a police sketch... You would, could and can. Take a look at police sketch software, it is a very manual version of this process. There are a very limited number of potential variations of the human face, that is why eigenfaces work for facial recognition. Consider the scenario where you have a photo of a human face, where one half is occluded. In manually reconstructing the image, you wouldn't place the reconstructed eye an inch above the chin - because human faces don't work that way. Neural networks pick up on that. There is software that tailors use in fitting suits, where a few measurements (like weight and arm length) can be used to extrapolate the rest of the measurements (like chest size and torso length). This works because of the limited number of potential human dimensions. As far as use of these techniques for evidence... I'd actually prefer it to reliance on eye witness accounts, as the algos are open to exact measurement - unlike most of the other stuff that passes for "criminal science" (humans are still in the loop for fingerprint analysis, wtf?).
- Daiz 11y agoWaifu2x is not actually the first or only image scaler to use neural networks - NNEDI3[1], an Avisynth[2] filter used for deinterlacing can also do really nice image upscaling (and it's a lot faster than waifu2x). Here's an example of what it can do to the images in the blogpost: Image 1: http://i.imgur.com/4cXr51v.png http://i.imgur.com/4cXr51v.png Image 2: http://i.imgur.com/PZAXeM8.png http://i.imgur.com/PZAXeM8.png It doesn't come with any noise reduction, but nothing stops you from doing that separately from the upscaling process itself, and that way you should be able to control it better anyway (I find the reduction options provided by waifu2x really aggressive even with the low setting, it just kills tons of detail). As a sidenote, when talking about something like image scaling, it would be a good idea to avoid saying something like "image scaled 2x (normally)" as there are lots of ways to scale images and what's "normal" can vary a lot depending on what you're using. [1] http://bengal.missouri.edu/~kes25c/ http://bengal.missouri.edu/~kes25c/ [2] http://avisynth.org http://avisynth.org
- jeresig 11y agoNNEDI3 is fantastic - thank you for providing a link and some samples! You're absolutely right that I shouldn't have said "normal". I update the post to clarify that this was using "OSX Preview". I did some hunting but didn't find any obvious pointers as to which algorithm they're using. If anyone knows offhand I'll be happy to include it!
- Daiz 11y agoWell, OSX Preview seems to be doing something interesting as I can't seem to find an exact match with some quick attempts, but whatever method they use it looks rather similar to Lanczos scaling.
- jonah 11y agoTalk with the imgix.com folks about the CoreImage stuff. They're using the built-in re-sampling in their product. Also chat with @deepbluecea who's done a lot of image processing stuff, including for Apple.
- 11y ago
- deepnet 11y agoGreat use case, upscaling print thumbnails. Norman Tasfi made a Neural Net upscaler for Flipoard http://engineering.flipboard.com/2015/05/scaling-convnets/ http://engineering.flipboard.com/2015/05/scaling-convnets/ I expect video upscaling next.
- the8472 11y ago> I expect video upscaling next. There is a directshow filter (madvr[1]) for windows that already offers a neural network scaler {NNEDI3, simpler network than waifu2x) in realtime. [1] http://forum.doom9.org/showthread.php?t=146228 http://forum.doom9.org/showthread.php?t=146228
- ma2rten 11y agoInteresting... If a Neural Network can be used for up scaling a video it means that you need to send less data over the wire for getting the same quality. This means neural networks can be used as a compression algorithm.
- nrmn 11y agofrom the flipboard article: "The final use case that we thought of was saving bandwidth. A smaller image could be sent to the client which would run a client side version of this model to gain a larger image." Can be applied to gifs and videos, but this really depends on the usage case and if the client would tolerate such a thing.
- Daiz 11y agoNNEDI3, which I mentioned in another comment, can be used for video upscaling, and was in fact built for video processing in the first place.
- georgehm 11y agofor comparison sake, can someone share the time taken to upscale some of these images?
- gburt 11y agoRelated, this really cool project: http://research.microsoft.com/en-us/um/people/kopf/pixelart/ http://research.microsoft.com/en-us/um/people/kopf/pixelart/
- the8472 11y agoThe NN was explicitly trained for artifact-free PNG sources of anime fanart. Which it handles quite well according to my own testing[1] Its benefits are questionable if used on anything else. I've also tested it on anime screenshots and in that case it's it pretty much is en par there with NNEDI3 (which is computationally much cheaper) because real world encodes actually have compression artifacts and those get scaled too if you disable noise reduction or everything is smoothed out too much if you leave it on. So if you want to use it on anything else you really do have to retrain the NN first, otherwise you get results you could also achieve by other means (e.g. warpsharp, NNEDI or Photoshop Topaz) Also, waifu2x only scales luma. Its chroma handling is just regular upscaling (whatever imagemagick uses by default. I think), so even that part could be improved. [1] http://forum.doom9.org/showpost.php?p=1722990&postcount=3 http://forum.doom9.org/showpost.php?p=1722990&postcount=3
- deleted 11y ago[deleted]
- CarVac 11y agoIt performs better on larger features. Anime is almost never drawn with finer detail than the output resolution, so artifacts are not a problem. This is a low resolution scan of something with very fine detail, something which it is not trained on.
- Joona 11y agoI did a quick comparison between ImageMagick and Waifu2x using a common anime-style image: http://imgur.com/a/teKVY http://imgur.com/a/teKVY
- yohui 11y agoI'd just like to express my appreciation for Waifu2x's informative name. More projects could do with such evocative labels.
- pjc50 11y agoMy neural network sarcasm detector is confused by this post. I was going to complain about it being the same kind of dim unintentional sexism as the original choice of Lena as reference image.
- JulianMorrison 11y agoI don't buy "unintentional".
- yohui 11y agoIt was a backhanded compliment. The name is a bit... unprofessional. On the other hand, it does help you grok its function, and I suspect the 'memorable' name is at least partially responsible for its popularity.
- Gravityloss 11y agoThe scans look to have jpeg artifacts? If you really are working with the original source, you should rescan to png or tiff or even just higher rate jpeg?