5 ms·

Exploiting Aliasing for Manga Restoration

86 points by lnyan 6y ago

Isamu 6y ago
Cool approach, and I am glad to find out there’s a manga dataset out there for academic use! http://www.manga109.org/en/ http://www.manga109.org/en/
- flakiness 6y ago
  Yeah, I hope it were available without contacting them, but it's probably too much to ask.
  - chocolatkey 6y ago
    As someone who gained access to this dataset, I will say that the image resolution is disappointing. Maybe that's what inspired this paper
  - mullr 6y ago
    They are willing to share the dataset, just tell them what you want to do with it.
- uyt 6y ago
  https://www.gwern.net/Danbooru2020 https://www.gwern.net/Danbooru2020 is a much bigger dataset but way less cleaned.
polishdude20 6y ago
Wondering how this compares to just a sharpening filter?
cleansingfire 6y ago
While the aliased sample is sharper, I experience an unpleasant artifact in that version based on the halftone dots lining up with pixels, with an effect like grid illusion. https://en.wikipedia.org/wiki/Grid_illusion?wprov=sfti1The https://en.wikipedia.org/wiki/Grid_illusion?wprov=sfti1The blurry character of the original is also unpleasant, but the aliased version is hard for me to look at. I'm interested to know if anyone else experiences this.
- xwdv 6y ago
  I agree, it was hard for me to tell exactly what was the restored version as both images had unpleasant flaws.
- lupire 6y ago
  It's not even an illusion; it looks bad because they have a terrible tile for the dithering. Should be easy to fix in a postprocessing step after the AI.
Groxx 6y ago
I kinda wonder how this would compare to an upscale and sharpen - a good amount of these screentone patterns are solid blacks on a consistent white or gray, which seems like it should work fairly well. Or maybe that'd round too much off - this is doing a pretty good job of keeping line-quality intact. That said, this is an interesting technique, and looks pretty good in the end... but the minor misalignments / pattern-jitter in some areas would probably bug me more than the blurry image, tbh. Seems like that could be improved somehow though, maybe by modifying the pattern it decides on with something similar but not original-pixel-aligned? --- edit: after writing the above and looking back at it a third or fourth time: I've changed my mind, the patterns this is producing will very likely look better than a sharpen when they're closer together or more heavily aliased. They're "plausible" and still look like patterns, sharpens have some terrible edge cases on stuff like the remote(?)'s frame. Maybe they just need some more examples / side-by-sides? I imagine more will be in the final paper, whenever that's linked.
crazygringo 6y ago
This is really clever! I love whenever it's possible to upsample/restore media due to known constraints in the original -- in this case, how screens work. Something analagous I've been waiting for is regenerating old scratchy piano recordings. Piano is unexpectedly simple compared to other instruments -- the only inputs are really note down + speed, note up, sustain pedal pressure, and (less frequently) soft pedal pressure. Seems like you should be able to turn any solo piano recording, no matter how degraded, into a relatively lossless MIDI representation, then re-record that replayed physically (via motors, which exist already) on a modern piano, or even just synthesized, trying to be as true to the original piano's characteristics as possible. Losing literally none of the artistry. It seems like this should be "easy" for piano in a way that it isn't, for example, with violin which has so many more complicated characteristics of pitch, timbre, bowing, vibrato, etc.
- cclark00 6y ago
  This has been done in various forms! One interesting one is shown in https://www.youtube.com/watch?v=6hv2zh_Z0Io https://www.youtube.com/watch?v=6hv2zh_Z0Io Sergei Rachmaninoff recorded a 78 album simultaneously with a piano roll of several pieces - thus capturing the 'keystrokes' and the audio of intonation of the master's hand. The piano roll was converted to an automatic reproducing piano (super high end player piano, a Bosendorfer 290SE) and massaged by an expert to sound almost exactly like the 78. Then it was re-recorded in modern fidelity, playing back from the 290SE. It is exciting to see and hear a 290SE (re)play in person, but a little weird in a concert setting with no pianist to watch.
  - crazygringo 6y ago
    That's amazing, thanks! I was even thinking that Rachmaninoff's recordings would be where I'd start. It's like a ghost playing. Absolutely crazy to hear his touch but with modern fidelity.
- zitterbewegung 6y ago
  A piano can be intentionally and or unintentionally not in tune. Every Piano has its own unique sound (due to manufacturer and also form factor) and is played in a place where temperature can have another affect in the sound. But it looks like already people have attempted the transcription strategy you describe https://www.google.com/search?hl=en&q=machine%20learning%20note%20recogntion%20piano https://www.google.com/search?hl=en&q=machine%20learning%20n...
Nondescript04 6y ago
Interesting. I myself have over the past month trained a simple 2X ESRGAN upscaling model for manga, so I did a quick comparison to see how it performs[1]. [1]https://i.imgur.com/9jnVPBK.png https://i.imgur.com/9jnVPBK.png
- salamandersauce 6y ago
  I like this considerably better. The text in particular is not blurry like the original scan nor is it pixelated like their example. Could something like this work on modest hardware in a real time fashion once the model has been trained? Or is something best ran beforehand?
  - Nondescript04 6y ago
    For ESRGAN I would say there is still ways to go for it to be useable in realtime. The shown image at an input resolution of 309x237 pixels takes about 7 seconds to process with my own model on a GTX1080.

86 points by lnyan 6y ago

Cool approach, and I am glad to find out there’s a manga dataset out there for academic use! http://www.manga109.org/en/ http://www.manga109.org/en/

flakiness 6y ago
Yeah, I hope it were available without contacting them, but it's probably too much to ask.
- chocolatkey 6y ago
  As someone who gained access to this dataset, I will say that the image resolution is disappointing. Maybe that's what inspired this paper
- mullr 6y ago
  They are willing to share the dataset, just tell them what you want to do with it.
uyt 6y ago
https://www.gwern.net/Danbooru2020 https://www.gwern.net/Danbooru2020 is a much bigger dataset but way less cleaned.

polishdude20 6y ago

Wondering how this compares to just a sharpening filter?

cleansingfire 6y ago

While the aliased sample is sharper, I experience an unpleasant artifact in that version based on the halftone dots lining up with pixels, with an effect like grid illusion. https://en.wikipedia.org/wiki/Grid_illusion?wprov=sfti1The https://en.wikipedia.org/wiki/Grid_illusion?wprov=sfti1The blurry character of the original is also unpleasant, but the aliased version is hard for me to look at. I'm interested to know if anyone else experiences this.

xwdv 6y ago
I agree, it was hard for me to tell exactly what was the restored version as both images had unpleasant flaws.
lupire 6y ago
It's not even an illusion; it looks bad because they have a terrible tile for the dithering. Should be easy to fix in a postprocessing step after the AI.

Groxx 6y ago

I kinda wonder how this would compare to an upscale and sharpen - a good amount of these screentone patterns are solid blacks on a consistent white or gray, which seems like it should work fairly well. Or maybe that'd round too much off - this is doing a pretty good job of keeping line-quality intact. That said, this is an interesting technique, and looks pretty good in the end... but the minor misalignments / pattern-jitter in some areas would probably bug me more than the blurry image, tbh. Seems like that could be improved somehow though, maybe by modifying the pattern it decides on with something similar but not original-pixel-aligned? --- edit: after writing the above and looking back at it a third or fourth time: I've changed my mind, the patterns this is producing will very likely look better than a sharpen when they're closer together or more heavily aliased. They're "plausible" and still look like patterns, sharpens have some terrible edge cases on stuff like the remote(?)'s frame. Maybe they just need some more examples / side-by-sides? I imagine more will be in the final paper, whenever that's linked.

crazygringo 6y ago

This is really clever! I love whenever it's possible to upsample/restore media due to known constraints in the original -- in this case, how screens work. Something analagous I've been waiting for is regenerating old scratchy piano recordings. Piano is unexpectedly simple compared to other instruments -- the only inputs are really note down + speed, note up, sustain pedal pressure, and (less frequently) soft pedal pressure. Seems like you should be able to turn any solo piano recording, no matter how degraded, into a relatively lossless MIDI representation, then re-record that replayed physically (via motors, which exist already) on a modern piano, or even just synthesized, trying to be as true to the original piano's characteristics as possible. Losing literally none of the artistry. It seems like this should be "easy" for piano in a way that it isn't, for example, with violin which has so many more complicated characteristics of pitch, timbre, bowing, vibrato, etc.

cclark00 6y ago
This has been done in various forms! One interesting one is shown in https://www.youtube.com/watch?v=6hv2zh_Z0Io https://www.youtube.com/watch?v=6hv2zh_Z0Io Sergei Rachmaninoff recorded a 78 album simultaneously with a piano roll of several pieces - thus capturing the 'keystrokes' and the audio of intonation of the master's hand. The piano roll was converted to an automatic reproducing piano (super high end player piano, a Bosendorfer 290SE) and massaged by an expert to sound almost exactly like the 78. Then it was re-recorded in modern fidelity, playing back from the 290SE. It is exciting to see and hear a 290SE (re)play in person, but a little weird in a concert setting with no pianist to watch.
- crazygringo 6y ago
  That's amazing, thanks! I was even thinking that Rachmaninoff's recordings would be where I'd start. It's like a ghost playing. Absolutely crazy to hear his touch but with modern fidelity.
zitterbewegung 6y ago
A piano can be intentionally and or unintentionally not in tune. Every Piano has its own unique sound (due to manufacturer and also form factor) and is played in a place where temperature can have another affect in the sound. But it looks like already people have attempted the transcription strategy you describe https://www.google.com/search?hl=en&q=machine%20learning%20note%20recogntion%20piano https://www.google.com/search?hl=en&q=machine%20learning%20n...

Nondescript04 6y ago

Interesting. I myself have over the past month trained a simple 2X ESRGAN upscaling model for manga, so I did a quick comparison to see how it performs[1]. [1]https://i.imgur.com/9jnVPBK.png https://i.imgur.com/9jnVPBK.png

salamandersauce 6y ago
I like this considerably better. The text in particular is not blurry like the original scan nor is it pixelated like their example. Could something like this work on modest hardware in a real time fashion once the model has been trained? Or is something best ran beforehand?
- Nondescript04 6y ago
  For ESRGAN I would say there is still ways to go for it to be useable in realtime. The shown image at an input resolution of 309x237 pixels takes about 7 seconds to process with my own model on a GTX1080.