6 ms·

Colorful Image Colorization

113 points by stared 11y ago

ilovefood 11y ago
Seriously?? This is like the uber of cool or the spotify of awesome!! There's a link in this blog http://tinyclouds.org/colorize/ http://tinyclouds.org/colorize/ where you can get the trained model for Tensorflow. I'll try this out. MIND = Blown.
- StavrosK 11y ago
  Seriously what the hell? How does it even know that the people in the Cartier-Bresson photo are black? I have no idea how it's doing that stuff.
  - JonnieCache 11y ago
    How do you know they're black?
    - StavrosK 11y ago
      Advanced face recognition wetware.
  - IanCal 11y ago
    It doesn't really need to, it just needs to expect it to be skin. I'll probably get some terms wrong here as it's been a little while since I did this kind of thing but: When you look at skin, people seem to have very different colourings. I've got freckles so I'm even patchy, and depending on the time of year will be lighter or darker. However, skin colour when you remove brightness becomes much more standard. There's remarkably little variation between a normalised colour of black and white skin, black skin is simply darker. Look at the water by the faces, it's quite a generically caucasian colour. That's because it's just lighter. Here's some old stuff I did with finger pointing detection: https://figshare.com/articles/Finger_pointing_detection/953171 https://figshare.com/articles/Finger_pointing_detection/9531... https://ndownloader.figshare.com/files/3155228 https://ndownloader.figshare.com/files/3155228 <- A plot of measured skin colours https://ndownloader.figshare.com/files/3155291 https://ndownloader.figshare.com/files/3155291
    - StavrosK 11y ago
      Ah, yes, it looks like you're right. Looks like it just used a generic "skin" colour and the luminosity in the photo made it look "right" for black skin.
    - simcop2387 11y ago
      That makes a lot of sense given that skin is colored by melanin. I would expect some small differences due to the fact that lighter skin colors are a bit more transparent but that they largely have the same base is pretty neat.
vmasto 11y ago
Previous HN discussion on this https://news.ycombinator.com/item?id=10864801 https://news.ycombinator.com/item?id=10864801
- stared 11y ago
  No. It is a different post, on a different research, by different people (on the same topic). I consider the earlier result is of much worse quality. (Saw it, but I was hardly impressed.)
- transcranial 11y ago
  Interestingly, this paper cites that previous submission: "2. Dahl, R.: Automatic colorization. In: http://tinyclouds.org/colorize/ http://tinyclouds.org/colorize/. (2016)" They even included it in their comparisons (Table 1; Figure 5).
hashmymustache 11y ago
Very impressive. Is it possible to combine this with a semantic analyzer in some way? Seems like having a first pass through one and then serving that output on top of raw image pixels could reduce the potential color space and maybe blunt training set biases? To wit, I know nothing about these things.
chias 11y ago
This is very cool. Aside from how impressive this is from a technical standpoint, I love that this algorithm seems to be a little colorblind in the human sense -- looking at the military trucks and uniforms in particular, it seems to have trouble with reds vs greens.
visarga 11y ago
This technique could be applied to archive b/w photos and footage. In the same line of ideas, it's a pity that some classical music recordings are very distorted. Some have been recorded by famous players who are long dead and carry important significance. Maybe there is a way to transform noisy recordings into ones that sound as if they were recorded in a modern studio. They could first try to emulate noise and distortions and generate a dataset of distorted recordings, then train a NN to reconstruct the original.
- stepvhen 11y ago
  I, for one, would not appreciate it if entire swaths of b/w photographs were needlessly colorized for "archiving", especially the Ansel Adams. His photography was too good for the exact medium he chose, and colorizing them detracts from the image. Colorization is super cool and whatnot, but I don't think it should be applied to art.
  - deleted 11y ago
    [deleted]
sarreph 11y ago
My first thought was "I hope this is someone showing off their photo-editing skills because if an algorithm did this then my mind is fully blown"... Either this or it's an advanced April Fool's release (with back-dating and references)!
oceanofsolaris 11y ago
This is rather impressive. Similarly to the cited work http://tinyclouds.org/colorize/ http://tinyclouds.org/colorize/, it does however seem to have a slight bias towards brownish colors (giving many images a sepia look). One thing very noticeable in both works is also the tendency to overshoot borders when coloring. In many images it can be seen that the color of the main subject bleeds into the background. A good example is e.g. the fifth picture in http://richzhang.github.io/colorization/resources/images/exs_sel_aa.jpg http://richzhang.github.io/colorization/resources/images/exs... , where parts of the sky get assigned the same color as the mesa. This looks like something that should be fixable if one tries to detect these kind of edges and gives less weight to information from the other side of such an edge. Does anyone know what the differences between this and the previous neural net approach are on a technical level?
- neckro23 11y ago
  The Ansel Adams examples are interesting because AA very deliberately manipulated the relative intensity of tones in his photographs. In fact, I'm not sure how applicable the current training method (desaturate a color photo, colorize it, compare it to the original) is to actual B&W photographs. Black-and-white photography is not a simple matter of desaturation.
tyingq 11y ago
It's impressive, but they do mention that the examples on the front page are where it did well. You can see some of the other results, including many there didn't work well at all, here: http://richzhang.github.io/colorization/resources/imagenet_comparison.html http://richzhang.github.io/colorization/resources/imagenet_c...
- web007 11y ago
  They have the "best" and "worst" matches at the bottom of the page. I thought the "worst" set was pretty good, and I wonder if correcting for white balance would have given them a HUGE benefit. The majority of the errors look like daylight-for-tungsten correction, just tweak the yellows to blues FTW.
  - tyingq 11y ago
    It seems to guess red at odd times, like http://richzhang.github.io/colorization/resources/images/imagenet_comparison//ours/9839.jpg http://richzhang.github.io/colorization/resources/images/ima... Is that white balance related?
dharma1 11y ago
Very impressive
ickwabe 11y ago
This is very impressive but I'll be that person. I shudder when I see someone/something colorize the works of masters of black-and-white like Ansel Adams or Henri Cartier-Bresson. Just a horrible mangling of their work. Adams for example could have used color, and did take many hundreds of color photos[1]. But in his work for display he deliberately and consciously chose black-and-white and put in much effort in the lab to get the final images just they way he wanted them. So while I truly appreciate the technological achievement here, I still cringe at the what I see as distorting the artists' work. [1]http://www.smithsonianmag.com/arts-culture/ansel-adams-in-color-145315674/?no-ist http://www.smithsonianmag.com/arts-culture/ansel-adams-in-co...
- mistercow 11y ago
  I really don't think the intention is to say "look we improved this!", but rather to have a visually striking (since the original photo is visually striking) result that doesn't have a "ground truth color" to compare it to.
  - ickwabe 11y ago
    Yes of course and I appreciate that. Just my opinion but in that case I would not have used images with such artistic weight. There certainly plenty black and white images to choose from.
saurik 11y ago
http://richzhang.github.io/colorization/resources/imagenet_comparison.html http://richzhang.github.io/colorization/resources/imagenet_c... This is amazing. Looking at the example comparison list, I love seeing the algorithm "correct" the color based on its biases. "No, people do not have pink hair: I made you blond." "Shoes are always a shade of brown (including black); the idea of shoes that are pink, purple, or even grey, morally offends me." "I don't understand why you would paint the frame of this window that crazy shade of green... let's go with a dark stain; doesn't that look classier?" "I've seen this insect before: I don't remember its name, but it was definitely brown, so stop trying to trick me into thinking it is bright blue with your silly Photoshop shenanigans." "Also: I've seen muzzles before, and they are made out of rusted metal; I don't know where you got this blue rubber-coated comfort muzzle, but back in my day we didn't have those extravagances." "On a similar note: I don't understand how your watch is glowing blue... back lights are supposed to be a dull and mottled yellow." "One red stripe on your truck is enough to make the point: three red stripes makes you look like you have no taste :/." "And why would you go to the trouble of forcing people to take a photo in front of a custom backdrop and not take the opportunity to print the name of your company in an eye-catching color, such as red?" "I am pretty certain all radiators everywhere are colored using the same aging white paint that has started browining from the heat; it gives a comfortable feeling of familiarity that your bright green radiator lacks." "Mittens are supposed to be festive, so why did you get grey ones?! I've colored them red: that should help."
cesarbs 11y ago
Am I the only one confused by the title? I honestly thought it would be an April Fool's joke :) The title makes me think it's about coloring colored images.

113 points by stared 11y ago

Seriously?? This is like the uber of cool or the spotify of awesome!! There's a link in this blog http://tinyclouds.org/colorize/ http://tinyclouds.org/colorize/ where you can get the trained model for Tensorflow. I'll try this out. MIND = Blown.

StavrosK 11y ago
Seriously what the hell? How does it even know that the people in the Cartier-Bresson photo are black? I have no idea how it's doing that stuff.
- JonnieCache 11y ago
  How do you know they're black?
  - StavrosK 11y ago
    Advanced face recognition wetware.
- IanCal 11y ago
  It doesn't really need to, it just needs to expect it to be skin. I'll probably get some terms wrong here as it's been a little while since I did this kind of thing but: When you look at skin, people seem to have very different colourings. I've got freckles so I'm even patchy, and depending on the time of year will be lighter or darker. However, skin colour when you remove brightness becomes much more standard. There's remarkably little variation between a normalised colour of black and white skin, black skin is simply darker. Look at the water by the faces, it's quite a generically caucasian colour. That's because it's just lighter. Here's some old stuff I did with finger pointing detection: https://figshare.com/articles/Finger_pointing_detection/953171 https://figshare.com/articles/Finger_pointing_detection/9531... https://ndownloader.figshare.com/files/3155228 https://ndownloader.figshare.com/files/3155228 <- A plot of measured skin colours https://ndownloader.figshare.com/files/3155291 https://ndownloader.figshare.com/files/3155291
  - StavrosK 11y ago
    Ah, yes, it looks like you're right. Looks like it just used a generic "skin" colour and the luminosity in the photo made it look "right" for black skin.
  - simcop2387 11y ago
    That makes a lot of sense given that skin is colored by melanin. I would expect some small differences due to the fact that lighter skin colors are a bit more transparent but that they largely have the same base is pretty neat.

vmasto 11y ago

Previous HN discussion on this https://news.ycombinator.com/item?id=10864801 https://news.ycombinator.com/item?id=10864801

stared 11y ago
No. It is a different post, on a different research, by different people (on the same topic). I consider the earlier result is of much worse quality. (Saw it, but I was hardly impressed.)
transcranial 11y ago
Interestingly, this paper cites that previous submission: "2. Dahl, R.: Automatic colorization. In: http://tinyclouds.org/colorize/ http://tinyclouds.org/colorize/. (2016)" They even included it in their comparisons (Table 1; Figure 5).

hashmymustache 11y ago

Very impressive. Is it possible to combine this with a semantic analyzer in some way? Seems like having a first pass through one and then serving that output on top of raw image pixels could reduce the potential color space and maybe blunt training set biases? To wit, I know nothing about these things.

chias 11y ago

This is very cool. Aside from how impressive this is from a technical standpoint, I love that this algorithm seems to be a little colorblind in the human sense -- looking at the military trucks and uniforms in particular, it seems to have trouble with reds vs greens.

visarga 11y ago

This technique could be applied to archive b/w photos and footage. In the same line of ideas, it's a pity that some classical music recordings are very distorted. Some have been recorded by famous players who are long dead and carry important significance. Maybe there is a way to transform noisy recordings into ones that sound as if they were recorded in a modern studio. They could first try to emulate noise and distortions and generate a dataset of distorted recordings, then train a NN to reconstruct the original.

stepvhen 11y ago
I, for one, would not appreciate it if entire swaths of b/w photographs were needlessly colorized for "archiving", especially the Ansel Adams. His photography was too good for the exact medium he chose, and colorizing them detracts from the image. Colorization is super cool and whatnot, but I don't think it should be applied to art.
- deleted 11y ago
  [deleted]

sarreph 11y ago

My first thought was "I hope this is someone showing off their photo-editing skills because if an algorithm did this then my mind is fully blown"... Either this or it's an advanced April Fool's release (with back-dating and references)!

oceanofsolaris 11y ago

This is rather impressive. Similarly to the cited work http://tinyclouds.org/colorize/ http://tinyclouds.org/colorize/, it does however seem to have a slight bias towards brownish colors (giving many images a sepia look). One thing very noticeable in both works is also the tendency to overshoot borders when coloring. In many images it can be seen that the color of the main subject bleeds into the background. A good example is e.g. the fifth picture in http://richzhang.github.io/colorization/resources/images/exs_sel_aa.jpg http://richzhang.github.io/colorization/resources/images/exs... , where parts of the sky get assigned the same color as the mesa. This looks like something that should be fixable if one tries to detect these kind of edges and gives less weight to information from the other side of such an edge. Does anyone know what the differences between this and the previous neural net approach are on a technical level?

neckro23 11y ago
The Ansel Adams examples are interesting because AA very deliberately manipulated the relative intensity of tones in his photographs. In fact, I'm not sure how applicable the current training method (desaturate a color photo, colorize it, compare it to the original) is to actual B&W photographs. Black-and-white photography is not a simple matter of desaturation.

tyingq 11y ago

It's impressive, but they do mention that the examples on the front page are where it did well. You can see some of the other results, including many there didn't work well at all, here: http://richzhang.github.io/colorization/resources/imagenet_comparison.html http://richzhang.github.io/colorization/resources/imagenet_c...

web007 11y ago
They have the "best" and "worst" matches at the bottom of the page. I thought the "worst" set was pretty good, and I wonder if correcting for white balance would have given them a HUGE benefit. The majority of the errors look like daylight-for-tungsten correction, just tweak the yellows to blues FTW.
- tyingq 11y ago
  It seems to guess red at odd times, like http://richzhang.github.io/colorization/resources/images/imagenet_comparison//ours/9839.jpg http://richzhang.github.io/colorization/resources/images/ima... Is that white balance related?

dharma1 11y ago

Very impressive

ickwabe 11y ago

This is very impressive but I'll be that person. I shudder when I see someone/something colorize the works of masters of black-and-white like Ansel Adams or Henri Cartier-Bresson. Just a horrible mangling of their work. Adams for example could have used color, and did take many hundreds of color photos[1]. But in his work for display he deliberately and consciously chose black-and-white and put in much effort in the lab to get the final images just they way he wanted them. So while I truly appreciate the technological achievement here, I still cringe at the what I see as distorting the artists' work. [1]http://www.smithsonianmag.com/arts-culture/ansel-adams-in-color-145315674/?no-ist http://www.smithsonianmag.com/arts-culture/ansel-adams-in-co...

mistercow 11y ago
I really don't think the intention is to say "look we improved this!", but rather to have a visually striking (since the original photo is visually striking) result that doesn't have a "ground truth color" to compare it to.
- ickwabe 11y ago
  Yes of course and I appreciate that. Just my opinion but in that case I would not have used images with such artistic weight. There certainly plenty black and white images to choose from.

saurik 11y ago

http://richzhang.github.io/colorization/resources/imagenet_comparison.html http://richzhang.github.io/colorization/resources/imagenet_c... This is amazing. Looking at the example comparison list, I love seeing the algorithm "correct" the color based on its biases. "No, people do not have pink hair: I made you blond." "Shoes are always a shade of brown (including black); the idea of shoes that are pink, purple, or even grey, morally offends me." "I don't understand why you would paint the frame of this window that crazy shade of green... let's go with a dark stain; doesn't that look classier?" "I've seen this insect before: I don't remember its name, but it was definitely brown, so stop trying to trick me into thinking it is bright blue with your silly Photoshop shenanigans." "Also: I've seen muzzles before, and they are made out of rusted metal; I don't know where you got this blue rubber-coated comfort muzzle, but back in my day we didn't have those extravagances." "On a similar note: I don't understand how your watch is glowing blue... back lights are supposed to be a dull and mottled yellow." "One red stripe on your truck is enough to make the point: three red stripes makes you look like you have no taste :/." "And why would you go to the trouble of forcing people to take a photo in front of a custom backdrop and not take the opportunity to print the name of your company in an eye-catching color, such as red?" "I am pretty certain all radiators everywhere are colored using the same aging white paint that has started browining from the heat; it gives a comfortable feeling of familiarity that your bright green radiator lacks." "Mittens are supposed to be festive, so why did you get grey ones?! I've colored them red: that should help."

cesarbs 11y ago

Am I the only one confused by the title? I honestly thought it would be an April Fool's joke :) The title makes me think it's about coloring colored images.