3 ms·
The 'averaging' problem is interesting and I think related to the Euclidean-style loss function: it is correctly minimizing the color distance between its gray-
by gwern 11y ago
The 'averaging' problem is interesting and I think related to the Euclidean-style loss function: it is correctly minimizing the color distance between its gray-brown guess and all the possible shades? If that's what's going on, then messing with the architecture won't fix it because it's already reaching the 'right' answer and what is necessary is a better loss... Perhaps something like DCGAN where the loss function is another CNN which is being trained to guess whether an image is original or generated? (So you'd feed the BW image into the generator, which converts it into a colorized image; the colorized and the true color images are both fed into the discriminator, which tries to guess which is the real one. Then error is backpropagated into both.)
- darkmighty 11y agoI too am fascinated by this problem... Your adversarial proposal is very interesting. Let me try to confront the fundamental problem directly. Even if you're given a perfect probability distribution over the space of images the solution wasn't obvious for me, mostly because we're used to thinking of a "best estimate". The first thing you think of is giving the least-squares estimate (the average), but MMSE exhibits the problem shown. So you might instead try a maximum likelihood estimate; but this too has problems: imagine every car is a sightly different shade of blue (none are quite the same, maybe the manufacturing is unreliable), except 1 in a million cars are red, but the red is very consistent. The ML estimate will pick the red car, which of course is unrealistic. The optimal solution is simply drawing from the underlying distribution, instead of relying on a deterministic "best estimate": an outside observer won't be able to distinguish your generated samples from the true distribution. That's why the "Adversarial discriminator" should work. I wonder if there exists a cost function that directly promotes sampling from the underlying distribution without needing the adversarial approach...
- Houshalter 11y agoYou could have the net predict an entire distribution for each pixel. Like a mixture of gaussians or something. But then sampling from it would be incorrect. E.g. it might not know if the car is blue or red, so half the pixels would randomly be red, and the other have blue. It would look terrible. Somewhere, the neural net needs to decide "this car is going to be blue" and then be consistent with that. Adversarial nets allow that, by having random inputs. One of the inputs to the NN is a random number, and that random number might determine if the car is going to be blue or red this time. The cool thing about this is that it allows you to generate multiple samples. You can generate 10 different images and select the best one. And the adversarial nets should learn to approximate the true distribution as closely as possible. And I don't think there is any other method that can do that. Another idea would to just have a loss function that doesn't punish it for getting a wrong color. But rewards it only when it gets very close to the right color. This way the algorithm doesn't worry about producing muddy brown colors when it isn't' sure, it just goes with a best guess.
- darkmighty 11y agoWhen I referred to sampling I was talking about the joint distribution of all pixels (hence "space of images"), which would work fine. But I suspect predicting distributions is impractical, it may be better to use methods that sample directly without ever explicitly finding the distribution. You do need a source of entropy to perform the sampling. This amount should be more than a minimum given by how precisely you want to sample from the continuous source, related to the Kullback-Leibler divergence of the distribution.
- Houshalter 11y agoWell in theory, the adversarial nets should learn to model the distribution perfectly. But there might be a way to do it directly. You could train an NN to produce samples from a random input source, just like the adversarial nets. But unlike the adversarial nets, the inputs don't need to be random. You could train another NN to predict a distribution of what they should be. And then instead of adversarial training, just regular training to predict the exact pixels, and backproping all the way through.
- deleted 11y ago[deleted]
- Houshalter 11y agoOne other problem is that "real" black and white images don't come from the same distribution as greyscaled images. The training data is just regular photos that have had all the color channels combined and averaged together. Sometimes it's a weighted average, like green is given more weight than blue, because human eyes see green better. However real black and white photos represent the actual intensity of the light waves. Black and white photos might be sensitive to light humans can't even see, or weight different "colors" very differently. Even two colors that appear the same to the human eye might be different intensities and frequencies, and so produce a different black and white photo. I don't know how much of an effect this would have. But real black and white photos definitely look different than greyscaled color photos, and it might lead the NN to guessing the wrong color information.
- derefr 11y agoInteresting point; the "perceptual brightness" formula used in grayscaling calculations is usually this: sqrt(0.299r² + 0.587g² + 0.114b²) Real "photosensitizing" chemicals, on the other hand, have spectral response curves, like this one[1] for the silver iodide used in daguerrotypes. (The formula above just being an approximation of the summed spectral response curves of the three cones in the eye.) There's a big difference, like you said, but it's not an impossible problem; we should be able to create a greyscaling formula for any photographic process with a known response curve. (And honestly, I have no reason to think this hasn't already been done in the creation of an Instagram filter at some point.) [1] https://books.google.ca/books?id=9IpaIAcgthQC&lpg=PA119&ots=zEWTsY30T3&dq=silver%20iodide%20photosensitivity%20spectrum&pg=PA120#v=onepage&q=silver%20iodide%20photosensitivity%20spectrum&f=false https://books.google.ca/books?id=9IpaIAcgthQC&lpg=PA119&ots=...
- Houshalter 11y agoWell not necessarily. If the response curves are different than that of a digital camera, then some information will be lost. In fact the response curves are different even for different cameras. But you could probably get a closer approximation using a better formula.
- 11y ago
- eutectic 11y agoThis would also hopefully solve the 'splotchiness' seen in some of the images, which presumably comes from the network optimising each pixel independently without caring about continuity.
- gwern 11y agoHere's someone using DCGAN to fix blur: https://swarbrickjones.wordpress.com/2016/01/13/enhancing-images-using-deep-convolutional-generative-adversarial-networks-dcgans/ https://swarbrickjones.wordpress.com/2016/01/13/enhancing-im... I wonder if it would work for colorizing...