12 ms·
Color-Diffusion: using diffusion models to colorize black and white images
- snvzz 3y agoAll the examples are portraits of people. I have to wonder whether it works well with anything else.
- erwannmillon 3y agotrained on celebA, so no, but you could for sure train this on a more varied dataset
- Eisenstein 3y agoWould it be as simple as feeding it a bunch of decolorized images along with the originals?
- atorodius 3y agoyes, so infinite training data. but the challenge will be scaling to large resolutions and getting global consistency
- drapado 3y agoI guess you can always use a two-stage process. First colorize, then upscale
- atorodius 3y agoyeah, you can use SOTA super res, but that tends to be generative too (even diffusion based on its own, or more commonly based on GANs). it can be a challenge to synthesize the right high res details. but that’s basically the stable diffusion paper (diffusion in latent space plus GAN superres)
- erwannmillon 3y agoYeah, if you have a high res image, you can get color info at super low-res and then regenerate the colors at high res with another model. (though this isn't an efficient approach at all) https://github.com/TencentARC/T2I-Adapter https://github.com/TencentARC/T2I-Adapter i've also seen a controlnet do this.
- jrockway 3y agoIs that challenging? Humans have awful color resolution perception, so even if you have a huge black-and-white image, people would think it looks right with even with very low-resolution color information. Or, if the AI hallucinates a lot of high frequency color noise, it wouldn't be noticable. Wikipedia has a great example image here: https://en.wikipedia.org/wiki/Chroma_subsampling https://en.wikipedia.org/wiki/Chroma_subsampling. Most people would say all of them looked fine at 1:1 resolution.
- atorodius 3y agoI meant more from a comoute standpoint, the models are expensive to run full res
- jrockway 3y agoI see what you mean. I think that you can happily scale the B&W image down, run the model, and then scale the chroma information back up. Something I was thinking about after writing the comment is that the model is probably trained on chroma-subsampled images. Digital cameras do it with the bayer filter, and video cameras add 4:2:0 subsampling or similar subsampling as they compress the image. So the AI is probably biased towards "look like this photo was taken with a digital camera" versus "actually reconstruct the colors of the image". What effect this actually has, I don't know!
- atorodius 3y agogood point, I hadn’t realized that you only need to predict chroma! That actully greatly simplifies things re. chroma subsampling in training data: this is actually a big problem and a good generative model will absolutely learn to predict chroma subsampled values (or JPEG artifacts even!). you can get around it by applying random downscaling with antialiasing during training.
- erwannmillon 3y agobasically the training works as follows: Take a color image in RGB. Convert it to LAB. This is an alternative color space where the first channel is a greyscale image, and two channels that represent the color information. In a traditional pixel-space (non latent) diffusion model, you noise all the RGB channels and train a Unet to predict the noise at a given timestep. When colorizing an image, the Unet always "knows" the black and white image (i.e the L channel). This implementation only adds noise to the color channels, while keeping the L channel constant. So to train the model, you need a dataset of colored images. They would be converted to LAB, and the color channels would be noised. You can't train on decolorized images, because the neural network needs to learn how to predict color with a black and white image as context. Without color info, the model can't learn.
- coldtea 3y ago>You can't train on decolorized images, because the neural network needs to learn how to predict color with a black and white image as context. Without color info, the model can't learn. I think the parent means with delocorized images used to test the success and guide the training (since they can be readily compared with the colored image they resulted from which would be the perfect result). Not to use decolorized images alone to train for coloring (which doesn't even make sense).
- atorodius 3y agoYou can take arbitrary images and convert them to grayscale for training, and do conditional diffusion
- bemusedthrow75 3y agoBut convert them to grayscale how? Black and white film doesn't have one single colour sensitivity. Play around with something like DxO FilmPack sometime (it has excellent measurement-based representations of black and white film stocks). It's a much more complex problem than it might seem on the surface.
- 3y ago
- bemusedthrow75 3y agoColourising old photographs is the banal apotheosis application of diffusion AI. It's the pinnacle of the whole thing: "imagine it for me in a way that conforms to my contemporary expectations". If you're going to colourise images, have the decency to do it by hand. If possible on a print with brushes. Edit: didn't think this would be popular. Maybe it's the historical photography nerd in me, but colourising images without effort and thought is like smashing vintage glass windows for the fun of it: cultural vandalism.
- crazygringo 3y agoIf you're going to write code, have the decency to do it on punch cards. If possible by hand punching, rather than using a keypunch machine.
- bemusedthrow75 3y agoThis isn't the point I am making. The point I am making is that colourisation is subjective art, and that alone. Colourisation cannot fail to enforce contemporary biases based on poor understanding of the materials. It will darken or lighten skin inappropriately, and mislead in any number of ways. Doing it by hand (in photoshop or on a print) acknowledges the inherent bias that is involved in colourisation. Automating it is banal at best and dangerous at worst; colourised images risk distorting history.
- coldtea 3y ago>Automating it is banal at best and dangerous at worst; colourised images risk distorting history Well, faces still have a certain tint, the sky is mostly blue, the grass green, water is blue, mud pools are brown, the ground too, a lot of historical fabrics are certain inherent colors, known flowers have known colors, brownstones have red/brown color. A lot of it, is just not that subjective. Besides different color film stock (or camera sensor "color science") can already result in dozens of widely different colorings of the same exactly scene.
- 3y ago
- buildbot 3y agoDoes it work on arbitrary image sizes? One of the nice features of the somewhat old Deoldify colorizer is support for any resolution. It actually does better than photoshops colorization: https://blog.maxg.io/colorizing-infrared-images-with-photoshop/ https://blog.maxg.io/colorizing-infrared-images-with-photosh... Edit - technically, I suppose, the way Deoldify works is by rendering the color at a low resolution and then applying the filter to a higher resolution using OpenCV. I think the same sub-sampling approach could work here...
- erwannmillon 3y agoTechnically yes, the encoder and unet are convolutional and support arbitrary input sizes, but the model was trained at 64x64px bc of compute limitations. You could probably resume the training from a 64x64 resolution checkpoint and train at a higher resolution. But like most diffusion models, they don't generalize very well to resolutions outside of their training dataset
- aziaziazi 3y agoHow much would it cost to colorize a movie with a fork of this?
- morelisp 3y ago[flagged]
- NBJack 3y agoI think the bigger question is would it be stable enough. Many SD like models struggle with consistency across multiple images (i.e. frames) even when content doesn't change much. Would he a cool problem to see tackled.
- erwannmillon 3y agotemporal coherence is def an issue with these types of models, though I haven't tested it out with ColorDiffusion. Assuming you're not doing anything autoregressive (from frame to frame) to do temporal coherence, you can also parallelize the colorization of each frame, which would affect cost. Tbh most cost effective would be a conditional GAN though
- lajamerr 3y agoChange up the model. That allows it to see previous frames and 1-2 future frames. Then train the model on movies that are color and then turn them black and white. That way you can train temporal coherence.
- leetharris 3y agoQuick math: 24 frames per second * 60 seconds per minute * 90 minute movie length = 129600 frames If you could get cost to a penny per frame, about $13k? But I'd bet you could easily get it an order of magnitude less in terms of cost. So $1500 or so? And that's assuming you do 100% of frames and don't have any clever tricks there.
- 3y ago
- data-ottawa 3y agoOff topic: this has an absolutely 90’s sci-fi movie effect watching the gifs, it’s funny how the tech just wound up looking like that.
- erwannmillon 3y agohahaha it reminded me of some "zoom and enhance" stuff when I was making the animations
- nerdponx 3y agoLooks like something you'd see in an X Files episode.
- barrkel 3y agoIt reminded me of the days of antenna pass-through VCR players, where you had to tune into your VCR's broadcast signal when you couldn't use SCART.
- Cthulhu_ 3y agoI never knew that was a thing, today I learned. I was spoiled with our first VCR having SCART already :p. And an IR remote! We could put the antenna cable into the VCR then use the remote to change channels (all three) instead of having to walk up to the TV. (this was late 80's, maybe early 90's; I wonder if we were late with things like that)
- jurassic 3y agoThis is a cool party trick, but I don't see a need for this in any real applications. Black and white is its own art form, and a lot of really great black and white images would look like absolute garbage if you could convert them to color. This is because the things that make a great black and white image (dramatic contrasts, emphasis on shape/geometry, texture, etc) can lose a lot of their impact when you introduce color. Our aesthetic tolerance for contrast seems significantly reduced in color because our expectations for the image are more anchored in how things look in the real world. And colors which can be very pleasing in some images are just distracting in others. So all this is to say.... I don't think there would be commercial demand to, say, "upgrade" classic movies with color. Those films were shot by cinematographers who were steeped in the black & white medium and made lighting and compositional choices that take greatest advantage of those creative limitations.
- pythonguython 3y agoCounterexample: They Shall Not Grow Old, a WW1 documentary film with mostly colorized footage with recreated audio. The film was commercially successful and I found it to be a great watch.
- dragonwriter 3y ago> I don't think there would be commercial demand to, say, "upgrade" classic movies with color. There was, and maybe there will be again once we get far enough from the consumer burnout from the absolute deluge of that in, mostly, the 1980s-1990s. https://en.m.wikipedia.org/wiki/List_of_black-and-white_films_that_have_been_colorized https://en.m.wikipedia.org/wiki/List_of_black-and-white_film...
- MrVandemar 3y agoSome of the old Doctor Who stories that were filmed in colour they only have black and white copies of. The colourisations have been ... very good, better than I would have thought, but not perfect. Could be an a good application.
- deleted 3y ago[deleted]
- 3y ago
- ChrisArchitect 3y agoAuthor's writeup on this from May: https://medium.com/@erwannmillon/color-diffusion-colorizing-black-and-white-images-with-diffusion-models-269828f71c81 https://medium.com/@erwannmillon/color-diffusion-colorizing-...
- erwannmillon 3y agoBtw, I did this in pixel space for simplicity, cool animations, and compute costs. Would be really interesting to do this as an LDM (though of course you can't really do the LAB color space thing, unless you maybe train an AE specifically for that color space. ) I was really interested in how color was represented in latent space and ran some experiments with VQGAN clip. You can actually do a (not great) colorization of an image by encoding it w/ VQGAN, and using a prompt like "a colorful image of a woman". Would be fun to experiment with if anyone wants to try, would love to see any results if someone wants to build
- xigency 3y agoQuestion, how long did it take to train this model and what hardware did you use?
- erwannmillon 3y agoTook a lot of failed experiments, the model would keep converging to greyscale / sepia images. Think one of the ways I fixed was by adding an greyscale encoder to the arch. Used its output embedding as additional conditioning. Can't remember if I only added it to the Unet input or injected it during various stages of the unet down pass.
- erwannmillon 3y agoThink the final training run was only a couple hours on a Colab V100
- carbocation 3y ago> I did this in pixel space for simplicity, cool animations, and compute costs A slight nitpick, wouldn't doing diffusion in the latent space be cheaper?
- ShamelessC 3y agoNot necessarily if you don’t already have a pretrained autoencoder.
- asciimov 3y agoI’m not a fan of b&w colorization. Often the colors are wrong, either outright color errors (like choices for clothing or cars) or often not taking in to account lighting conditions (late in day shadows but midday brightness). Then there is the issue of B&W movies. Using this kind of tech might not give pleasing results as the colors used for sets and outfits were chosen to work well for film contrast and not for story accuracy. That “blue” dress might really be green. (Please, just leave B&W movies the way they are.)
- zamadatix 3y agoI think colorization with some effort put in can be pretty decent. E.g. I prefer the 2007 colorization of It's a Wonderful Life to the original. It's never perfect but I don't think that's a prerequisite to being better. Some will always disagree though. About every completely automated colorized video tends to be pretty bad though. Particularly the YouTube "8k colorized interpolated" kind of low effort channels where they just let them pump out without caring if it's actually any good.
- imoverclocked 3y agoI think keeping the art as it was produced is important but there is also a good history of modifying art to produce new art too. In the digital age, we aren’t losing the original art so it seems even stranger to be against modification of the “original.” However, just applying a simple filter (or single transform without effort) definitely feels derivative to me.
- SiempreViernes 3y agoAdditionally, colorisations very commonly present themselves as showing a "more true" version of how things looked and not as creative art projects.
- esquivalience 3y agoI have always viewed them as "more relatable". People may well be biased to think of more relatable things as more true, but I don't think that is the fault of the colourisation or how it is presented.
- iamflimflam1 3y agoI wonder if this can be used for color correction in videos.
- quickthrower2 3y agoThat is so 2023 and so 1993 at the same time!
- realusername 3y agoIs there anything that exists right now with diffusion models to improve poor VHS coloring? The coloring does exist so I would not want to replace a red shirt by a blue shirt for example but it's just not very accurate.
- amelius 3y agoCould this type of modeling be used to generate segmentations of images?