12 ms·

Show HN: I made a new AI colorizer

472 points by emilwallner 4y ago

Hi HN, I’m Emil, the maker behind Palette. I’ve been tinkering with AI and colorization for about five years. This is my latest colorization model. It’s a text-based AI colorizer, so you can edit the colorizations with natural language. To make it easier to use, I also automatically create captions and generate filters. Let me know what you think. You can see some of my results on my reddit page: https://www.reddit.com/user/emilwallner/?sort=top https://www.reddit.com/user/emilwallner/?sort=top

nextaccountic 4y ago
Can I paint some colors as a starting point (like img2img) and let it finish the rest? What about a conversation, like, "the dress isn't blue, it should be orange", doing on top of previous prompt?
- emilwallner 4y ago
  not yet, these are great suggestions. it's always a dilemma to add features to mitigate the performance of a weak model, instead of making a better model. most of the problems go away with a better language and colorization model, and many model-specific features are made in vain
  - nextaccountic 4y ago
    I think there are two uses for an AI colorizer. One is to generate a color image that looks great, another is to generate an image that accurately reflects the true color of things. A better AI model helps a lot with the first goal, but help only so far with the second one. Truth to be told, there is a lot of contextual color information in black and white photos that an AI model can exploit; but nothing beats someone that knows, for sure, the color of the dress of someone in the photo. I mean, take a look at https://www.reddit.com/r/ColorizedHistory/ https://www.reddit.com/r/ColorizedHistory/ - some of those color artists do a lot of research to know the exact shade of green of the military uniform of some country in the 19th century, and things like that, just to have an accurate reference. So I think that the ability of directing the color output (either by rejecting a color textually, or by painting over the figure with a starting point - even if maybe I'm not painting with the exact tone or texture but a rough color that should help the AI to figure out the details) is essential for a colorization product, even if the model is flawless!
    - nextaccountic 4y ago
      Actually, thinking of /r/colorizedhistory, here's a comparison of a professional colorization and palette.fm https://www.reddit.com/r/ColorizedHistory/comments/y5mfqu/palestinian_woman_wedad_khuri_in_finely/ https://www.reddit.com/r/ColorizedHistory/comments/y5mfqu/pa... vs https://www.reddit.com/r/ColorizedHistory/comments/y5mfqu/palestinian_woman_wedad_khuri_in_finely/isnaa9c/ https://www.reddit.com/r/ColorizedHistory/comments/y5mfqu/pa... My concern here isn't that the professional photo has higher quality (in this case it has, but give it some time - months or years - and maybe technology will catch up). It's that sometimes we already know the right color, while the AI must always guess
      - emilwallner 4y ago
        true, thanks for adding context!
- w_for_wumbo 4y ago
  So you can kind of do that, the first step creates a description, and the second step colorizes using the description. So you can modify the text for various degrees of success and specify 'orange dress within the text'
deleted 4y ago
[deleted]
speedgoose 4y ago
How does it work? Do you write scientific papers?
- emilwallner 4y ago
  I haven't made a write-up of this yet, I still need to figure out how to self-fund it. I normally do more layman scientific writing like this: https://artsandculture.google.com/story/the-klimt-color-enigma/SQWxuZfE5ki3mQ?hl=en https://artsandculture.google.com/story/the-klimt-color-enig...
  - fritztastic 4y ago
    Amazing work! Thanks for sharing!
WrtCdEvrydy 4y ago
Wow, any way to run this locally? I love this technology but it would feel kinda terrible to upload a whole bunch of stuff to your site and exploit your generosity.
- emilwallner 4y ago
  not for now. i might make a local version in the future, but for now, enjoy!
- seanw444 4y ago
  Yeah agreed. I feel a bit icky running computationally-expensive stuff like this on someone else's dime. Give us a crypto address or something to donate to! Preferably not PayPal, though, if my say mattered.
aix1 4y ago
Quite an incredible compilation of photos on your reddit page, thanks for sharing.
- emilwallner 4y ago
  thanks, that means a lot!
MitPitt 4y ago
This is great. I'd also love to have the option to manually draw guiding colors over parts of the image, for even more control.
waltbosz 4y ago
Very cool. Your results look much better than the washed out AI colorization that I've seen in the past. I think you could charge money for this service. A suggestion for something fun to do / marketing tool: recolorize this video frame-by-frame https://www.youtube.com/watch?v=hZ1OgQL9_Cw https://www.youtube.com/watch?v=hZ1OgQL9_Cw Feature request: colorize B&W comic books. I really want to create a full color book of Calvin & Hobbes comics. (Not for publication)
- emilwallner 4y ago
  Thanks! yeah, a few people have used the tool to colorize videos, frame by frame. For example Lord of the flies (1963): https://www.dailymotion.com/video/x8eiho4 https://www.dailymotion.com/video/x8eiho4 Although, I'd recommend colorizing a few key frames and then use https://github.com/zhangmozhe/Deep-Exemplar-based-Video-Colorization https://github.com/zhangmozhe/Deep-Exemplar-based-Video-Colo... Cool, yeah, my next model will be better for comic books. You can also use the 'Surprise Me' button in the editor and you'll get some decent results.
tanvach 4y ago
Nice UI, and the approach works quite well (occasional wrong choice of palette compared to ground truth). It’s kind of fun to turn my photos into B/W and run through the model to try and guess the output. Do you have a privacy policy for the uploaded photos? I’m not keen on uploading anything important without knowing how it’s stored or will be used in the future.
- emilwallner 4y ago
  cheers! i'm working on a proper privacy policy. I don't store any images that are uploaded. i use google analytics and mixpanel to store user interactions.
nakedgremlin 4y ago
Thank you for providing pre-set examples on your site. That's one of the major barriers on these types of tools to "test" especially when you're not sure where/how/if the images are public.
shubhamjain 4y ago
Awesome job! This makes me a bit envious. I am hobbyist colorizer [1] who did it the old school way (with tools like Affinity Photo and lot of manual work). I tried de-oldify and other tools but I justified my work thinking how horrible they were but pallette.fm is way too good. Not sure if I would find the motivation to restore old photos anymore. Glad that this was just a small hobby for me and I had just started learning the ropes. But I would be dead scared if I was a professional[2] who did this full-time. Is this what AI taking your job feels like? [1]: https://shubhamjain.co/experiments/ https://shubhamjain.co/experiments/ [2]: https://www.youtube.com/watch?v=vubuBrcAwtY https://www.youtube.com/watch?v=vubuBrcAwtY
- emilwallner 4y ago
  ty! i'm glad you enjoy the tool. As for whether or not this tool could replace a professional colorist, I think it depends on the specific project. For some projects, Palette could do a great job of automatically colorizing photos, and for others, a professional colorist would still be necessary to get the best results. Especially when the projects require historical accuracy or a high aesthetic standard. it also makes colorization more accessible, which leads to more opportunities to refine results manually or say print the results.
  - Valgrim 4y ago
    Even if the results are not 100% historically perfect, 99% of the hard job is done. Skin tones, plants, skies, water... they all look incredibly good, and the object segmentation is almost perfect too. The tool seems to struggle with fabrics, but that part is by far the easiest to fix with a traditional photo editor. Congrats man. You made my mom happy this evening. Please keep a free tier on your tool.
    - throwaway14356 4y ago
      its fast and looks absolutely stunning but if you've done enough hand work you can still spot flaws (that are easy to correct by hand). The easiest to see is probably the use of black in parts of the face that are not dark shadows. Its slightly under-colored, it had to draw the line some place. It looks like black smudge's left and right of the nose. Then again, few/no one would notice without looking for it. edit: if a human made these this comment would not exist. It makes A fun angle of AI, it can take criticism and do useful things with it. Many humans fail that challange.
JoeAltmaier 4y ago
I love the way it finds features and labels the picture. It thinks I am a meteorologist!
brontosaurusrex 4y ago
All I can say is: very very nice (Testing on some 3d renders and some bw photos). p.s. Is output image width limited to 1920px?
- emilwallner 4y ago
  ty!! yes, 1920x1920. i'm working on an unlimited resolution option, but it will take a few weeks.
writeslowly 4y ago
I tried it on the photo linked below and it makes the string instruments look like they're made of brass. Not trying to bash it since I thought it was pretty impressive overall, but I'm curious about what leads to this type of failure. https://commons.wikimedia.org/wiki/File:Jazzing_orchestra_1921.png https://commons.wikimedia.org/wiki/File:Jazzing_orchestra_19...
- emilwallner 4y ago
  thanks for the feedback! it's made out of two models, one model creates a caption and the second model takes the caption and the black and white image and colorizes it. if you click on the edit button you can see the text that generated that colorization. if the text is incorrect, you can edit the text and recolorize it. this often leads to a better result, however, some cases are still hard, especially damaged photos.
  - cercatrova 4y ago
    I assume you're using CLIP or BLIP for the text generation of the model, and then img2img or something like that for the colorization. What model are you using for the latter colorization?
- learndeeply 4y ago
  > Vivid Natural — A slide of jam, jazzman, hook, band, pipe, and coil. Small contrasting details in natural colors. vivid natural. This one generates the string instruments with the correct colors.
FL33TW00D 4y ago
Excellent work - enjoyed the rick roll. Would you mind sharing how you've done the GPU deployment?
- emilwallner 4y ago
  lol, thanks! onnx, docker, and fastAPI on CPUs with AWS fargate. although i'm switching to GPUs in a month or two, so if you have any suggestions let me know.
- punkspider 4y ago
  I've also been searching for GPU solutions for potential small projects like this, and so far banana.dev and runpod.io seem promising.
kurtreed 4y ago
I haven't tried any other colorizer tools but this one works very well for me. Thank you.
callumprentice 4y ago
Thank you so much Emil - I have some old photos B/W of my dad who is no longer with us and have wanted to colorize them somehow for years. The results from just dropping the photos on your page - no tweaking or whatnot - are incredible and have me in tears. Amazing work.
- nathell 4y ago
  Comments like this are what make me come back to HN. I hope the memories are kept alive.
  - callumprentice 4y ago
    Thank you so much. My daughter is away at camp - I can't wait to show her tomorrow too when she gets back.
- loceng 4y ago
  I can't remember the name but there's a Reddit subreddit where people can post photos to request colorization, so that may be an option if you're curious what some passionate volunteers might do for you.
  - callumprentice 4y ago
    Great suggestion - thank you.
BizarroLand 4y ago
Just a quick heads up, when I ran the file and then opened the result in irfanview, I get a warning that the file is a JPG file with an incorrect extension. You may want to check your encoding settings to make sure everything gels together. Otherwise, great job! This is pretty nice stuff and way better than I could do on my own!
shisisms 4y ago
This is incredibly well done! Congrats. It’s such a solid layer on top of recent developments and offers instant value. Wish you much success!!
learndeeply 4y ago
Can you summarize the approach you took on creating the model? Really curious!
zxcvbn4038 4y ago
Wow! I am impressed. I uploaded a WWII photo of airmen posing in front of their plane and got an amazing result. I’m not certain if WWII era life vests were bright yellow, that seems out of place, but it’s a good guess if not.
- karamanolev 4y ago
  Found some that are yellow, not sure how bright we're talking - https://www.ima-usa.com/products/original-u-s-wwii-usaaf-mae-west-pneumatic-life-vest-with-dye-marker-dated-april-28-1944?variant=18851658596421 https://www.ima-usa.com/products/original-u-s-wwii-usaaf-mae...
  - zxcvbn4038 4y ago
    Jaw dropping, that's the approximate shade it used in the picture. Here is the B&W photo I tested with (I think "base" or "ambient historic" palettes are best): https://1.bp.blogspot.com/-ppnklwWgzIE/TYub-kkx8cI/AAAAAAAAAnw/vGLIaSHh5wU/s1600/b17_with_crew_01.jpg https://1.bp.blogspot.com/-ppnklwWgzIE/TYub-kkx8cI/AAAAAAAAA... Here is an similar, actual color, photo: http://4.bp.blogspot.com/-gzlivCEVuMk/VPzsJUVlYMI/AAAAAAADm_g/XtTqfgCDXdA/s1600/Rare+Color+Photographs+from+World+War+II+(27).jpg http://4.bp.blogspot.com/-gzlivCEVuMk/VPzsJUVlYMI/AAAAAAADm_...
  - zxcvbn4038 4y ago
    It also did really well with the different skin tones of the Tuskegee Airmen: https://i.pinimg.com/originals/21/7e/16/217e161c9e6cfe74861b2d12ec3b7b6f.jpg https://i.pinimg.com/originals/21/7e/16/217e161c9e6cfe74861b... It did not do so well with this grainy photo of soldiers w/ a captured German flag: https://i.redd.it/hbus98dh8sfz.jpg https://i.redd.it/hbus98dh8sfz.jpg If you use the "vintage charm" filter it does a good job with the flag raising at Iwo Jima: https://blog.uspatriottactical.com/wp-content/uploads/2020/02/Raising_the_Flag_on_Iwo_Jima-scaled.jpg https://blog.uspatriottactical.com/wp-content/uploads/2020/0...
kmnc 4y ago
This works amazingly well, just putting in some of my favorite black and white scenes from movies... wow! Now I need some kind of tool that can do this on every frame of a black and white movie. I cannot wait to be able to watch some old classics fully colorized.
surfsvammel 4y ago
This is magic… How the heck does it know that the walls of my home gym is light green?!? Impressive!
draugadrotten 4y ago
I laughed out loud reading your ABOUT page. Thank you!
knarf180 4y ago
Wow.
theden 4y ago
Wow I put a few b+w photos, initially shot in colour so I can compare how it did, and it more or less nailed it—great work. They were shot on a iphone, so I wonder if newer digitally-shot photos have more data for the app to parse, or if it's generally easier compared to film (and whether it uses any EXIF data)
- poulpy123 4y ago
  I expect the training to be done mostly on modern digital cameras
3xa 4y ago
Amazing. Colorized a rare picture of my grandfather. Thanks!!!
napier 4y ago
This is a top tier side project.
poniko 4y ago
Any chance you would license the model for us to use with pixlr.com?
- emilwallner 4y ago
  i'm planning on doing some sort of API, feel free to ping me at emil@palette[dot]fm with the constraints you have
3pt14159 4y ago
I thought from the title it was some JavaScript to auto-colorize essentially any programming language and was excited, because that stuff is annoying to implement for any new programming language out there. Still cool though.
logifail 4y ago
(Genuine) question, and no snark intended: why do so many people want to change history? Black and white photos are - for many of us - part of our shared historical record. Is there really a need to improve (=change) them? Can't we appreciate them exactly as they are, without modifications?
- croes 4y ago
  Some use picture to imagine how life was back then. Life wasn't black and white. Same with the facial expressions in old photos. People look pretty serious but that's just because of the technology pf photography back then. People were as silly and joyful as nowadays.
  - logifail 4y ago
    > Some use picture to imagine how life was back then. Life wasn't black and white. Life (time) doesn't tend to stand still, either, yet we are able to appreciate photographs. Would we really benefit from old photographs being AI-animated into "videos"? Perhaps I'm showing my age, but the older I get the more I feel at one with life's imperfections. I'm fine without filters and HDR ... or colour ... or motion.
    - kevmo314 4y ago
      > Would we really benefit from old photographs being AI-animated into "videos"? Yeah that would be pretty cool.
    - croes 4y ago
      >Would we really benefit from old photographs being AI-animated into "videos"? Some people seem to think that https://news.ycombinator.com/item?id=26279372 https://news.ycombinator.com/item?id=26279372
    - npteljes 4y ago
      I think that the filters, HDR, or the guesswork restorations of old media are similarly life's imperfections - just that the life is more current, compared to the original date of taking these photos.
smortaz 4y ago
tested on a few pics. excellent colorization. pls provide patreon etc link so the site stays up! thxxx
holoduke 4y ago
It's funny to let it operate on a pile of random Lego blocks of different colours. Funny how it sometimes uses a completely different colour. In some other cases it guesses it completely right. Sometimes you also see a from one color to another color gradient, while the real thing has one color with a shadow gradient. How large is your trainingsset?
Wistar 4y ago
This is quite amazing, and useful. Great work.
djleni 4y ago
Heh, a lot of the results are more visually appealing than the real landscape I shot in B&W.
hilyen 4y ago
I ran a test with a photo I took if people are curious. It does a great job for guessing, but of course there is room for improvement. https://i.postimg.cc/Y9dH4GxW/palette-fm-test.jpg https://i.postimg.cc/Y9dH4GxW/palette-fm-test.jpg
- mfgs 4y ago
  Cool. Interesting to note that the original is not necessarily the "correct" real world color palette (it's accuracy is subject to camera settings, sensor response, lighting, etc), so the colorized version could in theory be even better than the original in some or all areas of the photo.
  - npteljes 4y ago
    As well as there are multiple desaturation methods, GIMP alone offers at least three, and I'm sure that could influence the outcome as well. https://docs.gimp.org/2.8/en/gimp-tool-desaturate.html https://docs.gimp.org/2.8/en/gimp-tool-desaturate.html
  - hilyen 4y ago
    100% I think my original photo is a bit oversaturated. The real issues I see are where it makes blue things brown and red things green.
- batuhandirek 4y ago
  I love how the emphasis on initial letters are understood and colored red!
akouris 4y ago
Amazing work congratulations!
DerWOK 4y ago
Thanks, this is awesome. Colorized a few last images of my WWII grandfather who died in this war. One observation: on download the MIME type seems to by PNG format, but the binary file arriving is actually a (lossy compressed) JPEG. But anyhow! Great work!!
littlestymaar 4y ago
Unfortunately, this has the common bias[1] with IA coloring of making old stuff look dull, reinforcing the Hollywood cliché[2]. The past was actually often very colorful. [1]: https://nitter.lacontrevoie.fr/gwenckatz/status/1381652071695351810#m https://nitter.lacontrevoie.fr/gwenckatz/status/138165207169... [2]: see this video by a movie props maker about why Hollywood movies make old things look the way they do in movie https://youtu.be/mF1VFlCnLQ4?t=434 https://youtu.be/mF1VFlCnLQ4?t=434
meerita 4y ago
This blew my mind. Amazing.
vanillax 4y ago
Open Source? Would love to see how this works and also the website.
cercatrova 4y ago
Nice, maybe I can turn the movie The Lighthouse from black and white into a color motion picture. I hear color "movies" as they call 'em are what's hot these days. Original: https://i.imgur.com/9CZ7vk1.png https://i.imgur.com/9CZ7vk1.png Base Palette: https://i.imgur.com/iTq5H9W.jpeg https://i.imgur.com/iTq5H9W.jpeg
- bwoodward 4y ago
  It looks pretty great, actually -- https://i.imgur.com/fxZz6f7.jpg https://i.imgur.com/fxZz6f7.jpg (Base), https://i.imgur.com/2JAHnbe.jpg https://i.imgur.com/2JAHnbe.jpg (Vivid Natural)
  - cercatrova 4y ago
    I tried it as well (edited my comment above), I like the Base Palette option better, it looks more realistic compared to Vivid Natural which looks like a WW2 movie filter with the brown atmosphere.
atum47 4y ago
that's incredible. congratulations on the great work and thanks for sharing with us.
jbellis 4y ago
I was actually testing Photoshop's new colorizing filter this week, so this is timely. My first impression of Palette in comparison is that it does a comparably good job of colorizing. Sometimes Photoshop does better, sometimes Palette -- usually the skin tones is the hardest part, and one does a noticeably better job than the other. The natural language modifications available in Palette ("his shirt is light blue") are super useful. Well done.
w_for_wumbo 4y ago
This is absolutely mind-blowing to me, is this using an API of any sort? I'd love to be able to take something like my City's archive of Black & White images and colorize them.
zimpenfish 4y ago
Tried a few old family photos and it's done a good job - mostly better than the other ones I've tried. Excellent work.
wigster 4y ago
very cool. just coloured some old family tree pics and they look great. thanks.
jetrink 4y ago
I've tried a few of my own black and white photos and the description of each starts with "a stock photo of a..." Is this a comment on my style of photography or is this always added for the base palette prompt? (The colorized images have been very impressive.)
krisdol 4y ago
I had some black and white photos of my passed-away tuxedo cat, fully expecting this to produce a terrible result, but instead I was blown away by accurate and subtle colors added. Pinkish nose, ears, green eyes all came out of the photo.
default-user 4y ago
I've never posted here before but this demands a sincere "thank you". I just colorised some family photos from the 1930s to the 80s and the results are heartwarming. Thank you for making this and making it free and, most importantly, easy to use.
waffletower 4y ago
This is a really impressive colorizer. Don't feel guilty monetizing.
dirtyid 4y ago
Incredible service. Just annoyed once I show it to my folks I'll be dragged into multi hour tutorial session on how to scan / upload old albums. Also curious which "filter" they feel will most reflect the subjective reality of the past.
w_for_wumbo 4y ago
I immediately was limiting this to think it would just colorize B&W images, but you can use this on images with colour to use natural language to modify the colours of an existing image too!
amelius 4y ago
It would be nice if you could put hex colors in the prompts. E.g. instead of saying "man wearing blue shirt", you could say "man wearing #0010e4 shirt".
afalzetti 4y ago
Wow, it’s awesome! What technologies does it use? I’ve used to restore the only photo my wife has of her father who died when she was 3yo. She almost cried seeing the colori ed photo, thank you
FpUser 4y ago
I tried some of my old university photos. Totally amazing. Thank you so much for a wonderful work
jaflo 4y ago
How are the prompts/captions generated? Also I love that you're using Svelte!
adeptima 4y ago
Hi Emil, thank you for this!
chicob 4y ago
Today I was thinking of some sort of NN trained for colorizing photos, and then I get here and there's an AI right there... This tool works really well. The text prompt helps, too. The suggested colors leak to surrounding objects (I forced a red skirt on a traditional costume I knew was probably red), but it looks really good. Did you decolorize pictures originally in color in order to train the model?
kristopolous 4y ago
I gave it ansel adams photos which it did really well on. Then I tried some famous national geographic photos, like the these: https://www.nationalgeographic.com/photography/article/meet-grandfather-flash-the-pioneer-of-wildlife-photography https://www.nationalgeographic.com/photography/article/meet-... still impressed I decided to go really old, so I tried "Valley of the Shadow of Death" from the 1850s where we get some red and blue cannonballs, but that's fine. 1838 Louis Daguerre photo of paris: https://9ol.es/1838.png https://9ol.es/1838.png 1906 Kite aerial of san francisco after the earthquake: http://9ol.es/1906.png http://9ol.es/1906.png 1895 Montparnasse train wreck: http://9ol.es/1895.png http://9ol.es/1895.png 1920 Wall Street bombing: http://9ol.es/1920.png http://9ol.es/1920.png 1920s Park Row skyscrapers: http://9ol.es/parkrow.png http://9ol.es/parkrow.png
wfme 4y ago
Looks great and works particularly well. A couple of bug reports: - Clicking on the P logo on the home page takes me to a page with what looks to be an example, but none of the filters ever load. - Navigating directly to https://palette.fm/color/filters https://palette.fm/color/filters returns a 500.
nine_k 4y ago
Colorizing b/w photos is one thing; these are limited representation of colorful reality. Now let's colorize b/w works of Albrecht Dürer, or Francisco Goya.
WheelsAtLarge 4y ago
The ease of use,speed and accuracy are impressive. I tried doing a colorization by hand a few years ago using photoshop. It took a long time and it was not as good. Over the years, I've read that a products that's 10x better will be a game changer. I think you have, at least, a 10x better product. Congratulations! I'll be using your service for my B/W prints. Great job!
puttycat 4y ago
This is wonderful. Works really well. Thank you!
fuzzythinker 4y ago
Nice. Combine with restoration like this https://www.reddit.com/r/MachineLearning/comments/j01y9u/d_bringing_old_photos_back_to_life_microsofts/ https://www.reddit.com/r/MachineLearning/comments/j01y9u/d_b... would bring life to many old photos.
Jack_rando_fang 4y ago
First time getting rick rolled in plain text :)
asimpleusecase 4y ago
Is there a way I could contact you directly?
diamondage 4y ago
How does this differ from deoldify?
TomMasz 4y ago
I tried one of my recent B&W film images. It basically recreated the original scene (a graffitied building). Nice!

472 points by emilwallner 4y ago

nextaccountic 4y ago

Can I paint some colors as a starting point (like img2img) and let it finish the rest? What about a conversation, like, "the dress isn't blue, it should be orange", doing on top of previous prompt?

emilwallner 4y ago
not yet, these are great suggestions. it's always a dilemma to add features to mitigate the performance of a weak model, instead of making a better model. most of the problems go away with a better language and colorization model, and many model-specific features are made in vain
- nextaccountic 4y ago
  I think there are two uses for an AI colorizer. One is to generate a color image that looks great, another is to generate an image that accurately reflects the true color of things. A better AI model helps a lot with the first goal, but help only so far with the second one. Truth to be told, there is a lot of contextual color information in black and white photos that an AI model can exploit; but nothing beats someone that knows, for sure, the color of the dress of someone in the photo. I mean, take a look at https://www.reddit.com/r/ColorizedHistory/ https://www.reddit.com/r/ColorizedHistory/ - some of those color artists do a lot of research to know the exact shade of green of the military uniform of some country in the 19th century, and things like that, just to have an accurate reference. So I think that the ability of directing the color output (either by rejecting a color textually, or by painting over the figure with a starting point - even if maybe I'm not painting with the exact tone or texture but a rough color that should help the AI to figure out the details) is essential for a colorization product, even if the model is flawless!
  - nextaccountic 4y ago
    Actually, thinking of /r/colorizedhistory, here's a comparison of a professional colorization and palette.fm https://www.reddit.com/r/ColorizedHistory/comments/y5mfqu/palestinian_woman_wedad_khuri_in_finely/ https://www.reddit.com/r/ColorizedHistory/comments/y5mfqu/pa... vs https://www.reddit.com/r/ColorizedHistory/comments/y5mfqu/palestinian_woman_wedad_khuri_in_finely/isnaa9c/ https://www.reddit.com/r/ColorizedHistory/comments/y5mfqu/pa... My concern here isn't that the professional photo has higher quality (in this case it has, but give it some time - months or years - and maybe technology will catch up). It's that sometimes we already know the right color, while the AI must always guess
    - emilwallner 4y ago
      true, thanks for adding context!
w_for_wumbo 4y ago
So you can kind of do that, the first step creates a description, and the second step colorizes using the description. So you can modify the text for various degrees of success and specify 'orange dress within the text'

deleted 4y ago

[deleted]

speedgoose 4y ago

How does it work? Do you write scientific papers?

emilwallner 4y ago
I haven't made a write-up of this yet, I still need to figure out how to self-fund it. I normally do more layman scientific writing like this: https://artsandculture.google.com/story/the-klimt-color-enigma/SQWxuZfE5ki3mQ?hl=en https://artsandculture.google.com/story/the-klimt-color-enig...
- fritztastic 4y ago
  Amazing work! Thanks for sharing!

WrtCdEvrydy 4y ago

Wow, any way to run this locally? I love this technology but it would feel kinda terrible to upload a whole bunch of stuff to your site and exploit your generosity.

emilwallner 4y ago
not for now. i might make a local version in the future, but for now, enjoy!
seanw444 4y ago
Yeah agreed. I feel a bit icky running computationally-expensive stuff like this on someone else's dime. Give us a crypto address or something to donate to! Preferably not PayPal, though, if my say mattered.

aix1 4y ago

Quite an incredible compilation of photos on your reddit page, thanks for sharing.

emilwallner 4y ago
thanks, that means a lot!

MitPitt 4y ago

This is great. I'd also love to have the option to manually draw guiding colors over parts of the image, for even more control.

waltbosz 4y ago

Very cool. Your results look much better than the washed out AI colorization that I've seen in the past. I think you could charge money for this service. A suggestion for something fun to do / marketing tool: recolorize this video frame-by-frame https://www.youtube.com/watch?v=hZ1OgQL9_Cw https://www.youtube.com/watch?v=hZ1OgQL9_Cw Feature request: colorize B&W comic books. I really want to create a full color book of Calvin & Hobbes comics. (Not for publication)

emilwallner 4y ago
Thanks! yeah, a few people have used the tool to colorize videos, frame by frame. For example Lord of the flies (1963): https://www.dailymotion.com/video/x8eiho4 https://www.dailymotion.com/video/x8eiho4 Although, I'd recommend colorizing a few key frames and then use https://github.com/zhangmozhe/Deep-Exemplar-based-Video-Colorization https://github.com/zhangmozhe/Deep-Exemplar-based-Video-Colo... Cool, yeah, my next model will be better for comic books. You can also use the 'Surprise Me' button in the editor and you'll get some decent results.

tanvach 4y ago

Nice UI, and the approach works quite well (occasional wrong choice of palette compared to ground truth). It’s kind of fun to turn my photos into B/W and run through the model to try and guess the output. Do you have a privacy policy for the uploaded photos? I’m not keen on uploading anything important without knowing how it’s stored or will be used in the future.

emilwallner 4y ago
cheers! i'm working on a proper privacy policy. I don't store any images that are uploaded. i use google analytics and mixpanel to store user interactions.

nakedgremlin 4y ago

Thank you for providing pre-set examples on your site. That's one of the major barriers on these types of tools to "test" especially when you're not sure where/how/if the images are public.

shubhamjain 4y ago

Awesome job! This makes me a bit envious. I am hobbyist colorizer [1] who did it the old school way (with tools like Affinity Photo and lot of manual work). I tried de-oldify and other tools but I justified my work thinking how horrible they were but pallette.fm is way too good. Not sure if I would find the motivation to restore old photos anymore. Glad that this was just a small hobby for me and I had just started learning the ropes. But I would be dead scared if I was a professional[2] who did this full-time. Is this what AI taking your job feels like? [1]: https://shubhamjain.co/experiments/ https://shubhamjain.co/experiments/ [2]: https://www.youtube.com/watch?v=vubuBrcAwtY https://www.youtube.com/watch?v=vubuBrcAwtY

emilwallner 4y ago
ty! i'm glad you enjoy the tool. As for whether or not this tool could replace a professional colorist, I think it depends on the specific project. For some projects, Palette could do a great job of automatically colorizing photos, and for others, a professional colorist would still be necessary to get the best results. Especially when the projects require historical accuracy or a high aesthetic standard. it also makes colorization more accessible, which leads to more opportunities to refine results manually or say print the results.
- Valgrim 4y ago
  Even if the results are not 100% historically perfect, 99% of the hard job is done. Skin tones, plants, skies, water... they all look incredibly good, and the object segmentation is almost perfect too. The tool seems to struggle with fabrics, but that part is by far the easiest to fix with a traditional photo editor. Congrats man. You made my mom happy this evening. Please keep a free tier on your tool.
  - throwaway14356 4y ago
    its fast and looks absolutely stunning but if you've done enough hand work you can still spot flaws (that are easy to correct by hand). The easiest to see is probably the use of black in parts of the face that are not dark shadows. Its slightly under-colored, it had to draw the line some place. It looks like black smudge's left and right of the nose. Then again, few/no one would notice without looking for it. edit: if a human made these this comment would not exist. It makes A fun angle of AI, it can take criticism and do useful things with it. Many humans fail that challange.

JoeAltmaier 4y ago

I love the way it finds features and labels the picture. It thinks I am a meteorologist!

brontosaurusrex 4y ago

All I can say is: very very nice (Testing on some 3d renders and some bw photos). p.s. Is output image width limited to 1920px?

emilwallner 4y ago
ty!! yes, 1920x1920. i'm working on an unlimited resolution option, but it will take a few weeks.

writeslowly 4y ago

I tried it on the photo linked below and it makes the string instruments look like they're made of brass. Not trying to bash it since I thought it was pretty impressive overall, but I'm curious about what leads to this type of failure. https://commons.wikimedia.org/wiki/File:Jazzing_orchestra_1921.png https://commons.wikimedia.org/wiki/File:Jazzing_orchestra_19...

emilwallner 4y ago
thanks for the feedback! it's made out of two models, one model creates a caption and the second model takes the caption and the black and white image and colorizes it. if you click on the edit button you can see the text that generated that colorization. if the text is incorrect, you can edit the text and recolorize it. this often leads to a better result, however, some cases are still hard, especially damaged photos.
- cercatrova 4y ago
  I assume you're using CLIP or BLIP for the text generation of the model, and then img2img or something like that for the colorization. What model are you using for the latter colorization?
learndeeply 4y ago
> Vivid Natural — A slide of jam, jazzman, hook, band, pipe, and coil. Small contrasting details in natural colors. vivid natural. This one generates the string instruments with the correct colors.

FL33TW00D 4y ago

Excellent work - enjoyed the rick roll. Would you mind sharing how you've done the GPU deployment?

emilwallner 4y ago
lol, thanks! onnx, docker, and fastAPI on CPUs with AWS fargate. although i'm switching to GPUs in a month or two, so if you have any suggestions let me know.
punkspider 4y ago
I've also been searching for GPU solutions for potential small projects like this, and so far banana.dev and runpod.io seem promising.

kurtreed 4y ago

I haven't tried any other colorizer tools but this one works very well for me. Thank you.

callumprentice 4y ago

Thank you so much Emil - I have some old photos B/W of my dad who is no longer with us and have wanted to colorize them somehow for years. The results from just dropping the photos on your page - no tweaking or whatnot - are incredible and have me in tears. Amazing work.

nathell 4y ago
Comments like this are what make me come back to HN. I hope the memories are kept alive.
- callumprentice 4y ago
  Thank you so much. My daughter is away at camp - I can't wait to show her tomorrow too when she gets back.
loceng 4y ago
I can't remember the name but there's a Reddit subreddit where people can post photos to request colorization, so that may be an option if you're curious what some passionate volunteers might do for you.
- callumprentice 4y ago
  Great suggestion - thank you.

BizarroLand 4y ago

Just a quick heads up, when I ran the file and then opened the result in irfanview, I get a warning that the file is a JPG file with an incorrect extension. You may want to check your encoding settings to make sure everything gels together. Otherwise, great job! This is pretty nice stuff and way better than I could do on my own!

shisisms 4y ago

This is incredibly well done! Congrats. It’s such a solid layer on top of recent developments and offers instant value. Wish you much success!!

learndeeply 4y ago

Can you summarize the approach you took on creating the model? Really curious!

zxcvbn4038 4y ago

Wow! I am impressed. I uploaded a WWII photo of airmen posing in front of their plane and got an amazing result. I’m not certain if WWII era life vests were bright yellow, that seems out of place, but it’s a good guess if not.

karamanolev 4y ago
Found some that are yellow, not sure how bright we're talking - https://www.ima-usa.com/products/original-u-s-wwii-usaaf-mae-west-pneumatic-life-vest-with-dye-marker-dated-april-28-1944?variant=18851658596421 https://www.ima-usa.com/products/original-u-s-wwii-usaaf-mae...
- zxcvbn4038 4y ago
  Jaw dropping, that's the approximate shade it used in the picture. Here is the B&W photo I tested with (I think "base" or "ambient historic" palettes are best): https://1.bp.blogspot.com/-ppnklwWgzIE/TYub-kkx8cI/AAAAAAAAAnw/vGLIaSHh5wU/s1600/b17_with_crew_01.jpg https://1.bp.blogspot.com/-ppnklwWgzIE/TYub-kkx8cI/AAAAAAAAA... Here is an similar, actual color, photo: http://4.bp.blogspot.com/-gzlivCEVuMk/VPzsJUVlYMI/AAAAAAADm_g/XtTqfgCDXdA/s1600/Rare+Color+Photographs+from+World+War+II+(27).jpg http://4.bp.blogspot.com/-gzlivCEVuMk/VPzsJUVlYMI/AAAAAAADm_...
- zxcvbn4038 4y ago
  It also did really well with the different skin tones of the Tuskegee Airmen: https://i.pinimg.com/originals/21/7e/16/217e161c9e6cfe74861b2d12ec3b7b6f.jpg https://i.pinimg.com/originals/21/7e/16/217e161c9e6cfe74861b... It did not do so well with this grainy photo of soldiers w/ a captured German flag: https://i.redd.it/hbus98dh8sfz.jpg https://i.redd.it/hbus98dh8sfz.jpg If you use the "vintage charm" filter it does a good job with the flag raising at Iwo Jima: https://blog.uspatriottactical.com/wp-content/uploads/2020/02/Raising_the_Flag_on_Iwo_Jima-scaled.jpg https://blog.uspatriottactical.com/wp-content/uploads/2020/0...

kmnc 4y ago

This works amazingly well, just putting in some of my favorite black and white scenes from movies... wow! Now I need some kind of tool that can do this on every frame of a black and white movie. I cannot wait to be able to watch some old classics fully colorized.

surfsvammel 4y ago

This is magic… How the heck does it know that the walls of my home gym is light green?!? Impressive!

draugadrotten 4y ago

I laughed out loud reading your ABOUT page. Thank you!

knarf180 4y ago

Wow.

theden 4y ago

Wow I put a few b+w photos, initially shot in colour so I can compare how it did, and it more or less nailed it—great work. They were shot on a iphone, so I wonder if newer digitally-shot photos have more data for the app to parse, or if it's generally easier compared to film (and whether it uses any EXIF data)

poulpy123 4y ago
I expect the training to be done mostly on modern digital cameras

3xa 4y ago

Amazing. Colorized a rare picture of my grandfather. Thanks!!!

napier 4y ago

This is a top tier side project.

poniko 4y ago

Any chance you would license the model for us to use with pixlr.com?

emilwallner 4y ago
i'm planning on doing some sort of API, feel free to ping me at emil@palette[dot]fm with the constraints you have

3pt14159 4y ago

I thought from the title it was some JavaScript to auto-colorize essentially any programming language and was excited, because that stuff is annoying to implement for any new programming language out there. Still cool though.

logifail 4y ago

(Genuine) question, and no snark intended: why do so many people want to change history? Black and white photos are - for many of us - part of our shared historical record. Is there really a need to improve (=change) them? Can't we appreciate them exactly as they are, without modifications?

croes 4y ago
Some use picture to imagine how life was back then. Life wasn't black and white. Same with the facial expressions in old photos. People look pretty serious but that's just because of the technology pf photography back then. People were as silly and joyful as nowadays.
- logifail 4y ago
  > Some use picture to imagine how life was back then. Life wasn't black and white. Life (time) doesn't tend to stand still, either, yet we are able to appreciate photographs. Would we really benefit from old photographs being AI-animated into "videos"? Perhaps I'm showing my age, but the older I get the more I feel at one with life's imperfections. I'm fine without filters and HDR ... or colour ... or motion.
  - kevmo314 4y ago
    > Would we really benefit from old photographs being AI-animated into "videos"? Yeah that would be pretty cool.
  - croes 4y ago
    >Would we really benefit from old photographs being AI-animated into "videos"? Some people seem to think that https://news.ycombinator.com/item?id=26279372 https://news.ycombinator.com/item?id=26279372
  - npteljes 4y ago
    I think that the filters, HDR, or the guesswork restorations of old media are similarly life's imperfections - just that the life is more current, compared to the original date of taking these photos.

smortaz 4y ago

tested on a few pics. excellent colorization. pls provide patreon etc link so the site stays up! thxxx

holoduke 4y ago

It's funny to let it operate on a pile of random Lego blocks of different colours. Funny how it sometimes uses a completely different colour. In some other cases it guesses it completely right. Sometimes you also see a from one color to another color gradient, while the real thing has one color with a shadow gradient. How large is your trainingsset?

Wistar 4y ago

This is quite amazing, and useful. Great work.

djleni 4y ago

Heh, a lot of the results are more visually appealing than the real landscape I shot in B&W.

hilyen 4y ago

I ran a test with a photo I took if people are curious. It does a great job for guessing, but of course there is room for improvement. https://i.postimg.cc/Y9dH4GxW/palette-fm-test.jpg https://i.postimg.cc/Y9dH4GxW/palette-fm-test.jpg

mfgs 4y ago
Cool. Interesting to note that the original is not necessarily the "correct" real world color palette (it's accuracy is subject to camera settings, sensor response, lighting, etc), so the colorized version could in theory be even better than the original in some or all areas of the photo.
- npteljes 4y ago
  As well as there are multiple desaturation methods, GIMP alone offers at least three, and I'm sure that could influence the outcome as well. https://docs.gimp.org/2.8/en/gimp-tool-desaturate.html https://docs.gimp.org/2.8/en/gimp-tool-desaturate.html
- hilyen 4y ago
  100% I think my original photo is a bit oversaturated. The real issues I see are where it makes blue things brown and red things green.
batuhandirek 4y ago
I love how the emphasis on initial letters are understood and colored red!

akouris 4y ago

Amazing work congratulations!

DerWOK 4y ago

Thanks, this is awesome. Colorized a few last images of my WWII grandfather who died in this war. One observation: on download the MIME type seems to by PNG format, but the binary file arriving is actually a (lossy compressed) JPEG. But anyhow! Great work!!

littlestymaar 4y ago

Unfortunately, this has the common bias[1] with IA coloring of making old stuff look dull, reinforcing the Hollywood cliché[2]. The past was actually often very colorful. [1]: https://nitter.lacontrevoie.fr/gwenckatz/status/1381652071695351810#m https://nitter.lacontrevoie.fr/gwenckatz/status/138165207169... [2]: see this video by a movie props maker about why Hollywood movies make old things look the way they do in movie https://youtu.be/mF1VFlCnLQ4?t=434 https://youtu.be/mF1VFlCnLQ4?t=434

meerita 4y ago

This blew my mind. Amazing.

vanillax 4y ago

Open Source? Would love to see how this works and also the website.

cercatrova 4y ago

Nice, maybe I can turn the movie The Lighthouse from black and white into a color motion picture. I hear color "movies" as they call 'em are what's hot these days. Original: https://i.imgur.com/9CZ7vk1.png https://i.imgur.com/9CZ7vk1.png Base Palette: https://i.imgur.com/iTq5H9W.jpeg https://i.imgur.com/iTq5H9W.jpeg

bwoodward 4y ago
It looks pretty great, actually -- https://i.imgur.com/fxZz6f7.jpg https://i.imgur.com/fxZz6f7.jpg (Base), https://i.imgur.com/2JAHnbe.jpg https://i.imgur.com/2JAHnbe.jpg (Vivid Natural)
- cercatrova 4y ago
  I tried it as well (edited my comment above), I like the Base Palette option better, it looks more realistic compared to Vivid Natural which looks like a WW2 movie filter with the brown atmosphere.

atum47 4y ago

that's incredible. congratulations on the great work and thanks for sharing with us.

jbellis 4y ago

I was actually testing Photoshop's new colorizing filter this week, so this is timely. My first impression of Palette in comparison is that it does a comparably good job of colorizing. Sometimes Photoshop does better, sometimes Palette -- usually the skin tones is the hardest part, and one does a noticeably better job than the other. The natural language modifications available in Palette ("his shirt is light blue") are super useful. Well done.

w_for_wumbo 4y ago

This is absolutely mind-blowing to me, is this using an API of any sort? I'd love to be able to take something like my City's archive of Black & White images and colorize them.

zimpenfish 4y ago

Tried a few old family photos and it's done a good job - mostly better than the other ones I've tried. Excellent work.

wigster 4y ago

very cool. just coloured some old family tree pics and they look great. thanks.

jetrink 4y ago

I've tried a few of my own black and white photos and the description of each starts with "a stock photo of a..." Is this a comment on my style of photography or is this always added for the base palette prompt? (The colorized images have been very impressive.)

krisdol 4y ago

I had some black and white photos of my passed-away tuxedo cat, fully expecting this to produce a terrible result, but instead I was blown away by accurate and subtle colors added. Pinkish nose, ears, green eyes all came out of the photo.

default-user 4y ago

I've never posted here before but this demands a sincere "thank you". I just colorised some family photos from the 1930s to the 80s and the results are heartwarming. Thank you for making this and making it free and, most importantly, easy to use.

waffletower 4y ago

This is a really impressive colorizer. Don't feel guilty monetizing.

dirtyid 4y ago

Incredible service. Just annoyed once I show it to my folks I'll be dragged into multi hour tutorial session on how to scan / upload old albums. Also curious which "filter" they feel will most reflect the subjective reality of the past.

w_for_wumbo 4y ago

I immediately was limiting this to think it would just colorize B&W images, but you can use this on images with colour to use natural language to modify the colours of an existing image too!

amelius 4y ago

It would be nice if you could put hex colors in the prompts. E.g. instead of saying "man wearing blue shirt", you could say "man wearing #0010e4 shirt".

afalzetti 4y ago

Wow, it’s awesome! What technologies does it use? I’ve used to restore the only photo my wife has of her father who died when she was 3yo. She almost cried seeing the colori ed photo, thank you

FpUser 4y ago

I tried some of my old university photos. Totally amazing. Thank you so much for a wonderful work

jaflo 4y ago

How are the prompts/captions generated? Also I love that you're using Svelte!

adeptima 4y ago

Hi Emil, thank you for this!

chicob 4y ago

Today I was thinking of some sort of NN trained for colorizing photos, and then I get here and there's an AI right there... This tool works really well. The text prompt helps, too. The suggested colors leak to surrounding objects (I forced a red skirt on a traditional costume I knew was probably red), but it looks really good. Did you decolorize pictures originally in color in order to train the model?

kristopolous 4y ago

I gave it ansel adams photos which it did really well on. Then I tried some famous national geographic photos, like the these: https://www.nationalgeographic.com/photography/article/meet-grandfather-flash-the-pioneer-of-wildlife-photography https://www.nationalgeographic.com/photography/article/meet-... still impressed I decided to go really old, so I tried "Valley of the Shadow of Death" from the 1850s where we get some red and blue cannonballs, but that's fine. 1838 Louis Daguerre photo of paris: https://9ol.es/1838.png https://9ol.es/1838.png 1906 Kite aerial of san francisco after the earthquake: http://9ol.es/1906.png http://9ol.es/1906.png 1895 Montparnasse train wreck: http://9ol.es/1895.png http://9ol.es/1895.png 1920 Wall Street bombing: http://9ol.es/1920.png http://9ol.es/1920.png 1920s Park Row skyscrapers: http://9ol.es/parkrow.png http://9ol.es/parkrow.png

wfme 4y ago

Looks great and works particularly well. A couple of bug reports: - Clicking on the P logo on the home page takes me to a page with what looks to be an example, but none of the filters ever load. - Navigating directly to https://palette.fm/color/filters https://palette.fm/color/filters returns a 500.

nine_k 4y ago

Colorizing b/w photos is one thing; these are limited representation of colorful reality. Now let's colorize b/w works of Albrecht Dürer, or Francisco Goya.

WheelsAtLarge 4y ago

The ease of use,speed and accuracy are impressive. I tried doing a colorization by hand a few years ago using photoshop. It took a long time and it was not as good. Over the years, I've read that a products that's 10x better will be a game changer. I think you have, at least, a 10x better product. Congratulations! I'll be using your service for my B/W prints. Great job!

puttycat 4y ago

This is wonderful. Works really well. Thank you!

fuzzythinker 4y ago

Nice. Combine with restoration like this https://www.reddit.com/r/MachineLearning/comments/j01y9u/d_bringing_old_photos_back_to_life_microsofts/ https://www.reddit.com/r/MachineLearning/comments/j01y9u/d_b... would bring life to many old photos.

Jack_rando_fang 4y ago

First time getting rick rolled in plain text :)

asimpleusecase 4y ago

Is there a way I could contact you directly?

diamondage 4y ago

How does this differ from deoldify?

TomMasz 4y ago

I tried one of my recent B&W film images. It basically recreated the original scene (a graffitied building). Nice!