17 ms·
Deep Photo Style Transfer
- boxcardavin 10y agoThis is super fun and impressive, I'm going to get it going on my machine and start playing immediately.
- starik36 10y agoA write-up on how to get it going would be appreciated.
- taherchhabra 10y ago+1
- oarfish 10y agoDon't get your hopes up just yet. It's not usable in its current state, unless you have Matlab (octave doesn't work, see closed+open Github issues). There are efforts under way to port the matlab code to python, but i haven't tried the fork yet. Also, processing a single image on a K80 takes several hours, no comparison to neural style. Likewise, generating the laplacians also takes a loooong time, at least with octave. Maybe this will be accelerated in an python version.
- sixQuarks 10y agoWhat was up with the "apple" photo?
- arketyp 10y agoPresumably it is a demonstration of the network extracting the texture of an image, ignoring "spatiality", such that when a style/texture transfer is applied on one and the same image, the texture may very well order up differently than on the original image. More specifically, ignoring spatiality is the effect of matching the response of the covariance matrix of the feature vectors, which was the key insight in the original style transfer paper by Gatys.
- GordonS 10y agoAll 3 looked the same to me, but I guessed it was because I'm colour blind. It's not just that then?
- pveierland 10y agoThe paper states that it shows swapping of texture between apples by using a manual segmentation input. A manual mask is also used to transfer the fireball texture to the perfume bottle.
- teded32 10y agoI think the input and the style are the same photo, which is odd. But if you look at the output; you can see that even though the colors have switched, the patterns and shapes have stayed the same. The middle apple has dots on it, but it is now green. The right apple is kinda spotty, but it is now red.
- camtarn 10y agoIf you look at the URLs, most of them have the input in the 'examples/input/' dir, and the reference in the 'examples/style/' dir. The apple reference is in the 'examples/input/' dir. Perhaps that really does mean that the input and style were identical, and they deliberately re-used the same image URL, but to me it feels like a copy-and-paste error :)
- jackcarter 10y agoThe "style" image (tar23.png) is a symlink to the input image[0], so I guess it's intentional. I'm still not sure why that ended up swapping the colors in the output, though. [0] https://github.com/luanfujun/deep-photo-styletransfer/tree/master/examples/style https://github.com/luanfujun/deep-photo-styletransfer/tree/m...
- veli_joza 10y agoSome impressive results! I would like to see some heavily stylized examples like achieving Sin City and Scanner Darkly visuals. Was it really necessary to involve Matlab, Python and Lua?
- hayksaakian 10y agonow if you implemented this as a web app, i'd be sharing it with everybody really cool samples, would be cool to upload my own photos and run it through this
- Joeboy 10y agoThere's http://neuralstyle.com/ http://neuralstyle.com/ , based on different code that does a similar thing . It's quite GPU intensive so it takes a while and I think the free options are limited.
- Sunset 10y agoI can donate free time on a GeForce 1060 6GB and a 12core XEON. If anyone wants to make a web frontend for the code in the linked repo.
- sho12121 10y agoI'd like to spend a bit of time on this and try it out! Write me. Would need a MatLab license to use on the server.
- lucidrains 10y agoUgh Matlab. Someone needs to rewrite it in Julia
- lucidrains 10y agoPing me too, I'd be happy to help build whatever
- jshmrsn 10y agoIt almost causes me goosebumps to think about how if someone asked me "imagine this clear sky, but with this red sunset over here" I could very plausibly come up with a similar result as shown in these examples. The transfer of the flame to the blue perfume bottle looks like a very practical way to prototype marketing images.
- aorth 10y agoYes! I imagine being like, "Damn, I always wanted to see Venice when it was snowing, but alas it was sunny during my short visit."
- paradite 10y agoOr even better, snow in a tropical country, which can never happen in real life.
- maury91 10y agoYou are wrong it happened this year, http://www.italymagazine.com/sites/default/files/styles/940xauto/public/neve_sardegna_alghero.jpg?itok=NqGQKioM http://www.italymagazine.com/sites/default/files/styles/940x...
- wapz 10y agoUnfortunately I don't think this will work with snow. It changes the colors of the objects and the snow too will be changed. That being said, being able to change the sky color and building colors looks so amazing.
- gedy 10y agoWow. Our kids may be able to walk around with glasses and see the world styled in real-time as they please - perpetual sunshine, gloom, night, whatever..
- gmt2027 10y agoRose tinted glasses to shelter them from the legacy of a devastated planet. How about the Black Mirror episode Men Against Fire for a real dystopian vision of this future.
- daxelrod 10y agoThe excellent short film MORE by Mark Osborne is about similar reality-editing glasses: https://www.youtube.com/watch?v=cCeeTfsm8bk https://www.youtube.com/watch?v=cCeeTfsm8bk
- drusepth 10y agoI actually really hope this will turn out to be true. I'm unfamiliar with the processing power required to run this on a single image, let alone a video feed. Does anyone have any back of the napkin estimations that might show just how far away AR glasses are from this?
- coldtea 10y ago>I actually really hope this will turn out to be true. Yeah, screw reality.
- drusepth 10y agoReality is nice. Being able to augment it in various ways would also be nice, and has a ton of applications across a ton of fields. AR games and hollywood filters are the obvious go-tos, but I wouldn't be surprised to see seemingly random industries like therapy see benefits (a few applications that come to mind are brightening up the world to combat SAD, tinting views to ease aggression, or even Black Mirror-esque "blocking" of people who upset or anger you IRL if you want to trend towards dystopian tropes). Personally, I just want to up the world's saturation to max and chill with rainbows and bright colors while I'm otherwise productive. Seems like having fun filters would (or could) make life mildly better for many people going about their usual day.
- npgatech 10y agoI think it would be amazing if Adobe incorporated some of these projects (Neural Style, etc.) as part of their "Creative Cloud" offerings...actually compute it in the cloud and return the result back to Photoshop.
- ttoinou 10y agoThat would be great. I was actually thinking of coding a plugin like that
- phkahler 10y agocode one for darktable or gimp.
- Jhsto 10y agoI don't know whether you noticed it, but the paper reveals it is co-authored by people from Adobe. There very well might be intentions towards that later on.
- ttoinou 10y agoThat might take years
- coldtea 10y agoAdobe has incorporated research stuff they've shown to the public as mere papers/demos into PS in just 1-2 versions several times.
- dagw 10y agoAdobe moves pretty fast when they want to. Take content aware fill for example. It went from a presentation at ACM SIGGRAPH to Adobe hiring some of the people who wrote the paper to a working feature in Photoshop in about a year.
- ttoinou 10y ago@dagwTrue @coldtea True However the technology is not ready yet. Takes too long to render* and needs too much VRAM (=> limited resolution, varying on the current GPU). But a consumer-grade plugin with an online renderer might be feasible for now. * Fast solutions include a pre-training step that can take days. Would be weird to ask the user to do this
- thecopy 10y agoReally cool! This could prove useful for interior designing.
- agumonkey 10y agoI feel overwhelmed by the domain; as an ex photoshop addict, it's already above "complex job" level. I wonder if people feel bad about AI "stealing their hobs" (not a joke).
- hnarayanan 10y agoIf they feel this way, they could try to learn more about this new world. It is fun and opens up so many possibilities! Here is an (in progress) explanation I am working on aimed at beginners that might help: https://harishnarayanan.org/writing/artistic-style-transfer/ https://harishnarayanan.org/writing/artistic-style-transfer/
- sah2ed 10y agoNice work! Definitely bookmarking.
- intoverflow2 10y agoI feel Photoshop/Illustrator and 2D design in general is one of the only industries that has been pretty much stagnant for 15 years or so due to monopolies. Compare it to 3D where artists are drowning in new tools and techniques every year, any movement in this space is a good thing. Worth remembering there was a time when the heal brush didn't exist, a time when the clone brush didn't exist and even a time where the physical airbrush didn't exist.
- agumonkey 10y agoIllustration doesn't need much tech. It has more to do with human emotions and symbols.
- intoverflow2 10y agoFYI although it's called Illustrator it's used for everything from actual illustration to iconography, to web/app design to even basic type layout.
- bedros 10y agoawesome, wish there's no matlab requirement; can the matlab code be converted to python?
- johndough 10y agoPython is Turing Complete, so yes.
- prashnts 10y agoWhile correct, I think GP is more interested in knowing if there're sufficient libraries available in Python so porting MATLAB code to Python is easy. Looking at the source, almost everything in MATLAB can be ported though.
- notthemessiah 10y agoI'm sure it can, but I personally am going to go for MXNet in Julia (which is syntactically similar to MatLab) if I find the time. http://mxnet.io/ http://mxnet.io/
- oarfish 10y agoSomeone did it already: martinbenson/deep-photo-styletransfer
- VMG 10y agoPremium Instagram filters as-a-service?
- drodil 10y agoSo cool :)
- dvcrn 10y agoThis is super impressive and something that I didn't think would be possible without someone very skilled in photoshop going over the images. As a photo enthusiast, I am very excited about this, but also a little worried that soon very simple apps are capable of doing the craziest of edits through the power of neural nets. Imagine the next 'deep beauty transfer', able to copy perfect skin from a model onto everyone, making everything a little more fake and less genuine. The engineer in me now wants to understand how to build something like this from scratch but I think I'm probably lacking the math skills necessary.
- peteretep 10y agoI suspect it's not going to be long until there's a rather creepy "nudity from source image" generator.
- DanielBMarkham 10y agoThere's your killer app. Feed it a million or so porn images of people with all kinds of different body types. Then have it guess the closest match. Finally run this. Presto change-o! It's those x-rays glasses kids everywhere wanted for years. I could easily imagine it being done live-motion and 3-d. Run the whole thing on a set of AR glasses.
- peteretep 10y agoOr morph a face onto existing footage. I'm surprised this doesn't more obviously exist already, although I guess I haven't been actively looking for it.
- DanielBMarkham 10y agoThe tech is there to do a rough approximation of a dozen combinations of this. I could imagine an intermediate step where a wire-frame mesh is constructed around the image. As I understand (And I know nothing of this stuff), there was already an app to take a picture of breasts and jiggle them. I think it's just that nobody has put all of the pieces together yet. (Or if they have, the mainstream media hasn't heard about it)
- johndough 10y agoDoes anyone know how long it takes per image? The only piece of information I could find was that the run script uses 8 GPUs, which suggests that it takes a while.
- jjcm 10y agoProbably a significantly long time. That said, I do wonder if you need to actually run this at the resolution of the output image. Since really this is just changing the tones in the image and not altering the details, you could probably optimize the algorithm heavily such that it works for high resolution images quickly. As an example, I used frequency separation to split the detail layer from the tone layer in the original, high resolution stock photo of SF. From there I took the lower res (25% size) output from this script and used it as my tone layer. The results are OK: http://i.imgur.com/oakLUiE.jpg http://i.imgur.com/oakLUiE.jpg It has the same overall tones, and some of the sharpness is preserved at a high resolution. My 30 second approach suffers from some edge glow, but I'm sure it could be greatly improved in an efficient, automated way.
- oarfish 10y agoWith the linked implementation (there is at least 1 fork rewriting the matlab bits in python) the three steps needed per image amount to probably over an hour on one k80. However, you probably need the -backend cudnn and -cudnn_autotune flags not to run out of memory. Downscaling the images from currently 700 to say 500 significantly speeds up the process as well. Still, it definitely takes longer than neural style.
- vwbuwvb 10y agonotice how the lighting in the style-transferred images isn't physically plausible according to the target images. For example, in the 5th example the house is still lit as in the original, as if by sunlight not by spotlights. Maybe that's why they've chosen to showcase examples with flat or ambiguous lighting (like the nighttime scenes or the autumn scene). The DNN doesn't model the physical reality of the scene, it doesn't get that it's a 3d world, it simply transports a high-dimensional vector (the 2d image) from one space to another. What our imaginations do is map that 2d image back into 3d before transforming it.
- phkahler 10y agoAnd the city images that are styled into night were originally taken near dusk so the lights inside the buildings were already on but not very pronounced. It will not take a full daylight scene and decide where to turn the lights on at night.
- jjcm 10y agoOne interesting thing that may help the final output quality is preserving the detail layer of the original image, and then applying that to the output image. Here's my quick attempt at it: http://dev.jjcm.org/tonetransfer/ http://dev.jjcm.org/tonetransfer/ I basically just used a technique called frequency separation - it's extremely quick with the most computational part being a gaussian blur, and it allows you to separate detail from tone into two separate layers. From there I just took the detail layer of the original image and applied it to the tone layer of the output image.
- omegote 10y agoThat's pretty good, a nice improvement. Frequency separation is one of the most common retouching techniques, but works wonders in this case as well.
- epigramx 10y agoTheir implementation appears to lose a lot of detail in many examples. This is especially true on human faces or pictures that include human faces. Can you reproduce an improvement there?
- jjcm 10y agoI didn't see any examples of human faces, can you link me to one? Happy to try it out and see the results.
- woliveirajr 10y agoI really like decomposing an image using wavelets (more than 2 layers, in general I use 7). You can make some good corrections on skin imperfections in the middle layers, keepin it natural (high frequencies untouched). I'm thinking how it would be if I could you your method (or the OP) on the lower layers, which mainly control colors...
- hamilyon2 10y agoThis is where neural network-generated images might start feeding into other neural networks. Imagine a limited dataset of tagged pictures and a vast number of styles. We could on-demand generate permutations and train network that recognize them much more accurately than they would be able to do with original limited dataset
- gtani 10y agoThat's funny, i was just going over OS X makefile with somebody yesterday and i came here to look for the thunderbolt external GPU adapter. The repo author is super responsive if you're RAM constrained: https://www.reddit.com/r/CUDA/comments/61gpzd/make_error_when_trying_to_compile_cu_file_on_osx/ https://www.reddit.com/r/CUDA/comments/61gpzd/make_error_whe... and https://github.com/luanfujun/deep-photo-styletransfer/issues/5 https://github.com/luanfujun/deep-photo-styletransfer/issues... ________________ similar project https://github.com/jcjohnson/neural-style https://github.com/jcjohnson/neural-style
- gtani 10y agooh, that similar project repo seems to be the inspiration/genesis of this thread's repo. Also that repo's author did the Stanford lecture comparing theano, tensorflow, torch which is very worth digging out of the various archives
- kowdermeister 10y agoNow imagine what we could do if it were computed in real time, 60FPS, 100x resolution/detail, embeded in AR or contact lenses. Yep, sci-fi. But I often imagine that sometime soon I'll have a server farm in my pocket (I know I already has via the cloud, but it's a whole new game if you can do the computation on-site, low latency)
- deleted 10y ago[deleted]
- tabeth 10y agoWow, this is ridiculously impressive. I wonder what the audio equivalent of this would be. Anyone know how long this takes per image?
- visarga 10y ago> I wonder what the audio equivalent of this would be. Voice transfer. Imagine your words in the voice of any celeb.
- zild3d 10y agoor timbre transfer in general. Drum recording on a cheap beginner drumset -> high end Tama kit
- isseu 10y agoSame rhythm with a different instrument? The problem is that audio is very dense for this (now)
- guepe 10y agoFunny I saw a very similar attempt at Fitchburg Art Museum (MA) over the week-end. I seem to recall this was an MIT-project: is this related in some way ? It was coming with a rather complex (at least at first sight) interface allowing quite a bit of operations/transformations, which does not appear here, so this might be a separate attempt. Looks like this thing is "in the air".
- markab21 10y agoIt seems this could have application in conceptual interior design. Think room makeovers.
- wyldfire 10y agoWow -- lua, Python, matlab and CUDA. I see now that the image segmentation is probably the key element to getting these stunning results. Other style transfers I've seen took abstract elements from the donor picture, but this really captures the donor and transforms the recipient significantly.
- vwcx 10y agoI am a professional photo editor for a major magazine. Despite what the title sounds like, I spend much of my day sending photographers on assignment and sourcing images rather than manipulating files. I often wonder how my career will evolve in ten years; the writing is on the wall for my job, given the state of the media market. Looking at this, I now see a future as a forensic imaging specialist. At some point, the algorithm will get pretty damn good, and it will be my job to look for tells -- cracks in the generated image where reality doesn't quite line up. The question will be whether I am seeking out these abnormalities to cover them up or to call them out.
- lucidrains 10y agoyou may need to train some powerful discriminator networks to do that job instead of relying on the one in your head soon..
- hoschicz 10y ago"forensic imaging specialists" already exist in modern GAN networks and are called discriminators. They examine how likely is it that they would be from a human and then provide feedback to the generator network on what a human would do differently.
- saurabh20n 10y agoVery interesting. I work in AI. Curious about trends in your industry. Please ping me at the email address in my profile.
- rl3 10y agoSomeone should create an entire film using this technique. Shoot two films in parallel with similar scenes, then see what happens when they're blended. Give it a creepy and/or surrealist plot so the ethereal-looking output suits the film. Perhaps the visuals could be the result of viewing the world through a robot or AI with imperfect cognition. Would be an interesting twist on the old unreliable narrator trope.
- draw_down 10y agoThis is so cool, such an interesting and great idea. Really impressive and well-done.
- Cofike 10y agoHoly cow, this is amazing. The results are way better than I would ever expect as possible.
- pseudobry 10y agoFor anyone interested in a sci-fi book about how a society filled with this kind of technology might work, I recommend reading Rainbow's End by Vernor Vinge. https://en.wikipedia.org/wiki/Rainbows_End https://en.wikipedia.org/wiki/Rainbows_End
- double051 10y agoThose results look amazing! I'm curious how well it holds up at higher resolutions and closer inspection. Also, there doesn't seem to be any mention of the distribution license for the project. Would any of the maintainers be able to add a license to the repository? Thanks!
- oarfish 10y agothere is an issue discussing that; the problem is that the work was done at adobe labs, so they will determine the license.
- rcthompson 10y agoThis is really cool, but the only thing I can think right now, the question that's eating away at my soul, is: why did it color-cycle the apples?
- isoprophlex 10y agoThere are also semantic masks applied (can be found in the github repo), I think that's how the Colorado swapping is done.
- rcthompson 10y agoAh, I see. They're right here: Input mask: https://github.com/luanfujun/deep-photo-styletransfer/blob/master/examples/segmentation/in23.png https://github.com/luanfujun/deep-photo-styletransfer/blob/m... Output mask: https://github.com/luanfujun/deep-photo-styletransfer/blob/master/examples/segmentation/tar23.png https://github.com/luanfujun/deep-photo-styletransfer/blob/m... So, presumably these masks were drawn manually. The three different colors in the input mask tell it to train 3 different models from the masked regions, and then the same three colors in the output mask tell it where to apply each trained model. So it trains a model for each apple and then applies that model to the next apple over. It's not just randomly swapping colors on the apples.
- alanbernstein 10y agoI'm afraid the answer, like the answers to so many other soul-eating questions, is that it's random.
- deleted 10y ago[deleted]
- jtraffic 10y agoI wondered what was new about this particular implementation (since I've seen several others, notably the one this code is partially based on, from Justin Johnson). From the paper: "Our contribution is to constrain the transformation from the input to the output to be locally affine in colorspace, and to express this constraint as a custom CNN layer through which we can backpropagate. We show that this approach successfully suppresses distortion and yields satisfying photorealistic style transfers in a broad variety of scenarios, including transfer of the time of day, weather, season, and artistic edits" I was skeptical at first (even posted then deleted a sort of negative comment), but now that I read this, I see the value. The images are much more crisp and distortion free.
- folli 10y agoAnyone's got a link to a good write-up on how such a style transfer works on a high level?
- mistercow 10y agoJeez, how many style transfer papers have there been in the last year? It's awesome, but what an odd thing to become its own subfield.
- nvs 10y agoThis is amazing
- a7958479 10y agoA ball
- deleted 10y ago[deleted]