7 ms·
Turning two-bit doodles into fine artworks with deep neural networks
- nuclai 11y ago(Author here.) For details, the research paper is linked on the GitHub page: http://arxiv.org/abs/1603.01768 http://arxiv.org/abs/1603.01768 For a video and higher-level overview see my article from yesterday: http://nucl.ai/blog/neural-doodles/ http://nucl.ai/blog/neural-doodles/ Questions welcome!
- ThePhysicist 11y agoYou should make an app for that (seriously)!
- option_greek 11y agoOr an online generator!
- Untit1ed 11y agoI can't believe he posted it up on github before doing this - there's so much potential for this to go viral once it's packaged with a doodling app. EDIT: Actually reading more closely I guess 10 minutes on a machine with a decent GPU is a lot of server load :|.
- nuclai 11y agoThe research is based on work I did writing and improving @DeepForger (http://twitter.com/deepforger http://twitter.com/deepforger), an online service for "basic" style transfer. The GitHub is a standalone version for learning and education, which doesn't do HD rendering as well yet and uses a bit more memory. The positive side, however, is that opening up the source code makes these ideas progress faster! We'll try to integrate the idea of semantic style transfer into @DeepForger in the future, but this require quite a bit of work to get it to reliably understand portraits or landscapes without anyone's intervention. The fact it does require these semantic maps for all images makes it less straightforward to release as a service.
- TuringTest 11y agoOne question, is the semantic map created on the fly at the same time as the final image is composed, or are the maps pre-computed?
- nuclai 11y agoThe semantic map remains static during the optimization, so it can be provided as a pre-computation (e.g. pixel labeling, semantic segmentation, etc.) or done by hand. The ones in the repository are done manually, but now experimenting with other algorithms. Anything that returns a bitfield or masks can be used!
- mozumder 11y agoWhy CUDA instead of OpenCL? Asking because I'm about to start a GPU compute driven project.
- nuclai 11y agoAlmost nobody in deep learning uses OpenCL. All the DL frameworks primarily focus on CUDA and that's where you get the best performance. OpenCL is off the beaten path and you pay for it in every way: support, performance, reliability. AMD is going to support CUDA somehow too, I think that's a sign they admitted defeat on OpenCL for this.
- amelius 11y agoWhat data has been used to train the neural network?
- nuclai 11y agoIt's a pre-trained network on image classification dataset from 2014 called ImageNet. The network is called VGG, paper is here: http://arxiv.org/abs/1409.1556 http://arxiv.org/abs/1409.1556 There's no additional training apart from that. The neural network is used to extract patterns (grain/texture/style) and a separate optimization tries to reproduce them as appropriate.
- amelius 11y agoInteresting. If A is the input image, and B is the style image, then from which of those two images is the NN extracting patterns? And how is the other image used to get the desired effect? Just trying to get a birds-eye view of the algorithm :)
- nuclai 11y agoBoth images have their patterns extracted by the NN, and the optimization then tries to match the best patches from one image with the other, performing gradient descent to adjust the pixel values from a random start image.
- Dowwie 11y agoHave you run children's paintings through this yet?
- nuclai 11y agoNo, do you have any good ones? As long as entire sections are colored (not just lines), and those colors match with the annotations of another image, it should work fine!
- hcrisp 11y agoAnother suggestion: try running a copy of Tolkien's Middle-earth map to transfer the style to a more detailed USGS-style map [1]. [1] e.g. https://www.google.com/search?q=usgs+map&safe=active&client=ms-android-uscellular-us&biw=360&bih=559&prmd=ismvn&source=lnms&tbm=isch&sa=X&ved=0ahUKEwim-Kr4nrbLAhXL1CYKHQ_wC88Q_AUIBigB#imgrc=N1UT1Hw5ptb_bM%3A https://www.google.com/search?q=usgs+map&safe=active&client=...
- tjaad 11y agoWould this work with photos?
- nuclai 11y agoYou can specify two pairs of images (content+annotation) and it'll transfer the style from one to another as consistently as possible. The down side is that you need to find an algorithm, neural network, or person to create the annotations. (We're working on training one for portraits only.) These examples are in the paper above, direct link for convenience: https://twitter.com/alexjc/status/705784566657720320 https://twitter.com/alexjc/status/705784566657720320 https://twitter.com/alexjc/status/705811208901939200 https://twitter.com/alexjc/status/705811208901939200
- wyldfire 11y agoYeah, here's a similar neural network image analogy one that used photos with interesting results: https://github.com/awentzonline/image-analogies https://github.com/awentzonline/image-analogies
- bd 11y agoThese are really cool. Though if you were, like me, puzzled how could some really complex and coherent features come from those simple drawings / masks, have a look at the original paintings that were used as sources and compare them with generated images: Original #1: https://github.com/alexjc/neural-doodle/blob/master/samples/Monet.jpg https://github.com/alexjc/neural-doodle/blob/master/samples/... Generated #1: https://github.com/alexjc/neural-doodle/blob/master/docs/Coastline_example.png https://github.com/alexjc/neural-doodle/blob/master/docs/Coa... Original #2: https://github.com/alexjc/neural-doodle/blob/master/samples/Renoir.jpg https://github.com/alexjc/neural-doodle/blob/master/samples/... Generated #2: https://github.com/alexjc/neural-doodle/blob/master/docs/Landscape_example.png https://github.com/alexjc/neural-doodle/blob/master/docs/Lan... So those new generated images are structurally very similar to the original sources. Neural net seems to be good at "reshuffling" of the sources. That's probably how things like reflections on the water got there, even if not present in the doodles.
- thenomad 11y agoAha. I was wondering about that. I've done some experimentation with neural network-based style transfer (this one: https://github.com/jcjohnson/neural-style_ https://github.com/jcjohnson/neural-style_), and the results that I got pointed strongly to the same effect: it works well if the two images (source for style and source for content) are very similar in framing, composition and subject, and very badly if they're wildly different. Having said that, this algorithm seems to be MUCH better than the one I tried at transferring style. I'd have expected those paintings to transfer to the doodles much worse than they did. But don't expect to take a portrait doodle and a landscape source and have it come out well :)
- TheOtherHobbes 11y agoThe "Semantic" tag is misleading, because human perception parses lighting and textures cues in 2D images as 3D hinting. Representational art is all about modelling, highlighting and/or transforming the hinting, depending on the level of abstraction. E.g. if you look at portraits, the pen/brush strokes usually emphasise 3D structures. This code does a little of that, but the model is extremely crude compared to the models the human brain uses. For genuine semantic perception you'd have to duplicate - and maybe improve - the human model. I doubt you can do that in 2D, because the human model is trained by years of genuine 3D perception. That's not to sound negative - I think this is very impressive visually. But it could be taken further.
- pygy_ 11y agoI'd love/dread to see this this kind of work (neural nets run in reverse mode) applied to voices and accents. You could credibly put any words in the mouth of anyone.
- datr 11y agoOn the positive side you might end up with a Culture type situation where it's impossible to blackmail anyone due to it not being possible to verify the authenticity of any evidence.
- BoppreH 11y agoUnless an all seeing, trusted Mind steps in and vouches for it. NSA for notary?
- mdasen 11y agoThis basically already exists. Siri and similar TTS voices today are generated off of a lot of recorded speech from a person. There's a lot to get right for it to sound natural, not just hit the phonemes. You have to deal with the transitions between phonemes, declination, etc. I've even seen a demo converting one person's voice to another (without going through text) trying to preserve the pattern (pauses, stresses, etc.). It was kinda cool, but you wouldn't think it was the other person in a genuine way.
- matheweis 11y ago
- MichaelBurge 11y agoVery interesting! The thing that amazes me most about these neural network projects is how small the source usually is compared to what they're doing. Your doodle.py is only 453 lines.
- afandian 11y agoI imagine between SciKit[0], Thenano[1] and Lasagne[2], the total size is a little north of 453 lines. [0] http://scikit-image.org/ http://scikit-image.org/ [1] https://github.com/Theano/Theano https://github.com/Theano/Theano [2] https://github.com/Lasagne/Lasagne.git https://github.com/Lasagne/Lasagne.git
- mkj 11y agoIn coming years this will create a very strange reality combined with improving VR tech...
- wslh 11y agoExciting! Where can we find image databases for this?
- nuclai 11y agoYou can use any image as source, but to create annotations you have to do that yourself currently. Using simple segmentation libraries (or clustering) can do a good job for certain images, or look at better solutions for semantic segmentation: http://gitxiv.com/search/?q=segmentation http://gitxiv.com/search/?q=segmentation
- mhurron 11y agoFinally a way to draw things without learning how to draw. I'll be famous!
- Angostura 11y agoLooked at the images and honestly thought that someone had posted an April fools joke a few weeks early. Amazing.
- beeswax 11y agoThat's pretty cool. Might speed up asset creation for games by orders of magnitude: Train with concept art, generate the variations via these networks; adds consistency to the output and helps loosen the asset bottleneck / content treadmill esp for smaller studios/individuals.
- logicrook 11y agoNo, it's a gross misunderstanding of what is concept art. A concept art piece is about the idea, not the style; if you take a famous protagonist, say batman, you can have it drawn in a medieval, realistic, sci-fi version, drawn in a stylized, realistic, cartoon way; in each case you will recognize him because the idea, the shape language, have nothing to do with the style of the drawing. Even for illustrative work, where it can give you a good base, it still sucks, because for actual painters this step (thumbnailing) is actually the quickest; most of the time-consuming painting process is 'finishing', or 'detailing' the rough. However, where it's great is in giving the ability to inexperienced people to paint well. The hard part of the painting is getting the lightning, color scheme, perspective right, but the finishing process is quite mechanical. So it could ease the outsourcing of some art assets creation.
- beeswax 11y agoyeah you are right in terms of concept art; I figured that unfortunately more often than not the lines between concept, mood, detailing etc tend to be blurred depending on who looks at it; Also this is not going to replace individual character design or other specific assets, but might remove scalability issues w/ project that require a huge number of different backgrounds, texture variations. Of course the originial craft to producing high quality output is still needed - and just one image is not going to be sufficient anyway. As you mentioned, I can also think of giving lesser experienced folks the ability to tinker with scene setup, dimensions, ratios etc and get faster 'final' results, although the 'old school' approach to getting those right before actually detailing something is quite important.
- logicrook 11y ago
- ogreveins 11y agoI played with something similar for a while, https://github.com/jcjohnson/neural-style https://github.com/jcjohnson/neural-style What I've found so far is that it takes a while to get good results like something that looks like its own creation instead of an overlap of pictures. There's no exact way to do this. If you modify existing artwork it works well enough since the source is already somewhat divorced from reality but photos are difficult. When it works it's amazing though.
- nuclai 11y agoFrom that perspective, this research is two steps further than Neural Style, I wrote about it yesterday here: http://nucl.ai/blog/neural-doodles/ http://nucl.ai/blog/neural-doodles/ First, the paper I call "Neural Patches" (Li, January 2016) makes it possible to apply context-sensitive style, so you have more control how things map from one image to another. Second, we added extra annotations (which you can specify by hand or from a segmentation algorithm) that helps you control exactly how you want the styles to map. We call that "semantic style transfer" (Champandard, March 2016). You're right about it being hard otherwise, it was for many months and that's what pushed this particular line of research! Try it and see ;-)
- wimagguc 11y agoThis reminds me of "If Edison didn't invent the light bulb, someone else would have: there were thousands of other engineers experimenting with the exact same thing, a natural next step after electricity came about" (-- paraphrased from Kevin Kelly)
- SixSigma 11y agoOne was called Swan, Edison tried to sue him for patent infringement but Edison's lawyers warned him about prior art, so instead he negotiated a joint venture. You may remember the "Mazda" brand of bulbs https://en.wikipedia.org/wiki/Edison_and_Swan_Electric_Light_Company https://en.wikipedia.org/wiki/Edison_and_Swan_Electric_Light...
- 11y ago
- intrasight 11y agoNow please combine this with TiltBrush
- api 11y agoThis project should be named Bob Ross.
- deleted 11y ago[deleted]
- deleted 11y ago[deleted]