4 ms·

Real-time neural style transfer is not new; in the past year there have been several academic papers [1-4] on this topic and several open-source code releases:

by jcjohns 10y ago

Real-time neural style transfer is not new; in the past year there have been several academic papers [1-4] on this topic and several open-source code releases: https://github.com/jcjohnson/fast-neural-style https://github.com/jcjohnson/fast-neural-style https://github.com/DmitryUlyanov/texture_nets https://github.com/DmitryUlyanov/texture_nets https://github.com/chuanli11/MGANs https://github.com/chuanli11/MGANs Neural style blending is also not new; I did it more than a year ago using optimization-based method: https://github.com/jcjohnson/neural-style#multiple-style-images https://github.com/jcjohnson/neural-style#multiple-style-ima... The novelty of this work is a clever way for training a single network that can apply many different styles; existing methods for real-time style transfer train separate networks per style. Their method also allows for real-time style blending, which is very cool and to my knowledge has not been done before. (Disclaimer: I'm the author of [2]) [1] Ulyanov et al, "Texture Networks: Feed-forward Synthesis of Textures and Stylized Images", ICML 2016 [2] Johnson et al, "Perceptual Losses for Real-Time Style Transfer and Super-Resolution", ECCV 2016 [3] Li and Wand, "Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks", ECCV 2016 [4] Ulyanov et al, "Instance Normalization: The Missing Ingredient for Fast Stylization", arXiv 2016

domparise 10y ago
Also check out Prisma app for a slick mobile-friendly UI & subset of pre-selected art styles. (layman's application of this tech)
- tedmiston 10y ago
  Prisma's Instagram account is fantastic - https://www.instagram.com/prisma/ https://www.instagram.com/prisma/
- feelix 10y ago
  Don't forget everyone that is not Prisma that does the same thing, sometimes better. Here's mine. It runs on Mac & Windows, runs locally on all its styles unlike Prisma, runs HD images, and can process unlimited video: http://macdaddy.io/Style/ http://macdaddy.io/Style/
dharma1 10y ago
Thanks for releasing your code for this! Also really enjoy your cs231n lectures on youtube. There is some grain/small repeating pattern in many of the style transfer pictures I've seen - could it be an artifact of this? http://distill.pub/2016/deconv-checkerboard/ http://distill.pub/2016/deconv-checkerboard/
- jcjohns 10y ago
  Yes, I think that is a likely explanation. Also note that Vincent Dumoulin is an author of both the deconv-checkerboard blog post and the new paper from Google, and that the new Google paper uses the upsample+convolution technique suggested by the deconv-checkerboard blog post.
dinedal 10y ago
Hey jcjohns, fan of your work. I've noticed that your project for fast-neural-style does instance normalization over batch normalization. Batch normalization has the benefit that you can merge the gamma & beta into a convolutional layer on the forward pass, which makes it a lot faster by allowing you to skip a step when building the styled images using a trained model. Can the same be done with instance normalization? I didn't see a formula in the paper but I would think so, since they are fairly closely related.
- jcjohns 10y ago
  I've found that instance normalization usually gives better results so I prefer it over batch normalization. With batch norm you learn four scalars per convolutional feature map: mu (mean), sigma (stddev), alpha (scale) and beta (shift). During training, mu and sigma are estimated from data statistics; during testing they are constants, either estimated from the entire training set or computed as a running mean during training. At test time the batch norm operation is then alpha * (x - mu) / sigma + beta, which is a linear operation since everything but x is constant; since it is linear it can be merged into a convolutional layer. With instance norm, mu and sigma are estimated from data statistics during both training and testing; this means that the test-time forward pass is nonlinear, so it cannot be merged into a convolution (which is linear).
  - dinedal 10y ago
    Awesome, thanks for your response!

by jcjohns 10y ago

domparise 10y ago

Also check out Prisma app for a slick mobile-friendly UI & subset of pre-selected art styles. (layman's application of this tech)

tedmiston 10y ago
Prisma's Instagram account is fantastic - https://www.instagram.com/prisma/ https://www.instagram.com/prisma/
feelix 10y ago
Don't forget everyone that is not Prisma that does the same thing, sometimes better. Here's mine. It runs on Mac & Windows, runs locally on all its styles unlike Prisma, runs HD images, and can process unlimited video: http://macdaddy.io/Style/ http://macdaddy.io/Style/

dharma1 10y ago

Thanks for releasing your code for this! Also really enjoy your cs231n lectures on youtube. There is some grain/small repeating pattern in many of the style transfer pictures I've seen - could it be an artifact of this? http://distill.pub/2016/deconv-checkerboard/ http://distill.pub/2016/deconv-checkerboard/

jcjohns 10y ago
Yes, I think that is a likely explanation. Also note that Vincent Dumoulin is an author of both the deconv-checkerboard blog post and the new paper from Google, and that the new Google paper uses the upsample+convolution technique suggested by the deconv-checkerboard blog post.

dinedal 10y ago

Hey jcjohns, fan of your work. I've noticed that your project for fast-neural-style does instance normalization over batch normalization. Batch normalization has the benefit that you can merge the gamma & beta into a convolutional layer on the forward pass, which makes it a lot faster by allowing you to skip a step when building the styled images using a trained model. Can the same be done with instance normalization? I didn't see a formula in the paper but I would think so, since they are fairly closely related.

jcjohns 10y ago
I've found that instance normalization usually gives better results so I prefer it over batch normalization. With batch norm you learn four scalars per convolutional feature map: mu (mean), sigma (stddev), alpha (scale) and beta (shift). During training, mu and sigma are estimated from data statistics; during testing they are constants, either estimated from the entire training set or computed as a running mean during training. At test time the batch norm operation is then alpha * (x - mu) / sigma + beta, which is a linear operation since everything but x is constant; since it is linear it can be merged into a convolutional layer. With instance norm, mu and sigma are estimated from data statistics during both training and testing; this means that the test-time forward pass is nonlinear, so it cannot be merged into a convolution (which is linear).
- dinedal 10y ago
  Awesome, thanks for your response!