9 ms·

Supercharging Style Transfer

314 points by mrry 10y ago

flamtap 10y ago
Soon, I will be able to sound so snooty by saying "oh that? Yeah, it's a Monet pastiche of a picture I took."
- DiabloD3 10y ago
  Great, what I've always wanted, more Monet. Now only if I could have less problems...
- sp332 10y ago
  There was a twitter bot that would do this for you. Unfortunately it got to many copyright complaints and was shut down. Here's one it did for me: https://twitter.com/sep332/status/705882039720013824 https://twitter.com/sep332/status/705882039720013824 (scroll down for the result)
designium 10y ago
Eventually, they will be able to transfer music and song style from one artist to others too. I wonder what is the point to have artists in the future and how the artists can survive? Will artist become template markers?
- gallerdude 10y ago
  "I'll take a Wes Anderson movie about volcanoes, with Bill Murray starring in it, computer."
- idoh 10y ago
  Musicians have always been about the performance more than the creation of the music. A ton of musicians do not write their own music, for instance, but are still really popular.
- grandalf 10y ago
  Interesting. Your comment gets at the question of what constitutes art. I think there are many forms and purposes of art, but the kind we most often consider authentic usually seems in many ways crude compared to what preceded it: - Jazz began as something that sounded cacophonous compared to the more obvious order of hymns and classical music. Jazz transformed what had been brief moments of dissonance and arrhythmia into elaborate intereleavings of tension and resolution, structure and chaos. - Rap began by taking the repetition and rhyme of poetry and the rhythms of african music and created something many would not classify as music when they first heard it. - In order to be deemed fresh and worthy of recognition, fiction must not feel like a rehash of the great classics. In most cases the best new fiction smashes conventions in at least one narrow way. I'd argue that an unattributed work from a great author would not likely be publishable today because of that. Quality is not just subjective, it is relative. - In fashion, the moment a style is obvious it becomes outdated and means something quite different. In some cases the latter can be marketed successfully, but it is hard to think of it as art by that point. Think of the way iconic things felt when they were new (and ugly to most people at that moment) and someone was making them iconic. So while these algorithms create believable surface features mimicing established instances of "art" (they might persuade me that a short story was written by Fitzgerald when in fact it had been written by a grad student and run through the algorithm) they won't necessarily make it a good short story. But in another year researchers will likely understand what makes humans consider things stylistically novel and then all the above points will be irrelevant.
- tlrobinson 10y ago
  > I wonder what is the point to have artists in the future and how the artists can survive? I feel like that's a bit like someone in the 1800's asking if artists will still exist once photography technology improves. If art creation is completely monopolized by AI then I think it's likely most of the rest of our jobs will be too.
- j1vms 10y ago
  > (...) how the artists can survive? Let's take "survive" to mean survive financially based on one's ability to produce art with commercial value during their lifetimes. As others do, I think art is much more than than the piece itself, through whatever senses we use to perceive it. It is also the story of its creator and the wider context of its creation, as well as history of its movement through the world including those of its various owners. I think this leaves tremendous room for human artists today and far into the future. Further, I believe some artists, and there certainly exists those who have tried already, will direct machine learning based art, adapting it to be more likely to succeed based on those factors I indicated.
purple-dragon 10y ago
So is this the early warning that Prisma's market ownership of this effect will soon come to an end?
- minimaxir 10y ago
  Prisma never had "ownership" of this effect; the papers of Neural Style transfer have been out for months, with this Google method being another iteration. Prisma was just the first polished app released using style transfer. (However, since the technique is public, it will make an acquisition of Prisma less worthwhile)
  - purple-dragon 10y ago
    Sure, but I was talking about market ownership, i.e., the most market share for users looking to achieve this type of effect, not IP ownership.
    - singham 10y ago
      Of what use is market ownership when no one is going to buy you. Facebook is already working on its own style transfer algos.
      - purple-dragon 10y ago
        Yeah, see my original comment at the top of the thread.
- Eridrus 10y ago
  Prisma was a land grab because the original authors didn't make an app. Hell I started working on my own version at the time since it was such an obvious app. The original authors finally got around to making their DeepArt app, so we have at least two players in this space atm. I think either of these two players will incorporate this research, but there's no clear incentive for anyone else to try and get in on this market now. It's not clear that there's a whole lot of money to be made here, but who knows maybe usage will explode. I mostly expect an acquisition.
scarmig 10y ago
Imagine this used to create a film noir style transfer style network. Throw it onto Daydream. You could have a game where you're Sam Spade, walking around 1930's San Francisco. Or maybe it's a gloomy day on Ocean Beach. Make a sunny day filter, put it on Daydream, and you're ready for a picnic!
- grandalf 10y ago
  Protip: Often when it's overcast and unpleasant in the Marina it's beautiful at Ocean Beach (and vice versa).
hammock 10y ago
Did I miss the part of the article where they create a Monet style from multiple Monet works and apply it, rather than using the style of an individual painting? I thought that was the point of this. They sort of do it in the last image, I guess. Seems to pick up so much actual color, though! (as opposed to perhaps the quality of the color or how it's used, which is just as characteristic of a particular style as the color itself)
- snarf21 10y ago
  I agree. While this is super impressive stuff it seems to match palette far more than style. Van gogh has a distinct style even though not all of his paintings have the same color palette.
- webmaven 10y ago
  There has been other recent work (Preserving Color in Neural Artistic Style Transfer) that allowed control, separate from the "style", over how much of the original painting's color palette to use: https://arxiv.org/pdf/1606.05897v1.pdf https://arxiv.org/pdf/1606.05897v1.pdf It isn't too hard to extrapolate from this to using separate images as sources for the style and palette, and from there to transforming the "content" image's color palette to resemble the palette of the "style" image in terms of saturation, value, contrast, etc. (but not hue) by manipulating histograms, and then using that result instead of either the "content" image's or the "style" image's palettes. Because if you have a photo of a bright red cardinal and a desaturated pastel drawing of a pale blue boat, the result you are likely to be most pleased with would be a pastel cardinal that is, well, pink, and not a pale blue cardinal. Of course, the harder version of this would need to recognize the semantics of both the "content" and the "style" image in order to construct a more meaningful palette to be applied (which is what you're getting at), and that same semantic information will also help with transferring the style more meaningfully (eg. crude palette-knife houses in the background but small brushed strokes depicting the people in the foreground).
koolba 10y ago
I'll be the first one to admit I know next to nothing about art, but some of those generated paintings are quite beautiful. The "Starry Night" driven ones in particular. Very cool research.
newsignup 10y ago
There is a Twitter Bot[1] which does this too: https://twitter.com/neuralpainter https://twitter.com/neuralpainter You need to tweet a photo to this handle and it replies with painting. 1. He's a friend, also a SaaS: http://neuralpainting.co/ http://neuralpainting.co/
- tedmiston 10y ago
  That's a pretty cool use of a twitter bot.
- webmaven 10y ago
  There is also DeepForger (been borked for a while though): https://deepforger.com https://deepforger.com https://twitter.com/DeepForger https://twitter.com/DeepForger
tyre 10y ago
Notice how the transferred images have none of the power of the originals. Take the first set of images, in the style of “Head of a Clown”, by Georges Rouault. Not only does it not recreate the brush-strokes, but the content of the restyled image doesn't fit the artist's style. In the case of Munch's The Scream, and Starry Night, and every Monet, the subject matter is as important as the image. Maybe you'll be able to mask your Facebook profile in a combination of Murakami and Pissaro, but that's not the extent of their genius. Similar things are attempted in music. While we can pretty programmatically formulate pop music, deeper meaning eludes machines. Bob Dylan didn't just write great lyrics, he tapped into the zeitgeist of the era in which he wrote. When Kendrick Lamar or Beyonce write about black struggle in America, that's in the context of a larger societal conversation. I don't believe that neural networks will ever be artists because they work on copying existing patterns. Until there is an element of randomness fueled by context, they aren't creating anything new. That's where genius lies.
- MichaelGG 10y ago
  Surely those aren't good ones to compare against due to bias. A more fair way to judge would be to have people rate unknown artists' work as well as the machine output. Then see how that pans out over time. Judging from some of the stuff on display in galleries right now, I'd be shocked if machines can't outperform or match. And they'll only get better, right?
- minimaxir 10y ago
  > Until there is an element of randomness fueled by context, they aren't creating anything new. That argument could be said about any innovation. The Google paper is an iteration on neural style transfer; just because it doesn't mimic the "soul" of art doesn't mean the field is not worth pursuing.
- rando832 10y ago
  > Notice how the transferred images have none of the power of the originals. Partly because they are like 200x200 pixel images. There is no power in the original either that way. And I've been seen a ton of these posts like "look at these beautiful images", and the images are absolutely tiny. It's really ridiculous and annoying.
kirykl 10y ago
I'd like to seem the same thing done for food. Show it how to cook a cake. 'Feed' it a large set of edible ingredients. Enjoy random cakes in various styles
- andrewvijay 10y ago
  Love that idea!
crispyambulance 10y ago
These are thrilling developments but every time I see examples they're relatively "low-res" images. I would love to be able to zoom in and see whats going on, but I suspect that we're seeing the output exactly as it was rendered by the ML processing. Is that the case? Actual paintings have brush strokes and even surface effects that aren't visible until you're looking very up-close or at a high resolution image. This is a key feature of the enjoyment paintings that often gets lost on people. I guess the ultimate "turing test" for this stuff would be to have the system literally paint these images. That is, with actual paint and a paint brush, NOT with pixels. This would sort of give a really fresh and exciting twist of meaning to the title of the old essay by Walter Benjamin "Art in the age of mechanical reproduction."
jcjohns 10y ago
Real-time neural style transfer is not new; in the past year there have been several academic papers [1-4] on this topic and several open-source code releases: https://github.com/jcjohnson/fast-neural-style https://github.com/jcjohnson/fast-neural-style https://github.com/DmitryUlyanov/texture_nets https://github.com/DmitryUlyanov/texture_nets https://github.com/chuanli11/MGANs https://github.com/chuanli11/MGANs Neural style blending is also not new; I did it more than a year ago using optimization-based method: https://github.com/jcjohnson/neural-style#multiple-style-images https://github.com/jcjohnson/neural-style#multiple-style-ima... The novelty of this work is a clever way for training a single network that can apply many different styles; existing methods for real-time style transfer train separate networks per style. Their method also allows for real-time style blending, which is very cool and to my knowledge has not been done before. (Disclaimer: I'm the author of [2]) [1] Ulyanov et al, "Texture Networks: Feed-forward Synthesis of Textures and Stylized Images", ICML 2016 [2] Johnson et al, "Perceptual Losses for Real-Time Style Transfer and Super-Resolution", ECCV 2016 [3] Li and Wand, "Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks", ECCV 2016 [4] Ulyanov et al, "Instance Normalization: The Missing Ingredient for Fast Stylization", arXiv 2016
- domparise 10y ago
  Also check out Prisma app for a slick mobile-friendly UI & subset of pre-selected art styles. (layman's application of this tech)
  - tedmiston 10y ago
    Prisma's Instagram account is fantastic - https://www.instagram.com/prisma/ https://www.instagram.com/prisma/
  - feelix 10y ago
    Don't forget everyone that is not Prisma that does the same thing, sometimes better. Here's mine. It runs on Mac & Windows, runs locally on all its styles unlike Prisma, runs HD images, and can process unlimited video: http://macdaddy.io/Style/ http://macdaddy.io/Style/
- 10y ago
j2kun 10y ago
So I've been playing with style transfer for about six months. I even combined it with another technique to actually get something that I liked enough to have printed on canvas (https://github.com/j2kun/art-21-logo https://github.com/j2kun/art-21-logo) And as much as I love the math, the computer science, and the tech behind it, I have to admit that the novelty wore off quick. Not only does a style-transferred image not have the power of the original, as another commenter said (we can debate about why), but the longer you look at it the more you see the ugly artefacts. I understand it's not perfect, and that there will be improvements, but to me that's what's primarily holding back style transfer. Not runtime or memory constraints, but whatever first principles need to change to get a global unity of style in the image. Of course, if you introduce the human hand to fix the artefacts (which I think is great, and I know people are working on this), then style transfer can still be super useful. I can definitely see this becoming a new photoshop/illustrator integration.
- eppeltert 10y ago
  Did you try putting more structure (concentric outlines or a gradient or something) in the input image? Seems like style transfer would fall flat on a binary image.
  - pierrec 10y ago
    Gradients in the source image tend to yield poor results. They often awkwardly reappear in the result, looking like an artifact doesn't fit with the style image at all. Source images in a posterized style, with large flat surfaces in unified colors, tend to yield great results with few artifacts. With that in mind, I find the raw "21" logo to be a rather good source image. Two colors is minimalist, but it's a choice. I'm sure GP went for simplicity quite deliberately.
    - j2kun 10y ago
      Definitely the original is good, too, and my example was a fun addition, not a replacement of the original logo :)
- 10y ago
bertiewhykovich 10y ago
While these results are certainly impressive, it's hard to say that they're aesthetically or artistically "good." I played around with Photoshop filters a bunch when I was younger -- there are (or used to be) a good handful of "artistic" filters, imitating watercolor, pointillism , ink drawing, etc. These style transfer results are reminiscent of nothing so much as those effects: they look superficially appealing, but have the same blind, dead quality. Maybe this is an indictment of "style" as it's typically understood? A certain brush technique does not an artwork make.
- romaniv 10y ago
  The moderately impressive part of these algorithms is the fact that they can "learn" a style from just one image. However, in reality they are not nearly as universal as these papers make them seem. Pretty much all my attempts to use these algorithms resulted in garbage with ugly artefacts. Which, actually, was pretty informative, since I saw exactly what kinds of features they pick up. I am disappointed that these blog posts don't include any "failed" images of that sort.
- TulliusCicero 10y ago
  More than likely you're just biased because you know ahead of time it was done by a computer. We could probably take one of the images, present it in the right context, like in an art gallery where we pretend that it was created by some fictional up-and-coming artist, and people would love it instead.
  - bertiewhykovich 10y ago
    People will love all sorts of stupid shit if it's put in a gallery. (Commentary on that phenomenon is, in fact, an entire art movement -- readymades.) There are two questions we should be asking here: 1) Does a positive gallery reception indicate artistic value? Presumably not. 2) If we assume that these style transfers have artistic value, is it similar to the value of the original paintings? This is a more difficult question, but the answer does seem to be "no."
1_2__3 10y ago
> The pastiches produced via this algorithm look spectacular Not really they don't, no. It's kind of cool, that's about it.
andrewvijay 10y ago
It's prisma with random images rather than filters?
eppeltert 10y ago
Still no continuity between video frames? Doesn't seem like it would be very difficult.
tedmiston 10y ago
This is cool for my Instagram account, and I do really like Prisma, but... Google and others do hard work in neural nets. I feel like there must be less obvious applications here than what's discussed in the blog post. Anyone have ideas? Does this go far beyond photos? Will I eventually be able to translate my ideas and thoughts into anyone's style programmatically? Could each user view the same content interpreted in a different style that they like best? Might I be able to extrapolate how a user might use my website from their clickstream stlye on many sites? etc
raldi 10y ago
I wonder what you get if you ask the algorithm to start with Starry Night and then apply the inversion of the Van Gogh style filter.
shirman 10y ago
Is it really possible to run tensorflow under the ios and swift? Really great article, btw.

314 points by mrry 10y ago

flamtap 10y ago

Soon, I will be able to sound so snooty by saying "oh that? Yeah, it's a Monet pastiche of a picture I took."

DiabloD3 10y ago
Great, what I've always wanted, more Monet. Now only if I could have less problems...
sp332 10y ago
There was a twitter bot that would do this for you. Unfortunately it got to many copyright complaints and was shut down. Here's one it did for me: https://twitter.com/sep332/status/705882039720013824 https://twitter.com/sep332/status/705882039720013824 (scroll down for the result)

designium 10y ago

Eventually, they will be able to transfer music and song style from one artist to others too. I wonder what is the point to have artists in the future and how the artists can survive? Will artist become template markers?

gallerdude 10y ago
"I'll take a Wes Anderson movie about volcanoes, with Bill Murray starring in it, computer."
idoh 10y ago
Musicians have always been about the performance more than the creation of the music. A ton of musicians do not write their own music, for instance, but are still really popular.
grandalf 10y ago
Interesting. Your comment gets at the question of what constitutes art. I think there are many forms and purposes of art, but the kind we most often consider authentic usually seems in many ways crude compared to what preceded it: - Jazz began as something that sounded cacophonous compared to the more obvious order of hymns and classical music. Jazz transformed what had been brief moments of dissonance and arrhythmia into elaborate intereleavings of tension and resolution, structure and chaos. - Rap began by taking the repetition and rhyme of poetry and the rhythms of african music and created something many would not classify as music when they first heard it. - In order to be deemed fresh and worthy of recognition, fiction must not feel like a rehash of the great classics. In most cases the best new fiction smashes conventions in at least one narrow way. I'd argue that an unattributed work from a great author would not likely be publishable today because of that. Quality is not just subjective, it is relative. - In fashion, the moment a style is obvious it becomes outdated and means something quite different. In some cases the latter can be marketed successfully, but it is hard to think of it as art by that point. Think of the way iconic things felt when they were new (and ugly to most people at that moment) and someone was making them iconic. So while these algorithms create believable surface features mimicing established instances of "art" (they might persuade me that a short story was written by Fitzgerald when in fact it had been written by a grad student and run through the algorithm) they won't necessarily make it a good short story. But in another year researchers will likely understand what makes humans consider things stylistically novel and then all the above points will be irrelevant.
tlrobinson 10y ago
> I wonder what is the point to have artists in the future and how the artists can survive? I feel like that's a bit like someone in the 1800's asking if artists will still exist once photography technology improves. If art creation is completely monopolized by AI then I think it's likely most of the rest of our jobs will be too.
j1vms 10y ago
> (...) how the artists can survive? Let's take "survive" to mean survive financially based on one's ability to produce art with commercial value during their lifetimes. As others do, I think art is much more than than the piece itself, through whatever senses we use to perceive it. It is also the story of its creator and the wider context of its creation, as well as history of its movement through the world including those of its various owners. I think this leaves tremendous room for human artists today and far into the future. Further, I believe some artists, and there certainly exists those who have tried already, will direct machine learning based art, adapting it to be more likely to succeed based on those factors I indicated.

purple-dragon 10y ago

So is this the early warning that Prisma's market ownership of this effect will soon come to an end?

minimaxir 10y ago
Prisma never had "ownership" of this effect; the papers of Neural Style transfer have been out for months, with this Google method being another iteration. Prisma was just the first polished app released using style transfer. (However, since the technique is public, it will make an acquisition of Prisma less worthwhile)
- purple-dragon 10y ago
  Sure, but I was talking about market ownership, i.e., the most market share for users looking to achieve this type of effect, not IP ownership.
  - singham 10y ago
    Of what use is market ownership when no one is going to buy you. Facebook is already working on its own style transfer algos.
    - purple-dragon 10y ago
      Yeah, see my original comment at the top of the thread.
Eridrus 10y ago
Prisma was a land grab because the original authors didn't make an app. Hell I started working on my own version at the time since it was such an obvious app. The original authors finally got around to making their DeepArt app, so we have at least two players in this space atm. I think either of these two players will incorporate this research, but there's no clear incentive for anyone else to try and get in on this market now. It's not clear that there's a whole lot of money to be made here, but who knows maybe usage will explode. I mostly expect an acquisition.

scarmig 10y ago

Imagine this used to create a film noir style transfer style network. Throw it onto Daydream. You could have a game where you're Sam Spade, walking around 1930's San Francisco. Or maybe it's a gloomy day on Ocean Beach. Make a sunny day filter, put it on Daydream, and you're ready for a picnic!

grandalf 10y ago
Protip: Often when it's overcast and unpleasant in the Marina it's beautiful at Ocean Beach (and vice versa).

hammock 10y ago

Did I miss the part of the article where they create a Monet style from multiple Monet works and apply it, rather than using the style of an individual painting? I thought that was the point of this. They sort of do it in the last image, I guess. Seems to pick up so much actual color, though! (as opposed to perhaps the quality of the color or how it's used, which is just as characteristic of a particular style as the color itself)

snarf21 10y ago
I agree. While this is super impressive stuff it seems to match palette far more than style. Van gogh has a distinct style even though not all of his paintings have the same color palette.
webmaven 10y ago
There has been other recent work (Preserving Color in Neural Artistic Style Transfer) that allowed control, separate from the "style", over how much of the original painting's color palette to use: https://arxiv.org/pdf/1606.05897v1.pdf https://arxiv.org/pdf/1606.05897v1.pdf It isn't too hard to extrapolate from this to using separate images as sources for the style and palette, and from there to transforming the "content" image's color palette to resemble the palette of the "style" image in terms of saturation, value, contrast, etc. (but not hue) by manipulating histograms, and then using that result instead of either the "content" image's or the "style" image's palettes. Because if you have a photo of a bright red cardinal and a desaturated pastel drawing of a pale blue boat, the result you are likely to be most pleased with would be a pastel cardinal that is, well, pink, and not a pale blue cardinal. Of course, the harder version of this would need to recognize the semantics of both the "content" and the "style" image in order to construct a more meaningful palette to be applied (which is what you're getting at), and that same semantic information will also help with transferring the style more meaningfully (eg. crude palette-knife houses in the background but small brushed strokes depicting the people in the foreground).

koolba 10y ago

I'll be the first one to admit I know next to nothing about art, but some of those generated paintings are quite beautiful. The "Starry Night" driven ones in particular. Very cool research.

newsignup 10y ago

There is a Twitter Bot[1] which does this too: https://twitter.com/neuralpainter https://twitter.com/neuralpainter You need to tweet a photo to this handle and it replies with painting. 1. He's a friend, also a SaaS: http://neuralpainting.co/ http://neuralpainting.co/

tedmiston 10y ago
That's a pretty cool use of a twitter bot.
webmaven 10y ago
There is also DeepForger (been borked for a while though): https://deepforger.com https://deepforger.com https://twitter.com/DeepForger https://twitter.com/DeepForger

tyre 10y ago

Notice how the transferred images have none of the power of the originals. Take the first set of images, in the style of “Head of a Clown”, by Georges Rouault. Not only does it not recreate the brush-strokes, but the content of the restyled image doesn't fit the artist's style. In the case of Munch's The Scream, and Starry Night, and every Monet, the subject matter is as important as the image. Maybe you'll be able to mask your Facebook profile in a combination of Murakami and Pissaro, but that's not the extent of their genius. Similar things are attempted in music. While we can pretty programmatically formulate pop music, deeper meaning eludes machines. Bob Dylan didn't just write great lyrics, he tapped into the zeitgeist of the era in which he wrote. When Kendrick Lamar or Beyonce write about black struggle in America, that's in the context of a larger societal conversation. I don't believe that neural networks will ever be artists because they work on copying existing patterns. Until there is an element of randomness fueled by context, they aren't creating anything new. That's where genius lies.

MichaelGG 10y ago
Surely those aren't good ones to compare against due to bias. A more fair way to judge would be to have people rate unknown artists' work as well as the machine output. Then see how that pans out over time. Judging from some of the stuff on display in galleries right now, I'd be shocked if machines can't outperform or match. And they'll only get better, right?
minimaxir 10y ago
> Until there is an element of randomness fueled by context, they aren't creating anything new. That argument could be said about any innovation. The Google paper is an iteration on neural style transfer; just because it doesn't mimic the "soul" of art doesn't mean the field is not worth pursuing.
rando832 10y ago
> Notice how the transferred images have none of the power of the originals. Partly because they are like 200x200 pixel images. There is no power in the original either that way. And I've been seen a ton of these posts like "look at these beautiful images", and the images are absolutely tiny. It's really ridiculous and annoying.

kirykl 10y ago

I'd like to seem the same thing done for food. Show it how to cook a cake. 'Feed' it a large set of edible ingredients. Enjoy random cakes in various styles

andrewvijay 10y ago
Love that idea!

crispyambulance 10y ago

These are thrilling developments but every time I see examples they're relatively "low-res" images. I would love to be able to zoom in and see whats going on, but I suspect that we're seeing the output exactly as it was rendered by the ML processing. Is that the case? Actual paintings have brush strokes and even surface effects that aren't visible until you're looking very up-close or at a high resolution image. This is a key feature of the enjoyment paintings that often gets lost on people. I guess the ultimate "turing test" for this stuff would be to have the system literally paint these images. That is, with actual paint and a paint brush, NOT with pixels. This would sort of give a really fresh and exciting twist of meaning to the title of the old essay by Walter Benjamin "Art in the age of mechanical reproduction."

jcjohns 10y ago

Real-time neural style transfer is not new; in the past year there have been several academic papers [1-4] on this topic and several open-source code releases: https://github.com/jcjohnson/fast-neural-style https://github.com/jcjohnson/fast-neural-style https://github.com/DmitryUlyanov/texture_nets https://github.com/DmitryUlyanov/texture_nets https://github.com/chuanli11/MGANs https://github.com/chuanli11/MGANs Neural style blending is also not new; I did it more than a year ago using optimization-based method: https://github.com/jcjohnson/neural-style#multiple-style-images https://github.com/jcjohnson/neural-style#multiple-style-ima... The novelty of this work is a clever way for training a single network that can apply many different styles; existing methods for real-time style transfer train separate networks per style. Their method also allows for real-time style blending, which is very cool and to my knowledge has not been done before. (Disclaimer: I'm the author of [2]) [1] Ulyanov et al, "Texture Networks: Feed-forward Synthesis of Textures and Stylized Images", ICML 2016 [2] Johnson et al, "Perceptual Losses for Real-Time Style Transfer and Super-Resolution", ECCV 2016 [3] Li and Wand, "Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks", ECCV 2016 [4] Ulyanov et al, "Instance Normalization: The Missing Ingredient for Fast Stylization", arXiv 2016

domparise 10y ago
Also check out Prisma app for a slick mobile-friendly UI & subset of pre-selected art styles. (layman's application of this tech)
- tedmiston 10y ago
  Prisma's Instagram account is fantastic - https://www.instagram.com/prisma/ https://www.instagram.com/prisma/
- feelix 10y ago
  Don't forget everyone that is not Prisma that does the same thing, sometimes better. Here's mine. It runs on Mac & Windows, runs locally on all its styles unlike Prisma, runs HD images, and can process unlimited video: http://macdaddy.io/Style/ http://macdaddy.io/Style/
10y ago

j2kun 10y ago

So I've been playing with style transfer for about six months. I even combined it with another technique to actually get something that I liked enough to have printed on canvas (https://github.com/j2kun/art-21-logo https://github.com/j2kun/art-21-logo) And as much as I love the math, the computer science, and the tech behind it, I have to admit that the novelty wore off quick. Not only does a style-transferred image not have the power of the original, as another commenter said (we can debate about why), but the longer you look at it the more you see the ugly artefacts. I understand it's not perfect, and that there will be improvements, but to me that's what's primarily holding back style transfer. Not runtime or memory constraints, but whatever first principles need to change to get a global unity of style in the image. Of course, if you introduce the human hand to fix the artefacts (which I think is great, and I know people are working on this), then style transfer can still be super useful. I can definitely see this becoming a new photoshop/illustrator integration.

eppeltert 10y ago
Did you try putting more structure (concentric outlines or a gradient or something) in the input image? Seems like style transfer would fall flat on a binary image.
- pierrec 10y ago
  Gradients in the source image tend to yield poor results. They often awkwardly reappear in the result, looking like an artifact doesn't fit with the style image at all. Source images in a posterized style, with large flat surfaces in unified colors, tend to yield great results with few artifacts. With that in mind, I find the raw "21" logo to be a rather good source image. Two colors is minimalist, but it's a choice. I'm sure GP went for simplicity quite deliberately.
  - j2kun 10y ago
    Definitely the original is good, too, and my example was a fun addition, not a replacement of the original logo :)
10y ago

bertiewhykovich 10y ago

While these results are certainly impressive, it's hard to say that they're aesthetically or artistically "good." I played around with Photoshop filters a bunch when I was younger -- there are (or used to be) a good handful of "artistic" filters, imitating watercolor, pointillism , ink drawing, etc. These style transfer results are reminiscent of nothing so much as those effects: they look superficially appealing, but have the same blind, dead quality. Maybe this is an indictment of "style" as it's typically understood? A certain brush technique does not an artwork make.

romaniv 10y ago
The moderately impressive part of these algorithms is the fact that they can "learn" a style from just one image. However, in reality they are not nearly as universal as these papers make them seem. Pretty much all my attempts to use these algorithms resulted in garbage with ugly artefacts. Which, actually, was pretty informative, since I saw exactly what kinds of features they pick up. I am disappointed that these blog posts don't include any "failed" images of that sort.
TulliusCicero 10y ago
More than likely you're just biased because you know ahead of time it was done by a computer. We could probably take one of the images, present it in the right context, like in an art gallery where we pretend that it was created by some fictional up-and-coming artist, and people would love it instead.
- bertiewhykovich 10y ago
  People will love all sorts of stupid shit if it's put in a gallery. (Commentary on that phenomenon is, in fact, an entire art movement -- readymades.) There are two questions we should be asking here: 1) Does a positive gallery reception indicate artistic value? Presumably not. 2) If we assume that these style transfers have artistic value, is it similar to the value of the original paintings? This is a more difficult question, but the answer does seem to be "no."

1_2__3 10y ago

> The pastiches produced via this algorithm look spectacular Not really they don't, no. It's kind of cool, that's about it.

andrewvijay 10y ago

It's prisma with random images rather than filters?

eppeltert 10y ago

Still no continuity between video frames? Doesn't seem like it would be very difficult.

tedmiston 10y ago

This is cool for my Instagram account, and I do really like Prisma, but... Google and others do hard work in neural nets. I feel like there must be less obvious applications here than what's discussed in the blog post. Anyone have ideas? Does this go far beyond photos? Will I eventually be able to translate my ideas and thoughts into anyone's style programmatically? Could each user view the same content interpreted in a different style that they like best? Might I be able to extrapolate how a user might use my website from their clickstream stlye on many sites? etc

raldi 10y ago

I wonder what you get if you ask the algorithm to start with Starry Night and then apply the inversion of the Van Gogh style filter.

shirman 10y ago

Is it really possible to run tensorflow under the ios and swift? Really great article, btw.