6 ms·

Image Analogies using Neural Networks

186 points by syrusakbary 11y ago

fchollet 11y ago
It's really neat stuff. The 3rd example is especially stunning. For more projects like this, you can also check out the neural style transfer implementation in Keras: https://github.com/fchollet/keras/blob/master/examples/neural_style_transfer.py https://github.com/fchollet/keras/blob/master/examples/neura... This is the script that OP's project was adapted from.
exit 11y ago
the third "analogy" leaves me speechless. i wonder how disruptive this could become to the graphic design industry. https://raw.githubusercontent.com/awentzonline/image-analogies/master/images/sugarskull-analogy.jpg https://raw.githubusercontent.com/awentzonline/image-analogi...
- LoSboccacc 11y ago
  unless someone goes and creates a easy to use standalone package, very little. this guy had the right idea http://ostagram.ru/static_pages/lenta?last_days=30 http://ostagram.ru/static_pages/lenta?last_days=30 but the service costs an arm and a leg to run on current cuda clouds first to get to a standalone python or ios version takes all :D
  - pierrec 11y ago
    The "original" cloud version is http://deepart.io http://deepart.io As for an easy standalone version, it wouldn't have huge potential, considering the inevitable GPU / computing power requirements.
    - LoSboccacc 11y ago
      eh most computer now have some good muscle. also it's an iterative process so you can show a meaningful progress bar to user even if it takes an hour or so people wouldn't complain. and for many it would take minutes - one just need to set the expectations straight.
      - pierrec 11y ago
        "By increasing the server memory to 27GB I manage to create 1024px images on CPU - seven hours per image" - http://spiceprogram.org/artcorn/ http://spiceprogram.org/artcorn/ And the GPU version requires amounts of video memory that generally do not exist for anything else than research or GPGPU purposes. With 4GB of VRAM you might get a 920x690 image processed in under 1 hour, using recent optimizations added to the neural-style repo. So a standalone iOS version seems pretty useless. A standalone Windows version might appeal to people with gaming rigs but without the technical knowledge or determination to install neural-style in its current form. It certainly has potential, just not as huge as the cloud version.
        mediocrejoker 11y ago
        If you were a serious graphic design professional might not you already have an i7 or a xeon rig with SLI already set up, and be accustomed to leaving a render running overnight?
    - mdda 11y ago
      This makes me a little sad, though : "University of Tübingen has a pending patent application for the Neural Art technology."
      - LoSboccacc 11y ago
        I wonder what google would have to say since it's built from deep dream experience. Also it was recently improved with newer tech which may very well be non infringing
  - lqdc13 11y ago
    For GPU, https://www.balticservers.com/gpu-servers https://www.balticservers.com/gpu-servers I used them for other things, and they delivered servers same day.
    - eripaha 11y ago
      EC2 g2.2xlarge is $0.1181 per Hour. That makes EC2 like 8x times cheaper. No worth wasting money to that company.
    - imaginenore 11y ago
      AWS GPU spot instances are better for something like this, and are probably cheaper.
      - LoSboccacc 11y ago
        the aws smallest instance comes at 450$/mo (about 300/mo paid upfront), the one he suggest comes at 200$/mo but the gpu seems cheaper. still one would need some sort of payout to keep it running, it's not exactly cheap and you need to shell 2k upfront
        simplyinfinity 11y ago
        aws spot instances can cost from $0.30 down to $0.10/hour, and if you have a fluctuating load with image generation, that can be a good price :)
        lqdc13 11y ago
        But they could terminate your instance at any time... Usually, training step is the one that takes the longest and that can take a whole day. Evaluation step can be done on CPU usually almost instantly. Not sure what the utility of spot instances is in deep learning. Not to mention code complexity and Dev-Ops investments to run on very transient hardware. Finally, you can't haggle with AWS, but you can basically name your price with private dedicated server providers like that.
  - Roritharr 11y ago
    This is amazing, why is there no western version of that service?
  - ohblahitsme 11y ago
    Man I love all the Kandinsky ones. My favorite kind of art.
- parksy 11y ago
  I'd love to see the result of rotoscoping with this technique. Looking at more examples, I'm not yet convinced this technique isn't straight-up magic: http://www.mrl.nyu.edu/projects/image-analogies/flightsim.html http://www.mrl.nyu.edu/projects/image-analogies/flightsim.ht...
  - ovi256 11y ago
    Looks like this could open a new era of videogames with synthetic textures, procedural maps and sandbox worlds.
  - echelon 11y ago
    I would absolutely love to see papers that apply techniques like this to video. I've got an interest in filmmaking and would have a great deal of fun processing video with something like this. Naively I imagine you could pipe every frame through an algorithm like this, but it likely won't have a smooth evolution over time. Maybe that isn't a problem?
    - awentzonline 11y ago
      Not quite the same thing, but here's the first animation I've seen made with it: https://twitter.com/memotv/status/706817101386936320 https://twitter.com/memotv/status/706817101386936320
amelius 11y ago
This made me think of the PatchMatch algorithm, see e.g. [1]. It is almost as if this is a generalization of that algorithm. It would be interesting to know if that is the case (for example, it would be interesting to know if the examples from the patchmatch papers can be reproduced with this image analogies algorithm). [1] https://vimeo.com/5024379 https://vimeo.com/5024379
- awentzonline 11y ago
  That's pretty cool. The patch matching in neural image analogies is based on this: http://arxiv.org/pdf/1601.04589v1.pdf http://arxiv.org/pdf/1601.04589v1.pdf
- kastnerkyle 11y ago
  The MRF loss is patch based, and adapted from CNNMRF [1]. Since the precursors to PatchMatch (as mentioned in the PatchMatch paper) were MRF + belief propagation based, I am pretty sure it could be done with some tweaking. These analogies seem quite similar to the "user constraints" PatchMatch allows to be set, though an explicit "be straight" constraint might be much more difficult to optimize. [1] https://github.com/chuanli11/CNNMRF https://github.com/chuanli11/CNNMRF
jawns 11y ago
Can someone who's more familiar with this stuff give me an idea of how to go about trying it? Is it possible to just spin up an Amazon EC2 instance and make it work? If so, how long would it take, and how much would it cost, to produce one hybrid image? (Let's say something small, like 640x480 pixels.) We're not talking hours of processing time per image, right?
- matsiyatzy 11y ago
  I've tried colorizing photos with this code (result here : https://twitter.com/matsiyatzy/status/706270684209680384 https://twitter.com/matsiyatzy/status/706270684209680384) using EC2 instances. It's pretty fast, around 10-15 minutes, though you'll definitely want GPU-instances with CuDNN drivers installed (which can be a bit tedious to set up).
RockyMcNuts 11y ago
See also http://ostagram.ru http://ostagram.ru for some very striking neural network art mashups.

186 points by syrusakbary 11y ago

It's really neat stuff. The 3rd example is especially stunning. For more projects like this, you can also check out the neural style transfer implementation in Keras: https://github.com/fchollet/keras/blob/master/examples/neural_style_transfer.py https://github.com/fchollet/keras/blob/master/examples/neura... This is the script that OP's project was adapted from.

exit 11y ago

the third "analogy" leaves me speechless. i wonder how disruptive this could become to the graphic design industry. https://raw.githubusercontent.com/awentzonline/image-analogies/master/images/sugarskull-analogy.jpg https://raw.githubusercontent.com/awentzonline/image-analogi...

LoSboccacc 11y ago
unless someone goes and creates a easy to use standalone package, very little. this guy had the right idea http://ostagram.ru/static_pages/lenta?last_days=30 http://ostagram.ru/static_pages/lenta?last_days=30 but the service costs an arm and a leg to run on current cuda clouds first to get to a standalone python or ios version takes all :D
- pierrec 11y ago
  The "original" cloud version is http://deepart.io http://deepart.io As for an easy standalone version, it wouldn't have huge potential, considering the inevitable GPU / computing power requirements.
  - LoSboccacc 11y ago
    eh most computer now have some good muscle. also it's an iterative process so you can show a meaningful progress bar to user even if it takes an hour or so people wouldn't complain. and for many it would take minutes - one just need to set the expectations straight.
    - pierrec 11y ago
      "By increasing the server memory to 27GB I manage to create 1024px images on CPU - seven hours per image" - http://spiceprogram.org/artcorn/ http://spiceprogram.org/artcorn/ And the GPU version requires amounts of video memory that generally do not exist for anything else than research or GPGPU purposes. With 4GB of VRAM you might get a 920x690 image processed in under 1 hour, using recent optimizations added to the neural-style repo. So a standalone iOS version seems pretty useless. A standalone Windows version might appeal to people with gaming rigs but without the technical knowledge or determination to install neural-style in its current form. It certainly has potential, just not as huge as the cloud version.
      - mediocrejoker 11y ago
        If you were a serious graphic design professional might not you already have an i7 or a xeon rig with SLI already set up, and be accustomed to leaving a render running overnight?
  - mdda 11y ago
    This makes me a little sad, though : "University of Tübingen has a pending patent application for the Neural Art technology."
    - LoSboccacc 11y ago
      I wonder what google would have to say since it's built from deep dream experience. Also it was recently improved with newer tech which may very well be non infringing
- lqdc13 11y ago
  For GPU, https://www.balticservers.com/gpu-servers https://www.balticservers.com/gpu-servers I used them for other things, and they delivered servers same day.
  - eripaha 11y ago
    EC2 g2.2xlarge is $0.1181 per Hour. That makes EC2 like 8x times cheaper. No worth wasting money to that company.
  - imaginenore 11y ago
    AWS GPU spot instances are better for something like this, and are probably cheaper.
    - LoSboccacc 11y ago
      the aws smallest instance comes at 450$/mo (about 300/mo paid upfront), the one he suggest comes at 200$/mo but the gpu seems cheaper. still one would need some sort of payout to keep it running, it's not exactly cheap and you need to shell 2k upfront
      - simplyinfinity 11y ago
        aws spot instances can cost from $0.30 down to $0.10/hour, and if you have a fluctuating load with image generation, that can be a good price :)
        lqdc13 11y ago
        But they could terminate your instance at any time... Usually, training step is the one that takes the longest and that can take a whole day. Evaluation step can be done on CPU usually almost instantly. Not sure what the utility of spot instances is in deep learning. Not to mention code complexity and Dev-Ops investments to run on very transient hardware. Finally, you can't haggle with AWS, but you can basically name your price with private dedicated server providers like that.
- Roritharr 11y ago
  This is amazing, why is there no western version of that service?
- ohblahitsme 11y ago
  Man I love all the Kandinsky ones. My favorite kind of art.
parksy 11y ago
I'd love to see the result of rotoscoping with this technique. Looking at more examples, I'm not yet convinced this technique isn't straight-up magic: http://www.mrl.nyu.edu/projects/image-analogies/flightsim.html http://www.mrl.nyu.edu/projects/image-analogies/flightsim.ht...
- ovi256 11y ago
  Looks like this could open a new era of videogames with synthetic textures, procedural maps and sandbox worlds.
- echelon 11y ago
  I would absolutely love to see papers that apply techniques like this to video. I've got an interest in filmmaking and would have a great deal of fun processing video with something like this. Naively I imagine you could pipe every frame through an algorithm like this, but it likely won't have a smooth evolution over time. Maybe that isn't a problem?
  - awentzonline 11y ago
    Not quite the same thing, but here's the first animation I've seen made with it: https://twitter.com/memotv/status/706817101386936320 https://twitter.com/memotv/status/706817101386936320

amelius 11y ago

This made me think of the PatchMatch algorithm, see e.g. [1]. It is almost as if this is a generalization of that algorithm. It would be interesting to know if that is the case (for example, it would be interesting to know if the examples from the patchmatch papers can be reproduced with this image analogies algorithm). [1] https://vimeo.com/5024379 https://vimeo.com/5024379

awentzonline 11y ago
That's pretty cool. The patch matching in neural image analogies is based on this: http://arxiv.org/pdf/1601.04589v1.pdf http://arxiv.org/pdf/1601.04589v1.pdf
kastnerkyle 11y ago
The MRF loss is patch based, and adapted from CNNMRF [1]. Since the precursors to PatchMatch (as mentioned in the PatchMatch paper) were MRF + belief propagation based, I am pretty sure it could be done with some tweaking. These analogies seem quite similar to the "user constraints" PatchMatch allows to be set, though an explicit "be straight" constraint might be much more difficult to optimize. [1] https://github.com/chuanli11/CNNMRF https://github.com/chuanli11/CNNMRF

jawns 11y ago

Can someone who's more familiar with this stuff give me an idea of how to go about trying it? Is it possible to just spin up an Amazon EC2 instance and make it work? If so, how long would it take, and how much would it cost, to produce one hybrid image? (Let's say something small, like 640x480 pixels.) We're not talking hours of processing time per image, right?

matsiyatzy 11y ago
I've tried colorizing photos with this code (result here : https://twitter.com/matsiyatzy/status/706270684209680384 https://twitter.com/matsiyatzy/status/706270684209680384) using EC2 instances. It's pretty fast, around 10-15 minutes, though you'll definitely want GPU-instances with CuDNN drivers installed (which can be a bit tedious to set up).

RockyMcNuts 11y ago

See also http://ostagram.ru http://ostagram.ru for some very striking neural network art mashups.