3 ms·

Utterly amazing what people are achieving with neural nets. The idea that 'style transfer' can be fit into an algorithm is slightly blowing my mind right now.

Utterly amazing what people are achieving with neural nets. The idea that 'style transfer' can be fit into an algorithm is slightly blowing my mind right now. The jumping fox video does looks a bit 'off' though, I think because the animation is kept the same and so it ends up looking too realistic for that style. Still this is early days!

pavlov 10y ago
To get an idea of how style transfer works, it can be useful to look at some less successful examples. Here's one from the same git repo: https://github.com/lengstrom/fast-style-transfer/blob/master/examples/results/chicago_wave.jpg https://github.com/lengstrom/fast-style-transfer/blob/master... (Original photo: https://github.com/lengstrom/fast-style-transfer/blob/master/examples/content/chicago.jpg https://github.com/lengstrom/fast-style-transfer/blob/master... ) With the "Great Wave" painting as the network's style input, the limitations of the technique become more apparent. It's clear that a human painter would never render the Chicago skyline in this way: there are incongruent little waves on buildings' edges and all over the sky. The antennas on top of the tallest tower are particularly revealing. The neural network just sees an area of higher local contrast, and has continued the same pattern that was applied in the sky at the top-right of the antennas but with more contrast applied. This doesn't make any sense for what's supposed to be a painting. There's no intelligence here, "just" pattern matching that can do a brilliant illusion of creative variance on the right kind of content. ("Just" in quotes because it's still a great achievement.)
- pizza 10y ago
  Now we need some kind of object recognition-enhanced version of style transfer that learns constraints on what "makes sense" given "sensible" labeled/captioned training examples!
  - ebalit 10y ago
    It has been done with manual segmentation [1]. And the results are mind blowing. There also a lot of work done on segmentation with neural nets [2, 3], so I wouldn't be surprised to see someone implementing this idea in the near future. 1: https://github.com/alexjc/neural-doodle https://github.com/alexjc/neural-doodle 2: https://arxiv.org/pdf/1605.06211.pdf https://arxiv.org/pdf/1605.06211.pdf 3: http://mi.eng.cam.ac.uk/projects/segnet/ http://mi.eng.cam.ac.uk/projects/segnet/

pavlov 10y ago

To get an idea of how style transfer works, it can be useful to look at some less successful examples. Here's one from the same git repo: https://github.com/lengstrom/fast-style-transfer/blob/master/examples/results/chicago_wave.jpg https://github.com/lengstrom/fast-style-transfer/blob/master... (Original photo: https://github.com/lengstrom/fast-style-transfer/blob/master/examples/content/chicago.jpg https://github.com/lengstrom/fast-style-transfer/blob/master... ) With the "Great Wave" painting as the network's style input, the limitations of the technique become more apparent. It's clear that a human painter would never render the Chicago skyline in this way: there are incongruent little waves on buildings' edges and all over the sky. The antennas on top of the tallest tower are particularly revealing. The neural network just sees an area of higher local contrast, and has continued the same pattern that was applied in the sky at the top-right of the antennas but with more contrast applied. This doesn't make any sense for what's supposed to be a painting. There's no intelligence here, "just" pattern matching that can do a brilliant illusion of creative variance on the right kind of content. ("Just" in quotes because it's still a great achievement.)

pizza 10y ago
Now we need some kind of object recognition-enhanced version of style transfer that learns constraints on what "makes sense" given "sensible" labeled/captioned training examples!
- ebalit 10y ago
  It has been done with manual segmentation [1]. And the results are mind blowing. There also a lot of work done on segmentation with neural nets [2, 3], so I wouldn't be surprised to see someone implementing this idea in the near future. 1: https://github.com/alexjc/neural-doodle https://github.com/alexjc/neural-doodle 2: https://arxiv.org/pdf/1605.06211.pdf https://arxiv.org/pdf/1605.06211.pdf 3: http://mi.eng.cam.ac.uk/projects/segnet/ http://mi.eng.cam.ac.uk/projects/segnet/