3 ms·

Very cool, has anyone tasked a painting robot with reproducing images generated from a neural network? I've seen plenty of both, but nothing so far merging the

by RenRav 7y ago

Very cool, has anyone tasked a painting robot with reproducing images generated from a neural network? I've seen plenty of both, but nothing so far merging the two.

chrischen 7y ago
The problem is current conv neural nets are extracting textures, which may not necessarily translate to brush strokes that translate to the same physical rendition. A new net would have to be trained on the mechanical processes of paintings. I built a robo painter that captures brush stroke movements: https://www.instapainting.com/blog/research/2015/08/23/ai-painter/ https://www.instapainting.com/blog/research/2015/08/23/ai-pa... With enough input data here a neural net could be trained to produce physical results.
- lichtenberger 7y ago
  However, how do you measure the quality of the painting accurately? I think in their first paper on e-David Oliver Deussen mentioned something like that for future research :-)
  - i_am_proteus 7y ago
    I hypothesize that a camera watching the robot paint and a conv net with the right loss function (the tuning of which would be the hard part) could do the trick.
  - chrischen 7y ago
    Painting through that motion capture robot isn't exactly like painting with a brush in hand... I've tried creating prototypes with analog motion capture arms where the artist can physically guide the arm. The artist guided version would be the source of truth used for training, and you can evaluate results with cameras.
- lichtenberger 7y ago
  By the way, great work... maybe you could learn from each other!? I guess they are very open. I studied at the University of Konstanz and saw their first version back in 2012 when I finished my master on a complete other topic I might add ;-)
edge17 7y ago
https://youtu.be/GrEttzMCneo?t=172 https://youtu.be/GrEttzMCneo?t=172
irrational 7y ago
Or, have painting robots reproduce images generated from a neural network, then feed those images back into the neural network. Rinse and repeat. I imagine the results would be similar to running a sentence through Google Translate over and over again.
gwern 7y ago
What about DeepMind's SPIRAL? They didn't just prototype it in software but ran it on an actual robot arm, IIRC.

by RenRav 7y ago

Very cool, has anyone tasked a painting robot with reproducing images generated from a neural network? I've seen plenty of both, but nothing so far merging the two.

chrischen 7y ago

The problem is current conv neural nets are extracting textures, which may not necessarily translate to brush strokes that translate to the same physical rendition. A new net would have to be trained on the mechanical processes of paintings. I built a robo painter that captures brush stroke movements: https://www.instapainting.com/blog/research/2015/08/23/ai-painter/ https://www.instapainting.com/blog/research/2015/08/23/ai-pa... With enough input data here a neural net could be trained to produce physical results.

lichtenberger 7y ago
However, how do you measure the quality of the painting accurately? I think in their first paper on e-David Oliver Deussen mentioned something like that for future research :-)
- i_am_proteus 7y ago
  I hypothesize that a camera watching the robot paint and a conv net with the right loss function (the tuning of which would be the hard part) could do the trick.
- chrischen 7y ago
  Painting through that motion capture robot isn't exactly like painting with a brush in hand... I've tried creating prototypes with analog motion capture arms where the artist can physically guide the arm. The artist guided version would be the source of truth used for training, and you can evaluate results with cameras.
lichtenberger 7y ago
By the way, great work... maybe you could learn from each other!? I guess they are very open. I studied at the University of Konstanz and saw their first version back in 2012 when I finished my master on a complete other topic I might add ;-)

edge17 7y ago

https://youtu.be/GrEttzMCneo?t=172 https://youtu.be/GrEttzMCneo?t=172

irrational 7y ago

Or, have painting robots reproduce images generated from a neural network, then feed those images back into the neural network. Rinse and repeat. I imagine the results would be similar to running a sentence through Google Translate over and over again.

gwern 7y ago

What about DeepMind's SPIRAL? They didn't just prototype it in software but ran it on an actual robot arm, IIRC.