6 ms·

Show HN: Robot that copies artist's exact strokes to replicate a painting in 3D

73 points by chrischen 11y ago

chrischen 11y ago
I'm having a problem smoothing the stepper motors with real time motion data. I was hoping someone on HN could point me in the right direction.
- tlb 11y ago
  You either need microstepping or servo motors with encoder feedback. Microstepping drives like [0] are buttery smooth. [0] http://www.geckodrive.com/gm215-step-motor-motion-controller.html http://www.geckodrive.com/gm215-step-motor-motion-controller...
  - chrischen 11y ago
    Yea I saw that stepper motor driver. It's $229, which is more than it cost to build this whole machine.
  - chrischen 11y ago
    In the video it's being driven at 1/8th step. I'm considering switching to two servo motors and using a polar system. The problem is I need it to also be fast enough to fee lag free for the artist.
    - jxramos 11y ago
      But why can't the artist do their work in real time and get feedback digitally from the monitor and you then playback their strokes at a later time to do all the rendering/plotting. I can see how they may like to use the plotter medium for their feedback but all the noise I would think would be distracting.
      - chrischen 11y ago
        That's definitely something we're working on. To be honest it's not very hard. I'd just modify the capture script to capture pen and pressure input and turn it into vectors, instead of capturing movement from the robot like it is now. The problem though is that the actual brush strokes are different from a brush stroke generated in Photoshop. Controlling the machine lets them see the final output in real time, so they can paint it more like a real physical painting, and then the robot can reproduce the physical painting. I think in the next demo it'll be more evident the difference between lines on the computer and physically rendered brush strokes.
        jxramos 11y ago
        Aha, yes in the prototyping stage that sort of immediate feedback would prove useful to converge upon the modeling necessary to properly translate the brush stroke pressure from the tablet into accurate saturation on the canvas. My comment was assuming that sort of translation was in place, but if you're discovering it then they would have to train the device. Which is an interesting thing, maybe apply machine learning to train the device much like how Andrew Ng and team trained that helicopter to fly upside down.
        chrischen 11y ago
        Also I tried to design this based on real-life applications. Digital artwork is never designed to be produced physically, and physical artwork is done physically with real-time feedback loop from observing the brushwork and colors. That being said, turning digital artwork into physical is still an interesting problem domain since artists can't be expected to all come wield my machine. I'd like an artist to be able to remotely work the machine in some manner.
    - 542458 11y ago
      You can get good 1/32 drivers like the DRV8825 (https://www.pololu.com/product/2133 https://www.pololu.com/product/2133) for not too much - I'm pretty confident they'll be able to do what you need (You can buy knockoffs for even cheaper). Even 1/16 is pretty dang smooth. It also might be that you're supplying too much or too little current to the motors - both will result in jerkiness and noise. If your current drivers are current limiting, I'd look into tuning that. If they aren't, the ones I linked are.
      - chrischen 11y ago
        They are current limiting and can go down to 1/16 step I believe. However, even at the current settings the motors can't keep up if the artist whips their arm from one side to the other.
- vanattab 11y ago
  If the problem is to smooth to input data coming from the Wacom device you can try using a low pass butterworth filter. https://en.wikipedia.org/wiki/Butterworth_filter https://en.wikipedia.org/wiki/Butterworth_filter I have used this to good effect for smoothing head tracker data.
  - chrischen 11y ago
    The problem isn't the wacom data. The issue is with moving the motor smoothly when the next step is uncertain.
    - tlarkworthy 11y ago
      delay it a bit then
- bliti 11y ago
  What type of drive gears/belts are you using?
  - chrischen 11y ago
    It's just a rubber belt hooked onto some stepper motors. No gears. The drivers are doing the micro stepping for me.
    - bliti 11y ago
      I've had luck using gears to introduce a reduction in movement by increasing the final drive ratio. The motors will move at faster speeds but the gears slow down the process and allow the movement to be more precise. Dunno how it would affect your robot since you have a direct 1:1 connection with the tablet.
      - chrischen 11y ago
        Yea that's the main problem. Ramping the motor is a solved problem, but because I'm getting real time data and latency is an issue, the microcontroller doesn't know if the next step is going to stop or change direction or speed up.
        bliti 11y ago
        What brand steppers do you have?
        chrischen 11y ago
        They're these. I don't think they have a brand. http://www.makeblock.cc/42byg-stepper-motor/ http://www.makeblock.cc/42byg-stepper-motor/
        bliti 11y ago
        I've never had good luck with the cheap ones. I use the numa with a built in transmission. Here is a link as a an example, I've never bought from the his place :http://omc-stepperonline.kancart.com/categories/4#!1 http://omc-stepperonline.kancart.com/categories/4#!1
- TheOtherHobbes 11y ago
  The usual solution is a proportional-integral-differential (PDI) controller driven by position measurements. The system has a resonant frequency that depends on the damping, mass of the motor, and brush friction. Those set a hard limit on the maximum possible speed. PDI will get you close to that speed, but there's always going to be a trade-off between speed and accuracy. From the test, you're still quite a way out from perfect repeatability. (I prefer the automated version, because the smoother strokes look slightly better.) I suspect you'll find that PDI with a thicker medium like oils/acrylics will be harder because the brush friction will be less consistent. And in fact real paintings are often created with a range of brush sizes and perhaps a range of palette knives - so building a robot to handle all of that isn't going to be trivial.
  - LukeShu 11y ago
    The first time you wrote "PDI", I assumed it was a typo for "PID", but you kept writing it. I'm curious, what's your background that you know it as "PDI"?
    - chucksmash 11y ago
      "The usual solution is a proportional-integral-differential (PDI) controller driven by position measurements."
sandebert 11y ago
Cool project. Also, I like how the sounds the robot makes would fit perfectly in just about any horror movie. Example: https://youtu.be/ip5Bt_-Yxsc?t=7m https://youtu.be/ip5Bt_-Yxsc?t=7m
noahbradley 11y ago
Now you're talking. I can't wait to print out nicely formed brushstrokes from a digital painting. Something along the lines of embedding a bump map into the 2d image to give subtle depth information. Let me know if anyone needs an artist to do the art side of this.
- chrischen 11y ago
  Can you email chris@instapainting.com? If you're in SF would love to have you come in to try the full colored version.
fictivmade 11y ago
Since the strokes are recorded, is it translated into positional coordinates and movements? Is it possible to adjust the "code" post-recording to perfect a replicate or add changes? Cool project!
- chrischen 11y ago
  Yes. You can fully manipulate the raw data afterwards to change it. I actually truncated the recording to remove the test strokes at the beginning.
Raed667 11y ago
As a kind I dreamed of a project like this, I got told "why would we need this when we have printers". It was fifteen years ago.
salgernon 11y ago
It doesn't do pressure, but this is along the same lines: http://shop.evilmadscientist.com/productsmenu/605 http://shop.evilmadscientist.com/productsmenu/605 And if you've got an old serial plotter (or 3 or 7) that can emulate or do HPGL: http://music.columbia.edu/cmc/chiplotle/ http://music.columbia.edu/cmc/chiplotle/ There's something absolutely magical about plotters...
- joshu 11y ago
  The watercolor bot can do pressure.
cocoflunchy 11y ago
Not sure from the video, does the artist look at the robot when painting or at a screen? I feel like it's going to be really hard to paint that slow, so it might be better to simulate the whole thing (which will be way closer to real time) and then reproduce it later on the robot (with some corrections if needed). But simulating might be difficult... or lead the artist to use brush strokes that are really hard to replicate. Cool project anyway! Let us know how it evolves :) Also just a heads up on the Amanufactory website the font in the Plans section is really hard to read, you might want to check that out ( http://i.imgur.com/Klvmorc.png http://i.imgur.com/Klvmorc.png ).
- chrischen 11y ago
  The artist looks at the robot. They were going extra slow to be cautious, but it can mimic movement pretty fast as long as you're not going across the canvas in a single stroke. Edit: Thanks, fixed the amanufactory.com bug.
hpoydar 11y ago
Very cool project. Can you give us some more details about the hardware and software you're using?
- chrischen 11y ago
  The hardware is http://www.makeblock.cc/xy-plotter-robot-kit/ http://www.makeblock.cc/xy-plotter-robot-kit/ sans electronics. I ended up purchasing the electronics anyways because they have a modified Arduino that uses RJ25, which makes the wiring much more stable (as opposed to pins stuck into an Arduino). If you got the individual pieces yourself or 3D printed it can be much cheaper. The main electrical components are: 1 x Arduino Uno (I used the modified Uno called "Orion" that has RJ2 ports) 2 x Stepper motors 1 x Servo motor 4 x microswitches as limit switches 2 x stepper motor drivers (they handle the microstepping) 3 x RJ25 breakout boards (for connecting the limit switches to RJ25 ports) This was a prototype to prove the concept. I'm going to experiment with 3D printed parts next in a new design. If you're feeding in coordinates as Gcode I would recommend loading Grbl. Keep in mind Grbl has specific pin requirements that make it incompatible with the Orion's RJ25 ports, so you'd have to wire a normal Arduino. Makeblock supplies Gcode firmware that works with the Orion. For the demo in the video, the firmware was custom to handle real-time input and recording of motion. On the host computer side, it was just a python script that interfaced with the Wacom tablet. Specifically: https://bitbucket.org/AnomalousUnderdog/pythonmactabletlib https://bitbucket.org/AnomalousUnderdog/pythonmactabletlib I also hacked support for the Myo armband and mouse support.
  - dharma1 11y ago
    thanks, this is very cool
joshu 11y ago
This looks familiar!
- chrischen 11y ago
  Heh. This is just the beginning. Finishing up something tonight to reply to one of your tweets with Cthulu.
knicholes 11y ago
This is cool. It reminds me of 3D printers, but just in two dimensions and with a paintbrush and paint instead of a hot-end/nozzle/filament. You could look at the firmware used to control the stepper motors/end stops that much of the reprap community is using at https://github.com/MarlinFirmware/Marlin https://github.com/MarlinFirmware/Marlin. This uses GCODE as well, and deals with acceleration around turns.
- chrischen 11y ago
  Yep looked into Grbl. Hadn't seen Marlin. I didn't use this for Live mode since they weren't really built for that, but they'll be used for the photo to painting part.
zemotion 11y ago
Super cool project! And damn unexpected to see Jean. Interested to see how the colored versions will look. Good luck moving forward!
schanq 11y ago
Reminds me of a project by Patrick Tresset @ Goldsmiths, UoL... https://www.youtube.com/watch?v=bbdQbyff_Sk https://www.youtube.com/watch?v=bbdQbyff_Sk http://research.gold.ac.uk/9255/ http://research.gold.ac.uk/9255/
geon 11y ago
Why not skip the tablet and draw directly with the plotter? You could probably even use the generated electric pulses from the steppers as your input.

73 points by chrischen 11y ago

I'm having a problem smoothing the stepper motors with real time motion data. I was hoping someone on HN could point me in the right direction.

tlb 11y ago
You either need microstepping or servo motors with encoder feedback. Microstepping drives like [0] are buttery smooth. [0] http://www.geckodrive.com/gm215-step-motor-motion-controller.html http://www.geckodrive.com/gm215-step-motor-motion-controller...
- chrischen 11y ago
  Yea I saw that stepper motor driver. It's $229, which is more than it cost to build this whole machine.
- chrischen 11y ago
  In the video it's being driven at 1/8th step. I'm considering switching to two servo motors and using a polar system. The problem is I need it to also be fast enough to fee lag free for the artist.
  - jxramos 11y ago
    But why can't the artist do their work in real time and get feedback digitally from the monitor and you then playback their strokes at a later time to do all the rendering/plotting. I can see how they may like to use the plotter medium for their feedback but all the noise I would think would be distracting.
    - chrischen 11y ago
      That's definitely something we're working on. To be honest it's not very hard. I'd just modify the capture script to capture pen and pressure input and turn it into vectors, instead of capturing movement from the robot like it is now. The problem though is that the actual brush strokes are different from a brush stroke generated in Photoshop. Controlling the machine lets them see the final output in real time, so they can paint it more like a real physical painting, and then the robot can reproduce the physical painting. I think in the next demo it'll be more evident the difference between lines on the computer and physically rendered brush strokes.
      - jxramos 11y ago
        Aha, yes in the prototyping stage that sort of immediate feedback would prove useful to converge upon the modeling necessary to properly translate the brush stroke pressure from the tablet into accurate saturation on the canvas. My comment was assuming that sort of translation was in place, but if you're discovering it then they would have to train the device. Which is an interesting thing, maybe apply machine learning to train the device much like how Andrew Ng and team trained that helicopter to fly upside down.
        chrischen 11y ago
        Also I tried to design this based on real-life applications. Digital artwork is never designed to be produced physically, and physical artwork is done physically with real-time feedback loop from observing the brushwork and colors. That being said, turning digital artwork into physical is still an interesting problem domain since artists can't be expected to all come wield my machine. I'd like an artist to be able to remotely work the machine in some manner.
  - 542458 11y ago
    You can get good 1/32 drivers like the DRV8825 (https://www.pololu.com/product/2133 https://www.pololu.com/product/2133) for not too much - I'm pretty confident they'll be able to do what you need (You can buy knockoffs for even cheaper). Even 1/16 is pretty dang smooth. It also might be that you're supplying too much or too little current to the motors - both will result in jerkiness and noise. If your current drivers are current limiting, I'd look into tuning that. If they aren't, the ones I linked are.
    - chrischen 11y ago
      They are current limiting and can go down to 1/16 step I believe. However, even at the current settings the motors can't keep up if the artist whips their arm from one side to the other.
vanattab 11y ago
If the problem is to smooth to input data coming from the Wacom device you can try using a low pass butterworth filter. https://en.wikipedia.org/wiki/Butterworth_filter https://en.wikipedia.org/wiki/Butterworth_filter I have used this to good effect for smoothing head tracker data.
- chrischen 11y ago
  The problem isn't the wacom data. The issue is with moving the motor smoothly when the next step is uncertain.
  - tlarkworthy 11y ago
    delay it a bit then
bliti 11y ago
What type of drive gears/belts are you using?
- chrischen 11y ago
  It's just a rubber belt hooked onto some stepper motors. No gears. The drivers are doing the micro stepping for me.
  - bliti 11y ago
    I've had luck using gears to introduce a reduction in movement by increasing the final drive ratio. The motors will move at faster speeds but the gears slow down the process and allow the movement to be more precise. Dunno how it would affect your robot since you have a direct 1:1 connection with the tablet.
    - chrischen 11y ago
      Yea that's the main problem. Ramping the motor is a solved problem, but because I'm getting real time data and latency is an issue, the microcontroller doesn't know if the next step is going to stop or change direction or speed up.
      - bliti 11y ago
        What brand steppers do you have?
        chrischen 11y ago
        They're these. I don't think they have a brand. http://www.makeblock.cc/42byg-stepper-motor/ http://www.makeblock.cc/42byg-stepper-motor/
        bliti 11y ago
        I've never had good luck with the cheap ones. I use the numa with a built in transmission. Here is a link as a an example, I've never bought from the his place :http://omc-stepperonline.kancart.com/categories/4#!1 http://omc-stepperonline.kancart.com/categories/4#!1
TheOtherHobbes 11y ago
The usual solution is a proportional-integral-differential (PDI) controller driven by position measurements. The system has a resonant frequency that depends on the damping, mass of the motor, and brush friction. Those set a hard limit on the maximum possible speed. PDI will get you close to that speed, but there's always going to be a trade-off between speed and accuracy. From the test, you're still quite a way out from perfect repeatability. (I prefer the automated version, because the smoother strokes look slightly better.) I suspect you'll find that PDI with a thicker medium like oils/acrylics will be harder because the brush friction will be less consistent. And in fact real paintings are often created with a range of brush sizes and perhaps a range of palette knives - so building a robot to handle all of that isn't going to be trivial.
- LukeShu 11y ago
  The first time you wrote "PDI", I assumed it was a typo for "PID", but you kept writing it. I'm curious, what's your background that you know it as "PDI"?
  - chucksmash 11y ago
    "The usual solution is a proportional-integral-differential (PDI) controller driven by position measurements."

sandebert 11y ago

Cool project. Also, I like how the sounds the robot makes would fit perfectly in just about any horror movie. Example: https://youtu.be/ip5Bt_-Yxsc?t=7m https://youtu.be/ip5Bt_-Yxsc?t=7m

noahbradley 11y ago

Now you're talking. I can't wait to print out nicely formed brushstrokes from a digital painting. Something along the lines of embedding a bump map into the 2d image to give subtle depth information. Let me know if anyone needs an artist to do the art side of this.

chrischen 11y ago
Can you email chris@instapainting.com? If you're in SF would love to have you come in to try the full colored version.

fictivmade 11y ago

Since the strokes are recorded, is it translated into positional coordinates and movements? Is it possible to adjust the "code" post-recording to perfect a replicate or add changes? Cool project!

chrischen 11y ago
Yes. You can fully manipulate the raw data afterwards to change it. I actually truncated the recording to remove the test strokes at the beginning.

Raed667 11y ago

As a kind I dreamed of a project like this, I got told "why would we need this when we have printers". It was fifteen years ago.

salgernon 11y ago

It doesn't do pressure, but this is along the same lines: http://shop.evilmadscientist.com/productsmenu/605 http://shop.evilmadscientist.com/productsmenu/605 And if you've got an old serial plotter (or 3 or 7) that can emulate or do HPGL: http://music.columbia.edu/cmc/chiplotle/ http://music.columbia.edu/cmc/chiplotle/ There's something absolutely magical about plotters...

joshu 11y ago
The watercolor bot can do pressure.

cocoflunchy 11y ago

Not sure from the video, does the artist look at the robot when painting or at a screen? I feel like it's going to be really hard to paint that slow, so it might be better to simulate the whole thing (which will be way closer to real time) and then reproduce it later on the robot (with some corrections if needed). But simulating might be difficult... or lead the artist to use brush strokes that are really hard to replicate. Cool project anyway! Let us know how it evolves :) Also just a heads up on the Amanufactory website the font in the Plans section is really hard to read, you might want to check that out ( http://i.imgur.com/Klvmorc.png http://i.imgur.com/Klvmorc.png ).

chrischen 11y ago
The artist looks at the robot. They were going extra slow to be cautious, but it can mimic movement pretty fast as long as you're not going across the canvas in a single stroke. Edit: Thanks, fixed the amanufactory.com bug.

hpoydar 11y ago

Very cool project. Can you give us some more details about the hardware and software you're using?

chrischen 11y ago
The hardware is http://www.makeblock.cc/xy-plotter-robot-kit/ http://www.makeblock.cc/xy-plotter-robot-kit/ sans electronics. I ended up purchasing the electronics anyways because they have a modified Arduino that uses RJ25, which makes the wiring much more stable (as opposed to pins stuck into an Arduino). If you got the individual pieces yourself or 3D printed it can be much cheaper. The main electrical components are: 1 x Arduino Uno (I used the modified Uno called "Orion" that has RJ2 ports) 2 x Stepper motors 1 x Servo motor 4 x microswitches as limit switches 2 x stepper motor drivers (they handle the microstepping) 3 x RJ25 breakout boards (for connecting the limit switches to RJ25 ports) This was a prototype to prove the concept. I'm going to experiment with 3D printed parts next in a new design. If you're feeding in coordinates as Gcode I would recommend loading Grbl. Keep in mind Grbl has specific pin requirements that make it incompatible with the Orion's RJ25 ports, so you'd have to wire a normal Arduino. Makeblock supplies Gcode firmware that works with the Orion. For the demo in the video, the firmware was custom to handle real-time input and recording of motion. On the host computer side, it was just a python script that interfaced with the Wacom tablet. Specifically: https://bitbucket.org/AnomalousUnderdog/pythonmactabletlib https://bitbucket.org/AnomalousUnderdog/pythonmactabletlib I also hacked support for the Myo armband and mouse support.
- dharma1 11y ago
  thanks, this is very cool

joshu 11y ago

This looks familiar!

chrischen 11y ago
Heh. This is just the beginning. Finishing up something tonight to reply to one of your tweets with Cthulu.

knicholes 11y ago

This is cool. It reminds me of 3D printers, but just in two dimensions and with a paintbrush and paint instead of a hot-end/nozzle/filament. You could look at the firmware used to control the stepper motors/end stops that much of the reprap community is using at https://github.com/MarlinFirmware/Marlin https://github.com/MarlinFirmware/Marlin. This uses GCODE as well, and deals with acceleration around turns.

chrischen 11y ago
Yep looked into Grbl. Hadn't seen Marlin. I didn't use this for Live mode since they weren't really built for that, but they'll be used for the photo to painting part.

zemotion 11y ago

Super cool project! And damn unexpected to see Jean. Interested to see how the colored versions will look. Good luck moving forward!

schanq 11y ago

Reminds me of a project by Patrick Tresset @ Goldsmiths, UoL... https://www.youtube.com/watch?v=bbdQbyff_Sk https://www.youtube.com/watch?v=bbdQbyff_Sk http://research.gold.ac.uk/9255/ http://research.gold.ac.uk/9255/

geon 11y ago

Why not skip the tablet and draw directly with the plotter? You could probably even use the generated electric pulses from the steppers as your input.