3 ms·

This and Obico/Spaghetti Detective are really neat. I've been wondering about a version that was aware of the intended model being produced and could detect fai

This and Obico/Spaghetti Detective are really neat. I've been wondering about a version that was aware of the intended model being produced and could detect failures by considering the exact 3D shape. It feels like it should be better at detecting something going wrong, but frankly the two cases you've got covered, spaghetti and shift, covers like...almost everything. It looks like this runs entirely on a Raspberry Pi 4? That's pretty neat. As far as I know, Obico needs a significantly meatier computer.

grepLeigh 4y ago
Thank you! Your instincts are totally right. I've tried a few experiments using 3D model source as a model input, and... The good: It's possible to apply a segmentation model and "mask" expected real-world shape based on a mask extracted from the 3D source files. The bad: 3D segmentation proposals are expensive to calculate/reduce (imagine rendering a point cloud). The 2D masking experiments I ran were low signal / high noise compared to 2D bounding boxes. The ugly: Your camera, printer, and slicer object rotation must all be very carefully calibrated and anchored. I have ideas about how to use stereoscopic cameras to run an automatic calibration routine, but haven't had time to roll out experiments. If you're curious, I gave a talk about PrintNanny's print quality model [1]. Also yep, I run a TensorFlow Lite model and post-processing directly on the Pi. I'm actually working on a blog post about this now! [2] Besides TensorFlow Lite, the key pieces of technology in the computer vision stack are gstreamer [3], gstreamer's Rust bindings [4], and nnstreamer [5]. [1] https://bit.ly/tf-everywhere-2021-print-nanny https://bit.ly/tf-everywhere-2021-print-nanny [2] https://twitter.com/grepLeigh/status/1553777548378587136 https://twitter.com/grepLeigh/status/1553777548378587136 [3] https://gstreamer.freedesktop.org/ https://gstreamer.freedesktop.org/ [4] https://crates.io/crates/gstreamer https://crates.io/crates/gstreamer [5] https://github.com/nnstreamer/nnstreamer https://github.com/nnstreamer/nnstreamer
- joshvm 4y ago
  On the calibration side, would it be possible to use fiducial markers (even a single Aruco) on the print bed to make an ortho projection of the print? You have the issue of print rotation relative to the bed, but I don't think there's an easy way around that if you allow removable beds. My gut feeling is that stereo would work but it'd be a pain to make reliable. ToF using something like a Kinect or RealSense might work at a coarse level (few mm).
  - grepLeigh 4y ago
    I did try inferring depth distance from strategically-placed round colored stickers. Using 1 fixed camera, I didn't see noteworthy results until I cranked up the resolution to 1080+ with FP64 inputs. I used a UNET architecture with skip connections between 1/4th, 1/8th, and 1/16th resolution samples. Diagrams here: https://photos.app.goo.gl/xLmE9Xp7UYmekBwV6 https://photos.app.goo.gl/xLmE9Xp7UYmekBwV6 Keep in mind that PrintNanny's video encoding and neural network pipelines are running on-device, so real-world results favor models that perform well when u8 quantized at 320p resolution. The constraints are part of the fun, but there is a bias towards techniques that retain signal when the model is compressed. Segmentation and depth models naturally lose fidelity in that scenario compared to a dead-simple CNN/SSD pumping out bounding box proposals.

grepLeigh 4y ago

Thank you! Your instincts are totally right. I've tried a few experiments using 3D model source as a model input, and... The good: It's possible to apply a segmentation model and "mask" expected real-world shape based on a mask extracted from the 3D source files. The bad: 3D segmentation proposals are expensive to calculate/reduce (imagine rendering a point cloud). The 2D masking experiments I ran were low signal / high noise compared to 2D bounding boxes. The ugly: Your camera, printer, and slicer object rotation must all be very carefully calibrated and anchored. I have ideas about how to use stereoscopic cameras to run an automatic calibration routine, but haven't had time to roll out experiments. If you're curious, I gave a talk about PrintNanny's print quality model [1]. Also yep, I run a TensorFlow Lite model and post-processing directly on the Pi. I'm actually working on a blog post about this now! [2] Besides TensorFlow Lite, the key pieces of technology in the computer vision stack are gstreamer [3], gstreamer's Rust bindings [4], and nnstreamer [5]. [1] https://bit.ly/tf-everywhere-2021-print-nanny https://bit.ly/tf-everywhere-2021-print-nanny [2] https://twitter.com/grepLeigh/status/1553777548378587136 https://twitter.com/grepLeigh/status/1553777548378587136 [3] https://gstreamer.freedesktop.org/ https://gstreamer.freedesktop.org/ [4] https://crates.io/crates/gstreamer https://crates.io/crates/gstreamer [5] https://github.com/nnstreamer/nnstreamer https://github.com/nnstreamer/nnstreamer

joshvm 4y ago
On the calibration side, would it be possible to use fiducial markers (even a single Aruco) on the print bed to make an ortho projection of the print? You have the issue of print rotation relative to the bed, but I don't think there's an easy way around that if you allow removable beds. My gut feeling is that stereo would work but it'd be a pain to make reliable. ToF using something like a Kinect or RealSense might work at a coarse level (few mm).
- grepLeigh 4y ago
  I did try inferring depth distance from strategically-placed round colored stickers. Using 1 fixed camera, I didn't see noteworthy results until I cranked up the resolution to 1080+ with FP64 inputs. I used a UNET architecture with skip connections between 1/4th, 1/8th, and 1/16th resolution samples. Diagrams here: https://photos.app.goo.gl/xLmE9Xp7UYmekBwV6 https://photos.app.goo.gl/xLmE9Xp7UYmekBwV6 Keep in mind that PrintNanny's video encoding and neural network pipelines are running on-device, so real-world results favor models that perform well when u8 quantized at 320p resolution. The constraints are part of the fun, but there is a bias towards techniques that retain signal when the model is compressed. Segmentation and depth models naturally lose fidelity in that scenario compared to a dead-simple CNN/SSD pumping out bounding box proposals.