5 ms·

Thank you! Your instincts are totally right. I've tried a few experiments using 3D model source as a model input, and... The good: It's possible to apply a se

Thank you! Your instincts are totally right. I've tried a few experiments using 3D model source as a model input, and... The good: It's possible to apply a segmentation model and "mask" expected real-world shape based on a mask extracted from the 3D source files. The bad: 3D segmentation proposals are expensive to calculate/reduce (imagine rendering a point cloud). The 2D masking experiments I ran were low signal / high noise compared to 2D bounding boxes. The ugly: Your camera, printer, and slicer object rotation must all be very carefully calibrated and anchored. I have ideas about how to use stereoscopic cameras to run an automatic calibration routine, but haven't had time to roll out experiments. If you're curious, I gave a talk about PrintNanny's print quality model [1]. Also yep, I run a TensorFlow Lite model and post-processing directly on the Pi. I'm actually working on a blog post about this now! [2] Besides TensorFlow Lite, the key pieces of technology in the computer vision stack are gstreamer [3], gstreamer's Rust bindings [4], and nnstreamer [5]. [1] https://bit.ly/tf-everywhere-2021-print-nanny https://bit.ly/tf-everywhere-2021-print-nanny [2] https://twitter.com/grepLeigh/status/1553777548378587136 https://twitter.com/grepLeigh/status/1553777548378587136 [3] https://gstreamer.freedesktop.org/ https://gstreamer.freedesktop.org/ [4] https://crates.io/crates/gstreamer https://crates.io/crates/gstreamer [5] https://github.com/nnstreamer/nnstreamer https://github.com/nnstreamer/nnstreamer

joshvm 4y ago
On the calibration side, would it be possible to use fiducial markers (even a single Aruco) on the print bed to make an ortho projection of the print? You have the issue of print rotation relative to the bed, but I don't think there's an easy way around that if you allow removable beds. My gut feeling is that stereo would work but it'd be a pain to make reliable. ToF using something like a Kinect or RealSense might work at a coarse level (few mm).
- grepLeigh 4y ago
  I did try inferring depth distance from strategically-placed round colored stickers. Using 1 fixed camera, I didn't see noteworthy results until I cranked up the resolution to 1080+ with FP64 inputs. I used a UNET architecture with skip connections between 1/4th, 1/8th, and 1/16th resolution samples. Diagrams here: https://photos.app.goo.gl/xLmE9Xp7UYmekBwV6 https://photos.app.goo.gl/xLmE9Xp7UYmekBwV6 Keep in mind that PrintNanny's video encoding and neural network pipelines are running on-device, so real-world results favor models that perform well when u8 quantized at 320p resolution. The constraints are part of the fun, but there is a bias towards techniques that retain signal when the model is compressed. Segmentation and depth models naturally lose fidelity in that scenario compared to a dead-simple CNN/SSD pumping out bounding box proposals.

joshvm 4y ago

On the calibration side, would it be possible to use fiducial markers (even a single Aruco) on the print bed to make an ortho projection of the print? You have the issue of print rotation relative to the bed, but I don't think there's an easy way around that if you allow removable beds. My gut feeling is that stereo would work but it'd be a pain to make reliable. ToF using something like a Kinect or RealSense might work at a coarse level (few mm).

grepLeigh 4y ago
I did try inferring depth distance from strategically-placed round colored stickers. Using 1 fixed camera, I didn't see noteworthy results until I cranked up the resolution to 1080+ with FP64 inputs. I used a UNET architecture with skip connections between 1/4th, 1/8th, and 1/16th resolution samples. Diagrams here: https://photos.app.goo.gl/xLmE9Xp7UYmekBwV6 https://photos.app.goo.gl/xLmE9Xp7UYmekBwV6 Keep in mind that PrintNanny's video encoding and neural network pipelines are running on-device, so real-world results favor models that perform well when u8 quantized at 320p resolution. The constraints are part of the fun, but there is a bias towards techniques that retain signal when the model is compressed. Segmentation and depth models naturally lose fidelity in that scenario compared to a dead-simple CNN/SSD pumping out bounding box proposals.