4 ms·

OctoPrint is what inspired me to create https://printnanny.ai/ https://printnanny.ai/ - I originally released PrintNanny as an OctoPrint plugin over holiday bre

OctoPrint is what inspired me to create https://printnanny.ai/ https://printnanny.ai/ - I originally released PrintNanny as an OctoPrint plugin over holiday break! I ended up rolling my own Linux distribution (PrintNanny OS) to roll out over-the-air updates and share files/jobs/profiles across a mixed fleet of printer controller softwares (Moonraker API/Mainsail UI, Repetier, OctoPrint).

sitkack 4y ago
This is awesome! You probably saved a bunch of people's lives.
- grepLeigh 4y ago
  Thank you! Mostly just saving a little weekend time.
CobrastanJorji 4y ago
This and Obico/Spaghetti Detective are really neat. I've been wondering about a version that was aware of the intended model being produced and could detect failures by considering the exact 3D shape. It feels like it should be better at detecting something going wrong, but frankly the two cases you've got covered, spaghetti and shift, covers like...almost everything. It looks like this runs entirely on a Raspberry Pi 4? That's pretty neat. As far as I know, Obico needs a significantly meatier computer.
- grepLeigh 4y ago
  Thank you! Your instincts are totally right. I've tried a few experiments using 3D model source as a model input, and... The good: It's possible to apply a segmentation model and "mask" expected real-world shape based on a mask extracted from the 3D source files. The bad: 3D segmentation proposals are expensive to calculate/reduce (imagine rendering a point cloud). The 2D masking experiments I ran were low signal / high noise compared to 2D bounding boxes. The ugly: Your camera, printer, and slicer object rotation must all be very carefully calibrated and anchored. I have ideas about how to use stereoscopic cameras to run an automatic calibration routine, but haven't had time to roll out experiments. If you're curious, I gave a talk about PrintNanny's print quality model [1]. Also yep, I run a TensorFlow Lite model and post-processing directly on the Pi. I'm actually working on a blog post about this now! [2] Besides TensorFlow Lite, the key pieces of technology in the computer vision stack are gstreamer [3], gstreamer's Rust bindings [4], and nnstreamer [5]. [1] https://bit.ly/tf-everywhere-2021-print-nanny https://bit.ly/tf-everywhere-2021-print-nanny [2] https://twitter.com/grepLeigh/status/1553777548378587136 https://twitter.com/grepLeigh/status/1553777548378587136 [3] https://gstreamer.freedesktop.org/ https://gstreamer.freedesktop.org/ [4] https://crates.io/crates/gstreamer https://crates.io/crates/gstreamer [5] https://github.com/nnstreamer/nnstreamer https://github.com/nnstreamer/nnstreamer
  - joshvm 4y ago
    On the calibration side, would it be possible to use fiducial markers (even a single Aruco) on the print bed to make an ortho projection of the print? You have the issue of print rotation relative to the bed, but I don't think there's an easy way around that if you allow removable beds. My gut feeling is that stereo would work but it'd be a pain to make reliable. ToF using something like a Kinect or RealSense might work at a coarse level (few mm).
    - grepLeigh 4y ago
      I did try inferring depth distance from strategically-placed round colored stickers. Using 1 fixed camera, I didn't see noteworthy results until I cranked up the resolution to 1080+ with FP64 inputs. I used a UNET architecture with skip connections between 1/4th, 1/8th, and 1/16th resolution samples. Diagrams here: https://photos.app.goo.gl/xLmE9Xp7UYmekBwV6 https://photos.app.goo.gl/xLmE9Xp7UYmekBwV6 Keep in mind that PrintNanny's video encoding and neural network pipelines are running on-device, so real-world results favor models that perform well when u8 quantized at 320p resolution. The constraints are part of the fun, but there is a bias towards techniques that retain signal when the model is compressed. Segmentation and depth models naturally lose fidelity in that scenario compared to a dead-simple CNN/SSD pumping out bounding box proposals.
cyphax 4y ago
I had planned to install OctoPrint on a Pi4 to leverage my printer as well but I would never have been able to conjure up something this extensive! Thank you for making it and sharing it!
- grepLeigh 4y ago
  Thanks! OctoPrint's plugin system is what helped me ship a prototype quickly. The project would have ended up in the elephant graveyard otherwise. All of the optimizations (building tuned Linux kernel, rolling a distro, porting code to Rust) would have been daunting without OctoPrint's maker community cheering me on and enthusiastically testing PrintNanny.
c_o_n_v_e_x 4y ago
Have you ever considered adding microphone support as an additional stream of data for diagnostics? Having sat next to my printer for many prints, seems like there are some audio cues you could tune in on.
- grepLeigh 4y ago
  I think your intuition is spot on (and I used to listen to my 3D printer before PrintNanny)! Microphone input is on my experiment list, but I don't think I'll execute on this idea any time soon. I think the mechanical failure detection is "good enough" at the moment, so I'm now focused on stream-lining the experience of managing lots of printers. Workflow automation (like printing shipping labels or batch QR code labels) and fleet management are at the top of my priority list. The AI is fun and flashy, but it turns out people just need a CRM for 3D printing.
hda111 4y ago
Do you use an OTA updater that is already available (open source) for Yocto Project or did you write your own?
- grepLeigh 4y ago
  I'm using SWUpdate. I distribute an image with a [boot, rootfsA, rootfsB, data] partition scheme [1], and subsequent updates target the non-active rootfs partition. You can even theme SwUpdate's CSS [2] - nifty! When I was assessing options for over-the-air updates, I evaluated everything between Mender.io (completely managed update service) to building a Coreos distro with OsTree-based patch system. So far Swupdate has felt like the right trade-off between customization and ease-of-use - plus the trail I've walked down feels well-traversed. [1] https://sbabic.github.io/swupdate/overview.html#double-copy https://sbabic.github.io/swupdate/overview.html#double-copy [2] https://twitter.com/grepLeigh/status/1545564418502668289 https://twitter.com/grepLeigh/status/1545564418502668289
alexk307 4y ago
This is a really cool project, well done!
- grepLeigh 4y ago
  Thank you!

sitkack 4y ago

This is awesome! You probably saved a bunch of people's lives.

grepLeigh 4y ago
Thank you! Mostly just saving a little weekend time.

CobrastanJorji 4y ago

This and Obico/Spaghetti Detective are really neat. I've been wondering about a version that was aware of the intended model being produced and could detect failures by considering the exact 3D shape. It feels like it should be better at detecting something going wrong, but frankly the two cases you've got covered, spaghetti and shift, covers like...almost everything. It looks like this runs entirely on a Raspberry Pi 4? That's pretty neat. As far as I know, Obico needs a significantly meatier computer.

grepLeigh 4y ago
Thank you! Your instincts are totally right. I've tried a few experiments using 3D model source as a model input, and... The good: It's possible to apply a segmentation model and "mask" expected real-world shape based on a mask extracted from the 3D source files. The bad: 3D segmentation proposals are expensive to calculate/reduce (imagine rendering a point cloud). The 2D masking experiments I ran were low signal / high noise compared to 2D bounding boxes. The ugly: Your camera, printer, and slicer object rotation must all be very carefully calibrated and anchored. I have ideas about how to use stereoscopic cameras to run an automatic calibration routine, but haven't had time to roll out experiments. If you're curious, I gave a talk about PrintNanny's print quality model [1]. Also yep, I run a TensorFlow Lite model and post-processing directly on the Pi. I'm actually working on a blog post about this now! [2] Besides TensorFlow Lite, the key pieces of technology in the computer vision stack are gstreamer [3], gstreamer's Rust bindings [4], and nnstreamer [5]. [1] https://bit.ly/tf-everywhere-2021-print-nanny https://bit.ly/tf-everywhere-2021-print-nanny [2] https://twitter.com/grepLeigh/status/1553777548378587136 https://twitter.com/grepLeigh/status/1553777548378587136 [3] https://gstreamer.freedesktop.org/ https://gstreamer.freedesktop.org/ [4] https://crates.io/crates/gstreamer https://crates.io/crates/gstreamer [5] https://github.com/nnstreamer/nnstreamer https://github.com/nnstreamer/nnstreamer
- joshvm 4y ago
  On the calibration side, would it be possible to use fiducial markers (even a single Aruco) on the print bed to make an ortho projection of the print? You have the issue of print rotation relative to the bed, but I don't think there's an easy way around that if you allow removable beds. My gut feeling is that stereo would work but it'd be a pain to make reliable. ToF using something like a Kinect or RealSense might work at a coarse level (few mm).
  - grepLeigh 4y ago
    I did try inferring depth distance from strategically-placed round colored stickers. Using 1 fixed camera, I didn't see noteworthy results until I cranked up the resolution to 1080+ with FP64 inputs. I used a UNET architecture with skip connections between 1/4th, 1/8th, and 1/16th resolution samples. Diagrams here: https://photos.app.goo.gl/xLmE9Xp7UYmekBwV6 https://photos.app.goo.gl/xLmE9Xp7UYmekBwV6 Keep in mind that PrintNanny's video encoding and neural network pipelines are running on-device, so real-world results favor models that perform well when u8 quantized at 320p resolution. The constraints are part of the fun, but there is a bias towards techniques that retain signal when the model is compressed. Segmentation and depth models naturally lose fidelity in that scenario compared to a dead-simple CNN/SSD pumping out bounding box proposals.

cyphax 4y ago

I had planned to install OctoPrint on a Pi4 to leverage my printer as well but I would never have been able to conjure up something this extensive! Thank you for making it and sharing it!

grepLeigh 4y ago
Thanks! OctoPrint's plugin system is what helped me ship a prototype quickly. The project would have ended up in the elephant graveyard otherwise. All of the optimizations (building tuned Linux kernel, rolling a distro, porting code to Rust) would have been daunting without OctoPrint's maker community cheering me on and enthusiastically testing PrintNanny.

c_o_n_v_e_x 4y ago

Have you ever considered adding microphone support as an additional stream of data for diagnostics? Having sat next to my printer for many prints, seems like there are some audio cues you could tune in on.

grepLeigh 4y ago
I think your intuition is spot on (and I used to listen to my 3D printer before PrintNanny)! Microphone input is on my experiment list, but I don't think I'll execute on this idea any time soon. I think the mechanical failure detection is "good enough" at the moment, so I'm now focused on stream-lining the experience of managing lots of printers. Workflow automation (like printing shipping labels or batch QR code labels) and fleet management are at the top of my priority list. The AI is fun and flashy, but it turns out people just need a CRM for 3D printing.

hda111 4y ago

Do you use an OTA updater that is already available (open source) for Yocto Project or did you write your own?

grepLeigh 4y ago
I'm using SWUpdate. I distribute an image with a [boot, rootfsA, rootfsB, data] partition scheme [1], and subsequent updates target the non-active rootfs partition. You can even theme SwUpdate's CSS [2] - nifty! When I was assessing options for over-the-air updates, I evaluated everything between Mender.io (completely managed update service) to building a Coreos distro with OsTree-based patch system. So far Swupdate has felt like the right trade-off between customization and ease-of-use - plus the trail I've walked down feels well-traversed. [1] https://sbabic.github.io/swupdate/overview.html#double-copy https://sbabic.github.io/swupdate/overview.html#double-copy [2] https://twitter.com/grepLeigh/status/1545564418502668289 https://twitter.com/grepLeigh/status/1545564418502668289

alexk307 4y ago

This is a really cool project, well done!

grepLeigh 4y ago
Thank you!