4 ms·

Great name for this kind of blog (pointersgonewild) > Long story short: a software bug caused the machine to occasionally give radiation doses that were someti

Great name for this kind of blog (pointersgonewild) > Long story short: a software bug caused the machine to occasionally give radiation doses that were sometimes hundreds of times greater than normal Oh my god. I have 15+ years working with "mission critical (software) infrastructure" and thought my job was important/hard because I played shepherd for production database fleets. Certain kinds of mistakes could bankrupt the business, so I had to make systems resilient to human errors. Today, I run a 3D printer software startup. Thinking deeply about the safety mechanisms I need to control machines running at 250-300°C. Precision errors when shooting radiation at a person is a whole different level. Wow.

masklinn 3y ago
The Therac-25 story is pretty wild, especially as it was a reuse (with modification) of an existing codebase, but for cost savings the -25 was designed without hardware interlocks, under the assumption that the software would run fine, as it had run fine on the -20. Turns out the software had run fine on the -6 -20 because of the mechanical interlocks which prevented it from critically fucking up, but there was no reporting built into the software interlocks, so no way to know when they'd triggered. And that was before additional modifications were added for the -25.
- anonymousiam 3y ago
  Not all that different from the Ariane-5 maiden launch, except for human lives being at stake vs. billions of Euros (adjusted for inflation). https://en.wikipedia.org/wiki/Ariane_flight_V88 https://en.wikipedia.org/wiki/Ariane_flight_V88
  - nirui 3y ago
    Well, for me, the level of anxiety is definitely different. I somehow have a few degrees of fear towards machines that could emit harmful level of ionizing radiation, and I can be nervous if the machine is operating on me (say doing X-Ray at the dentist). Plus, I also have some anxiety when handling concurrent programming, which according to the Wikipedia is what caused the error in Therac-25. So I guess I got double anxieties on that story.
jrockway 3y ago
I think your 3D printing company basically exists because of Therac-25 indifference in the 3D printing community. We could put motor position encoders on the axes to actually detect crashes. But those things are like $20 per axis, and your microcontroller needs a quadrature encoder, so we say "fuck it, we'll do it with OpenCV". It's good when it's good, I guess!
- Palomides 3y ago
  3d printers are very safe and reliable, mine detects crashes via the current through the motor controller with no extra hardware (not that it's ever crashed) cost/benefit analysis is the heart of engineering, and that applies to redundant safety and error detection hardware as well
- grepLeigh 3y ago
  Defects in 3D print jobs are usually usually related to model design, slicing parameter decisions, and occasionally a printer part needs replacement/servicing (like belts). So far, firmware crashing hasn't been an issue. I'm supporting open source firmwares though (Marlin, Klipper, and soon RepRap/Duet) so they're VERY battle-tested. Maybe this is an issue for proprietary closed-source firmware?
- horsawlarway 3y ago
  I mean... the ones that are any good are about twice that price. Ex: I personally prefer CUI encoders: https://www.cuidevices.com/amt-modular-encoders https://www.cuidevices.com/amt-modular-encoders and they run about $50 each (with bulk ordering able to get you closer to $40). But I think you're not really considering the problem space from all the appropriate angles. Ex: The main reasons you avoid putting encoders on a 3d printer... 1. Weight - Hands down the biggest in most cases (excluding CoreXY setups). Each of those CUI encoders is about 16 grams. Total moving weight matters a lot in printer design. Go too high and you have to bulk up every other part of the machine to account for it and reduce shake/wobble and ensure you can maintain the correct accel. Or you have to print much slower, which makes the machine less capable (and valuable). It also adds wear and tear and increases friction. 2. Safety - They aren't worth very much in terms of safety. You can detect a lot of bad things by just setting sane current limits in your motor controllers. Combined with the thermal runaway protection available in most firmware... that covers most of the same things you can detect with encoders. 3. Value - Steppers are already incredibly precise. There is limited value in adding the extra encoder feedback into the system. Most of the cases where it matters are already detectable (and without any need for OpenCV...). So encoders are nifty - but this isn't really the best case for using them. No big win and lots of downsides (not the least of which is the extra $150 for adding them - which is 50% of the total cost of my first printer).

masklinn 3y ago

The Therac-25 story is pretty wild, especially as it was a reuse (with modification) of an existing codebase, but for cost savings the -25 was designed without hardware interlocks, under the assumption that the software would run fine, as it had run fine on the -20. Turns out the software had run fine on the -6 -20 because of the mechanical interlocks which prevented it from critically fucking up, but there was no reporting built into the software interlocks, so no way to know when they'd triggered. And that was before additional modifications were added for the -25.

anonymousiam 3y ago
Not all that different from the Ariane-5 maiden launch, except for human lives being at stake vs. billions of Euros (adjusted for inflation). https://en.wikipedia.org/wiki/Ariane_flight_V88 https://en.wikipedia.org/wiki/Ariane_flight_V88
- nirui 3y ago
  Well, for me, the level of anxiety is definitely different. I somehow have a few degrees of fear towards machines that could emit harmful level of ionizing radiation, and I can be nervous if the machine is operating on me (say doing X-Ray at the dentist). Plus, I also have some anxiety when handling concurrent programming, which according to the Wikipedia is what caused the error in Therac-25. So I guess I got double anxieties on that story.

jrockway 3y ago

I think your 3D printing company basically exists because of Therac-25 indifference in the 3D printing community. We could put motor position encoders on the axes to actually detect crashes. But those things are like $20 per axis, and your microcontroller needs a quadrature encoder, so we say "fuck it, we'll do it with OpenCV". It's good when it's good, I guess!

Palomides 3y ago
3d printers are very safe and reliable, mine detects crashes via the current through the motor controller with no extra hardware (not that it's ever crashed) cost/benefit analysis is the heart of engineering, and that applies to redundant safety and error detection hardware as well
grepLeigh 3y ago
Defects in 3D print jobs are usually usually related to model design, slicing parameter decisions, and occasionally a printer part needs replacement/servicing (like belts). So far, firmware crashing hasn't been an issue. I'm supporting open source firmwares though (Marlin, Klipper, and soon RepRap/Duet) so they're VERY battle-tested. Maybe this is an issue for proprietary closed-source firmware?
horsawlarway 3y ago
I mean... the ones that are any good are about twice that price. Ex: I personally prefer CUI encoders: https://www.cuidevices.com/amt-modular-encoders https://www.cuidevices.com/amt-modular-encoders and they run about $50 each (with bulk ordering able to get you closer to $40). But I think you're not really considering the problem space from all the appropriate angles. Ex: The main reasons you avoid putting encoders on a 3d printer... 1. Weight - Hands down the biggest in most cases (excluding CoreXY setups). Each of those CUI encoders is about 16 grams. Total moving weight matters a lot in printer design. Go too high and you have to bulk up every other part of the machine to account for it and reduce shake/wobble and ensure you can maintain the correct accel. Or you have to print much slower, which makes the machine less capable (and valuable). It also adds wear and tear and increases friction. 2. Safety - They aren't worth very much in terms of safety. You can detect a lot of bad things by just setting sane current limits in your motor controllers. Combined with the thermal runaway protection available in most firmware... that covers most of the same things you can detect with encoders. 3. Value - Steppers are already incredibly precise. There is limited value in adding the extra encoder feedback into the system. Most of the cases where it matters are already detectable (and without any need for OpenCV...). So encoders are nifty - but this isn't really the best case for using them. No big win and lots of downsides (not the least of which is the extra $150 for adding them - which is 50% of the total cost of my first printer).