7 ms·

E-Stop and Fuel, software that keeps you awake at night

198 points by janvdberg 9y ago

hawktheslayer 9y ago
This was an enjoyable read. While I don't often lose sleep over my code, (it's my kids that cause that), I do often find that my mind is working on solving coding puzzles in my sleep as I will frequently wake up with a spark of insight.
fireismyflag 9y ago
Thank you for sharing this. I enjoyed imagining scenarios so different from the ones that rob MY sleep at night.
snerbles 9y ago
For modern industrial applications, the safety circuit is often (edit- see child comment's note on safety PLCs) managed by discrete safety relay hardware such as the AB GuardMaster or Pilz PNOZ. There's a good chance these weren't even available at the time of OP's application! A common configuration involves emergency stops, guard doors, light curtains, etc. being wired in a pair of loops with the relay. The relay continuously monitors both loops (usually with a phased pulse train), and any interruption or crossover will trip the unit. Only when the loop states return to nominal will the relay permit a reset to re-enable the outputs. The safety relay's outputs are generally connected to dumb hardware interlocks on the various dangerous bits of the machine.
- _trampeltier 9y ago
  No, most even small machines arrive with a kind of "Safety Integeated" right in the PLC. Even the smallest PLCs like a Siemens S7 1200 are now aviable with Safety Integrated. So Profi-Safe and ASi-Safe are very common. It does reduce A LOT of wiring. But of course brings new problems, for example RJ45 jack and plugs sometimes break the connection for a very short time if you touch it and you lose a safety packet over Profi-Net and .. boom .. emergency stop. Special if you use a lot of drives in a machine, any kind of Safety Integrated reduce the wiring a lot and makes cabinets much much smaller. But on the other side, for just once, yes I like Pilz PNOZ. Easy to use .. and I'm pretty sure you can buy a PNOZ even in 100 years.
  - snerbles 9y ago
    It needs a little extra to meet PLd/e, no? I usually work with Allen-Bradley, my Siemens experience is rather lacking. Most of our customers would run away screaming from anything that's not hardwired safety, and are too cheap to shell out for a fully-blown safety PLC like the GuardLogix.
    - _trampeltier 9y ago
      Yes of course it needs a bit of extra. But for a pretty "low" price these days. Special if you build the same machine many times, you can safe much wiring time. Most maintenance people like hardware solutions because they are more easy to bridge ... With a software solution you can show easy for each safety button/switch a message on a display. Do it with a wired safety.
- MarkSweep 9y ago
  As a programmer, I like this approach to human safety for robots. By putting electrical interlocks on the doors that expose humans to the robot you can make it impossible for a software error to hurt a human. For some applications where you need to have humans working in the same area with the robot things get a lot hard. You probably need some software involved in enforcing speed limits for robots. The compliance engineers I've talked to say getting safety certification for software is quite arduous. In this case the off-the-shelf solutions the parent child comments mention become valuable.
  - _trampeltier 9y ago
    Just slow down machines is not enought. I'm still stunning how much the operators trust on things like E-Stop and Safety-Doors. They open the doors, don't even look and go into the machine with there hands. Most people have no idea what's behind a E-Stop and Safety-Doors these days. Also how much paper work each time .. sadly the paper part grows very fast .. and does not really help at all. Just worthless paper.
    - bsder 9y ago
      > I'm still stunning how much the operators trust on things like E-Stop and Safety-Doors. They open the doors, don't even look and go into the machine with there hands. Wow. Where I am we sometimes do this, but never without real thought. And, generally, we revisit that part of the process again and again over time. One example: initial setup of a CNC machine part. If you don't have $50K-$100K, you are setting this up by hand. You are moving the cutting head while your hand is in the same space. If you screw up, it probably won't rip your hand off, but you will likely wind up with a solid, painful gash and it might break a small bone if you get really unlucky. People don't respect servos enough. They're remarkably powerful and probably moreso than you expect.
      - froindt 9y ago
        > initial setup of a CNC machine part. If you don't have $50K-$100K, you are setting this up by hand. Curious what specifically you need your hand inside for? Do you simply mean the machine is on (though entirely inactive) while putting the part in, touching off the part slipping the business card in and out, or something else entirely?
rodrigocoelho 9y ago
Can relate. I 'stayed awake at night' when I had to write a program to calculate salaries.
- alex_hitchins 9y ago
  Ha! I once wrote a program to calculate commission for the sales people at our company. I remember the director telling me he would love my numbers to be true, but thought it best I had another look. Fat fingered decimals could have resulted in some expensive commissions!
otakucode 9y ago
Rather than the Therac-25 debacle, I would more recommend looking into the Toyota 'unintended acceleration' case. And the legal fallout from that. Because it is terrifying. Toyota was essentially as grossly negligent as it is possible to be. And the result? The court said there existed no standards they could be held legally liable for violating. So your self driving car? It will be developed by junior developers hired as cheaply as possible, driven like slaves by business-oriented managers who only care about meeting schedules, not given the tools or information needed to do an effective job, with testing time cut short and any claimed 'industry standards' for safe coding ignored. The automotive industry had 90+ coding practices they list as either 'required' or 'recommended'. Toyota followed 4 of those in their code. And the court said this was OK. Do you think Toyota spent tens or hundreds of millions of dollars rebuilding their entire development infrastructure, hiring more competent software engineers, firing the business managers who got people killed by rushing an unsafe product to market, and putting the engineers in charge of all future decisions regarding scheduling and release? No, of course not. If anything, they probably saw it as carte blanche to make things worse.
- elcritch 9y ago
  Wasn’t the cause of the “unintended acceleration issue” hardware and driver error related rather than software related? That’s what I’ve read in one book about Toyota management (the Toyota Way) and also a Wikipedia entry on the topic says: > Toyota also claimed that no defects existed and that the electronic control systems within the vehicles were unable to fail in a way that would result in an acceleration surge. More investigations were made but were unsuccessful in finding any defect until April 2008, when it was discovered that the driver side trim on a 2004 Toyota Sienna could come loose and prevent the accelerator pedal from returning to its fully closed position.[4] Based on those two sources it seems the issue was hardware related, and Toyota may have tried papering over the matt issue. The faulty matt design issue doesn’t support your claim of shoddy software practices and hiring underpaid junior developers. That may still be the case but it appears not to have caused the SA issue.
  - ineedasername 9y ago
fnord77 9y ago
> a fuel estimation program for a small cargo airline > ...floating point computations Hello, rounding errors. Oh hell no.
- TylerE 9y ago
  Please. Don't kneejerk. Any floating point errors will be multiple orders of magnitude smaller than than the accuracy of either the fuel gauges or the pumps.
  - KirinDave 9y ago
    Not if you use multiplication and floats. We certainly cannot use it for money, for example. Most shops I know of have fancy decimal libraries they never get to use because an unsigned long unit of centicents is simpler, predictable and easy to encode and assume client software will have the same rounding strategy.
    - monob 9y ago
      Fixed point isn't any better than floating point. If anything errors in floating point numbers are graceful and give you a lot of leeway before they become catastrophic. Fixed point works until you hit 2(n-1) bits then probably breaks unexpectedly. Where n is the last exponent you have seen in business.
      - KirinDave 9y ago
        Fixed point libraries tend to be better because good ones can cleanly identify when they're unable to continue and often tell you why. Floating point libraries can't even do this.
    - kurtisc 9y ago
      Just like the variation in the output voltage of a pin might be amplified by an op-amp. These things are taken into account by engineers, it's their duty to mathematically prove they're not an issue and fix it if it is.
gonzo 9y ago
When I was the CTO & VP of Engineering for Wayport (public, mostly hotel Internet) we designed an Ethernet switch that could use Home PNA or Ethernet PHYs. (Later adapted to also offer VDSL to an in-room modem.) We also designed our own 802.11 access points. All of our competitors had at least one fire. In a hotel with hundreds of people asleep. It didn’t matter if they used commercial gear or not. Every one of them had a fire. We never had one, but I was obsessed with not hurting anyone because we had missed something. And yes, it kept me up at night.
- sleepychu 9y ago
  I'm not following, what caused the fire?
  - isatty 9y ago
    Heat, if I had to guess (and no I'm not being sarcastic). When you design your own hardware there's way too many things that can go wrong.
  - DiabloD3 9y ago
    I'm not him, but probably either shoddy PSUs and/or there were lithium batteries involved. It's the curse of "modern" equipment.
cube00 9y ago
> needed to be re-written because the source code could not be produced Is that a diplomatic way of saying someone lost the source code?
- asterius 9y ago
  They might not have paid for a source code licence. or they did, but they never made sure they had a copy, just left it with the developer. Surprsingly common for companies to get a big binder of paperwork, an installer disk, and consider it done.
draw_down 9y ago
I would never work on software whose failure mode could cause death or injury to humans, I simply would refuse. Our tools are shit and we all know it. And the really frightening thing is when you get to work in a place that everyone presumes is one of the “good ones” and you see that it’s still the same shit wall-to-wall. We don’t have anything like the rigor of real engineering fields. Of course that may be less the case for e.g. pacemaker software, but nonetheless we tend to lack the equipment and processes that help prevent dangerous failures.
kemonocode 9y ago
My own personal "staying awake at night" case is an application that connected to an ancient version of Banner (A big kludge of an ERP system for universities) to handle new applicants' enrollment process and billing. I'm rather skittish when directly handling money, doubly so when the code was all written in Perl (Which I had to learn in order to re-implement in far more readable PHP, the lesser of two evils, natch) and extremely poorly documented. In retrospective, I should not have accepted a job like that.
knodi123 9y ago
Geez, and I thought I was stressed after I wrote software that would automate computing someone's pay bonus based on efficiency metrics.
w_t_payne 9y ago
So how can we approach safety in a systematic manner? Clearly 'blame' isn't an appropriate response. It has to involve tooling.
- jacquesm 9y ago
  Procedures and documentation seem to work well for the aviation industry. Things will still go wrong, but only very rarely twice in the same way. It makes development a lot more expensive but it does work and probably is the only way that we are aware of right now that will get this done in a way that leads to acceptable outcomes. This leads to glacial progress but I find that is preferable over the 'move fast and break shit' mentality that pervades the software industry.
  - w_t_payne 9y ago
    I think that it leads to glacial progress only because it is done badly. (It is done badly pretty much everywhere). I'm trying to develop tooling to make developing safe software fast: https://github.com/wtpayne/hiai https://github.com/wtpayne/hiai (Long way from being finished, unfortunately).
- HeyLaughingBoy 9y ago
  Well before tooling is considered, it has to involve people and process. At the highest level, you must have a culture of "blame the process, not the people" or people will do what is natural when things go wrong: try to cover it up and avoid being blamed. There are procedures in various safety-conscious industries for handling this kind of development. I like that you used the word "systemic" because it is literally a systems issue, not a software, or electronics, or mechanical issue. The entire system has to be considered and analyzed for potential faults. I spent over a decade writing code for medical devices and while the software aspect of these systems was the most advanced in terms of development process (unlike what many on HN seem to think :-), everything we did had to be considered from a system perspective because even if the individual parts were designed properly, it was possible for the interactions between them to cause problems.
joezydeco 9y ago
Just a plug for the RISKS Digest, now in it's 32nd year of operation: http://catless.ncl.ac.uk/Risks/ http://catless.ncl.ac.uk/Risks/

198 points by janvdberg 9y ago

This was an enjoyable read. While I don't often lose sleep over my code, (it's my kids that cause that), I do often find that my mind is working on solving coding puzzles in my sleep as I will frequently wake up with a spark of insight.

fireismyflag 9y ago

Thank you for sharing this. I enjoyed imagining scenarios so different from the ones that rob MY sleep at night.

snerbles 9y ago

For modern industrial applications, the safety circuit is often (edit- see child comment's note on safety PLCs) managed by discrete safety relay hardware such as the AB GuardMaster or Pilz PNOZ. There's a good chance these weren't even available at the time of OP's application! A common configuration involves emergency stops, guard doors, light curtains, etc. being wired in a pair of loops with the relay. The relay continuously monitors both loops (usually with a phased pulse train), and any interruption or crossover will trip the unit. Only when the loop states return to nominal will the relay permit a reset to re-enable the outputs. The safety relay's outputs are generally connected to dumb hardware interlocks on the various dangerous bits of the machine.

_trampeltier 9y ago
No, most even small machines arrive with a kind of "Safety Integeated" right in the PLC. Even the smallest PLCs like a Siemens S7 1200 are now aviable with Safety Integrated. So Profi-Safe and ASi-Safe are very common. It does reduce A LOT of wiring. But of course brings new problems, for example RJ45 jack and plugs sometimes break the connection for a very short time if you touch it and you lose a safety packet over Profi-Net and .. boom .. emergency stop. Special if you use a lot of drives in a machine, any kind of Safety Integrated reduce the wiring a lot and makes cabinets much much smaller. But on the other side, for just once, yes I like Pilz PNOZ. Easy to use .. and I'm pretty sure you can buy a PNOZ even in 100 years.
- snerbles 9y ago
  It needs a little extra to meet PLd/e, no? I usually work with Allen-Bradley, my Siemens experience is rather lacking. Most of our customers would run away screaming from anything that's not hardwired safety, and are too cheap to shell out for a fully-blown safety PLC like the GuardLogix.
  - _trampeltier 9y ago
    Yes of course it needs a bit of extra. But for a pretty "low" price these days. Special if you build the same machine many times, you can safe much wiring time. Most maintenance people like hardware solutions because they are more easy to bridge ... With a software solution you can show easy for each safety button/switch a message on a display. Do it with a wired safety.
MarkSweep 9y ago
As a programmer, I like this approach to human safety for robots. By putting electrical interlocks on the doors that expose humans to the robot you can make it impossible for a software error to hurt a human. For some applications where you need to have humans working in the same area with the robot things get a lot hard. You probably need some software involved in enforcing speed limits for robots. The compliance engineers I've talked to say getting safety certification for software is quite arduous. In this case the off-the-shelf solutions the parent child comments mention become valuable.
- _trampeltier 9y ago
  Just slow down machines is not enought. I'm still stunning how much the operators trust on things like E-Stop and Safety-Doors. They open the doors, don't even look and go into the machine with there hands. Most people have no idea what's behind a E-Stop and Safety-Doors these days. Also how much paper work each time .. sadly the paper part grows very fast .. and does not really help at all. Just worthless paper.
  - bsder 9y ago
    > I'm still stunning how much the operators trust on things like E-Stop and Safety-Doors. They open the doors, don't even look and go into the machine with there hands. Wow. Where I am we sometimes do this, but never without real thought. And, generally, we revisit that part of the process again and again over time. One example: initial setup of a CNC machine part. If you don't have $50K-$100K, you are setting this up by hand. You are moving the cutting head while your hand is in the same space. If you screw up, it probably won't rip your hand off, but you will likely wind up with a solid, painful gash and it might break a small bone if you get really unlucky. People don't respect servos enough. They're remarkably powerful and probably moreso than you expect.
    - froindt 9y ago
      > initial setup of a CNC machine part. If you don't have $50K-$100K, you are setting this up by hand. Curious what specifically you need your hand inside for? Do you simply mean the machine is on (though entirely inactive) while putting the part in, touching off the part slipping the business card in and out, or something else entirely?

rodrigocoelho 9y ago

Can relate. I 'stayed awake at night' when I had to write a program to calculate salaries.

alex_hitchins 9y ago
Ha! I once wrote a program to calculate commission for the sales people at our company. I remember the director telling me he would love my numbers to be true, but thought it best I had another look. Fat fingered decimals could have resulted in some expensive commissions!

otakucode 9y ago

Rather than the Therac-25 debacle, I would more recommend looking into the Toyota 'unintended acceleration' case. And the legal fallout from that. Because it is terrifying. Toyota was essentially as grossly negligent as it is possible to be. And the result? The court said there existed no standards they could be held legally liable for violating. So your self driving car? It will be developed by junior developers hired as cheaply as possible, driven like slaves by business-oriented managers who only care about meeting schedules, not given the tools or information needed to do an effective job, with testing time cut short and any claimed 'industry standards' for safe coding ignored. The automotive industry had 90+ coding practices they list as either 'required' or 'recommended'. Toyota followed 4 of those in their code. And the court said this was OK. Do you think Toyota spent tens or hundreds of millions of dollars rebuilding their entire development infrastructure, hiring more competent software engineers, firing the business managers who got people killed by rushing an unsafe product to market, and putting the engineers in charge of all future decisions regarding scheduling and release? No, of course not. If anything, they probably saw it as carte blanche to make things worse.

elcritch 9y ago
Wasn’t the cause of the “unintended acceleration issue” hardware and driver error related rather than software related? That’s what I’ve read in one book about Toyota management (the Toyota Way) and also a Wikipedia entry on the topic says: > Toyota also claimed that no defects existed and that the electronic control systems within the vehicles were unable to fail in a way that would result in an acceleration surge. More investigations were made but were unsuccessful in finding any defect until April 2008, when it was discovered that the driver side trim on a 2004 Toyota Sienna could come loose and prevent the accelerator pedal from returning to its fully closed position.[4] Based on those two sources it seems the issue was hardware related, and Toyota may have tried papering over the matt issue. The faulty matt design issue doesn’t support your claim of shoddy software practices and hiring underpaid junior developers. That may still be the case but it appears not to have caused the SA issue.
- ineedasername 9y ago

fnord77 9y ago

> a fuel estimation program for a small cargo airline > ...floating point computations Hello, rounding errors. Oh hell no.

TylerE 9y ago
Please. Don't kneejerk. Any floating point errors will be multiple orders of magnitude smaller than than the accuracy of either the fuel gauges or the pumps.
- KirinDave 9y ago
  Not if you use multiplication and floats. We certainly cannot use it for money, for example. Most shops I know of have fancy decimal libraries they never get to use because an unsigned long unit of centicents is simpler, predictable and easy to encode and assume client software will have the same rounding strategy.
  - monob 9y ago
    Fixed point isn't any better than floating point. If anything errors in floating point numbers are graceful and give you a lot of leeway before they become catastrophic. Fixed point works until you hit 2(n-1) bits then probably breaks unexpectedly. Where n is the last exponent you have seen in business.
    - KirinDave 9y ago
      Fixed point libraries tend to be better because good ones can cleanly identify when they're unable to continue and often tell you why. Floating point libraries can't even do this.
  - kurtisc 9y ago
    Just like the variation in the output voltage of a pin might be amplified by an op-amp. These things are taken into account by engineers, it's their duty to mathematically prove they're not an issue and fix it if it is.

gonzo 9y ago

When I was the CTO & VP of Engineering for Wayport (public, mostly hotel Internet) we designed an Ethernet switch that could use Home PNA or Ethernet PHYs. (Later adapted to also offer VDSL to an in-room modem.) We also designed our own 802.11 access points. All of our competitors had at least one fire. In a hotel with hundreds of people asleep. It didn’t matter if they used commercial gear or not. Every one of them had a fire. We never had one, but I was obsessed with not hurting anyone because we had missed something. And yes, it kept me up at night.

sleepychu 9y ago
I'm not following, what caused the fire?
- isatty 9y ago
  Heat, if I had to guess (and no I'm not being sarcastic). When you design your own hardware there's way too many things that can go wrong.
- DiabloD3 9y ago
  I'm not him, but probably either shoddy PSUs and/or there were lithium batteries involved. It's the curse of "modern" equipment.

cube00 9y ago

> needed to be re-written because the source code could not be produced Is that a diplomatic way of saying someone lost the source code?

asterius 9y ago
They might not have paid for a source code licence. or they did, but they never made sure they had a copy, just left it with the developer. Surprsingly common for companies to get a big binder of paperwork, an installer disk, and consider it done.

draw_down 9y ago

I would never work on software whose failure mode could cause death or injury to humans, I simply would refuse. Our tools are shit and we all know it. And the really frightening thing is when you get to work in a place that everyone presumes is one of the “good ones” and you see that it’s still the same shit wall-to-wall. We don’t have anything like the rigor of real engineering fields. Of course that may be less the case for e.g. pacemaker software, but nonetheless we tend to lack the equipment and processes that help prevent dangerous failures.

kemonocode 9y ago

My own personal "staying awake at night" case is an application that connected to an ancient version of Banner (A big kludge of an ERP system for universities) to handle new applicants' enrollment process and billing. I'm rather skittish when directly handling money, doubly so when the code was all written in Perl (Which I had to learn in order to re-implement in far more readable PHP, the lesser of two evils, natch) and extremely poorly documented. In retrospective, I should not have accepted a job like that.

knodi123 9y ago

Geez, and I thought I was stressed after I wrote software that would automate computing someone's pay bonus based on efficiency metrics.

w_t_payne 9y ago

So how can we approach safety in a systematic manner? Clearly 'blame' isn't an appropriate response. It has to involve tooling.

jacquesm 9y ago
Procedures and documentation seem to work well for the aviation industry. Things will still go wrong, but only very rarely twice in the same way. It makes development a lot more expensive but it does work and probably is the only way that we are aware of right now that will get this done in a way that leads to acceptable outcomes. This leads to glacial progress but I find that is preferable over the 'move fast and break shit' mentality that pervades the software industry.
- w_t_payne 9y ago
  I think that it leads to glacial progress only because it is done badly. (It is done badly pretty much everywhere). I'm trying to develop tooling to make developing safe software fast: https://github.com/wtpayne/hiai https://github.com/wtpayne/hiai (Long way from being finished, unfortunately).
HeyLaughingBoy 9y ago
Well before tooling is considered, it has to involve people and process. At the highest level, you must have a culture of "blame the process, not the people" or people will do what is natural when things go wrong: try to cover it up and avoid being blamed. There are procedures in various safety-conscious industries for handling this kind of development. I like that you used the word "systemic" because it is literally a systems issue, not a software, or electronics, or mechanical issue. The entire system has to be considered and analyzed for potential faults. I spent over a decade writing code for medical devices and while the software aspect of these systems was the most advanced in terms of development process (unlike what many on HN seem to think :-), everything we did had to be considered from a system perspective because even if the individual parts were designed properly, it was possible for the interactions between them to cause problems.

joezydeco 9y ago

Just a plug for the RISKS Digest, now in it's 32nd year of operation: http://catless.ncl.ac.uk/Risks/ http://catless.ncl.ac.uk/Risks/