5 ms·

>I'm so happy when I read this that my work does not include decisions as grave as these. Yeah, I don't think I could take a (software) job where screwing up m

by avh02 9y ago

>I'm so happy when I read this that my work does not include decisions as grave as these. Yeah, I don't think I could take a (software) job where screwing up meant people die (or are even very seriously inconvenienced - e.g: airline booking systems), I don't want that responsibility on my head. My props to the people who do the countless jobs where messing up means terrible things (in and out of software), and not just lost revenue and 'downtime'.

jacquesm 9y ago
I've worked on software that controlled servo driven lathes and mills. I still know the e-stop function of that code by heart 30 years later, not because it was such an elegant or beautiful bit of code but mostly because I realized that any kind of fuck-up there would likely mean someone would lose a finger, a limb or even worse, and so I kept on re-reading it and inventing new ways to test it to ensure that it was solid. It definitely raises your level of awareness when you are working on stuff that has real-world consequences. I should probably do a write-up on how this worked, it was a pretty creative solution to a very hard problem and as far as I know it never failed to work in production. The machines are still in use in some places after all these years, never ceases to amaze me how long lived some of those projects were. Atari ST as the hardware, software in a combination of GFA basic and 68K assembler, just about every port on it used to drive some output or read a sensor.
- rublev 9y ago
  30 years? Damn. Can you go more in depth about that? What sort of systems and languages did you work with? How do you test it?
  - jacquesm 9y ago
    Atari ST, GFA basic, 68K assembler. Testing it was a long and labor intensive process with a first stage in simulation, then a second stage on a machine with all motors powered off except for the servos, then a run in 'air' but with a turning spindle and finally two real runs, one in aluminum and one in steel. The electronics were tested separately, we found that one of the early mobile phones (ATF-3) was really good at getting latches to trip that shouldn't trip. I learned a lot on that job, but it wasn't fun to work with the guy that ran that company, I wrote about this here: https://jacquesmattheij.com/the-start-up-from-hell https://jacquesmattheij.com/the-start-up-from-hell Apologies for the missing images, I still have to do some re-routing of my old gallery.
    - foobarian 9y ago
      Oh my, I grew up on GFA basic. When I first started it up and saw function collapsing I felt like I witnessed an alien ship landing. Good times.
- narvind 9y ago
  We are great fans of your lego sorting spectacle and your comment affirms my belief in your engineering prowess. You sir, are awesome!
- bpicolo 9y ago
  Would love to see a write up. The idea of writing code that could put someone's life on the line is totally terrifying to me.
  - digi_owl 9y ago
    That is why Torvalds gets riled up about kernel behavior, they just don't know where it may be used.
  - InternetOfStuff 9y ago
    > The idea of writing code that could put someone's life on the line is totally terrifying to me. That's very reasonable :-) I've worked on very safety-critical systems before. It's fun! For one thing, you get to do things right. Sure, there are deadlines, but nobody dares to tell you to leave things half-baked. If it takes time, then it take time. Shoo, manager. For another, you must dive deep into the behaviour of your embedded (mechanical/electrical/software) system and understand all its corner cases, even in the face of faults ("OK, so we're doing this operation, and now a cosmic ray hits and a crucial bit flips in RAM -- do we have a detection and safe reaction?") Apart from the fun-ness, realise you're not in it alone. You'll be supported by a bunch of your fellow engineers double-checking your work, testing the system, etc., so your confidence in the system is quite high. I like it.
- Someone 9y ago
  ”it was a pretty creative solution” Did that scare you at the time, or was it immediately clear it was the good kind of creative?
  - jacquesm 9y ago
    I tested the hell out of it to make sure that it worked as intended. Long story short, the hard part was to make sure that on powering up the system it wouldn't put random bits of output on the various ports which the hardware would then interpret as signals to make servos run and power up spindles. Initially I thought that the hard part was to make things stop, but that's actually rather easy, you just drop the power to a relay that is held 'on' by itself. That's irreversible from the software side, someone then needs to re-enable the power manually using the same mechanism (those mushroom switches stay down once you activate them). But when you do that - the re-enabling - you risk mayhem because the state of the machine is completely undefined. The trick to solving this was to confirm one-by-one that all outputs were in defined states and then outputting a sequence of bytes on a port to re-engage the relay, independent of whether the user had re-set the e-stop. The sequence was long enough that it could not happen by chance. It's this kind of subtle interaction between hardware and software where it is super easy to make a stupid mistake and to end up regretting it the rest of your life. Making sure that machine stayed off when it had to was super important, some of those servos were the size of buckets and would happily rip your arm off when powered up, the chuck on the largest machine that we did was 6 meters, to cut wheels for harbor cranes. Really nothing to fuck up with.
- taneq 9y ago
  > I still know the e-stop function of that code by heart 30 years later I'd be very interested in hearing more about this! I've recently re-designed the safety system for my company's product, and I chose to keep it purely hardware based (using safety relays) rather than involving any software, purely because it's so much harder to guarantee that software will be failsafe.
  - jacquesm 9y ago
    That's the perfect solution, and actually quite close to what I ended up doing. But you are still going to have to (1) detect the fact that there is an e-stop in progress, (2) restore the machinery to stable state afterwards and (3) lift the e-stop condition at some point. And all of those will have a software component. I've written a bit more about how it worked in another comment. The big thing to keep in mind here is that an ST is a very limited machine when it comes to IO and that just about every port was already occupied so I somehow had to make this work safely without having additional ports for e-stop i/o in the budget. If I had to do it again in the present I would never have consented to using such an I/O hamstrung platform to begin with and I would use an FPGA to do all the control logic.
    - taneq 9y ago
      Bit of a late response, but I've read the other comments plus your "startup from hell" blog post - man, what a ride! Thanks for telling the tale!
deleted 9y ago
[deleted]
kjs3 9y ago
Yeah, much better you grind out another twitter or fb clone or be the 100th X of Y out in SV. I'm glad the cowards stay out.

by avh02 9y ago

jacquesm 9y ago

I've worked on software that controlled servo driven lathes and mills. I still know the e-stop function of that code by heart 30 years later, not because it was such an elegant or beautiful bit of code but mostly because I realized that any kind of fuck-up there would likely mean someone would lose a finger, a limb or even worse, and so I kept on re-reading it and inventing new ways to test it to ensure that it was solid. It definitely raises your level of awareness when you are working on stuff that has real-world consequences. I should probably do a write-up on how this worked, it was a pretty creative solution to a very hard problem and as far as I know it never failed to work in production. The machines are still in use in some places after all these years, never ceases to amaze me how long lived some of those projects were. Atari ST as the hardware, software in a combination of GFA basic and 68K assembler, just about every port on it used to drive some output or read a sensor.

rublev 9y ago
30 years? Damn. Can you go more in depth about that? What sort of systems and languages did you work with? How do you test it?
- jacquesm 9y ago
  Atari ST, GFA basic, 68K assembler. Testing it was a long and labor intensive process with a first stage in simulation, then a second stage on a machine with all motors powered off except for the servos, then a run in 'air' but with a turning spindle and finally two real runs, one in aluminum and one in steel. The electronics were tested separately, we found that one of the early mobile phones (ATF-3) was really good at getting latches to trip that shouldn't trip. I learned a lot on that job, but it wasn't fun to work with the guy that ran that company, I wrote about this here: https://jacquesmattheij.com/the-start-up-from-hell https://jacquesmattheij.com/the-start-up-from-hell Apologies for the missing images, I still have to do some re-routing of my old gallery.
  - foobarian 9y ago
    Oh my, I grew up on GFA basic. When I first started it up and saw function collapsing I felt like I witnessed an alien ship landing. Good times.
narvind 9y ago
We are great fans of your lego sorting spectacle and your comment affirms my belief in your engineering prowess. You sir, are awesome!
bpicolo 9y ago
Would love to see a write up. The idea of writing code that could put someone's life on the line is totally terrifying to me.
- digi_owl 9y ago
  That is why Torvalds gets riled up about kernel behavior, they just don't know where it may be used.
- InternetOfStuff 9y ago
  > The idea of writing code that could put someone's life on the line is totally terrifying to me. That's very reasonable :-) I've worked on very safety-critical systems before. It's fun! For one thing, you get to do things right. Sure, there are deadlines, but nobody dares to tell you to leave things half-baked. If it takes time, then it take time. Shoo, manager. For another, you must dive deep into the behaviour of your embedded (mechanical/electrical/software) system and understand all its corner cases, even in the face of faults ("OK, so we're doing this operation, and now a cosmic ray hits and a crucial bit flips in RAM -- do we have a detection and safe reaction?") Apart from the fun-ness, realise you're not in it alone. You'll be supported by a bunch of your fellow engineers double-checking your work, testing the system, etc., so your confidence in the system is quite high. I like it.
Someone 9y ago
”it was a pretty creative solution” Did that scare you at the time, or was it immediately clear it was the good kind of creative?
- jacquesm 9y ago
  I tested the hell out of it to make sure that it worked as intended. Long story short, the hard part was to make sure that on powering up the system it wouldn't put random bits of output on the various ports which the hardware would then interpret as signals to make servos run and power up spindles. Initially I thought that the hard part was to make things stop, but that's actually rather easy, you just drop the power to a relay that is held 'on' by itself. That's irreversible from the software side, someone then needs to re-enable the power manually using the same mechanism (those mushroom switches stay down once you activate them). But when you do that - the re-enabling - you risk mayhem because the state of the machine is completely undefined. The trick to solving this was to confirm one-by-one that all outputs were in defined states and then outputting a sequence of bytes on a port to re-engage the relay, independent of whether the user had re-set the e-stop. The sequence was long enough that it could not happen by chance. It's this kind of subtle interaction between hardware and software where it is super easy to make a stupid mistake and to end up regretting it the rest of your life. Making sure that machine stayed off when it had to was super important, some of those servos were the size of buckets and would happily rip your arm off when powered up, the chuck on the largest machine that we did was 6 meters, to cut wheels for harbor cranes. Really nothing to fuck up with.
taneq 9y ago
> I still know the e-stop function of that code by heart 30 years later I'd be very interested in hearing more about this! I've recently re-designed the safety system for my company's product, and I chose to keep it purely hardware based (using safety relays) rather than involving any software, purely because it's so much harder to guarantee that software will be failsafe.
- jacquesm 9y ago
  That's the perfect solution, and actually quite close to what I ended up doing. But you are still going to have to (1) detect the fact that there is an e-stop in progress, (2) restore the machinery to stable state afterwards and (3) lift the e-stop condition at some point. And all of those will have a software component. I've written a bit more about how it worked in another comment. The big thing to keep in mind here is that an ST is a very limited machine when it comes to IO and that just about every port was already occupied so I somehow had to make this work safely without having additional ports for e-stop i/o in the budget. If I had to do it again in the present I would never have consented to using such an I/O hamstrung platform to begin with and I would use an FPGA to do all the control logic.
  - taneq 9y ago
    Bit of a late response, but I've read the other comments plus your "startup from hell" blog post - man, what a ride! Thanks for telling the tale!

deleted 9y ago

[deleted]

kjs3 9y ago

Yeah, much better you grind out another twitter or fb clone or be the 100th X of Y out in SV. I'm glad the cowards stay out.