9 ms·

I hope people are beginning to understand that the presence of software in any system is a liability.

by laretluval 8y ago

minedwiz 8y ago
Boeing 777 is also digital fly-by-wire and has never suffered an issue attributable to software error.
- gedy 8y ago
  "Well, I don’t think there is any question about it. It can only be attributable to human error."
glup 8y ago
After serverless comes codeless.
- lsllc 8y ago
  "low code" is a thing apparently.
- rpeden 8y ago
  Airbus has been been doing "codeless" development with Esterel products for quite a while. I put codeless in quotes, because I believe the tools to generate code at the end of the day - but it's not code that is written by humans, and it is generated based on formally verified models. I believe this is what Airbus uses: http://www.esterel-technologies.com/products/scade-suite/ http://www.esterel-technologies.com/products/scade-suite/
  - Glawen 8y ago
    Yes they generate c code which is compiled. I don't know if the model is formally verified, but the whole toolchain from esterel, including the library blocks are certified (aicraft, automitive, industrial and train safety levels) It does not mean that any software done with this tools is thus certified, you need to apply the safety methodology to certify it
  - glup 8y ago
    Very interesting! Trying to understand a bit better... does this mean that a human writes some logic, then it gets translated into a formally-verified model? Or these are pieces of verified code with strict combinatorial rules and the human does the composition?
cpcallen 8y ago
Indeed. And that is just an extension of the danger presented by electronic systems generally. When learning to fly sailplanes, I was always grateful that the "primitive" wood, steel and canvas gliders I flew had purely mechanical linkages between the controls and the control surfaces, and only one instrument that required electricity at all (and even that was duplicated by a purely mechanical version). Obviously automation has great advantages, but inevitably the increased complexity comes with risks which are more difficult to fully comprehend.
- mikeash 8y ago
  I’m also a glider pilot and I also appreciate the simplicity of the basic systems, but I feel like it’s worth noting that you’re far more likely to die in a glider crash than you are in a modern airliner full of complex software.
mikeash 8y ago
Software is like any other technology. It can be a liability or an asset or both depending on how it’s built. Sometimes software fails and sometimes it even kills people, but how many lives has it also saved because of things like collision avoidance and accurate navigation?
CodeSheikh 8y ago
Correction: "I hope people are beginning to understand that the presence of poorly-tested software in any system is a liability."
- collyw 8y ago
  Well tested software still has bugs. Ones that haven't been tested for.
  - CodeSheikh 8y ago
    So it makes it a not Well tested then if scenarios have not been tested. My point.
    - laretluval 8y ago
      Tautological.
    - quickthrower2 8y ago
      For all software that does something useful, there exist scenarios that are not tested.
anonu 8y ago
I think software has existed in computer systems for 7 or 8 decades now - and the pitfalls are well understood. I think the problem is one of simplicity vs complexity. New systems today rely on dozens - hundreds even - of packages and libraries - many of which are not written by the implementer. How can you ensure reliability in all these components? This is why engineering from first principles - even in today's software development environment - is important. Unfortunately, the trends are in the other direction.
- dmead 8y ago
  you would be wrong.
  - AnimalMuppet 8y ago
    That's not a very useful reply. Can you explain where anonu is wrong? Can you explain why? Can you give some evidence? A bare dismissal is like five-year-olds arguing: "Did too!" "Did not!" It's not useful to advance the conversation, and it's not effective at persuading people.
    - noir_lord 8y ago
      It's quite possible that software creates new failure modes (as programmers I'm sure we all feel that to be true some days) while at the same time preventing more. When the software fails (if it did) we assume the software makes it more dangerous because we have a concrete example of what happened but not what could have happened in it's absence. This is what may eventually stall self driving cars for many years, the first time one does the PR equivalent of smashing through the local orphanage. We as a society are notoriously poor at assessing systemic risks. Now to address your point about evidence, I'm curious how you could control for other confounding factors. Airliners are safer today than ever before (based on passenger miles) but how do you control for better engines, material science, procedures if you compare say 1979 to 2019.
      - AnimalMuppet 8y ago
jessaustin 8y ago
It's a liability when you're getting equipment serviced, whether there's software or not. We have a Cat 287B skid-steer. After a month of inactivity, it wouldn't come out of "park" mode. A mechanic lifted the cab, tested a couple of solenoids, and stated "must be a bad computer". This is an old machine, so I called Cat and confirmed that it doesn't actually have a computer. I took up the floorboard and found some wires that had been chewed by rodents. We're lucky we don't have a computer, because replacing it wouldn't have helped...
Someone1234 8y ago
My take-away is a little different: As software grows it is hard to quantify. If you have several small distinct systems, you can often quantify them down to inputs, outputs, and expected logic. Once systems get too complex, you have too many possible states, and bugs are harder to find (and the system cannot be fully defined). I often go back to the famous Therac-25 accident. The bug in Therac-25 existed LONG before the accident, but there were two systems interacting, one system checked the other system's output and threw away invalid state. Once those two systems were merged, it was only then that the bug turned into an accident. If aircraft stopped building monolithic software and instead built parts that ran on software, parts that could each be individually verified, it would likely result in safer software.
AnimalMuppet 8y ago
The trend over the last 20 years has been that 1: airliners are getting safer and safer and 2: airliners contain more and more software. True, the software introduces new failure modes. But the presence of software in airliners seems to be less a liability than the absence of software.
outworlder 8y ago
> I hope people are beginning to understand that the presence of software in any system is a liability. I do not understand this point of view. If you don't have software, you are going to replace it with something mechanical. Or with people. Or lose the functionality entirely. Do you want to bring back a 'flight engineer' to operate airliner engines? Do you want to bring back mechanical autopilots? Analog computers? Let's ditch GPS while we are at it (GPS failures can cause deaths, specially under poor visibility).
laretluval 8y ago
More precisely. For two systems offering the same functionality, the one that does so using less software should be preferred, because software of any nontrivial complexity is irreducibly buggy.

I hope people are beginning to understand that the presence of software in any system is a liability.

minedwiz 8y ago

Boeing 777 is also digital fly-by-wire and has never suffered an issue attributable to software error.

gedy 8y ago
"Well, I don’t think there is any question about it. It can only be attributable to human error."

glup 8y ago

After serverless comes codeless.

lsllc 8y ago
"low code" is a thing apparently.
rpeden 8y ago
Airbus has been been doing "codeless" development with Esterel products for quite a while. I put codeless in quotes, because I believe the tools to generate code at the end of the day - but it's not code that is written by humans, and it is generated based on formally verified models. I believe this is what Airbus uses: http://www.esterel-technologies.com/products/scade-suite/ http://www.esterel-technologies.com/products/scade-suite/
- Glawen 8y ago
  Yes they generate c code which is compiled. I don't know if the model is formally verified, but the whole toolchain from esterel, including the library blocks are certified (aicraft, automitive, industrial and train safety levels) It does not mean that any software done with this tools is thus certified, you need to apply the safety methodology to certify it
- glup 8y ago
  Very interesting! Trying to understand a bit better... does this mean that a human writes some logic, then it gets translated into a formally-verified model? Or these are pieces of verified code with strict combinatorial rules and the human does the composition?

cpcallen 8y ago

Indeed. And that is just an extension of the danger presented by electronic systems generally. When learning to fly sailplanes, I was always grateful that the "primitive" wood, steel and canvas gliders I flew had purely mechanical linkages between the controls and the control surfaces, and only one instrument that required electricity at all (and even that was duplicated by a purely mechanical version). Obviously automation has great advantages, but inevitably the increased complexity comes with risks which are more difficult to fully comprehend.

mikeash 8y ago
I’m also a glider pilot and I also appreciate the simplicity of the basic systems, but I feel like it’s worth noting that you’re far more likely to die in a glider crash than you are in a modern airliner full of complex software.

mikeash 8y ago

Software is like any other technology. It can be a liability or an asset or both depending on how it’s built. Sometimes software fails and sometimes it even kills people, but how many lives has it also saved because of things like collision avoidance and accurate navigation?

CodeSheikh 8y ago

Correction: "I hope people are beginning to understand that the presence of poorly-tested software in any system is a liability."

collyw 8y ago
Well tested software still has bugs. Ones that haven't been tested for.
- CodeSheikh 8y ago
  So it makes it a not Well tested then if scenarios have not been tested. My point.
  - laretluval 8y ago
    Tautological.
  - quickthrower2 8y ago
    For all software that does something useful, there exist scenarios that are not tested.

anonu 8y ago

I think software has existed in computer systems for 7 or 8 decades now - and the pitfalls are well understood. I think the problem is one of simplicity vs complexity. New systems today rely on dozens - hundreds even - of packages and libraries - many of which are not written by the implementer. How can you ensure reliability in all these components? This is why engineering from first principles - even in today's software development environment - is important. Unfortunately, the trends are in the other direction.

dmead 8y ago
you would be wrong.
- AnimalMuppet 8y ago
  That's not a very useful reply. Can you explain where anonu is wrong? Can you explain why? Can you give some evidence? A bare dismissal is like five-year-olds arguing: "Did too!" "Did not!" It's not useful to advance the conversation, and it's not effective at persuading people.
  - noir_lord 8y ago
    It's quite possible that software creates new failure modes (as programmers I'm sure we all feel that to be true some days) while at the same time preventing more. When the software fails (if it did) we assume the software makes it more dangerous because we have a concrete example of what happened but not what could have happened in it's absence. This is what may eventually stall self driving cars for many years, the first time one does the PR equivalent of smashing through the local orphanage. We as a society are notoriously poor at assessing systemic risks. Now to address your point about evidence, I'm curious how you could control for other confounding factors. Airliners are safer today than ever before (based on passenger miles) but how do you control for better engines, material science, procedures if you compare say 1979 to 2019.
    - AnimalMuppet 8y ago

jessaustin 8y ago

It's a liability when you're getting equipment serviced, whether there's software or not. We have a Cat 287B skid-steer. After a month of inactivity, it wouldn't come out of "park" mode. A mechanic lifted the cab, tested a couple of solenoids, and stated "must be a bad computer". This is an old machine, so I called Cat and confirmed that it doesn't actually have a computer. I took up the floorboard and found some wires that had been chewed by rodents. We're lucky we don't have a computer, because replacing it wouldn't have helped...

Someone1234 8y ago

My take-away is a little different: As software grows it is hard to quantify. If you have several small distinct systems, you can often quantify them down to inputs, outputs, and expected logic. Once systems get too complex, you have too many possible states, and bugs are harder to find (and the system cannot be fully defined). I often go back to the famous Therac-25 accident. The bug in Therac-25 existed LONG before the accident, but there were two systems interacting, one system checked the other system's output and threw away invalid state. Once those two systems were merged, it was only then that the bug turned into an accident. If aircraft stopped building monolithic software and instead built parts that ran on software, parts that could each be individually verified, it would likely result in safer software.

AnimalMuppet 8y ago

The trend over the last 20 years has been that 1: airliners are getting safer and safer and 2: airliners contain more and more software. True, the software introduces new failure modes. But the presence of software in airliners seems to be less a liability than the absence of software.

outworlder 8y ago

> I hope people are beginning to understand that the presence of software in any system is a liability. I do not understand this point of view. If you don't have software, you are going to replace it with something mechanical. Or with people. Or lose the functionality entirely. Do you want to bring back a 'flight engineer' to operate airliner engines? Do you want to bring back mechanical autopilots? Analog computers? Let's ditch GPS while we are at it (GPS failures can cause deaths, specially under poor visibility).

laretluval 8y ago

More precisely. For two systems offering the same functionality, the one that does so using less software should be preferred, because software of any nontrivial complexity is irreducibly buggy.