8 ms·

In their defense, probably every piece of software of any complexity at all has bugs waiting to be found, and it's not super surprising that they found some new

In their defense, probably every piece of software of any complexity at all has bugs waiting to be found, and it's not super surprising that they found some new ones while doing a rigorous testing regimen.

totalZero 7y ago
Shouldn't that rigorous testing regimen have preceded the sale and flight of these large passenger aircraft?
- onychomys 7y ago
  Sure. But I'd hope that they brought in new people, with a new outlook on things, and started all over from scratch. They're probably running mostly the same tests they were before the crashes, and also some new ones that people have dreamed up.
  - tremon 7y ago
    I think it's more likely that all the managers in the chain of control suddenly feel like their ass is on the line if another 737 MAX drops out of the sky, which is why they're now paying more attention to what their engineers have been telling them for years. Which is probably a good thing, except for the part that it took hundreds of people to die first to get here.
- kayfox 7y ago
  A regime which finds all the defects would cost billions on its own, only one such piece of software has been subject to such a regime and that's the Space Shuttle guidance program. Every other piece of software is tested to levels set by the aeronautical industry and regulators, which may not include all possible scenarios. You will note, some of these scenarios are essentially fuzzing the memory of the flight computer and seeing what happens, ideally most bit flips will either be detected or be minor, but some can end up causing issues. I'm not sure if it would be possible to explore all the branches in the software in any sort of reasonable amount of time. https://en.wikipedia.org/wiki/Qantas_Flight_72 https://en.wikipedia.org/wiki/Qantas_Flight_72
  - imglorp 7y ago
    That cost is already baked into aviation hardware. It's why fasteners are so expensive, for example, because the cost of certifying, tracking their history, and quantifying their quality is already amortized into each bolt. Flight software development lifecycle--and regs as you point out--still needs to catch up.
    - m4rtink 7y ago
      Actually, for fasteners there is often a material difference - cheaper fasteners are cut to shape while better fasteners are forged into shape, resulting in a much stronger part due to the resulting grain flow.
  - andbberger 7y ago
    What's this about the space shuttle? I'm intrigued. I thought that the paris metro was the posterchild of formal verification methods.
    - salawat 7y ago
      The software that went into the Space Shuttle was coded to CMM level 5; this is a level of Quality Assurance that would make even the most Iron Fisted, detail oriented QA begin to sweat at the pile of paperwork and pressure ahead of them. It's a completely different experience that likely only a handful of modern testers would even contemplate nowadays, but has always been a shining example of something to aspire to be part of one day for me. It isn't at all easy, and basically requires the discipline to understand, document, and justify every single line of code within the context of the overarching system. No business wants to deal with it, and it is no surprise that only publically funded projects tend to get anywhere close. https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20110014946.pdf https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/201100... https://developers.slashdot.org/story/00/05/19/050258/space-shuttle-software-not-for-hacks https://developers.slashdot.org/story/00/05/19/050258/space-...
  - tgflynn 7y ago
    > A regime which finds all the defects would cost billions on its own, only one such piece of software has been subject to such a regime and that's the Space Shuttle guidance program. Do you have a source for that claim ? I'm skeptical that even a testing process costing billions of dollars could claim to cover all possible inputs and states for a software system of any complexity.
johannes1234321 7y ago
As a plane crash costs lives of a few hundred people the criteria should be different from "normal" software. And for most parts and most history software in aeronautics was created with special care and tooling and under research of provability.
- organsnyder 7y ago
  In the case of MCAS, that's likely true: the software was behaving exactly per spec; the problem was in the earlier requirements-definition phase.
  - salawat 7y ago
    A grievous failure in Systems Engineering and FMEA, I believe is the industry term.
coldpie 7y ago
In their defense, the software industry is a complete joke in terms of quality control. I hope that this and the spate of ransomware will wake the industry up to realize we need new processes, languages and tooling to make software provably correct. It's clear we can't use our existing languages and tooling to make high quality software. Realistically though, that's not going to happen. The smart move is to eliminate as much software from your life as possible. It's only going to get worse as decades of laziness catches up to us.
- m_fayer 7y ago
  Watching Android descend from a place of dorky stability to sleek sealed glitch-city, kinda makes me think it's the institutions, as opposed to the tooling, that's behind the plunge in the quality of mainstream software.
  - peteradio 7y ago
    Feed me features I shit backlogs.
    - salawat 7y ago
      I couldn't help but laugh, because it is so damn true. It's taken me a while to come to the realization that all project management is is a backlog manufacturing and prioritization layer that operates on top of actual software engineers. Most just want to implement things, and don't care what it is they are implementing as long as they are getting paid.
  - shadowgovt 7y ago
    Was Android the stable one? I recall relative to iOS that it was the one likely to be eating its entire battery in the early iterations.
  - jsight 7y ago
    I can't agree. I've been using Android since the Nexus One days, and it is generally much more stable for me than some of those early devices.
simion314 7y ago
>and it's not super surprising that they found some new ones while doing a rigorous testing regimen. reply So they did not do enough testing before, just the happy case and never considered sensor failures. I am now wondering if they tested the rest of the software or FAA will just look into MCAS and ignore all the rest.
anshou- 7y ago
When some software is being built the constraints are much more rigorous, but it all comes down to the organization. The guidelines established by NASA for their space program are a good example of rigorous requirements. Boeing should not be unfamiliar with this concept and can certainly afford to do this properly. This move was motivated purely by greed.
skissane 7y ago
> it's not super surprising that they found some new ones while doing a rigorous testing regimen Why didn’t they find these bugs during the plane’s development and certification? It appears the testing then was less rigorous. They should extend the same rigorous testing to their other aircraft models also.
Findeton 7y ago
Actually there are two types of software failures: design failures and implementation failures. Design failures are harder, but we already have ways of completely avoiding implementation failures, with systems that prove mathematically that the software implements the design. NASA for example uses those in many occasions. Boeing doesn't, obviously, and there's no valid excuse.
nradov 7y ago
The Space Shuttle flight computer software was effectively bug free. Only a tiny number of defects were found post release, and none impacted safety.

totalZero 7y ago

Shouldn't that rigorous testing regimen have preceded the sale and flight of these large passenger aircraft?

onychomys 7y ago
Sure. But I'd hope that they brought in new people, with a new outlook on things, and started all over from scratch. They're probably running mostly the same tests they were before the crashes, and also some new ones that people have dreamed up.
- tremon 7y ago
  I think it's more likely that all the managers in the chain of control suddenly feel like their ass is on the line if another 737 MAX drops out of the sky, which is why they're now paying more attention to what their engineers have been telling them for years. Which is probably a good thing, except for the part that it took hundreds of people to die first to get here.
kayfox 7y ago
A regime which finds all the defects would cost billions on its own, only one such piece of software has been subject to such a regime and that's the Space Shuttle guidance program. Every other piece of software is tested to levels set by the aeronautical industry and regulators, which may not include all possible scenarios. You will note, some of these scenarios are essentially fuzzing the memory of the flight computer and seeing what happens, ideally most bit flips will either be detected or be minor, but some can end up causing issues. I'm not sure if it would be possible to explore all the branches in the software in any sort of reasonable amount of time. https://en.wikipedia.org/wiki/Qantas_Flight_72 https://en.wikipedia.org/wiki/Qantas_Flight_72
- imglorp 7y ago
  That cost is already baked into aviation hardware. It's why fasteners are so expensive, for example, because the cost of certifying, tracking their history, and quantifying their quality is already amortized into each bolt. Flight software development lifecycle--and regs as you point out--still needs to catch up.
  - m4rtink 7y ago
    Actually, for fasteners there is often a material difference - cheaper fasteners are cut to shape while better fasteners are forged into shape, resulting in a much stronger part due to the resulting grain flow.
- andbberger 7y ago
  What's this about the space shuttle? I'm intrigued. I thought that the paris metro was the posterchild of formal verification methods.
  - salawat 7y ago
    The software that went into the Space Shuttle was coded to CMM level 5; this is a level of Quality Assurance that would make even the most Iron Fisted, detail oriented QA begin to sweat at the pile of paperwork and pressure ahead of them. It's a completely different experience that likely only a handful of modern testers would even contemplate nowadays, but has always been a shining example of something to aspire to be part of one day for me. It isn't at all easy, and basically requires the discipline to understand, document, and justify every single line of code within the context of the overarching system. No business wants to deal with it, and it is no surprise that only publically funded projects tend to get anywhere close. https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20110014946.pdf https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/201100... https://developers.slashdot.org/story/00/05/19/050258/space-shuttle-software-not-for-hacks https://developers.slashdot.org/story/00/05/19/050258/space-...
- tgflynn 7y ago
  > A regime which finds all the defects would cost billions on its own, only one such piece of software has been subject to such a regime and that's the Space Shuttle guidance program. Do you have a source for that claim ? I'm skeptical that even a testing process costing billions of dollars could claim to cover all possible inputs and states for a software system of any complexity.

johannes1234321 7y ago

As a plane crash costs lives of a few hundred people the criteria should be different from "normal" software. And for most parts and most history software in aeronautics was created with special care and tooling and under research of provability.

organsnyder 7y ago
In the case of MCAS, that's likely true: the software was behaving exactly per spec; the problem was in the earlier requirements-definition phase.
- salawat 7y ago
  A grievous failure in Systems Engineering and FMEA, I believe is the industry term.

coldpie 7y ago

In their defense, the software industry is a complete joke in terms of quality control. I hope that this and the spate of ransomware will wake the industry up to realize we need new processes, languages and tooling to make software provably correct. It's clear we can't use our existing languages and tooling to make high quality software. Realistically though, that's not going to happen. The smart move is to eliminate as much software from your life as possible. It's only going to get worse as decades of laziness catches up to us.

m_fayer 7y ago
Watching Android descend from a place of dorky stability to sleek sealed glitch-city, kinda makes me think it's the institutions, as opposed to the tooling, that's behind the plunge in the quality of mainstream software.
- peteradio 7y ago
  Feed me features I shit backlogs.
  - salawat 7y ago
    I couldn't help but laugh, because it is so damn true. It's taken me a while to come to the realization that all project management is is a backlog manufacturing and prioritization layer that operates on top of actual software engineers. Most just want to implement things, and don't care what it is they are implementing as long as they are getting paid.
- shadowgovt 7y ago
  Was Android the stable one? I recall relative to iOS that it was the one likely to be eating its entire battery in the early iterations.
- jsight 7y ago
  I can't agree. I've been using Android since the Nexus One days, and it is generally much more stable for me than some of those early devices.

simion314 7y ago

>and it's not super surprising that they found some new ones while doing a rigorous testing regimen. reply So they did not do enough testing before, just the happy case and never considered sensor failures. I am now wondering if they tested the rest of the software or FAA will just look into MCAS and ignore all the rest.

anshou- 7y ago

When some software is being built the constraints are much more rigorous, but it all comes down to the organization. The guidelines established by NASA for their space program are a good example of rigorous requirements. Boeing should not be unfamiliar with this concept and can certainly afford to do this properly. This move was motivated purely by greed.

skissane 7y ago

> it's not super surprising that they found some new ones while doing a rigorous testing regimen Why didn’t they find these bugs during the plane’s development and certification? It appears the testing then was less rigorous. They should extend the same rigorous testing to their other aircraft models also.

Findeton 7y ago

Actually there are two types of software failures: design failures and implementation failures. Design failures are harder, but we already have ways of completely avoiding implementation failures, with systems that prove mathematically that the software implements the design. NASA for example uses those in many occasions. Boeing doesn't, obviously, and there's no valid excuse.

nradov 7y ago

The Space Shuttle flight computer software was effectively bug free. Only a tiny number of defects were found post release, and none impacted safety.