4 ms·

> Investigators have traced the problem to a faulty starter-generator, but have been unable to pinpoint why it goes haywire or devise a permanent fix. I find i

> Investigators have traced the problem to a faulty starter-generator, but have been unable to pinpoint why it goes haywire or devise a permanent fix. I find it incredible that they know where the problem is, but not how to fix it. Get 100-1000 of them and start stress testing them in a lab. This stuff isn't rocket science.

UnoriginalGuy 11y ago
I do love when people imply a "lab" is this mythical thing that can, through no effort, reproduce any likely or unlikely condition that one could ever expect to encounter. They should just print out a sign that says "lab" put it on an empty room, place the starter motor in the middle of it, and then suddenly they would know why it was broken. It is in a lab after all! This isn't rocket science! Or if one in an empty room fails put 100-1000 into the empty room, it has GOT to work!!! It is too easy to fail! Being in a lab is all that it takes! I'd suggest you watch this[0], this issue was literally costing human lives, they knew there was a rudder issue, they placed it in a lab, testing it for thousands of hours and ultimately found it by a little luck. It moved our understanding of failure states forward. There's no reason to assume that finding the issue with the starter motor is easy or trivial. No amount of lab hand waving changes that, it will take just brute effort. [0] https://www.youtube.com/watch?v=36mg8BNwBT4 https://www.youtube.com/watch?v=36mg8BNwBT4
- msandford 11y ago
  Yes, clearly it's not just "dump motors in room, get results", but rather that it's usually very doable to generate failures. There are only so many reasons that they can fail and it shouldn't be too difficult to generate the circumstances that cause those failures. Obviously figuring out which circumstances are the ones that are actually responsible for failures will be more difficult. But it's not too hard to do some sensitivity analysis to get some clues. If they're highly sensitive to having too much current drawn then that might make you suspect that somehow, somewhere the system's power demands are overwhelming the generator and that's what is causing the failure. The point behind "it's not rocket science" is that rockets are very, very expensive to make and so you can't just test as many of them as you'd like, for as long as you'd like, until you through trial and error reproduce the conditions that caused the failure. You also don't get the rocket back when it either succeeds or fails. But in this case the starter/generators are cheap, and they are able to in many cases sort through the debris of the crash. So that does make it qualitatively and quantitatively different than rocket science.
  - TallGuyShort 11y ago
    Sounds exactly like why people tell me it shouldn't take so long to get bug-free software.
    - msandford 11y ago
      You can't see software. You can't touch it. It can spontaneously change shape, form and function in dramatic ways if a single memory bit gets corrupted. That's very, very different than physical objects.
      - irishcoffee 11y ago
        Not when software controls hardware, which it does in this case.
        msandford 11y ago
        Please point me to the software that controls the alternator/generator/starter motor. I'm not talking about the whole thing, just the part whose failure is causing the larger failure. When software controls hardware, yes things can get weirder. But the software can't cause magical things to happen; the atoms don't rearrange themselves. Bit flips tend to be much more like magic, if applied to the real world. A wrench doesn't stop working because you swap out a single atom. But software can, if you flip the wrong bit.
      - TallGuyShort 11y ago
        On the other hand, I can relatively (meaning compared to physical aircraft) easily and inexpensively replicate thousands of copies of my prototypes and automate exhaustive testing. Complex systems are just complex - very few things are actually as trivial as a bystander may think.
      - thaumasiotes 11y ago
        > It can spontaneously change shape, form and function in dramatic ways if a single memory bit gets corrupted. > That's very, very different than physical objects. True... spontaneous changes to the structure of physical objects are quite a bit more likely. One of software's greatest features is that it's not subject to the physical wear that physical tools experience.
- the-dude 11y ago
  Video not available in the Netherlands.
sk8ingdom 11y ago
Ultimately the issue here is budget and time. With aircraft there are TONS of variables: temperature, pressure, humidity, vibration environment, dust. Even once you THINK you know all the variables, reproducing the problem in the lab might not happen--then after hundreds of hours of testing you're still at square one.
CamperBob2 11y ago
I find it incredible, too. As UnoriginalGuy says, lab tests don't always tell you what's going wrong in the field. But the fact that they don't know what's happening (if that's true at all, some other posts have called the whole article into question) is still the designers' fault. With modern hardware, the most important thing -- the thing you start building on Day #1 of the project -- is built-in diagnostics. In the case of something like the 737 rudder problem, those planes were designed 40 years ago, in an era when it wasn't practical to record every bit of operating data in real time, or add cheap, lightweight sensors to everything in sight. That's not true anymore. Whether they're building a drone, a fighter plane, a spacecraft, or even a modern passenger car, there is no excuse for requiring people to stand around some wreckage scratching their heads. The onboard controllers should be able to tell them exactly what went wrong. If they can't, then that's the problem to be solved first.
- metaphor 11y ago
  I'm having a difficult time imagining anything "cheap" about a complex sensor diagnostic system integrated into a defense aerospace platform with big daddy ACAT 1D program status whose acquisition strategy revolved around cost reduction.
  - CamperBob2 11y ago
    Well, here we're seeing the cost of not adding that stuff.
- toss1 11y ago
  Yes, we do now have far better onboard sensor suites and logging than in decades past. And this can provide incredible amounts of information, and get us closer to the source of issues much faster. Many times, it makes the issue immediately obvious and documented. HOWEVER! Do NOT make assumptions and fool yourself into thinking this is a cure-all. There are many problems that, while the logged sensor data narrows it down and provides clues, it can sometimes either not provide the critical clues, or can even be misleading. Yes, sometimes adding more sensors focused on the issue may help, but not always. Sometimes it is just down to the hard work of finding the peculiar circumstances and interactions that cause previously unknown failure modes. Been there, done that (UAV & motorsport systems), seen lots of very smart & experienced engineers stumped for way longer than you'd expect.

UnoriginalGuy 11y ago

I do love when people imply a "lab" is this mythical thing that can, through no effort, reproduce any likely or unlikely condition that one could ever expect to encounter. They should just print out a sign that says "lab" put it on an empty room, place the starter motor in the middle of it, and then suddenly they would know why it was broken. It is in a lab after all! This isn't rocket science! Or if one in an empty room fails put 100-1000 into the empty room, it has GOT to work!!! It is too easy to fail! Being in a lab is all that it takes! I'd suggest you watch this[0], this issue was literally costing human lives, they knew there was a rudder issue, they placed it in a lab, testing it for thousands of hours and ultimately found it by a little luck. It moved our understanding of failure states forward. There's no reason to assume that finding the issue with the starter motor is easy or trivial. No amount of lab hand waving changes that, it will take just brute effort. [0] https://www.youtube.com/watch?v=36mg8BNwBT4 https://www.youtube.com/watch?v=36mg8BNwBT4

msandford 11y ago
Yes, clearly it's not just "dump motors in room, get results", but rather that it's usually very doable to generate failures. There are only so many reasons that they can fail and it shouldn't be too difficult to generate the circumstances that cause those failures. Obviously figuring out which circumstances are the ones that are actually responsible for failures will be more difficult. But it's not too hard to do some sensitivity analysis to get some clues. If they're highly sensitive to having too much current drawn then that might make you suspect that somehow, somewhere the system's power demands are overwhelming the generator and that's what is causing the failure. The point behind "it's not rocket science" is that rockets are very, very expensive to make and so you can't just test as many of them as you'd like, for as long as you'd like, until you through trial and error reproduce the conditions that caused the failure. You also don't get the rocket back when it either succeeds or fails. But in this case the starter/generators are cheap, and they are able to in many cases sort through the debris of the crash. So that does make it qualitatively and quantitatively different than rocket science.
- TallGuyShort 11y ago
  Sounds exactly like why people tell me it shouldn't take so long to get bug-free software.
  - msandford 11y ago
    You can't see software. You can't touch it. It can spontaneously change shape, form and function in dramatic ways if a single memory bit gets corrupted. That's very, very different than physical objects.
    - irishcoffee 11y ago
      Not when software controls hardware, which it does in this case.
      - msandford 11y ago
        Please point me to the software that controls the alternator/generator/starter motor. I'm not talking about the whole thing, just the part whose failure is causing the larger failure. When software controls hardware, yes things can get weirder. But the software can't cause magical things to happen; the atoms don't rearrange themselves. Bit flips tend to be much more like magic, if applied to the real world. A wrench doesn't stop working because you swap out a single atom. But software can, if you flip the wrong bit.
    - TallGuyShort 11y ago
      On the other hand, I can relatively (meaning compared to physical aircraft) easily and inexpensively replicate thousands of copies of my prototypes and automate exhaustive testing. Complex systems are just complex - very few things are actually as trivial as a bystander may think.
    - thaumasiotes 11y ago
      > It can spontaneously change shape, form and function in dramatic ways if a single memory bit gets corrupted. > That's very, very different than physical objects. True... spontaneous changes to the structure of physical objects are quite a bit more likely. One of software's greatest features is that it's not subject to the physical wear that physical tools experience.
the-dude 11y ago
Video not available in the Netherlands.

sk8ingdom 11y ago

Ultimately the issue here is budget and time. With aircraft there are TONS of variables: temperature, pressure, humidity, vibration environment, dust. Even once you THINK you know all the variables, reproducing the problem in the lab might not happen--then after hundreds of hours of testing you're still at square one.

CamperBob2 11y ago

I find it incredible, too. As UnoriginalGuy says, lab tests don't always tell you what's going wrong in the field. But the fact that they don't know what's happening (if that's true at all, some other posts have called the whole article into question) is still the designers' fault. With modern hardware, the most important thing -- the thing you start building on Day #1 of the project -- is built-in diagnostics. In the case of something like the 737 rudder problem, those planes were designed 40 years ago, in an era when it wasn't practical to record every bit of operating data in real time, or add cheap, lightweight sensors to everything in sight. That's not true anymore. Whether they're building a drone, a fighter plane, a spacecraft, or even a modern passenger car, there is no excuse for requiring people to stand around some wreckage scratching their heads. The onboard controllers should be able to tell them exactly what went wrong. If they can't, then that's the problem to be solved first.

metaphor 11y ago
I'm having a difficult time imagining anything "cheap" about a complex sensor diagnostic system integrated into a defense aerospace platform with big daddy ACAT 1D program status whose acquisition strategy revolved around cost reduction.
- CamperBob2 11y ago
  Well, here we're seeing the cost of not adding that stuff.
toss1 11y ago
Yes, we do now have far better onboard sensor suites and logging than in decades past. And this can provide incredible amounts of information, and get us closer to the source of issues much faster. Many times, it makes the issue immediately obvious and documented. HOWEVER! Do NOT make assumptions and fool yourself into thinking this is a cure-all. There are many problems that, while the logged sensor data narrows it down and provides clues, it can sometimes either not provide the critical clues, or can even be misleading. Yes, sometimes adding more sensors focused on the issue may help, but not always. Sometimes it is just down to the hard work of finding the peculiar circumstances and interactions that cause previously unknown failure modes. Been there, done that (UAV & motorsport systems), seen lots of very smart & experienced engineers stumped for way longer than you'd expect.