3 ms·

Why would the program react like that to a SINGLE wrong signal that disagrees with everything else and produce a signal that cannot do anything good in any circ

Why would the program react like that to a SINGLE wrong signal that disagrees with everything else and produce a signal that cannot do anything good in any circumstances? This just smells like a truly naive piece of implementation. There should be layers upon layers of safeties to prevent this dumb thing from happening. The computer should know the position, orientation and velocity of the rocket at any point in time and new signals should be interpreted in the context of what the computer already knows and in context of what other sensors are telling. It is not like the rocket can turn itself around in 1ms and if it does there probably isn't much it can do anyway. This suggests to me the problem is not the bug, it is the overall quality of development.

amelius 4y ago
I'm also wondering if they had a simulation environment, and why it didn't catch the problem.
- twawaaay 4y ago
  Pretty much. The trajectory is preplanned anyway. I would expect nothing less than every launch to be ran through simulated environment multiple times if only to catch wrong launch information. Also this FP to integer does not smell any better. Anybody who's been interested in programming for any length of time will learn this is just a bad idea asking for trouble. If I was tech lead for the project I would definitely make sure there is static analysis that prevents these kinds of conversions from happening.
  - allenrb 4y ago
    And you'd think catching incorrect launch information would be an obvious focal point, and yet... https://spaceflightnow.com/2018/02/23/investigators-say-erroneous-navigation-input-led-ariane-5-rocket-off-course/ https://spaceflightnow.com/2018/02/23/investigators-say-erro...
    - twawaaay 4y ago
      Yup. Definitely seems like they have basic quality issue.
dahart 4y ago
The article mentions there was a duplicate redundant signal, and they both agreed with each other. The problem indeed wasn’t the bug, you’re right. It was the assumption somewhere along the way that this older system couldn’t cause damage, the failure to review the fixed point range. It’s easy to judge with hindsight though. Every rocket development project that has ever happened on the planet has had unintended explosions and accidents, and they are always staffed with brilliant people. Layers of safety at some level might only make the problem harder, more code just adds more complexity and failure points. Since there is an uncountable number of ways for something seemingly innocuous to break a rocket, and until we’ve all tried making rockets, it’s probably best to take away the lesson that fundamentally making rockets is highly prone to catastrophic failure.
- twawaaay 4y ago
  Of course there was a redundant signal (because the contract probably required it). What I meant was other signals -- other information about state of the rocket. There is lots of sensors in these things if only so that you can figure out when stuff goes wrong. > It’s easy to judge with hindsight though. Every rocket development project that has ever happened on the planet has had unintended explosions and accidents, No, that's bad excuse. Accidents do happen but can only be excused when a reasonable effort to prevent it has been taken. For example, Challenger disaster resulted in such a huge shakeup in Nasa exactly because reasonable precautions were NOT taken vs. other accidents which were truly unforeseen and were due to lack of knowledge/experience. The question is about taking reasonable precautions. It is reasonable effort to design a system that will drive multiple half a billion dollar rockets with some level of care to ignore absolutely idiotic signals. I do it on my home projects and at work with non safety critical applications. Why can't they do it for such a critical project?
  - dahart 4y ago
    > Why can’t they do it for such a critical project? They can, they did, and they still exploded a rocket! Again, hindsight is 20/20. This question isn’t very reasonable to ask this way (with incredulity) until you’ve successfully built several rockets yourself.
deleted 4y ago
[deleted]

amelius 4y ago

I'm also wondering if they had a simulation environment, and why it didn't catch the problem.

twawaaay 4y ago
Pretty much. The trajectory is preplanned anyway. I would expect nothing less than every launch to be ran through simulated environment multiple times if only to catch wrong launch information. Also this FP to integer does not smell any better. Anybody who's been interested in programming for any length of time will learn this is just a bad idea asking for trouble. If I was tech lead for the project I would definitely make sure there is static analysis that prevents these kinds of conversions from happening.
- allenrb 4y ago
  And you'd think catching incorrect launch information would be an obvious focal point, and yet... https://spaceflightnow.com/2018/02/23/investigators-say-erroneous-navigation-input-led-ariane-5-rocket-off-course/ https://spaceflightnow.com/2018/02/23/investigators-say-erro...
  - twawaaay 4y ago
    Yup. Definitely seems like they have basic quality issue.

dahart 4y ago

The article mentions there was a duplicate redundant signal, and they both agreed with each other. The problem indeed wasn’t the bug, you’re right. It was the assumption somewhere along the way that this older system couldn’t cause damage, the failure to review the fixed point range. It’s easy to judge with hindsight though. Every rocket development project that has ever happened on the planet has had unintended explosions and accidents, and they are always staffed with brilliant people. Layers of safety at some level might only make the problem harder, more code just adds more complexity and failure points. Since there is an uncountable number of ways for something seemingly innocuous to break a rocket, and until we’ve all tried making rockets, it’s probably best to take away the lesson that fundamentally making rockets is highly prone to catastrophic failure.

twawaaay 4y ago
Of course there was a redundant signal (because the contract probably required it). What I meant was other signals -- other information about state of the rocket. There is lots of sensors in these things if only so that you can figure out when stuff goes wrong. > It’s easy to judge with hindsight though. Every rocket development project that has ever happened on the planet has had unintended explosions and accidents, No, that's bad excuse. Accidents do happen but can only be excused when a reasonable effort to prevent it has been taken. For example, Challenger disaster resulted in such a huge shakeup in Nasa exactly because reasonable precautions were NOT taken vs. other accidents which were truly unforeseen and were due to lack of knowledge/experience. The question is about taking reasonable precautions. It is reasonable effort to design a system that will drive multiple half a billion dollar rockets with some level of care to ignore absolutely idiotic signals. I do it on my home projects and at work with non safety critical applications. Why can't they do it for such a critical project?
- dahart 4y ago
  > Why can’t they do it for such a critical project? They can, they did, and they still exploded a rocket! Again, hindsight is 20/20. This question isn’t very reasonable to ask this way (with incredulity) until you’ve successfully built several rockets yourself.

deleted 4y ago

[deleted]