6 ms·

It's a question of proper sensor fusion. The way it's typically done is with some form of bayesian prior + a robust estimator that can handle the case where you

by rsp1984 10y ago

It's a question of proper sensor fusion. The way it's typically done is with some form of bayesian prior + a robust estimator that can handle the case where your sensors give you disagreeing evidence of the true state. So you have a prediction function that takes as input your previous estimated state and some form of "common sense" that's hard-coded or learned and which outputs a prediction of the current state. Then you look at your current measurements and if one of it disagrees to much it gets downweighted or outright discarded. The updated estimate is then computed from the prediction and the current measurements.

grondilu 10y ago
Neat. There actually is a wikipedia entry about sensor fusion: https://en.wikipedia.org/wiki/Sensor_fusion https://en.wikipedia.org/wiki/Sensor_fusion TIL, thanks.
redcalx 10y ago
Indeed. I.e. have some probabilistic model of the 'world' and the probe's status within it (position, speed, rotation, etc). The sensor information then 'merely' informs that model, so e.g. if some sensor or piece of code says the probe has instantaneously jumped to a new location some distance away then the model can just disregard that as not conforming to anything like what was expected (the set of priors on how the probe moves, or essentially the laws of physics). Or... run the flight controller through a 'fuzzer' that throws lots of unexpected scenarios at it. To be honest I'm rather disappointed that a project at this level has succumbed to a bug like this. It's a few thousand dollars of additional dev effort versus the whole project failing; classic small certain initial cost versus large possible future cost (see Taleb's Antifragile, systemic failure of financial markets, etc.) Ho hum.
- ansible 10y ago
  It's a few thousand dollars of additional dev effort versus the whole project failing; classic small certain initial cost versus large possible future cost... We in hindsight, now know that a little extra development could have avoided the problem. The trick is trying to figure out where to spend your limited resources as the project is going. There are lots of areas that might have caused problems. There may well be other flaws in the system that would also have caused a failure on descent.
  - redcalx 10y ago
    > The trick is trying to figure out where to spend your limited resources as the project is going. In a situation such as this where you don't have the funds to test everything to the level you would like and the cost of a failure is large, I wonder if a reasonable compromise is to hire a smallish team of 'adversaries'. I.e. their full time job is to try and find issues with other people's work; and to ask what could go wrong, or, what is likely to to go wrong - let's go see if we can break that. I also find the idea of a risk register useful. I.e. during development you'll often have thoughts about whether particular edge cases are truly being handled properly, but often the day to day cost versus benefit decisions mean you can't explore all of these issues. If however you log them then this can serve as a source of ideas for an adversarial team, or indeed your own investigations if you happen to have bits of free time for testing.
    - plafl 10y ago
      Engineering simulators are used for this purpose, I have worked on them, and seen them being used precisely for trying to find unexpected system behaviors under sensor or other subsystem failures (valves, computers, motors, power,...). This is quite costly. In commercial aviation there are teams for developing the simulations and separate teams just trying all kind of strange scenarios. In less critical projects, say for example drones, I have seen it done too although with smaller teams. As I see it with the limited information we have right now it seems that either the error was made when writing the requirements for the IMU or the IMU wasn't the correct one. Having worked writing navigation software I cannot help siding with the software developers. The IMU is usually the central piece of the navigation system, at best they could have flagged it as failed but that is usually way more improbable that the altimeter failing. And even if they discarded the IMU measures I don't know what other sensors could be used to get at least the vehicle attitude. I think that after the saturation they were doomed and a software solution is not realistic.
      - redcalx 10y ago
        > software solution is not realistic. I would expect such a role to be holistic; to consider the entire system and ways it could fail. The boundaries between sub-systems would be high on the list of targets; software-software, software-hardware, hardware-hardware. It's not a matter of siding with anyone, it's about accepting that an error has been made at some level, and asking if there are general purpose strategies for increasing the probability of identifying errors without having to build a simulation. Simulations of course have all the same problems - they're complex and therefore expensive and will contain defects. Regarding the other point about probabilistic models compensating for sensor failures; it depends on which sensors fail and when. Parts of a descent are highly predictable, so e.g. if you know height and you're in free-fall then you can model pretty accurately where you're likely to be in +t seconds, and can use that as a prior for sensory input. OTOH if you're in the final landing stage using retro-thrusters then feedback sensors are paramount to doing landing correctly. I accept that a probabilistic model would be more expensive to develop; initially anyway, and therefore may be unrealistic with the resources available.
JohnStrange 10y ago
Yes, but maybe they should have implemented a more fault-tolerant system. With three different methods of determining altitude during decent, this problem shouldn't have occurred.
Tomrn 10y ago
This reminds me of something I read a couple of years ago about an earth-orbit satellite collecting O-zone data.. when it encountered the 'hole' it ended up throwing away most of the data since it failed its 'common sense' test. Wish I could find the link. In the case of an altimeter it makes sense to be aggressive with the 'common-sense' component to throw out bad data. But with other things like temperature, you wouldn't want to throw out data that could look wrong, but could actually be caused by some unknown phenomenon (like say, superheated gas pockets on mars) as that's exactly the type of weird stuff you'd be interested in. I guess the problem is distinguishing [data that differs greatly from what was expected] and [data from a sensor malfunctions]
- pvg 10y ago
  It's just ozone, not a kind of zone.

by rsp1984 10y ago

grondilu 10y ago

Neat. There actually is a wikipedia entry about sensor fusion: https://en.wikipedia.org/wiki/Sensor_fusion https://en.wikipedia.org/wiki/Sensor_fusion TIL, thanks.

redcalx 10y ago

Indeed. I.e. have some probabilistic model of the 'world' and the probe's status within it (position, speed, rotation, etc). The sensor information then 'merely' informs that model, so e.g. if some sensor or piece of code says the probe has instantaneously jumped to a new location some distance away then the model can just disregard that as not conforming to anything like what was expected (the set of priors on how the probe moves, or essentially the laws of physics). Or... run the flight controller through a 'fuzzer' that throws lots of unexpected scenarios at it. To be honest I'm rather disappointed that a project at this level has succumbed to a bug like this. It's a few thousand dollars of additional dev effort versus the whole project failing; classic small certain initial cost versus large possible future cost (see Taleb's Antifragile, systemic failure of financial markets, etc.) Ho hum.

ansible 10y ago
It's a few thousand dollars of additional dev effort versus the whole project failing; classic small certain initial cost versus large possible future cost... We in hindsight, now know that a little extra development could have avoided the problem. The trick is trying to figure out where to spend your limited resources as the project is going. There are lots of areas that might have caused problems. There may well be other flaws in the system that would also have caused a failure on descent.
- redcalx 10y ago
  > The trick is trying to figure out where to spend your limited resources as the project is going. In a situation such as this where you don't have the funds to test everything to the level you would like and the cost of a failure is large, I wonder if a reasonable compromise is to hire a smallish team of 'adversaries'. I.e. their full time job is to try and find issues with other people's work; and to ask what could go wrong, or, what is likely to to go wrong - let's go see if we can break that. I also find the idea of a risk register useful. I.e. during development you'll often have thoughts about whether particular edge cases are truly being handled properly, but often the day to day cost versus benefit decisions mean you can't explore all of these issues. If however you log them then this can serve as a source of ideas for an adversarial team, or indeed your own investigations if you happen to have bits of free time for testing.
  - plafl 10y ago
    Engineering simulators are used for this purpose, I have worked on them, and seen them being used precisely for trying to find unexpected system behaviors under sensor or other subsystem failures (valves, computers, motors, power,...). This is quite costly. In commercial aviation there are teams for developing the simulations and separate teams just trying all kind of strange scenarios. In less critical projects, say for example drones, I have seen it done too although with smaller teams. As I see it with the limited information we have right now it seems that either the error was made when writing the requirements for the IMU or the IMU wasn't the correct one. Having worked writing navigation software I cannot help siding with the software developers. The IMU is usually the central piece of the navigation system, at best they could have flagged it as failed but that is usually way more improbable that the altimeter failing. And even if they discarded the IMU measures I don't know what other sensors could be used to get at least the vehicle attitude. I think that after the saturation they were doomed and a software solution is not realistic.
    - redcalx 10y ago
      > software solution is not realistic. I would expect such a role to be holistic; to consider the entire system and ways it could fail. The boundaries between sub-systems would be high on the list of targets; software-software, software-hardware, hardware-hardware. It's not a matter of siding with anyone, it's about accepting that an error has been made at some level, and asking if there are general purpose strategies for increasing the probability of identifying errors without having to build a simulation. Simulations of course have all the same problems - they're complex and therefore expensive and will contain defects. Regarding the other point about probabilistic models compensating for sensor failures; it depends on which sensors fail and when. Parts of a descent are highly predictable, so e.g. if you know height and you're in free-fall then you can model pretty accurately where you're likely to be in +t seconds, and can use that as a prior for sensory input. OTOH if you're in the final landing stage using retro-thrusters then feedback sensors are paramount to doing landing correctly. I accept that a probabilistic model would be more expensive to develop; initially anyway, and therefore may be unrealistic with the resources available.

JohnStrange 10y ago

Yes, but maybe they should have implemented a more fault-tolerant system. With three different methods of determining altitude during decent, this problem shouldn't have occurred.

Tomrn 10y ago

This reminds me of something I read a couple of years ago about an earth-orbit satellite collecting O-zone data.. when it encountered the 'hole' it ended up throwing away most of the data since it failed its 'common sense' test. Wish I could find the link. In the case of an altimeter it makes sense to be aggressive with the 'common-sense' component to throw out bad data. But with other things like temperature, you wouldn't want to throw out data that could look wrong, but could actually be caused by some unknown phenomenon (like say, superheated gas pockets on mars) as that's exactly the type of weird stuff you'd be interested in. I guess the problem is distinguishing [data that differs greatly from what was expected] and [data from a sensor malfunctions]

pvg 10y ago
It's just ozone, not a kind of zone.