11 ms·
Schiaparelli landing investigation makes progress
- MrBuddyCasino 10y agoWhen merged into the navigation system, the erroneous information generated an estimated altitude that was negative – that is, below ground level. I wonder if this was a sensor fusion problem or a pedestrian integer overflow.
- blutack 10y agoI would hope that it was a sensor fusion error. IMU saturation is an expected condition, especially in this type of extreme real-world environment. It's absolutely normal to temporarily saturate IMU sensors on drones during hard landings for example and saturation values are generally clear in the documentation. A large part of the difficulty in position and attitude estimators is in rejecting glitching or erroneous data. From the description (specifically that the duration of the saturation was an issue), it would sound like the position estimator (most likely a kalman filter) did not reject or distrust saturation values properly and converged on a bad solution. It appears that any strategy in place to (for example) reset the estimator state if it diverged from more reliable sensors (such as the radar altimeter), failed or were not included. It would also appear that the case where the IMU saturated during descent for over 1 second was not properly tested, as they were able to reproduce the issue in simulation. This must be disappointing for the ESA but at least it was found now 'in beta' rather than in the relatively more important next lander.
- votingprawn 10y agoWhilst incredibly disappointing for ESA, this is unfortunately not their first IMU-failure rodeo. An Ariane 5 launch back in 1996 [0] suffered a catstrophic failure after the inertial reference units gave bad data, and the flight control computer accepted it as gospel. It is sad that they might have lost another platform due to a lack of appropriate range / saturation checking, especially as there was a radar altimeter onboard telling them they weren't underground. [0] https://en.wikipedia.org/wiki/Cluster_(spacecraft)#Launch_failure https://en.wikipedia.org/wiki/Cluster_(spacecraft)#Launch_fa...
- creshal 10y agoAriane is built by Airbus, Schiaparelli was built by Thales Alenia, two completely independent companies. Not sure why there's supposed to be a correlation, other than both were contracted by ESA.
- acqq 10y agoNobody claimed the same company did it, just that there are similarities in the way the software errors caused the destruction.
- votingprawn 10y ago> both were contracted by ESA. Yes I'm aware of how ESA works and who built what. But I would have hoped the common connection would have greater instilled the need for checking for these sorts of errors, if the Schiaparelli incident is as blutack theorised.
- projektinjohto 10y agoIt says on Thales webpage that their IMU is already onboard on Ariane 5. I don't know what IMU was on Schiaparelli. https://www.thalesgroup.com/en/worldwide/aerospace/topaxyz-inertial-measurement-unit-imu https://www.thalesgroup.com/en/worldwide/aerospace/topaxyz-i...
- Gibbon1 10y ago> Kalman filter I hesitate to speculate but the book I read that talked of Kalman filters and two people I know who messed with them said they are opaque and tricky to get right[1]. Hard to understand what the filter constants mean, and buggy software implementations 'sort of work' then break. [1] Convinced me to use something else.
- planteen 10y agoNonsense, Kalman filters are well understood and widely used in spacecraft attitude dynamics and control systems. In fact, I'd argue that a Kalman filter is industry standard for that work. It's typical on teams doing GNC (guidance, navigation, and control) to have a PhD or two in control theory to make sure you get things right. Most aerospace engineers with a GNC background know a lot about Kalmam filters. Examples: http://www.techbriefs.com/component/content/article/ntb/webcasts/on-demand-webinars/23852 http://www.techbriefs.com/component/content/article/ntb/webc... http://digitalcommons.usu.edu/smallsat/2016/TS10AdvTech2/5/ http://digitalcommons.usu.edu/smallsat/2016/TS10AdvTech2/5/
- acqq 10y agoThe simpler statement would be: the lander started to spin after the parachute opened, and the sensor properly detected that it rotated a lot, giving the maximum value as an output, but the rotation lasted for at least one second and we made the rest of software without having any idea that this could happen, so this one sensor output confused the whole lander. Or even shorter "nobody here expected it could spin the whole second." Once the model is so simple, there are many ways to program it wrongly. To compare, NASA's Mars Climate Orbiter https://en.wikipedia.org/wiki/Mars_Climate_Orbiter https://en.wikipedia.org/wiki/Mars_Climate_Orbiter burned close to Mars in 1998 because the "ground based computer" programmed per contract by Lockheed used lbf·s values instead of agreed N·s values. https://en.wikipedia.org/wiki/Pound_(force) https://en.wikipedia.org/wiki/Pound_(force) The values generated on the ground were 4.45 times bigger. Space missions are risky.
- fest 10y agoIMO it's more likely that accelerometer instead of gyroscope would saturate, and the sentence that explains IMU is shortened too much.
- acqq 10y agoMaybe. The only thing I can conclude from the description is just that the lander was simply spinning too long (too long as in "the whole second") for their software to handle the correct information from the sensors. Now it sounds too simple. But it's good that they made such an experiment before sending the whole rover, planned for 2020. https://en.wikipedia.org/wiki/ExoMars_(rover) https://en.wikipedia.org/wiki/ExoMars_(rover)
- raverbashing 10y agoMight be a sensor range issue
- kashkhan 10y agoMaybe next time have a better system design so one instrument/sensor error/failure/loss doesn't lead to total loss of unit.
- w_t_payne 10y agoMaybe rather than a design issue this can be seen as a development process and engineering management issue. I expect that a top level system model should reduce the risk of this sort of error. Of course, it is hard to justify the cost of developing these models without being able to amortize that cost over multiple products / missions, so product line oriented engineering management and strong leadership from the sponsor has a big role to play as well.
- kashkhan 10y agoErrors in INS are common. Saturation is common. What isn't common is a bad design that isn't failsafe or failop for a single failure.
- LoSboccacc 10y agoRemember, spacecrafts have weight constraints that limits redundancy options. Besides this was a 'tracing bullet' designed to test an unproven procedure before committing to a more expensive mission, so there's that.
- kashkhan 10y agoDesigning a reliable system that is failure resistant is what aerospace design is all about. With INS fail op design is the norm. Yes accidents still happen for common mode failures, but those are supposed to be worked out before committing hundreds of millions of euros. Costly mistakes happen when people take unjustified shortcuts. Like not testing Hubble's mirror on the ground, and ending up with a myopic telescope in orbit that then needs another billion dollar mission to correct.
- LoSboccacc 10y ago
- acdx 10y agoSo their model of the spacecraft behavior allowed for the possibility of the altitude instantly changing from 3.7km to a negative value. Seems like poor design.
- isoprophlex 10y ago"A large volume of data recovered from the Mars lander shows that the atmospheric entry and associated braking occurred exactly as expected" In every communication and media event they keep putting a positive spin on what now turns out to be in essence a preventable design error...
- TeMPOraL 10y agoThey have to, unfortunately, because general public (including journalists) doesn't understand that mistakes happen, and ESA being seriously underfunded as it is, cannot afford bad publicity. Also, this was a test flight designed to gather data about the performance of the orbiter/lander platform, so it did perform its mission and according to the collected data, the platform behaves mostly as expected.
- baq 10y agothey're fighting tooth and nail for this mission's successor (which will be the real deal vs what was effectively an expensive acceptance test run) to not be canceled, i.e. politics.
- manarth 10y agoTheir system design accepted data from a number of sensors, and allowed for the possibility of sensor error. In a hypothetical scenario, if one of those altitude sensors used barometric pressure, then a piece of grit or ice could have blocked the air intake, allowing the sensor to record a lower pressure than expected. If that grit or ice were suddenly blown away, then the sensor could record an instant change from an incorrect value (3.7km altitude) to a correct value. Even a negative altitude isn't necessarily incorrect, if the "altitude" is based on a mean ground level. For example, the shore of the Dead Sea is at an altitude of -423m [1]. The challenge with designing something like this is to imagine all the possible ways it could fail, all the possible misleading sensors inputs, and an appropriate action for each of them. [1] https://en.wikipedia.org/wiki/List_of_places_on_land_with_elevations_below_sea_level https://en.wikipedia.org/wiki/List_of_places_on_land_with_el...
- xgbi 10y agoWhat is fantastic here is that the telemetry was received for nearly the entire descent, up to a few seconds before the crash. The could reproduce the exact same scenario in simulation by replaying the telemetry as inputs to their firmware and see what happened in the software. This is tremendous advantage for debugging, it's like a wireshark replay of network packets for debugging an error in the TCP stack. This orbiter/descender architecture is really paying off for future missions. If the orbiter hadn't been there, earth wouldn't have received any telemetry at all.. Let's hope that this "real world" telemetry is saved for simulating successive versions of the guidance software of future probes.
- fest 10y agoI've worked on sensor fusion systems and I fully agree. Replaying the sensor inputs and evaluating new estimated state is a really good way of debugging failures (because you can't just stop the system mid-air and evaluate internal state). It also helps with regression test suite and trying out new algorithms quickly.
- deleted 10y ago[deleted]
- vvanders 10y agoWe'd do the same for debugging networked games. Incredibly useful. If you build it right you also get a replay feature for almost free.
- ragebol 10y agoIf you use ROS to do robotics, there's ROS Bags [0] that also allow this. Very very useful. [0] http://wiki.ros.org/rosbag http://wiki.ros.org/rosbag
- mikepurvis 10y agoThere's a lot of interest in supplementary tooling around this format, too. See MARV robotics (formerly bagbunker), about which there was a ROSCon talk this fall: https://github.com/ternaris/marv-robotics https://github.com/ternaris/marv-robotics https://vimeo.com/187705380 https://vimeo.com/187705380
- yummybear 10y agoGood work diagnosing the error. A reliable diagnostic is a great outcome of a bad situation. At least now, measures can be taken to eliminate this error from future scenarios.
- fest 10y agoWhile it may sound that the failure was caused by a rookie mistake- I'm sure the on-board systems were designed and developed by people who know what they are doing. It must be hard for all the people involved in Schiaparelli development. Sometimes I wonder whether majority of software errors are caused only by multiple layers of people having incorrect / unchallenged / unjustified (and mostly implicit) assumptions about something.
- monk_e_boy 10y agoSeems strange that a function looked at the data and thought the machine was underground didn't throw that data out. Wouldn't they have a model of expected data and anything that is wildly out be removed? Fall back to an estimation?
- fest 10y agoThey probably have. A large problem in sensor fusion is: you don't always know which sensor you can trust. For example, in this case it probably would have helped to reject IMU and rely on doppler radar altimeter. But what if it's doppler radar that for some reason return an incorrect value that indicates that vehicle is a lot higher than IMU?
- TheOtherHobbes 10y agoIt's reasonable to assume that if you're still moving and apparently underground there's a problem with the sensors, and therefore it's best not to do anything rash - like starting the landing sequence. This does sound like a rookie mistake, and I'll bet there's a dev somewhere saying "Oh crap" a lot because of it. It's also a hardware design problem. NASA's human missions have a weight budget that includes multiple redundant systems. I have no idea if this probe had three of everything, but missions that do are more likely to be successful. More subtly, I'm curious if the sensor failed or maxed out. The kinds of forces that would max out the sensor might also max out the probe. So it's possible it was already doomed after they were encountered.
- grondilu 10y ago> the erroneous information generated an estimated altitude that was negative – that is, below ground level. Kind of sad to see that computers nowadays still lack what we call common sense. I mean the machine received negative altimeter data but if it had been able to just look around, it could have seen that this data was erroneous. Hell, considering the descent profile was carefully prepared, the computer should also have noticed that it was way too early to reach the ground and should have concluded that something was wrong with the altitude measure.
- ygra 10y agoWell, common sense in a way would be enabled with assertions. But even if you know that you cannot be below ground, how would you recover? Your sensors clearly tell you that you're below ground, so what is the course of action? You have no way of knowing where you are, you just know that you don't know. It doesn't really change anything in the outcome. Sure, maybe you could go from altitude measurements to time measurements and most likely still leave a crater since it's not exact enough (there's a reason for the internal measurement instead of simpler ways, after all). It would also require you to anticipate the malfunction in which case you already know that things like that could happen, in which case you'd likely determine how it could happen and how to avoid it and continue having useful data. Sounds to me like a better solution than trying to recover without data.
- grondilu 10y agoFrom the article: > This in turn successively triggered a premature release of the parachute and the backshell, a brief firing of the braking thrusters and finally activation of the on-ground systems as if Schiaparelli had already landed. It would have been better to do nothing, in the hope that the altimeter data will resume to reasonable values, especially since the article mentions that the error is not supposed to last long: > Its output was generally as predicted except for this event, which persisted for about one second – longer than would be expected.
- WalterBright 10y agoThere are several ways. First off, ignore the erroneous data. Then, estimate the altitude by other methods. As you mentioned, one way would be to integrate time and velocity. All inputs and outputs should be run through sanity checks, with a plan for when they go wrong. Essentially, when you've got a system that can't be fixed, plan for failure of each and every component (in isolation).
- xer 10y agoHow could there not be a test case against negative attitude during descent?
- tie_ 10y agoFirst integer overflow to (literally) hit Mars!
- dudeonthenet 10y agoWhat i'm wondering is why is it that when the Inertial Measurement Unit erroneous information was fed into the navigation system, and resulted in a negative estimated altitude, didn't the mars lander also have an accelerometer that could tell "hey, we're still accelerating, we're not stationary" and thus attempt some form of recovery of the navigation system.. perhaps reset the IMU or something? Reread data from the IMU after a certain delay? I'm also quite confident that even before commencing entry, they had some idea from simulations as to how much the descent should take. Surely there's something wrong f you read negative altitude after just half the time required for landing.
- programmer_dude 10y agoWhat's the difference between an IMU and an accelerometer?
- dudeonthenet 10y agoSorry, what I meant to say was a redundant IMU that could be read in specific cases like this one.
- planteen 10y agoIMU means gyro, that is, it measures rotational rates. Accelerometers measure translational rates.
- votingprawn 10y ago> IMU means gyro In aerospace the phrase IMU typically refers to the combination of accelerometers and gyroscopes.
- fghgfdfg 10y agoBut you don't know which IMU is misreporting. Now, given the nature of the situation you could probably say it's best to err on the side of acceleration. If you've stopped accelerating and haven't initiated landing procedures you're either wrong or somehow already landed and can afford to wait. It's also not clear to me that this was an issue of erroneous data. All they said so far was that a rotational sensor maxed out for about a second. I don't think a simple delay-and-retry would have sufficed here. When attempting to land a second seems like a pretty long wait.
- radarsat1 10y agoI'm still trying to understand whether this was a software or a hardware issue.. I mean, it's true that perhaps the software should have been better at rejecting the saturated data, but if an IMU persists in a saturated state for 1 second, that's a whole lot of time that the software just cannot know what is happening. Granted, if it predicts an impossible configuration due to this, perhaps some different action should have been taken, like waiting for higher confidence and a more close-to-expected attitude estimate, but by then perhaps it would be too late. I mean, the thing is falling from the sky.. perhaps the "safe" thing was to deploy anyway. (Even if it didn't work.) I just don't know.
- cleeus 10y agonegative altitude ... so integer overflow it was ...
- verytrivial 10y agoSo, spend more money on the sensor simulator's chaos monkey. Having worked on ESA commissioned ground systems emulation, I know this is easier said than done. Indeed getting anything done is hard.
- junke 10y ago> Indeed getting anything done is hard. Why? Too much Red tape?
- verytrivial 10y agoBasically. Imagine how NASA splits up contracts for a large project, then put each of the companies in a different country.
- mirekrusin 10y agoif (altitude < 0) { return ERR_INVALID_ALTITUDE; }
- SeanDav 10y agoOf course it is easy to be wise after the fact, but surely this is the sort of thing that should have been exhaustively tested (What happens if one sensor malfunctions and sends incorrect data)? Massive cudo's for being public about the issue. It is not easy to talk about mistakes, especially those that appear to be silly in hindsight. EDIT Interesting to see that I am getting downvoted multiple times on this. I don't really care about a few downvotes but would be interested to find out why.
- msravi 10y agoIt isn't too clear from the description, but isn't non-saturation of the inertial measurement responsible, rather than saturation? I'd expect that some computation did not saturate its output, and the result overflowed (and hence became negative), which in turn was fed into altitude measurement.
- blondie9x 10y agoTo prevent an issue if IMU is saturated, couldn't they just add an and clause to conditional to prevent parachute release? altitude > 0 and time since entry > x then release parachute? Entry counter triggered once temperature reaches certain threshold or another sensor triggers entry.
- thwd 10y agoyour user name suits you.
- hatsunearu 10y agoJust by inspection that looks like that will have a ton of unwanted side effects... How do you define as "entry"? What if the triggering of the "entry" timer fails?
- blondie9x 10y agoYou can have multiple sensors start the timer.
- krenoten 10y agoThis feels like the kind of bug I catch early in implementation using quickcheck x_x Is property based testing used in the development of these kinds of systems?
- sitkack 10y agoWhich would totally work if you have a super high resolution full system simulation. Hardware, software, mechanical and electrical interconnections all need to be simulated for issues like this to be found automatically.
- UhUhUhUh 10y agoI don't understand why/how a feed of IMU data (gyro + probably accelerometers) could over-ride a Doppler altimeter that "functioned properly"... Any aircraft that would use inertial data to determine altitude would be in very deep trouble on Earth as well. I don't get it.
- SNSE 10y agoWhy is rotation data being used to calculate altitude? And why hasn't anybody asked that question yet? It's the first question I thought to myself. It's a shame such an expensive piece of equipment and years of work and waiting around were obliterated by a bug that should have been caught.