9 ms·

While it may sound that the failure was caused by a rookie mistake- I'm sure the on-board systems were designed and developed by people who know what they are d

by fest 10y ago

While it may sound that the failure was caused by a rookie mistake- I'm sure the on-board systems were designed and developed by people who know what they are doing. It must be hard for all the people involved in Schiaparelli development. Sometimes I wonder whether majority of software errors are caused only by multiple layers of people having incorrect / unchallenged / unjustified (and mostly implicit) assumptions about something.

monk_e_boy 10y ago
Seems strange that a function looked at the data and thought the machine was underground didn't throw that data out. Wouldn't they have a model of expected data and anything that is wildly out be removed? Fall back to an estimation?
- fest 10y ago
  They probably have. A large problem in sensor fusion is: you don't always know which sensor you can trust. For example, in this case it probably would have helped to reject IMU and rely on doppler radar altimeter. But what if it's doppler radar that for some reason return an incorrect value that indicates that vehicle is a lot higher than IMU?
  - TheOtherHobbes 10y ago
    It's reasonable to assume that if you're still moving and apparently underground there's a problem with the sensors, and therefore it's best not to do anything rash - like starting the landing sequence. This does sound like a rookie mistake, and I'll bet there's a dev somewhere saying "Oh crap" a lot because of it. It's also a hardware design problem. NASA's human missions have a weight budget that includes multiple redundant systems. I have no idea if this probe had three of everything, but missions that do are more likely to be successful. More subtly, I'm curious if the sensor failed or maxed out. The kinds of forces that would max out the sensor might also max out the probe. So it's possible it was already doomed after they were encountered.
    - junke 10y ago
      > This does sound like a rookie mistake, and I'll bet there's a dev somewhere saying "Oh crap" a lot because of it. Howard Wolowitz.
- amq 10y ago
  Byzantine Generals
blutack 10y ago
Just so people are aware that ESA weren't necessarily directly responsible for writing the software, the RTPU (the computer handling the landing sequencing) seems to have been outsourced to a company called Terma (via Thales). I can't figure out if Terma wrote the landing software, but it seems likely from their press release [1] and they have the relevant specialties. 1: https://www.terma.com/space/exomars/ https://www.terma.com/space/exomars/
- Fjolle 10y ago
  Terma has said to danish media that they only developed the hardware, and had nothing to do with the software. The software might have been outsourced to another company.
  - gaius 10y ago
    When will people learn not to outsource anything important?!
    - NikolaeVarius 10y ago
      Seriously. Look at NASA, they definitely haven't outsourced anything at all. They use all in house hardware and software to launch satellites or send astronauts to the ISS.
      - Gmo 10y ago
        Not to be snarky, but NASA also had its fair share of failures. I mean ... it's rocket science after all.
      - kuschku 10y ago
        Eh, is this sarcasm? NASA has outsourced things since the original Apollo program, massively. NASA has never brought a single astronaut into space on a fully NASA-made rocket. Especially with parts like these, they're commonky outsourced.
wott 10y ago
> I'm sure the on-board systems were designed and developed by people who know what they are doing. I have been working in safety-related domains, where each failure could involve the death of several hundred people. It wasn't pretty: everything was sub-contracted to death, and there were a minority of people who really knew what they were doing, a majority who was so-so and didn't give a fuck, and still a fair part who was notoriously incompetent. So, the minority of competent and concerned people cannot always make up for the others, and when themselves fail, there is almost no one to make up for them. Oh, and I was working in a company that was supposed to produce better quality than others. And it did :-( So, for something like ESA where life is not even concerned, I can't imagine it is any better if stuff is outsourced, and since nowadays almost everything everywhere is outsourced, I assume ESA does it too. > Sometimes I wonder whether majority of software errors are caused only by multiple layers of people having incorrect / unchallenged / unjustified (and mostly implicit) assumptions about something. Yes and no. This causes a category of mistakes but on the other hand, it prevents another category. When someone knows the full system, he makes a whole lot of assumptions: it doesn't matter if this function cannot handle this or that case because it will never be called this way, it cannot happen, so I won't handle those cases and I won't even check them. And then, later, something changes in an upper function or system, the assumption is not valid any more, and boom. When you have no idea what the system is, you just stick to the function definition and have it handle all the weird cases. You don't make assumptions. You can still make mistakes, though, some of them caused by the lack of global understanding of the system because the specification writer had those assumptions in mind and did not write them down in the specification of this function because they were obvious to him or they were already written in another part of the specification.
- acqq 10y ago
  > And then, later, something changes in an upper function or system, the assumption is not valid any more, and boom. That's exactly how Ariane 5 crashed in 1996: https://en.wikipedia.org/wiki/Cluster_(spacecraft)#Launch_failure https://en.wikipedia.org/wiki/Cluster_(spacecraft)#Launch_fa... "Specifically, the Ariane 5's greater horizontal acceleration caused the computers in both the back-up and primary platforms to crash and emit diagnostic data misinterpreted by the autopilot as spurious position and velocity data. Pre-flight tests had never been performed on the inertial platform under simulated Ariane 5 flight conditions so the error was not discovered before launch. During the investigation, a simulated Ariane 5 flight was conducted on another inertial platform. It failed in exactly the same way as the actual flight units."
mannykannot 10y ago
The IMU saturated, yet its outputs continued to be used - i.e. a failure to handle an exception. In some sense, it was a rookie mistake. Formal systems verification has reached the point where this sort of thing should be caught, if you are prepared to take the fairly considerable effort that adopting these methods requires.

by fest 10y ago

monk_e_boy 10y ago

Seems strange that a function looked at the data and thought the machine was underground didn't throw that data out. Wouldn't they have a model of expected data and anything that is wildly out be removed? Fall back to an estimation?

fest 10y ago
They probably have. A large problem in sensor fusion is: you don't always know which sensor you can trust. For example, in this case it probably would have helped to reject IMU and rely on doppler radar altimeter. But what if it's doppler radar that for some reason return an incorrect value that indicates that vehicle is a lot higher than IMU?
- TheOtherHobbes 10y ago
  It's reasonable to assume that if you're still moving and apparently underground there's a problem with the sensors, and therefore it's best not to do anything rash - like starting the landing sequence. This does sound like a rookie mistake, and I'll bet there's a dev somewhere saying "Oh crap" a lot because of it. It's also a hardware design problem. NASA's human missions have a weight budget that includes multiple redundant systems. I have no idea if this probe had three of everything, but missions that do are more likely to be successful. More subtly, I'm curious if the sensor failed or maxed out. The kinds of forces that would max out the sensor might also max out the probe. So it's possible it was already doomed after they were encountered.
  - junke 10y ago
    > This does sound like a rookie mistake, and I'll bet there's a dev somewhere saying "Oh crap" a lot because of it. Howard Wolowitz.
amq 10y ago
Byzantine Generals

blutack 10y ago

Just so people are aware that ESA weren't necessarily directly responsible for writing the software, the RTPU (the computer handling the landing sequencing) seems to have been outsourced to a company called Terma (via Thales). I can't figure out if Terma wrote the landing software, but it seems likely from their press release [1] and they have the relevant specialties. 1: https://www.terma.com/space/exomars/ https://www.terma.com/space/exomars/

Fjolle 10y ago
Terma has said to danish media that they only developed the hardware, and had nothing to do with the software. The software might have been outsourced to another company.
- gaius 10y ago
  When will people learn not to outsource anything important?!
  - NikolaeVarius 10y ago
    Seriously. Look at NASA, they definitely haven't outsourced anything at all. They use all in house hardware and software to launch satellites or send astronauts to the ISS.
    - Gmo 10y ago
      Not to be snarky, but NASA also had its fair share of failures. I mean ... it's rocket science after all.
    - kuschku 10y ago
      Eh, is this sarcasm? NASA has outsourced things since the original Apollo program, massively. NASA has never brought a single astronaut into space on a fully NASA-made rocket. Especially with parts like these, they're commonky outsourced.

wott 10y ago

> I'm sure the on-board systems were designed and developed by people who know what they are doing. I have been working in safety-related domains, where each failure could involve the death of several hundred people. It wasn't pretty: everything was sub-contracted to death, and there were a minority of people who really knew what they were doing, a majority who was so-so and didn't give a fuck, and still a fair part who was notoriously incompetent. So, the minority of competent and concerned people cannot always make up for the others, and when themselves fail, there is almost no one to make up for them. Oh, and I was working in a company that was supposed to produce better quality than others. And it did :-( So, for something like ESA where life is not even concerned, I can't imagine it is any better if stuff is outsourced, and since nowadays almost everything everywhere is outsourced, I assume ESA does it too. > Sometimes I wonder whether majority of software errors are caused only by multiple layers of people having incorrect / unchallenged / unjustified (and mostly implicit) assumptions about something. Yes and no. This causes a category of mistakes but on the other hand, it prevents another category. When someone knows the full system, he makes a whole lot of assumptions: it doesn't matter if this function cannot handle this or that case because it will never be called this way, it cannot happen, so I won't handle those cases and I won't even check them. And then, later, something changes in an upper function or system, the assumption is not valid any more, and boom. When you have no idea what the system is, you just stick to the function definition and have it handle all the weird cases. You don't make assumptions. You can still make mistakes, though, some of them caused by the lack of global understanding of the system because the specification writer had those assumptions in mind and did not write them down in the specification of this function because they were obvious to him or they were already written in another part of the specification.

acqq 10y ago
> And then, later, something changes in an upper function or system, the assumption is not valid any more, and boom. That's exactly how Ariane 5 crashed in 1996: https://en.wikipedia.org/wiki/Cluster_(spacecraft)#Launch_failure https://en.wikipedia.org/wiki/Cluster_(spacecraft)#Launch_fa... "Specifically, the Ariane 5's greater horizontal acceleration caused the computers in both the back-up and primary platforms to crash and emit diagnostic data misinterpreted by the autopilot as spurious position and velocity data. Pre-flight tests had never been performed on the inertial platform under simulated Ariane 5 flight conditions so the error was not discovered before launch. During the investigation, a simulated Ariane 5 flight was conducted on another inertial platform. It failed in exactly the same way as the actual flight units."

mannykannot 10y ago

The IMU saturated, yet its outputs continued to be used - i.e. a failure to handle an exception. In some sense, it was a rookie mistake. Formal systems verification has reached the point where this sort of thing should be caught, if you are prepared to take the fairly considerable effort that adopting these methods requires.