4 ms·

Note that this robustness seems to refer to just the physical build, not that the code had a glitch yet somehow saved the day. > Flight Six ended with Ingenuit

by lucb1e 5y ago

Note that this robustness seems to refer to just the physical build, not that the code had a glitch yet somehow saved the day. > Flight Six ended with Ingenuity safely on the ground because a number of subsystems – the rotor system, the actuators, and the power system – responded to increased demands to keep the helicopter flying. In a very real sense, Ingenuity muscled through the situation, and while the flight uncovered a timing vulnerability that will now have to be addressed, it also confirmed the robustness of the system in multiple ways. The code being robust and self-correcting, rather than the increased demands coincidentally being within tolerance levels, would have been more interesting or laudable.

ethbr0 5y ago
Afaict, JPL and NASA (and subcontractors) have a very strong historical knowledge base of building maximally-robust systems with limited redundancy hardware (in the sense of not having an unlimited mass budget for more hardware). But then again, they've been working with semi-autonomous systems about as long as anybody. https://newatlas.com/apollo-11-guidance-computer/59766/ https://newatlas.com/apollo-11-guidance-computer/59766/
foobarbecue 5y ago
The fact that the code disables optical nav at low altitudes is also one of the reasons it landed safely.
- snewman 5y ago
  Yes, but in this case that was just good luck. As the article mentions, the low-altitude behavior is because of concerns regarding dust. It was not intended to provide robustness in the face of glitches in the optical navigation system.
  - shkkmo 5y ago
    Dust concerns were one reason, but avoiding glitches and ensuring the smoothness of position data is another. > Then, once the vehicle estimates that the legs are within 1 meter of the ground, the algorithms stop using the navigation camera and altimeter for estimation, relying on the IMU in the same way as on takeoff. As with takeoff, this avoids dust obscuration, but it also serves another purpose -- by relying only on the IMU, we expect to have a very smooth and continuous estimate of our vertical velocity, which is important in order to avoid detecting touchdown prematurely. https://mars.nasa.gov/technology/helicopter/status/298/what-were-learning-about-ingenuitys-flight-control-and-aerodynamic-performance/ https://mars.nasa.gov/technology/helicopter/status/298/what-...
  - qwertox 5y ago
    I wonder if, had Ingenuity disabled the visual system upon noticing timestamp conflicts, the use of the IMU alone would have made the entire situation a more safe one.
vzaliva 5y ago
> Note that this robustness seems to refer to just the physical build, not that the code had a glitch yet somehow saved the day. I understnad the missing video frame is not just physical build issue. This is also a video of video pipeline and speed estimation implementation.
solipsism 5y ago
would have been more interesting or laudable. I think it's interesting and laudable that the hardware coped. But yeah, on the face of it this sounds very much like brittle software. For 46 seconds the craft's performance suffered from a single frame being dropped. It sounds like either the calculations were smeared over way too much time, or (more likely) the code made too many assumptions that relied on every piece of the pipeline properly behaving, not only in the moment but also over the entire history of the flight. I'm speaking from extreme ignorance obviously. But this reminds me of a million code reviews I've done where I've asked developers to make fewer assumptions about the state of the system. Often the response is something like "but how could that ever happen?" And my response is always "i have no idea, but shit happens." I would love to see a postmortem that discussed the specifics of what went wrong in the software, and whether they can attribute the lack of robustness to system design flaws.
- qwertox 5y ago
  > This glitch caused a single image to be lost, but more importantly, it resulted in all later navigation images being delivered with inaccurate timestamps. It was not a single lost frame which caused this issue. The issue was that the glitch then corrupted the timestamp accuracy of the frames that followed. I guess the dropped frame was just a symptom, maybe an initial memory corruption or something like that. But it was not the fact that the frame dropped and that this missing piece of information then had the negative effects.
qwertox 5y ago
> The code being robust and self-correcting, rather than the increased demands coincidentally being within tolerance levels, would have been more interesting or laudable. This is the good thing about the overall outcome of this incident: They now have a chance to learn from this. It would be interesting to know from NASA if, had Ingenuity crashed and lost all COMMS, they would still know what was the cause of the issue. I mean the wrong timestamps of the images, not what caused them. It would also be interesting to know what caused them, if it was a bug in the software, or a particle flipping a bit in the timestamping-code.

by lucb1e 5y ago

ethbr0 5y ago

Afaict, JPL and NASA (and subcontractors) have a very strong historical knowledge base of building maximally-robust systems with limited redundancy hardware (in the sense of not having an unlimited mass budget for more hardware). But then again, they've been working with semi-autonomous systems about as long as anybody. https://newatlas.com/apollo-11-guidance-computer/59766/ https://newatlas.com/apollo-11-guidance-computer/59766/

foobarbecue 5y ago

The fact that the code disables optical nav at low altitudes is also one of the reasons it landed safely.

snewman 5y ago
Yes, but in this case that was just good luck. As the article mentions, the low-altitude behavior is because of concerns regarding dust. It was not intended to provide robustness in the face of glitches in the optical navigation system.
- shkkmo 5y ago
  Dust concerns were one reason, but avoiding glitches and ensuring the smoothness of position data is another. > Then, once the vehicle estimates that the legs are within 1 meter of the ground, the algorithms stop using the navigation camera and altimeter for estimation, relying on the IMU in the same way as on takeoff. As with takeoff, this avoids dust obscuration, but it also serves another purpose -- by relying only on the IMU, we expect to have a very smooth and continuous estimate of our vertical velocity, which is important in order to avoid detecting touchdown prematurely. https://mars.nasa.gov/technology/helicopter/status/298/what-were-learning-about-ingenuitys-flight-control-and-aerodynamic-performance/ https://mars.nasa.gov/technology/helicopter/status/298/what-...
- qwertox 5y ago
  I wonder if, had Ingenuity disabled the visual system upon noticing timestamp conflicts, the use of the IMU alone would have made the entire situation a more safe one.

vzaliva 5y ago

> Note that this robustness seems to refer to just the physical build, not that the code had a glitch yet somehow saved the day. I understnad the missing video frame is not just physical build issue. This is also a video of video pipeline and speed estimation implementation.

solipsism 5y ago

would have been more interesting or laudable. I think it's interesting and laudable that the hardware coped. But yeah, on the face of it this sounds very much like brittle software. For 46 seconds the craft's performance suffered from a single frame being dropped. It sounds like either the calculations were smeared over way too much time, or (more likely) the code made too many assumptions that relied on every piece of the pipeline properly behaving, not only in the moment but also over the entire history of the flight. I'm speaking from extreme ignorance obviously. But this reminds me of a million code reviews I've done where I've asked developers to make fewer assumptions about the state of the system. Often the response is something like "but how could that ever happen?" And my response is always "i have no idea, but shit happens." I would love to see a postmortem that discussed the specifics of what went wrong in the software, and whether they can attribute the lack of robustness to system design flaws.

qwertox 5y ago
> This glitch caused a single image to be lost, but more importantly, it resulted in all later navigation images being delivered with inaccurate timestamps. It was not a single lost frame which caused this issue. The issue was that the glitch then corrupted the timestamp accuracy of the frames that followed. I guess the dropped frame was just a symptom, maybe an initial memory corruption or something like that. But it was not the fact that the frame dropped and that this missing piece of information then had the negative effects.

qwertox 5y ago

> The code being robust and self-correcting, rather than the increased demands coincidentally being within tolerance levels, would have been more interesting or laudable. This is the good thing about the overall outcome of this incident: They now have a chance to learn from this. It would be interesting to know from NASA if, had Ingenuity crashed and lost all COMMS, they would still know what was the cause of the issue. I mean the wrong timestamps of the images, not what caused them. It would also be interesting to know what caused them, if it was a bug in the software, or a particle flipping a bit in the timestamping-code.