21 ms·

Mars Helicopter Lands Safely After Serious In-Flight Anomaly

369 points by tmfi 5y ago

teekert 5y ago
Here's a cool relevant podcast (from before flight 6 but gives some nice insights into what it takes to fly that Helicopter on Mars): [0] Description: Tim Canham, Mars Helicopter Operations Lead at NASA’s JPL joins us again to share technical details you've never heard about the Ingenuity Linux Copter on Mars. And the challenges they had to work around to achieve their five successful flights. [0]: https://linuxunplugged.com/406 https://linuxunplugged.com/406
londons_explore 5y ago
It looks like the real issue is "we wrote all the camera code ourself rather than using battle tested libraries". The android camera API for example will tag every frame with the timestamps and exposure settings it was captured with. Unit and integration tests can simulate dropped frames and verify the algorithm still runs as it should. I would guess they tried to go for direct camera hardware access and ended up writing their own logic to schedule frames, and it contained a bug.
- Clewza313 5y ago
  And where, pray tell, would you find battle tested libraries for flying a drone on Mars? I'm also reasonably certain that the drone isn't running Android, and that not doing so is the correct design choice.
  - rurounijones 5y ago
    To be Devil's advocate. OP was talking about a VGA camera API only and what an example of a battle-tested camera library already does. I don't think anyone is seriously suggesting the thing should run android. But if you are writing something yourself then looking at what existing libraries already (and more importantly their test suites!) do is a good idea. Maybe they did and maybe they didn't in this case, hard to know with the data available.
  - fjdncncndb 5y ago
    I would imagine you could reuse code from a flying drone on Earth.
    - gscott 5y ago
      Next time they need to send up gps sats to orbit Mars...
      - serf 5y ago
        visual odometry is a fairly common feature for earth-creations (even toy drones) operating in GPS-denied environments. We have plenty of faux-mars environments to practice these techniques on. NASA even has their own. many of the toy indoor-use drones use such techniques for attitude/altitude-hold features when under a roof.
    - ghhhhhk8899jj 5y ago
      There's no reason to assume that code would contain fewer bugs.
  - serf 5y ago
    >And where, pray tell, would you find battle tested libraries for flying a drone on Mars? this isn't magic. Image processing isn't a wholly unknown field when under the influence of Martian gravity. It's a staggering achievement, and a testament to humankind's ability to wrangle technology, but -- to be frank here -- visual odometry is technology from 1960s era spy planes, not beyond-human voodoo. Sometimes errors are stupid ones, even if the stupidity is causing problems a zillion miles away on distant unknown frontiers being first-explored by human kind. tl;dr : This visual odometry problem isn't a space problem, it's a computer science problem -- one that has been encountered by numerous people even here on earth. The repercussions of such a simple non-space-problem might very well turn into space problems when the craft crashes, however -- and that's a damn shame given that this particular set of problems has been encountered-and-fixed by numerous earthlings over and over again. super tl;dr : as an only half-informed-participant in this discussion I bet this problem is NIH-syndrome-related, a problem NASA has encountered a lot in the past, a problem that's entirely managerial and very hard to eradicate.
    - numpad0 5y ago
      Re:tl;dr: Snapdragon camera peripheral is probably not super often used on Earth to fly a drone
  - perryizgr8 5y ago
    According to Wikipedia: The helicopter uses a Qualcomm Snapdragon 801 processor with a Linux operating system. Considering Qualcomm never releases their kernel sources, I am really worried that NASA might have been forced to use a modified Android system.
    - ludocode 5y ago
      I mean they weren't "forced" to use anything. They chose the chip. If they had different requirements they could have chosen a different chip. Most likely they just used the specific kernel version with closed binary blobs from Qualcomm for a typical embedded Linux, not the full Android system.
- sgt 5y ago
  Yeah let's just find a battle tested library for camera code on Mars. And I'm sure you expect that to be available on npm as well.
  - 0xfaded 5y ago
    Id be happy to find one on earth. Go look at some driver code, it's not uncommon for timestamps to increment by a precomputed expected frame duration.
- mjfisher 5y ago
  Silly spacecraft engineers. If only they had the foresight to run "npm install -g ingenuity-camera-api"
- JosephRedfern 5y ago
  It does sound like something you’d expect unit and integration tests to catch, but I am not sure that necessitates the use of off the shelf libraries? In the context of a space helicopter, I’d guess that deep understanding and ownership of each software component is super important when you’re likely to have to support a debug it from millions of miles away. At this scale, and in this context, doesn’t “rolling your own” critical components, in some cases, make sense?
- Tarq0n 5y ago
  Scientific cameras are all like that. You get a half-baked SDK if you're lucky, then you have to implement and integrate everything yourself.
- pragma93 5y ago
  I can't say for the camera code, but it looks like a lot of the project is reliant on open source software: - https://github.com/nasa/fprime https://github.com/nasa/fprime - https://github.com/readme/nasa-ingenuity-helicopter https://github.com/readme/nasa-ingenuity-helicopter
zadler 5y ago
I don’t think the article mentions whether or not they can update the code remotely. Can they patch it? Edit: appears so: https://astronomynow.com/2021/04/11/nasa-delays-mars-helicopter-flight-to-troubleshoot-test-glitch/ https://astronomynow.com/2021/04/11/nasa-delays-mars-helicop...
- ArnoVW 5y ago
  OTA updates of the software are a hard requirement if your equipment is at 30 light minutes. Actually, I remember reading somewhere that more and more they use FPGA chips, allowing to even "change out the CPU" if needed. http://www.esa.int/Enabling_Support/Space_Engineering_Technology/Microelectronics/The_use_of_reprogrammable_FPGAs_in_space http://www.esa.int/Enabling_Support/Space_Engineering_Techno...
  - j4mie 5y ago
    Is it still called OTA if there isn't any air?
    - wereHamster 5y ago
      There is air for the fist and last few dozens of kilometers (at varying density)
    - samsari 5y ago
      Over the Aether?
      - hvidgaard 5y ago
        That isn't there either.
        jk7tarYZAQNpTQa 5y ago
        "The modern concept of the vacuum of space, confirmed every day by experiment, is a relativistic ether. But we do not call it this because it is taboo." https://en.wikipedia.org/wiki/Aether_theories#Quantum_vacuum https://en.wikipedia.org/wiki/Aether_theories#Quantum_vacuum
_Microft 5y ago
The article links to NASA but here it is anyways. Let me skim it and I might add some additional information here but go and check out the first picture there yourself - it is amazing. It could as well be an aerial photograph of a desert area on Earth by the looks of it. https://mars.nasa.gov/technology/helicopter/status/305/surviving-an-in-flight-anomaly-what-happened-on-ingenuitys-sixth-flight/ https://mars.nasa.gov/technology/helicopter/status/305/survi... OK, so from the NASA article, it seems that they use an inertial measurement unit [0] (think an accelerometer/rotational sensor similiar to what your smartphone uses) that reads out heading and acceleration 500 times per second and sums it up to get the the helicopter's position. This is "dead reckoning" [1]) which unfortunately suffers from accumulating errors. To work around these accumulating errors, they sync their prediction to camera data thirty times per second by predicting how features visible on the surface should have moved in this time. So, sensor fusion [2], in a sense. The images are delivered with timestamps (I would have been surprised if they were not, to be honest) but one of the images went missing and for some reason, its timestamp was not. (Variable not cleared? Are they maybe using parallel queues to match images to timestamps (would be very weird, imo but I am no helicopter scientist?)). Either way, the following timestamps were now attributed to wrong images. I really wonder how that happened and hope they expand on that later. As the system predicts how far a feature in an image should have moved between frames, things were now considerably off (example: from the IMU prediction, a feature should have moved 1 unit per second but with a missing frame and the previous timestamp, it now looks as if it moved 2 units in a second) and the state gets updated with this wrong information. Here is an interesting bit about safety margins from this article by the way: Despite encountering this anomaly, Ingenuity was able to maintain flight and land safely [...]. One reason it was able to do so is the considerable effort that has gone into ensuring that the helicopter’s flight control system has ample “stability margin”: We designed Ingenuity to tolerate significant errors without becoming unstable, including errors in timing. This built-in margin was not fully needed in Ingenuity’s previous flights, because the vehicle’s behavior was in-family with our expectations, but this margin came to the rescue in Flight Six. [0] https://en.wikipedia.org/wiki/Inertial_measurement_unit https://en.wikipedia.org/wiki/Inertial_measurement_unit [1] https://en.wikipedia.org/wiki/Dead_reckoning https://en.wikipedia.org/wiki/Dead_reckoning [2] https://en.wikipedia.org/wiki/Sensor_fusion https://en.wikipedia.org/wiki/Sensor_fusion
areoform 5y ago
Does anyone know if the image was lost due to routine reasons, or due to radiation causing a cascade failure?
- orbital-decay 5y ago
  Seems to be the main question with no clear answer yet. One of the purposes of Ingenuity was to demonstrate non-radhard electronics working on Mars, and a frame-dropping glitch could have happened anywhere from software to hardware.
- snewman 5y ago
  Whatever the issue, presumably it did not arise during testing. That seems to point toward something that would only happen on Mars... so, yeah, maybe radiation?
tgbugs 5y ago
This is the kind of failure mode that I imagine (now with hindsight) should be mitigated by (re)designing the system in such a way that the timestamp desync would always be detected because each part of the system should have its own internal model of the approximate times that it should be receiving from other systems and never blindly trust them. I imagine it is sort of like the organisms early in evolution that simply believed their sensory system when it had in fact been hijacked by something that wanted to eat them. There aren't many sensory systems with those issues these days because anything that doesn't have an internal model that it can use for comparison winds up being a snack or crashing on an alien planet.
- SV_BubbleTime 5y ago
  I struggle to think that JPL didn’t think of this already. That in the distance/time code they didn’t take timestamps of each successful frame vs just assuming it’s a 30hz camera so surely each frame will be 30hz! It seems to me, as someone who does similar embedded work, that if I would account for a missing frame failure, they surely would. I am a professional in embedded timing systems - I am guessing this isn’t the full story or it’s been paraphrased for normal people and something important was lost. EDIT: Better info below. The photo was lost but it’s timestamp was not. They apparently didn’t tightly couple the image and it’s metadata together (hashes... what are they for!?) so there was some sort of mess up there. Knew there was more to the story.
  - dmurray 5y ago
    I wonder if hashes are discouraged at JPL due to a historical focus on "correct, and provably correct" code. Hash collisions happen. Yes, you can make collisions arbitrarily unlikely, and there are so many situations where they come in useful, but I can imagine an engineering culture where they are just never an easy tool to reach for.
    - dnautics 5y ago
      You can axiomatize uniqueness of hashes in proof systems.
qwertox 5y ago
> Fortunately, JPL knows exactly what went wrong. > and while the flight uncovered a timing vulnerability that will now have to be addressed, it also confirmed the robustness of the system in multiple ways. This is so important. Imagine if it would have crashed and damaged communications, that would have been the end of it. This extra step in engineering effort really payed off and gave Ingenuity a second life, and NASA a really good opportunity to learn important lessons and do further debugging. Also, the article at mars.nasa.gov is a delight to read, with all the information one would like to know being in there. Not to mention that there is even a video for us to see.
- DrAwdeOccarim 5y ago
  I had the same response while reading it, too. Like, wow, this is telling me all the technical things I wanted to know about the situation. This is one reason I pay for IEEE, they do a great job relaying the facts in summary articles from the main source.
- dna_polymerase 5y ago
  For those on mobile: https://mars.nasa.gov/technology/helicopter/status/305/surviving-an-in-flight-anomaly-what-happened-on-ingenuitys-sixth-flight/ https://mars.nasa.gov/technology/helicopter/status/305/survi...
- lucb1e 5y ago
  Note that this robustness seems to refer to just the physical build, not that the code had a glitch yet somehow saved the day. > Flight Six ended with Ingenuity safely on the ground because a number of subsystems – the rotor system, the actuators, and the power system – responded to increased demands to keep the helicopter flying. In a very real sense, Ingenuity muscled through the situation, and while the flight uncovered a timing vulnerability that will now have to be addressed, it also confirmed the robustness of the system in multiple ways. The code being robust and self-correcting, rather than the increased demands coincidentally being within tolerance levels, would have been more interesting or laudable.
  - ethbr0 5y ago
FranOntanaya 5y ago
> 30hz I assume it was just a failure to record the frame, but it'd be quite something if it was a drop frame from 29.97 video.
- deleted 5y ago
  [deleted]
- jerf 5y ago
  Let's put a pin your prediction; it's definitely an interesting possibility! 54 seconds of 29.97 frames: 1618.38 frames 54 seconds of 30.00 frames: 1620.00 frames Toss in a bit of rounding errors and/or tolerances and it's definitely suspiciously close to where a 1 frame error appears between those two. I'd suggest that the evidence against it would be that while this may be the longest flight on Mars, it seems certain that they'd have taken longer flights on Earth than this. But still, the math nearly checks out.
  - elliekelly 5y ago
    The tech specs[1] give it a range of “up to 300m” but I wonder whether that range is based on the hardware or the hardware + software. [1]https://mars.nasa.gov/technology/helicopter/#Deployment https://mars.nasa.gov/technology/helicopter/#Deployment
  - LorenPechtel 5y ago
    Floating point--don't use it unless you truly, truly need it. Why is this not just frame number?
    - jerf 5y ago
      YouTube answer to why 29.97 exists: https://www.youtube.com/watch?v=3GJUM6pCpew https://www.youtube.com/watch?v=3GJUM6pCpew
irjustin 5y ago
Incredible! on so many levels. My favorite is we got details. Legitimate details that give us a pretty clear picture where it went wrong and how it can be corrected. No calling it "software glitch" and stopping there. Those kinds of articles frustrate me to no end. As a very small portion of the HN crowd, believe we are quite pleased.
- thanatos519 5y ago
  The most incredible thing for me about this glitch is that as far as I know it's the first serious error in the entire mission. The live coverage of the landing, and the combined video that followed, was utterly astounding and humbling. Dare mighty things, indeed! The communications from NASA's Mars teams has been incredibly thorough and transparent. That they share all of the raw images soon after they are received from Mars gives an incredible feeling of inclusion.
fnord77 5y ago
I'm surprised that this corner case - the loss of a single frame from the nav camera - wasn't tested.
- GuB-42 5y ago
  Maybe is was just "impossible". Hard real time systems are not supposed to lose frames, sometimes provably so. You have to make some assumptions. For example, that pure functions will always return the same result with the same arguments. In fact, in real life, it is not always true, especially in space where a cosmic ray can flip a bit. You may try do be defensive to account for an unreliable hardware, but you have to draw the line at some point. Yes, is should have been tested, but it doesn't surprise me that it wasn't even for a company with as good a track record as JPL.
  - deleted 5y ago
    [deleted]
  - fnord77 5y ago
    I'm sure this will be added to their set of test cases :)
    - GeorgeTirebiter 5y ago
      Corner cases are hard. Here's a little C program; try to figure out what it prints before you compile & run it. It computes a step size between two numbers, given a number of steps: #include <stdio.h> int f(unsigned n_steps, int from_val, int to_val){ int step_size = (from_val - to_val) / n_steps; return step_size; } int main(){ printf("f(3, 0, 30) = %d\n", f(3, 0, 30) ); printf("f(3, 30, 0) = %d\n", f(3, 30, 0) ); }
      - ordu 5y ago
        > Corner cases are hard. Yeah, they are, but your example is not good enough, it is like an example from a textbook on traps of C. Corner cases hard when you are trying to do something new, because if you did it before, you'd know the most of them. Or if someone else did it before and wrote a textbook. =)
mtreis86 5y ago
Summary of the issue from another page; "For the majority of time airborne, the downward-looking navcams takes 30 pictures a second of the Martian surface and immediately feeds them into the helicopter’s navigation system. Each time an image arrives, the navigation system’s algorithm performs a series of actions: First, it examines the timestamp that it receives together with the image in order to determine when the image was taken. Then, the algorithm makes a prediction about what the camera should have been seeing at that particular point in time, in terms of surface features that it can recognize from previous images taken moments before (typically due to color variations and protuberances like rocks and sand ripples). Finally, the algorithm looks at where those features actually appear in the image. The navigation algorithm uses the difference between the predicted and actual locations of these features to correct its estimates of position, velocity, and attitude. Approximately 54 seconds into the flight, a glitch occurred in the pipeline of images being delivered by the navigation camera. This glitch caused a single image to be lost, but more importantly, it resulted in all later navigation images being delivered with inaccurate timestamps. From this point on, each time the navigation algorithm performed a correction based on a navigation image, it was operating on the basis of incorrect information about when the image was taken. The resulting inconsistencies significantly degraded the information used to fly the helicopter, leading to estimates being constantly “corrected” to account for phantom errors. Large oscillations ensued."
- lucb1e 5y ago
  As for why this didn't crash the helicopter, the hardware managed to keep up with the executed corrections and oscillations: > Flight Six ended with Ingenuity safely on the ground because a number of subsystems – the rotor system, the actuators, and the power system – responded to increased demands to keep the helicopter flying. In a very real sense, Ingenuity muscled through the situation, and while the flight uncovered a timing vulnerability that will now have to be addressed, it also confirmed the robustness of the system in multiple ways.
geocrasher 5y ago
It's incredible that the thing works at all- but to add the resilience that they seem to have accidentally built in (switching from visual at landing) and it surviving such a wild flight envelope is just incredible. I bought a cheap drone on a common importing website and the thing likes to get lost even though it has GPS and is in the perfect clear. It flew into my truck, uncommanded, at full speed. On Earth. With GPS. So for this thing on another planet to do what it just did? Wow.
ashtonkem 5y ago
Looking forward to the first off planet NTSB report.
jerf 5y ago
Look at the quality of those images of the ground in that video. Those blades are whirring around at super-high rates to provide lift in the Martian atmosphere and taking 30 pictures per second, but in each exposure you can clearly see the shadow of the blades with very little blurring. And with ambient solar brightness quite a bit lower than Earth's "full sunlight", too. Those are quality cameras.
- spywaregorilla 5y ago
  Are these colors accurate to what we would see?
  - teraflop 5y ago
    The first image, from the Perseverance rover's MastCam-Z, is at least as much "true color" as any ordinary camera. (That is, the precise color might be slightly off due to differences between the camera's sensitivity and your monitor's spectrum, but it should be pretty close.) The video from Ingenuity itself is taken by a black-and-white camera and has no color information.
- LorenPechtel 5y ago
  When you're sending stuff to other planets the cost of what you're sending is usually a trivial part of the cost of the mission. So long as it's off the shelf and not heavier you send the best there is.
- jccooper 5y ago
  Omnivision OV7251: https://www.ovt.com/sensors/OV7251 https://www.ovt.com/sensors/OV7251 Looks to be ~$3 on Digikey: https://www.digikey.com/en/products/detail/omnivision-technologies-inc/OV07251-A35A-1F/7346051 https://www.digikey.com/en/products/detail/omnivision-techno...
- Robotbeat 5y ago
  Only a factor of 2.3 lower. Sunlight is extremely bright. Full sunlight on Mars is much brighter than a cloudy day on Earth and an order of magnitude brighter than, say, an office or bedroom on Earth
uwagar 5y ago
NASA needs some drama for new funding
kumarvvr 5y ago
Very interesting problem. I wonder, why not have a real time clock capture time stamps for different inputs for same frame, like one when written to memory, one when the trigger for the frame is done, etc and have a comparative analysis. Discard the process when there is a discrepancy, and re-start the captures from beginning, and go into a default hover state when such a fault is found.
kortex 5y ago
I feel this bug in my bones. I work a lot on WAMI/aerial imagery. I've dealt with this exact problem of timestamp-off-by-one-image. I have 9 cameras with different exposure times all triggered by the same hardware pulse. Fortunately my system is data-collect only so the worst outcome is some wrong filenames. I learned only way too late in the project that most GigeVision cameras emit timestamps that can be synched using NTP and/or PTP. The camera specs/manual said nothing about this. Only found it by accident looking through the GeniCam XML API. Moral of the story: ensure your cameras emit timestamps on device. Don't rely on system time.
- rurban 5y ago
  Unless you need to sync times. Such as with connected low power devices which need to wake up by themselves. (Like Mars devices running for more than a few years). You cannot wake them up via a ping. Device time is then unreliable, you need to sync with network time, and then distribute time changes to your measurements. But properly. Google/Amazon do have a similar problem with eventually consistent databases, but low power (NB-IoT) meshes are harder, as synced wakeup's are critical. Don't rely on device time.
beefman 5y ago
Better source: https://mars.nasa.gov/technology/helicopter/status/305/surviving-an-in-flight-anomaly-what-happened-on-ingenuitys-sixth-flight/ https://mars.nasa.gov/technology/helicopter/status/305/survi...
Baeocystin 5y ago
I wonder what the shutter speed for the nav camera used in that GIF. They said the frames themselves are captured at 30Hz, but for the shadow of the rotors to be frozen without blurring, the shutter must be much shorter. Does anyone here know?
GeorgeTirebiter 5y ago
Because of using multiple, independent sensors (camera in the air, IMU near the ground) a disaster was averted. This is an important lesson: do not rely totally on one sensing modality. ( I wish Elon would learn this lesson: https://jalopnik.com/teslas-removing-radar-for-semi-automated-driving-on-mod-1846976679 https://jalopnik.com/teslas-removing-radar-for-semi-automate... )
sam_bristow 5y ago
If you enjoy this route of analysis you might find NASA's public 'lessons leaned' database interesting. https://llis.nasa.gov/ https://llis.nasa.gov/

369 points by tmfi 5y ago

Here's a cool relevant podcast (from before flight 6 but gives some nice insights into what it takes to fly that Helicopter on Mars): [0] Description: Tim Canham, Mars Helicopter Operations Lead at NASA’s JPL joins us again to share technical details you've never heard about the Ingenuity Linux Copter on Mars. And the challenges they had to work around to achieve their five successful flights. [0]: https://linuxunplugged.com/406 https://linuxunplugged.com/406

londons_explore 5y ago

It looks like the real issue is "we wrote all the camera code ourself rather than using battle tested libraries". The android camera API for example will tag every frame with the timestamps and exposure settings it was captured with. Unit and integration tests can simulate dropped frames and verify the algorithm still runs as it should. I would guess they tried to go for direct camera hardware access and ended up writing their own logic to schedule frames, and it contained a bug.

Clewza313 5y ago
And where, pray tell, would you find battle tested libraries for flying a drone on Mars? I'm also reasonably certain that the drone isn't running Android, and that not doing so is the correct design choice.
- rurounijones 5y ago
  To be Devil's advocate. OP was talking about a VGA camera API only and what an example of a battle-tested camera library already does. I don't think anyone is seriously suggesting the thing should run android. But if you are writing something yourself then looking at what existing libraries already (and more importantly their test suites!) do is a good idea. Maybe they did and maybe they didn't in this case, hard to know with the data available.
- fjdncncndb 5y ago
  I would imagine you could reuse code from a flying drone on Earth.
  - gscott 5y ago
    Next time they need to send up gps sats to orbit Mars...
    - serf 5y ago
      visual odometry is a fairly common feature for earth-creations (even toy drones) operating in GPS-denied environments. We have plenty of faux-mars environments to practice these techniques on. NASA even has their own. many of the toy indoor-use drones use such techniques for attitude/altitude-hold features when under a roof.
  - ghhhhhk8899jj 5y ago
    There's no reason to assume that code would contain fewer bugs.
- serf 5y ago
  >And where, pray tell, would you find battle tested libraries for flying a drone on Mars? this isn't magic. Image processing isn't a wholly unknown field when under the influence of Martian gravity. It's a staggering achievement, and a testament to humankind's ability to wrangle technology, but -- to be frank here -- visual odometry is technology from 1960s era spy planes, not beyond-human voodoo. Sometimes errors are stupid ones, even if the stupidity is causing problems a zillion miles away on distant unknown frontiers being first-explored by human kind. tl;dr : This visual odometry problem isn't a space problem, it's a computer science problem -- one that has been encountered by numerous people even here on earth. The repercussions of such a simple non-space-problem might very well turn into space problems when the craft crashes, however -- and that's a damn shame given that this particular set of problems has been encountered-and-fixed by numerous earthlings over and over again. super tl;dr : as an only half-informed-participant in this discussion I bet this problem is NIH-syndrome-related, a problem NASA has encountered a lot in the past, a problem that's entirely managerial and very hard to eradicate.
  - numpad0 5y ago
    Re:tl;dr: Snapdragon camera peripheral is probably not super often used on Earth to fly a drone
- perryizgr8 5y ago
  According to Wikipedia: The helicopter uses a Qualcomm Snapdragon 801 processor with a Linux operating system. Considering Qualcomm never releases their kernel sources, I am really worried that NASA might have been forced to use a modified Android system.
  - ludocode 5y ago
    I mean they weren't "forced" to use anything. They chose the chip. If they had different requirements they could have chosen a different chip. Most likely they just used the specific kernel version with closed binary blobs from Qualcomm for a typical embedded Linux, not the full Android system.
sgt 5y ago
Yeah let's just find a battle tested library for camera code on Mars. And I'm sure you expect that to be available on npm as well.
- 0xfaded 5y ago
  Id be happy to find one on earth. Go look at some driver code, it's not uncommon for timestamps to increment by a precomputed expected frame duration.
mjfisher 5y ago
Silly spacecraft engineers. If only they had the foresight to run "npm install -g ingenuity-camera-api"
JosephRedfern 5y ago
It does sound like something you’d expect unit and integration tests to catch, but I am not sure that necessitates the use of off the shelf libraries? In the context of a space helicopter, I’d guess that deep understanding and ownership of each software component is super important when you’re likely to have to support a debug it from millions of miles away. At this scale, and in this context, doesn’t “rolling your own” critical components, in some cases, make sense?
Tarq0n 5y ago
Scientific cameras are all like that. You get a half-baked SDK if you're lucky, then you have to implement and integrate everything yourself.
pragma93 5y ago
I can't say for the camera code, but it looks like a lot of the project is reliant on open source software: - https://github.com/nasa/fprime https://github.com/nasa/fprime - https://github.com/readme/nasa-ingenuity-helicopter https://github.com/readme/nasa-ingenuity-helicopter

zadler 5y ago

I don’t think the article mentions whether or not they can update the code remotely. Can they patch it? Edit: appears so: https://astronomynow.com/2021/04/11/nasa-delays-mars-helicopter-flight-to-troubleshoot-test-glitch/ https://astronomynow.com/2021/04/11/nasa-delays-mars-helicop...

ArnoVW 5y ago
OTA updates of the software are a hard requirement if your equipment is at 30 light minutes. Actually, I remember reading somewhere that more and more they use FPGA chips, allowing to even "change out the CPU" if needed. http://www.esa.int/Enabling_Support/Space_Engineering_Technology/Microelectronics/The_use_of_reprogrammable_FPGAs_in_space http://www.esa.int/Enabling_Support/Space_Engineering_Techno...
- j4mie 5y ago
  Is it still called OTA if there isn't any air?
  - wereHamster 5y ago
    There is air for the fist and last few dozens of kilometers (at varying density)
  - samsari 5y ago
    Over the Aether?
    - hvidgaard 5y ago
      That isn't there either.
      - jk7tarYZAQNpTQa 5y ago
        "The modern concept of the vacuum of space, confirmed every day by experiment, is a relativistic ether. But we do not call it this because it is taboo." https://en.wikipedia.org/wiki/Aether_theories#Quantum_vacuum https://en.wikipedia.org/wiki/Aether_theories#Quantum_vacuum

_Microft 5y ago

The article links to NASA but here it is anyways. Let me skim it and I might add some additional information here but go and check out the first picture there yourself - it is amazing. It could as well be an aerial photograph of a desert area on Earth by the looks of it. https://mars.nasa.gov/technology/helicopter/status/305/surviving-an-in-flight-anomaly-what-happened-on-ingenuitys-sixth-flight/ https://mars.nasa.gov/technology/helicopter/status/305/survi... OK, so from the NASA article, it seems that they use an inertial measurement unit [0] (think an accelerometer/rotational sensor similiar to what your smartphone uses) that reads out heading and acceleration 500 times per second and sums it up to get the the helicopter's position. This is "dead reckoning" [1]) which unfortunately suffers from accumulating errors. To work around these accumulating errors, they sync their prediction to camera data thirty times per second by predicting how features visible on the surface should have moved in this time. So, sensor fusion [2], in a sense. The images are delivered with timestamps (I would have been surprised if they were not, to be honest) but one of the images went missing and for some reason, its timestamp was not. (Variable not cleared? Are they maybe using parallel queues to match images to timestamps (would be very weird, imo but I am no helicopter scientist?)). Either way, the following timestamps were now attributed to wrong images. I really wonder how that happened and hope they expand on that later. As the system predicts how far a feature in an image should have moved between frames, things were now considerably off (example: from the IMU prediction, a feature should have moved 1 unit per second but with a missing frame and the previous timestamp, it now looks as if it moved 2 units in a second) and the state gets updated with this wrong information. Here is an interesting bit about safety margins from this article by the way: Despite encountering this anomaly, Ingenuity was able to maintain flight and land safely [...]. One reason it was able to do so is the considerable effort that has gone into ensuring that the helicopter’s flight control system has ample “stability margin”: We designed Ingenuity to tolerate significant errors without becoming unstable, including errors in timing. This built-in margin was not fully needed in Ingenuity’s previous flights, because the vehicle’s behavior was in-family with our expectations, but this margin came to the rescue in Flight Six. [0] https://en.wikipedia.org/wiki/Inertial_measurement_unit https://en.wikipedia.org/wiki/Inertial_measurement_unit [1] https://en.wikipedia.org/wiki/Dead_reckoning https://en.wikipedia.org/wiki/Dead_reckoning [2] https://en.wikipedia.org/wiki/Sensor_fusion https://en.wikipedia.org/wiki/Sensor_fusion

areoform 5y ago

Does anyone know if the image was lost due to routine reasons, or due to radiation causing a cascade failure?

orbital-decay 5y ago
Seems to be the main question with no clear answer yet. One of the purposes of Ingenuity was to demonstrate non-radhard electronics working on Mars, and a frame-dropping glitch could have happened anywhere from software to hardware.
snewman 5y ago
Whatever the issue, presumably it did not arise during testing. That seems to point toward something that would only happen on Mars... so, yeah, maybe radiation?

tgbugs 5y ago

This is the kind of failure mode that I imagine (now with hindsight) should be mitigated by (re)designing the system in such a way that the timestamp desync would always be detected because each part of the system should have its own internal model of the approximate times that it should be receiving from other systems and never blindly trust them. I imagine it is sort of like the organisms early in evolution that simply believed their sensory system when it had in fact been hijacked by something that wanted to eat them. There aren't many sensory systems with those issues these days because anything that doesn't have an internal model that it can use for comparison winds up being a snack or crashing on an alien planet.

SV_BubbleTime 5y ago
I struggle to think that JPL didn’t think of this already. That in the distance/time code they didn’t take timestamps of each successful frame vs just assuming it’s a 30hz camera so surely each frame will be 30hz! It seems to me, as someone who does similar embedded work, that if I would account for a missing frame failure, they surely would. I am a professional in embedded timing systems - I am guessing this isn’t the full story or it’s been paraphrased for normal people and something important was lost. EDIT: Better info below. The photo was lost but it’s timestamp was not. They apparently didn’t tightly couple the image and it’s metadata together (hashes... what are they for!?) so there was some sort of mess up there. Knew there was more to the story.
- dmurray 5y ago
  I wonder if hashes are discouraged at JPL due to a historical focus on "correct, and provably correct" code. Hash collisions happen. Yes, you can make collisions arbitrarily unlikely, and there are so many situations where they come in useful, but I can imagine an engineering culture where they are just never an easy tool to reach for.
  - dnautics 5y ago
    You can axiomatize uniqueness of hashes in proof systems.

qwertox 5y ago

> Fortunately, JPL knows exactly what went wrong. > and while the flight uncovered a timing vulnerability that will now have to be addressed, it also confirmed the robustness of the system in multiple ways. This is so important. Imagine if it would have crashed and damaged communications, that would have been the end of it. This extra step in engineering effort really payed off and gave Ingenuity a second life, and NASA a really good opportunity to learn important lessons and do further debugging. Also, the article at mars.nasa.gov is a delight to read, with all the information one would like to know being in there. Not to mention that there is even a video for us to see.

DrAwdeOccarim 5y ago
I had the same response while reading it, too. Like, wow, this is telling me all the technical things I wanted to know about the situation. This is one reason I pay for IEEE, they do a great job relaying the facts in summary articles from the main source.
dna_polymerase 5y ago
For those on mobile: https://mars.nasa.gov/technology/helicopter/status/305/surviving-an-in-flight-anomaly-what-happened-on-ingenuitys-sixth-flight/ https://mars.nasa.gov/technology/helicopter/status/305/survi...
lucb1e 5y ago
Note that this robustness seems to refer to just the physical build, not that the code had a glitch yet somehow saved the day. > Flight Six ended with Ingenuity safely on the ground because a number of subsystems – the rotor system, the actuators, and the power system – responded to increased demands to keep the helicopter flying. In a very real sense, Ingenuity muscled through the situation, and while the flight uncovered a timing vulnerability that will now have to be addressed, it also confirmed the robustness of the system in multiple ways. The code being robust and self-correcting, rather than the increased demands coincidentally being within tolerance levels, would have been more interesting or laudable.
- ethbr0 5y ago

FranOntanaya 5y ago

> 30hz I assume it was just a failure to record the frame, but it'd be quite something if it was a drop frame from 29.97 video.

deleted 5y ago
[deleted]
jerf 5y ago
Let's put a pin your prediction; it's definitely an interesting possibility! 54 seconds of 29.97 frames: 1618.38 frames 54 seconds of 30.00 frames: 1620.00 frames Toss in a bit of rounding errors and/or tolerances and it's definitely suspiciously close to where a 1 frame error appears between those two. I'd suggest that the evidence against it would be that while this may be the longest flight on Mars, it seems certain that they'd have taken longer flights on Earth than this. But still, the math nearly checks out.
- elliekelly 5y ago
  The tech specs[1] give it a range of “up to 300m” but I wonder whether that range is based on the hardware or the hardware + software. [1]https://mars.nasa.gov/technology/helicopter/#Deployment https://mars.nasa.gov/technology/helicopter/#Deployment
- LorenPechtel 5y ago
  Floating point--don't use it unless you truly, truly need it. Why is this not just frame number?
  - jerf 5y ago
    YouTube answer to why 29.97 exists: https://www.youtube.com/watch?v=3GJUM6pCpew https://www.youtube.com/watch?v=3GJUM6pCpew

irjustin 5y ago

Incredible! on so many levels. My favorite is we got details. Legitimate details that give us a pretty clear picture where it went wrong and how it can be corrected. No calling it "software glitch" and stopping there. Those kinds of articles frustrate me to no end. As a very small portion of the HN crowd, believe we are quite pleased.

thanatos519 5y ago
The most incredible thing for me about this glitch is that as far as I know it's the first serious error in the entire mission. The live coverage of the landing, and the combined video that followed, was utterly astounding and humbling. Dare mighty things, indeed! The communications from NASA's Mars teams has been incredibly thorough and transparent. That they share all of the raw images soon after they are received from Mars gives an incredible feeling of inclusion.

fnord77 5y ago

I'm surprised that this corner case - the loss of a single frame from the nav camera - wasn't tested.

GuB-42 5y ago
Maybe is was just "impossible". Hard real time systems are not supposed to lose frames, sometimes provably so. You have to make some assumptions. For example, that pure functions will always return the same result with the same arguments. In fact, in real life, it is not always true, especially in space where a cosmic ray can flip a bit. You may try do be defensive to account for an unreliable hardware, but you have to draw the line at some point. Yes, is should have been tested, but it doesn't surprise me that it wasn't even for a company with as good a track record as JPL.
- deleted 5y ago
  [deleted]
- fnord77 5y ago
  I'm sure this will be added to their set of test cases :)
  - GeorgeTirebiter 5y ago
    Corner cases are hard. Here's a little C program; try to figure out what it prints before you compile & run it. It computes a step size between two numbers, given a number of steps: #include <stdio.h> int f(unsigned n_steps, int from_val, int to_val){ int step_size = (from_val - to_val) / n_steps; return step_size; } int main(){ printf("f(3, 0, 30) = %d\n", f(3, 0, 30) ); printf("f(3, 30, 0) = %d\n", f(3, 30, 0) ); }
    - ordu 5y ago
      > Corner cases are hard. Yeah, they are, but your example is not good enough, it is like an example from a textbook on traps of C. Corner cases hard when you are trying to do something new, because if you did it before, you'd know the most of them. Or if someone else did it before and wrote a textbook. =)

mtreis86 5y ago

Summary of the issue from another page; "For the majority of time airborne, the downward-looking navcams takes 30 pictures a second of the Martian surface and immediately feeds them into the helicopter’s navigation system. Each time an image arrives, the navigation system’s algorithm performs a series of actions: First, it examines the timestamp that it receives together with the image in order to determine when the image was taken. Then, the algorithm makes a prediction about what the camera should have been seeing at that particular point in time, in terms of surface features that it can recognize from previous images taken moments before (typically due to color variations and protuberances like rocks and sand ripples). Finally, the algorithm looks at where those features actually appear in the image. The navigation algorithm uses the difference between the predicted and actual locations of these features to correct its estimates of position, velocity, and attitude. Approximately 54 seconds into the flight, a glitch occurred in the pipeline of images being delivered by the navigation camera. This glitch caused a single image to be lost, but more importantly, it resulted in all later navigation images being delivered with inaccurate timestamps. From this point on, each time the navigation algorithm performed a correction based on a navigation image, it was operating on the basis of incorrect information about when the image was taken. The resulting inconsistencies significantly degraded the information used to fly the helicopter, leading to estimates being constantly “corrected” to account for phantom errors. Large oscillations ensued."

lucb1e 5y ago
As for why this didn't crash the helicopter, the hardware managed to keep up with the executed corrections and oscillations: > Flight Six ended with Ingenuity safely on the ground because a number of subsystems – the rotor system, the actuators, and the power system – responded to increased demands to keep the helicopter flying. In a very real sense, Ingenuity muscled through the situation, and while the flight uncovered a timing vulnerability that will now have to be addressed, it also confirmed the robustness of the system in multiple ways.

geocrasher 5y ago

It's incredible that the thing works at all- but to add the resilience that they seem to have accidentally built in (switching from visual at landing) and it surviving such a wild flight envelope is just incredible. I bought a cheap drone on a common importing website and the thing likes to get lost even though it has GPS and is in the perfect clear. It flew into my truck, uncommanded, at full speed. On Earth. With GPS. So for this thing on another planet to do what it just did? Wow.

ashtonkem 5y ago

Looking forward to the first off planet NTSB report.

jerf 5y ago

Look at the quality of those images of the ground in that video. Those blades are whirring around at super-high rates to provide lift in the Martian atmosphere and taking 30 pictures per second, but in each exposure you can clearly see the shadow of the blades with very little blurring. And with ambient solar brightness quite a bit lower than Earth's "full sunlight", too. Those are quality cameras.

spywaregorilla 5y ago
Are these colors accurate to what we would see?
- teraflop 5y ago
  The first image, from the Perseverance rover's MastCam-Z, is at least as much "true color" as any ordinary camera. (That is, the precise color might be slightly off due to differences between the camera's sensitivity and your monitor's spectrum, but it should be pretty close.) The video from Ingenuity itself is taken by a black-and-white camera and has no color information.
LorenPechtel 5y ago
When you're sending stuff to other planets the cost of what you're sending is usually a trivial part of the cost of the mission. So long as it's off the shelf and not heavier you send the best there is.
jccooper 5y ago
Omnivision OV7251: https://www.ovt.com/sensors/OV7251 https://www.ovt.com/sensors/OV7251 Looks to be ~$3 on Digikey: https://www.digikey.com/en/products/detail/omnivision-technologies-inc/OV07251-A35A-1F/7346051 https://www.digikey.com/en/products/detail/omnivision-techno...
Robotbeat 5y ago
Only a factor of 2.3 lower. Sunlight is extremely bright. Full sunlight on Mars is much brighter than a cloudy day on Earth and an order of magnitude brighter than, say, an office or bedroom on Earth

uwagar 5y ago

NASA needs some drama for new funding

kumarvvr 5y ago

Very interesting problem. I wonder, why not have a real time clock capture time stamps for different inputs for same frame, like one when written to memory, one when the trigger for the frame is done, etc and have a comparative analysis. Discard the process when there is a discrepancy, and re-start the captures from beginning, and go into a default hover state when such a fault is found.

kortex 5y ago

I feel this bug in my bones. I work a lot on WAMI/aerial imagery. I've dealt with this exact problem of timestamp-off-by-one-image. I have 9 cameras with different exposure times all triggered by the same hardware pulse. Fortunately my system is data-collect only so the worst outcome is some wrong filenames. I learned only way too late in the project that most GigeVision cameras emit timestamps that can be synched using NTP and/or PTP. The camera specs/manual said nothing about this. Only found it by accident looking through the GeniCam XML API. Moral of the story: ensure your cameras emit timestamps on device. Don't rely on system time.

rurban 5y ago
Unless you need to sync times. Such as with connected low power devices which need to wake up by themselves. (Like Mars devices running for more than a few years). You cannot wake them up via a ping. Device time is then unreliable, you need to sync with network time, and then distribute time changes to your measurements. But properly. Google/Amazon do have a similar problem with eventually consistent databases, but low power (NB-IoT) meshes are harder, as synced wakeup's are critical. Don't rely on device time.

beefman 5y ago

Better source: https://mars.nasa.gov/technology/helicopter/status/305/surviving-an-in-flight-anomaly-what-happened-on-ingenuitys-sixth-flight/ https://mars.nasa.gov/technology/helicopter/status/305/survi...

Baeocystin 5y ago

I wonder what the shutter speed for the nav camera used in that GIF. They said the frames themselves are captured at 30Hz, but for the shadow of the rotors to be frozen without blurring, the shutter must be much shorter. Does anyone here know?

GeorgeTirebiter 5y ago

Because of using multiple, independent sensors (camera in the air, IMU near the ground) a disaster was averted. This is an important lesson: do not rely totally on one sensing modality. ( I wish Elon would learn this lesson: https://jalopnik.com/teslas-removing-radar-for-semi-automated-driving-on-mod-1846976679 https://jalopnik.com/teslas-removing-radar-for-semi-automate... )

sam_bristow 5y ago

If you enjoy this route of analysis you might find NASA's public 'lessons leaned' database interesting. https://llis.nasa.gov/ https://llis.nasa.gov/