9 ms·

Mars helicopter employs advanced control techniques to survive in-flight anomaly

210 points by dougmany 5y ago

anticristi 5y ago
Time to set up a GPS constellation on Mars? Somehow forgot that this navigation aid -- that we take for granted on Earth -- is missing on this other planet.
- anvandare 5y ago
  Pretty sure SpaceX's first deliveries to Mars (orbit) will include just that. I think they could probably use Starlink satellites without much modification (most important one I can think of would be more panels to cope with the reduced solar irradiance)? The GPS constellation has 32 satellites. A GPS satellite weighs ~2000kg. Starship should get 100-150 tons to Mars - the math checks out! (Disclosure: I got all my rocket science from sending Kerbals to their doom.) So, MPS for Mars, LPS for Luna? Just imagine the amount and quality of rover footage we could get if each of them had (constant!) 1Gbps uplink to Earth...
  - TheAdamAndChe 5y ago
    It's funny/amazing to me how we lived without GPS for thousands of years, but it's become so integral to our society that it's one of the first things we're setting up there before visiting.
    - anvandare 5y ago
      > The first and most moral responsibility of Freeland will be establishing a decent brewery on the new planet. (Civilization: Beyond Earth - Hutama) Sure, our ancestors (who could make their way and thrive in the wild) would probably consider me (who gets lost the moment I make a wrong right turn and depends on the local supermarket) a complete idiot, especially since I couldn't recreate any of the technology my daily life so depends on... But I've got central heating and clean tap-water, so I'm happy with the exchange. The higher we climb the ladder of technological dependency, the more calamitous it becomes should we ever fall off. Best not to look down. (I think I'm supposed to bang rocks together to make fire, right?)
      - ghaff 5y ago
        >(I think I'm supposed to bang rocks together to make fire, right?) Won't work very well unless you have flint specifically :-) Even if you know the techniques, starting fire without any manufactured tools or manufactured tools to make the devices is really really hard.
        usepgp 5y ago
        Its possible to just spin a dry stick against another dry stick with your hands. takes < 5 minutes to get a fire going.
        ghaff 5y ago
        As someone who did this sort of things in Boy Scouts, this is not super-simple just using natural materials. Maybe you're superman but starting fires with just natural materials was actually very difficult for our ancestors for a very long time. And is difficult today.
        craftinator 5y ago
        I'd invite you to watch the show Alone, where many survival experts have had to give up in the first day or two because they opted not to bring a fire steel and don't want to freeze. You're right that it's possible, but how long it takes, and whether or not you can do it, are dependent on skill and environment. Wrong kind of wood? Screwed. Been raining for a day? Screwed. Sometimes this can be overcome, but you have to know what you're doing, and it can still take hours to generate enough heat to dry the wood until it's possible to friction combust it. Try it sometime, just bring a lighter with you.
        ghaff 5y ago
        A fire steel is probably the simplest low-tech way to start a fire. You still need some expertise like finding a mouse nest or other flammable material even if it's been reasonably dry. But friction methods are tough, even if you have a well-made spindle and fireboard--which of course require tools. You're not easily creating those without tools to make those tools. And, even then, it's not super easy especially in a non-optimum environment.
    - bombcar 5y ago
      GPS is also “easy” to setup if you’re already coming in from space. I wonder what the “useful minimum” for satellites would be if you wanted, say, relatively accurate positioning once every orbit or each day.
    - quesera 5y ago
      I get the humor, but if you think about it: Almost all of the technologies that we would need for survival and productivity on Mars are recently-invented.
    - closeparen 5y ago
      One of the main risks for early explorers was simply getting lost.
    - CamperBob2 5y ago
      The problem of determining position (specifically the longitude part; latitude is relatively easy) has been one of the most pressing technical problems in human history. See https://en.wikipedia.org/wiki/Longitude_rewards https://en.wikipedia.org/wiki/Longitude_rewards . No kingdoms offered huge prizes for the first electric light, or the first antibiotic, the first radio, or the first computer. But geolocation has always been considered a big deal, and it arguably wasn't well and truly solved until the advent of satellite navigation. It'd be surprising if it weren't being planned for the next generation of Mars orbiters.
  - the8472 5y ago
    > satellites without much modification (most important one I can think of would be more panels to cope with the reduced solar irradiance)? atomic clocks
    - gpm 5y ago
      Are those actually necessary? Not an expert, but the whole network is on communication (with perfect understanding of the satellites locations no less), shouldn't they be able to run an ntpd like protocol to stay in sync, and just sync ground clocks to "starling consensus time" instead of "real time".
      - cryptoz 5y ago
        There is a pretty decent description of the problem here: https://physicscentral.com/explore/writers/will.cfm https://physicscentral.com/explore/writers/will.cfm I'm also not an expert so I don't know if you can solve it with syncing. Maybe you can. But that doesn't sound like a "just" concept to me - any solution to this is going to be complicated.
    - creatornator 5y ago
      This got me interested so I brushed off my notes from grad school--GPS predecessors like the US Navy's Timation first used quartz oscillators, then graduated to atomic clocks in orbit. However, even now ground station atomic clocks issue periodic time corrections to satellites in orbit. Since we can get accurate time of flight to Mars, I imagine satellites without atomic clocks could be used, albeit with less stable pseudo-range estimates. This could be overcome by using more satellites, giving more favorable pseudo-range variances.
- aeternum 5y ago
  It might be enough for the rover to just drop a few radio beacons at various distances.
- idiotsecant 5y ago
  Yes, but wouldn't help with this particular problem. This was a roll/pitch issue, not fundamentally a position issue.
  - bonzini 5y ago
    It stemmed from incorrect estimation of the position and thus of the velocity, which the helicopter attempted to correct.
- LeifCarrotson 5y ago
  I think it would be an APS constellation, because Martian orbits like Areosynchronous get their prefix from the greek Ares. That would be really cool, I wonder if it would be faster and more precise because there would be negligible human-produced radio noise to interfere and the Martian ionosphere is much thinner, or if it would be worse because the thinner and weaker atmosphere and magnetic field don't protect the receivers from solar noise. I wonder if a rover could drop a few beacons for time-of-flight 2D triangulation, it's not like they're moving hundreds of miles away over the horizon.
  - pilsetnieks 5y ago
    But the G in the Global Positioning System doesn't stand for Geosynchronous...
    - Sharlin 5y ago
      Indeed satellite navigation sats aren’t even on geosynchronous orbits.
- JoeAltmaier 5y ago
  GPS doesn't have to be satellites. It can simply be beacons on the ground. Often used terrestrially to get a few more decimal points on a fix.
  - mfsch 5y ago
    Not sure what techniques exactly you’re referring to, but the commonly used techniques for GPS enhancement (e.g. RTK [1]) are not simple distance measurements between GPS receivers and base stations but rather both stations measuring the satellite signal and using the difference of the signal in both locations to improve position estimates. I imagine it’s possible to set up a ground-based network, but you would need a high density to cover large surfaces (you want to see at least four stations from every position). I also imagine that it would be difficult to get accurate vertical positions if the stations are all in the same horizontal plane. [1]: https://en.wikipedia.org/wiki/Real-time_kinematic_positioning https://en.wikipedia.org/wiki/Real-time_kinematic_positionin...
    - Gwypaas 5y ago
      With the fuzzy GPS signal you had DGPS correcting it by broadcasting the current change from a known point. [0] Before GPS LORAN [1] and it's versions was used relying on ground stations [0]: https://en.m.wikipedia.org/wiki/Differential_GPS https://en.m.wikipedia.org/wiki/Differential_GPS [1]: https://en.m.wikipedia.org/wiki/LORAN https://en.m.wikipedia.org/wiki/LORAN
  - squarefoot 5y ago
    They could use UWB positioning systems. Accuracy is still not the best, but power consumption is very low and range more than adequate for a small base camp like the landing area. Solar power alone should be enough to power each beacon during Mars' daytime. https://www.firaconsortium.org/discover/how-uwb-works https://www.firaconsortium.org/discover/how-uwb-works
dougmany 5y ago
After five successful flights, the Mars Helicopter had a minor incident during its sixth voyage that was fixed using advanced control systems.
iancarroll 5y ago
It’s cool to see how they compensated for the IMU inaccuracy. I bought an IMU off of Amazon once and tried to use it to measure the position of a steering wheel. This worked for one rotation, but as it mentions, it’s incredibly hard to compensate for the error margin — as you integrate more measurements, the error builds up irreconcilably to the point where it’s a useless instrument. I am sure the one on the Mars helicopter had more precision though :)
- dieortin 5y ago
  I might be wrong, but I think the data from the accelerometers would be enough to know the position of the steering wheel without any accumulating error. Of course, the gyro data can be incorporated to improve the system.
  - ganzuul 5y ago
    The accelerometer would be fine until you take a corner. Then things would get exciting.
njoubert 5y ago
I see no mention of "advanced control techniques"? Sounds like there is just a limit on roll/pitch angles and a limit on distance applied. Saying "advanced control techniques [for multirotors]" sets an expectation for something along the lines of Tedrake's Underactuated Control approaches: http://underactuated.mit.edu/ http://underactuated.mit.edu/
- idiotsecant 5y ago
  Sensor fusion is part of the control system. Sensor fusion that incorporates visual information is outside of the typical classical controls sphere and is likely considered part of an advanced controls segment of the devices firmware as opposed to the bog standard deterministic controls algorithm. You need not gatekeep here, this is a real term.
  - njoubert 5y ago
    The article title is still misleading. The sensor fusion was the root cause of the malfunction, it was not the advanced control technique that was deployed to survive the anomaly.
    - arbirk 5y ago
      Agree. It's interesting that they don't treat the optical and gyroscopic systems as two separate sensors with the possibility to disregard one if it disagrees too much. A quick restart of the visual system would have been the most optimal solution #2020-hindsight
      - afrodc_ 5y ago
        How would you know which one is wrong in this scenario?
        omeze 5y ago
        This reminds me of the adage “ Never go to sea with two chronometers; take one or three” [1] to avoid this exact conundrum [1] https://en.m.wikipedia.org/wiki/Triple_modular_redundancy#Chronometers https://en.m.wikipedia.org/wiki/Triple_modular_redundancy#Ch...
isatty 5y ago
Terrible article - rehashed old news with a new title and even though it’s on “control.com” has no mentions of what said advanced control techniques are. Atleast it has pictures.
kklisura 5y ago
Here's a research paper that might provide more information: "Vision-Based Navigation for the NASA Mars Helicopter" https://sci-hub.se/10.2514/6.2019-1411 https://sci-hub.se/10.2514/6.2019-1411 > This paper provides an overview of the Mars Helicopter navigation system, architecture, sensors, vision processing and state estimation algorithms.
- deleted 5y ago
  [deleted]
ealloc 5y ago
The article says the problem was a dropped frame from the camera, but that just further piques my curiosity: Presumably they use some kind of Kalman Filter, but those are easy to program to account for missing frames, or frames at non-discrete timepoints, perhaps even for screwy camera images if the programmer had a reasonable prior for the likelihood of it happening. Kalman Filters by design account for measurement error.
- jcims 5y ago
  It seems like the issue wasn’t that there was a dropped frame, it’s that the time slot for that frame got filled by the next frame, then every subsequent frame was off by one resulting in a persistent timestamp offset of the vision data from reality for the remainder of the flight. I didn’t read into it too much so I may not have all the details right, but I think this is the gist of it.
  - ealloc 5y ago
    That would be a straight-up, avoidable software/hardware bug: The incoming timestamp is incorrect, and garbage in is garbage out. That would make me curious how the timestamp error occurred: software, hardware? Camera or Navigation code? I assume they have very high standards, what was the process failure point?
    - cm2187 5y ago
      Or timezone bug! Always hard to test.
- tonyarkles 5y ago
  Thank you for your comment, because it triggered an interesting chain of thoughts about a semi-related problem I’m working on at work. Usually with a Kalman filter, you’re taking into account the spatial measurement error (gyro-measured roll rate error, accelerometer-measured acceleration error, etc) but I don’t think I’ve ever encountered a system that explicitly modelled sensor latency variation relative to timestamps. Based on the description of the problem they encountered here, I suspect what happened is that it lost a frame but didn’t adjust the “photo timestamps” appropriately; every frame that came along afterwards would have had an incorrect timestamp? Even if the Kalman filter was set up to handle “this photo was taken 20ms ago” when doing its forward integration, if they didn’t model “this photo was taken 50ms ago but is reporting that it was taken 20ms ago” then you’d pretty readily get the kinds of oscillation they were getting. Edit: yeah, just like the sibling comment said :)
jollybean 5y ago
Can anyone hint why they wouldn't use a gyro? When it's on land, they can make the gyro reliably point 'down'. Then at least during flight they know which way 'down' is. Would this be too fragile for Mars?
- yupper32 5y ago
  They have an inclinometer, so they know which way is down.
- tonyarkles 5y ago
  When the article refers to an IMU, they’re referring to a combination of sensors; most commonly on Earth that’d be a 3-axis gyro, a 3-axis accelerometer, and a 3-axis magnetometer (compass). The problem with just using that is accumulated error and drift. On super small aircraft like this one, we usually use MEMS parts instead of big spinning physical gyros, and they don’t have great long-term performance. MEMS Gyros measure angular rate, not absolute angle; to compute an actual angle, you’re taking the integral of the rate from t=0 to now. Any small errors in the measurements add up quickly to give you completely nonsensical results. For drones-on-Earth, we use a variation of the Kalman filter to combine short-term and long-term measurements. As an example, an accelerometer requires a double-integral to turn into position, so errors accumulate very quickly, but we can correct those errors using GPS. The accelerometer and its integrals give us really quick acceleration, velocity, and position updates (at, say 500hz), and then the GPS is used to correct the long-term position and velocity (at, say 5hz).
ncmncm 5y ago
The control system is clearly designed wrong. Navigation input should not be able to affect the closed-loop control system directly. It should affect only the calibration, incrementally. If that had been done, there would have been no flight instability, just disagreement between the IMU and navigation about how far they had flown. There are certainly people involved in the project who could have explained this to them. I hope they are learning fast.
- baybal2 5y ago
  > There are certainly people involved in the project who could have explained this to them. I hope they are learning fast. This is how nearly all modern high-end quadcopter drones fly - navigation using optical flow camera sensors short circuited directly into attitude/motor control loops. I guess there is not a small chance they entrusted helicopter autonomous operations programming to people with quadcopter background.
  - londons_explore 5y ago
    Put tape over the downward facing camera on almost any quadcopter and you'll soon find out why... It turns out sideways drift accumulates very quickly - and so quickly that unless you are a very practiced drone operator its very hard to compensate for by hand. GPS compensates somewhat, but obviously that isn't available on mars (or indoors).
  - ncmncm 5y ago
    It is one thing to feed averaged relative motion into the control system, entirely another to feed in absolute position. The flight instability in response to navigational position error is incontrovertible proof of a mistaken design. Fixing the off-by-one coding error just papers over the design error. Testing clearly failed to detect the mistake. I would not be at all surprised to learn that commercial quads share the design mistake. I was surprised to learn that NASA professionals copied it into a Mars probe. But, notably, not into the vehicle that delivered the lander.
HALtheWise 5y ago
I can't think of a polite way to say this, but as someone who professionally develops drone software, both of the software failures experienced by Ingenuity have been embarrassingly amateur at a technical level. The first failure, which delayed the initial spin test, was described as a "watchdog timeout", which for anyone not familiar with embedded development basically means the code crashed. We all write code that crashes, but I am having trouble thinking of an excuse to justify the fact that their code crashed before takeoff, on Mars, and they didn't see it coming. There is nothing about sitting on the ground on Mars that shouldn't have been tested repeatedly on earth, and testing in production is _really_ not the right way to do Aerospace development (although Boeing Starliner would beg to differ) Similarly, there are a huge number of things that can and will result in dropped frames when running Linux on a Qualcomm mobile chip, and having a software stack that infers frame timing purely from the sequence number is brittle, and would definitely not have passed code review and testing where I work (I actually checked, we do have a robust solution). If I had to guess, I suspect the root cause of the dropped frame wasn't actually anything exciting like a cosmic ray, but instead was some run-of-the-mill event that would have been caught by a couple hours of flight testing on Earth. Either way, it shouldn't have made it to Mars. I'm sure that there are a lot of great engineers working on the Ingenuity project that _don't_ write these sorts of bugs, and am glad that theae amateur fuckups (barely) haven't crashed the drone before it has been able to do some incredible technology demonstration work.
- fermuch 5y ago
  It might be related to the hostility of the environment. The chips aren't radiation proof, so it is expected to have some bit flops due to radiation.
  - x86_64Ubuntu 5y ago
    Why on earth wouldn't they use rad-proofed chips on a planet closer to the sun, with virtually no atmosphere compared to earth?
    - dieortin 5y ago
rsp1984 5y ago
Fusing video with IMU for navigation is called VIN (visual-inertial navigation) or VIO (visual-inertial odometry) and the field has made enormous progress over the last 10-15 years. It's the same technology that the iPhone uses for all its AR features. Dropped frames are one of the easiest things to handle. Yes, the visual feature tracking depends on frames of video but even cheap phone IMUs these days are good enough to dead-reckon for a second or two, especially when embedded into a sensor fusion framework, so the prediction errors resulting from a single lost frame should be very minimal and not enough to throw off the tracking. That's why I find it hard to believe that the VIN in use by the Mars Helicopter (part of a multi billion dollar program) wouldn't be able to deal with a dropped frame. It just doesn't add up. I suspect that the situation is much more complex than what the article suggests and that more things went wrong than just a dropped camera frame.
- ncmncm 5y ago
  It means that the people who know most about closed-loop vehicle control were not involved in the design of the copter. Such fragility is a really elementary design mistake any experienced engineer would not make. Certainly the vehicle that delivered the lander would not suffer from the same mistake. My interpretion is that the copter, as an inessential system component, was seen as an opportunity for junior people to get some end-to-end experience. I hope they are learning the right things.
  - ehsankia 5y ago
    The article makes it sound like after the missed frame, every subsequent frame had the wrong timestamp and was processed wrong, so it sounds like more of a bug. If the subsequent timestamps had the right timestamps, and the calculation was based on delta timestamps, then it would basically basically interpolate over the missing frame and then recover. The fact that every subsequent frame was assumed to be 33ms in the past was the issue here.
    - mikepurvis 5y ago
      And that being a system boundary thing, it’s easy to imagine it as an integration issue, where the controls side maybe assumed it would get a dummy frame or something.
jsrcout 5y ago
Article from the Mars Helicopter chief pilot: https://mars.nasa.gov/technology/helicopter/status/305/surviving-an-in-flight-anomaly-what-happened-on-ingenuitys-sixth-flight/ https://mars.nasa.gov/technology/helicopter/status/305/survi...
renewiltord 5y ago
NASA systems appear to have the property that they are both perfectly designed when HN commenters do not understand the code and amateurishly designed when they have errors. This bathtub style curve for perception of NASA design by HN commenters makes me question if the perception correlates with reality.
- kibwen 5y ago
  You're simply coming to the astute observation that any individual HN commenter who may very well be a genius WRT an extremely narrow area of expertise will be just as hopelessly clueless as the average person when it comes to any area outside of their expertise.
- stevenhuang 5y ago
  Perfectly designed when it's right and badly designed when it's wrong, you mean? Strange way to state a tautology to make yet another boring statement on ye olde HN commenter.
  - renewiltord 5y ago
    No, actually, I do not mean that. I mean exactly what I said.
heisenbit 5y ago
> the inertial measurement unit (IMU) and the navigational camera. The IMU measures acceleration in three dimensions, using data from several sensors to estimate altitude, velocity, and position. Even though this system samples at 500 Hz, the error would accumulate over time, causing the helicopter to become lost quickly. I understand the IMU is not an ideal input and integration over time leads to positional errors. But gyros are much better and the orientation of the drone in flight is paramount. What I wonder is what ‚advanced‘ control law allowed the drone to become unstable wrt. orientation when there was noisy positional input.
- jamiedamien 5y ago
  I suspect what happened is that the position and velocity estimates from the VIO system (cameras) were wrong (and perhaps wildly jumping) and since position and velocity errors are inputs to the attitude controller, the tilt oscillated as well. It's not that the IMU did a poor job estimating attitude (tilt), but rather that the attitude controller was asked to tilt in erratic directions to compensate for the erratic and incorrect position and velocity readings. It's quite hard to detect when position and velocities are "obviously incorrect", especially when they come from VIO, where the optimization result can jump around in non ideal conditions, so I'm not surprised there was not a more graceful anomaly detection.

210 points by dougmany 5y ago

Time to set up a GPS constellation on Mars? Somehow forgot that this navigation aid -- that we take for granted on Earth -- is missing on this other planet.

anvandare 5y ago
Pretty sure SpaceX's first deliveries to Mars (orbit) will include just that. I think they could probably use Starlink satellites without much modification (most important one I can think of would be more panels to cope with the reduced solar irradiance)? The GPS constellation has 32 satellites. A GPS satellite weighs ~2000kg. Starship should get 100-150 tons to Mars - the math checks out! (Disclosure: I got all my rocket science from sending Kerbals to their doom.) So, MPS for Mars, LPS for Luna? Just imagine the amount and quality of rover footage we could get if each of them had (constant!) 1Gbps uplink to Earth...
- TheAdamAndChe 5y ago
  It's funny/amazing to me how we lived without GPS for thousands of years, but it's become so integral to our society that it's one of the first things we're setting up there before visiting.
  - anvandare 5y ago
    > The first and most moral responsibility of Freeland will be establishing a decent brewery on the new planet. (Civilization: Beyond Earth - Hutama) Sure, our ancestors (who could make their way and thrive in the wild) would probably consider me (who gets lost the moment I make a wrong right turn and depends on the local supermarket) a complete idiot, especially since I couldn't recreate any of the technology my daily life so depends on... But I've got central heating and clean tap-water, so I'm happy with the exchange. The higher we climb the ladder of technological dependency, the more calamitous it becomes should we ever fall off. Best not to look down. (I think I'm supposed to bang rocks together to make fire, right?)
    - ghaff 5y ago
      >(I think I'm supposed to bang rocks together to make fire, right?) Won't work very well unless you have flint specifically :-) Even if you know the techniques, starting fire without any manufactured tools or manufactured tools to make the devices is really really hard.
      - usepgp 5y ago
        Its possible to just spin a dry stick against another dry stick with your hands. takes < 5 minutes to get a fire going.
        ghaff 5y ago
        As someone who did this sort of things in Boy Scouts, this is not super-simple just using natural materials. Maybe you're superman but starting fires with just natural materials was actually very difficult for our ancestors for a very long time. And is difficult today.
        craftinator 5y ago
        I'd invite you to watch the show Alone, where many survival experts have had to give up in the first day or two because they opted not to bring a fire steel and don't want to freeze. You're right that it's possible, but how long it takes, and whether or not you can do it, are dependent on skill and environment. Wrong kind of wood? Screwed. Been raining for a day? Screwed. Sometimes this can be overcome, but you have to know what you're doing, and it can still take hours to generate enough heat to dry the wood until it's possible to friction combust it. Try it sometime, just bring a lighter with you.
        ghaff 5y ago
        A fire steel is probably the simplest low-tech way to start a fire. You still need some expertise like finding a mouse nest or other flammable material even if it's been reasonably dry. But friction methods are tough, even if you have a well-made spindle and fireboard--which of course require tools. You're not easily creating those without tools to make those tools. And, even then, it's not super easy especially in a non-optimum environment.
  - bombcar 5y ago
    GPS is also “easy” to setup if you’re already coming in from space. I wonder what the “useful minimum” for satellites would be if you wanted, say, relatively accurate positioning once every orbit or each day.
  - quesera 5y ago
    I get the humor, but if you think about it: Almost all of the technologies that we would need for survival and productivity on Mars are recently-invented.
  - closeparen 5y ago
    One of the main risks for early explorers was simply getting lost.
  - CamperBob2 5y ago
    The problem of determining position (specifically the longitude part; latitude is relatively easy) has been one of the most pressing technical problems in human history. See https://en.wikipedia.org/wiki/Longitude_rewards https://en.wikipedia.org/wiki/Longitude_rewards . No kingdoms offered huge prizes for the first electric light, or the first antibiotic, the first radio, or the first computer. But geolocation has always been considered a big deal, and it arguably wasn't well and truly solved until the advent of satellite navigation. It'd be surprising if it weren't being planned for the next generation of Mars orbiters.
- the8472 5y ago
  > satellites without much modification (most important one I can think of would be more panels to cope with the reduced solar irradiance)? atomic clocks
  - gpm 5y ago
    Are those actually necessary? Not an expert, but the whole network is on communication (with perfect understanding of the satellites locations no less), shouldn't they be able to run an ntpd like protocol to stay in sync, and just sync ground clocks to "starling consensus time" instead of "real time".
    - cryptoz 5y ago
      There is a pretty decent description of the problem here: https://physicscentral.com/explore/writers/will.cfm https://physicscentral.com/explore/writers/will.cfm I'm also not an expert so I don't know if you can solve it with syncing. Maybe you can. But that doesn't sound like a "just" concept to me - any solution to this is going to be complicated.
  - creatornator 5y ago
    This got me interested so I brushed off my notes from grad school--GPS predecessors like the US Navy's Timation first used quartz oscillators, then graduated to atomic clocks in orbit. However, even now ground station atomic clocks issue periodic time corrections to satellites in orbit. Since we can get accurate time of flight to Mars, I imagine satellites without atomic clocks could be used, albeit with less stable pseudo-range estimates. This could be overcome by using more satellites, giving more favorable pseudo-range variances.
aeternum 5y ago
It might be enough for the rover to just drop a few radio beacons at various distances.
idiotsecant 5y ago
Yes, but wouldn't help with this particular problem. This was a roll/pitch issue, not fundamentally a position issue.
- bonzini 5y ago
  It stemmed from incorrect estimation of the position and thus of the velocity, which the helicopter attempted to correct.
LeifCarrotson 5y ago
I think it would be an APS constellation, because Martian orbits like Areosynchronous get their prefix from the greek Ares. That would be really cool, I wonder if it would be faster and more precise because there would be negligible human-produced radio noise to interfere and the Martian ionosphere is much thinner, or if it would be worse because the thinner and weaker atmosphere and magnetic field don't protect the receivers from solar noise. I wonder if a rover could drop a few beacons for time-of-flight 2D triangulation, it's not like they're moving hundreds of miles away over the horizon.
- pilsetnieks 5y ago
  But the G in the Global Positioning System doesn't stand for Geosynchronous...
  - Sharlin 5y ago
    Indeed satellite navigation sats aren’t even on geosynchronous orbits.
JoeAltmaier 5y ago
GPS doesn't have to be satellites. It can simply be beacons on the ground. Often used terrestrially to get a few more decimal points on a fix.
- mfsch 5y ago
  Not sure what techniques exactly you’re referring to, but the commonly used techniques for GPS enhancement (e.g. RTK [1]) are not simple distance measurements between GPS receivers and base stations but rather both stations measuring the satellite signal and using the difference of the signal in both locations to improve position estimates. I imagine it’s possible to set up a ground-based network, but you would need a high density to cover large surfaces (you want to see at least four stations from every position). I also imagine that it would be difficult to get accurate vertical positions if the stations are all in the same horizontal plane. [1]: https://en.wikipedia.org/wiki/Real-time_kinematic_positioning https://en.wikipedia.org/wiki/Real-time_kinematic_positionin...
  - Gwypaas 5y ago
    With the fuzzy GPS signal you had DGPS correcting it by broadcasting the current change from a known point. [0] Before GPS LORAN [1] and it's versions was used relying on ground stations [0]: https://en.m.wikipedia.org/wiki/Differential_GPS https://en.m.wikipedia.org/wiki/Differential_GPS [1]: https://en.m.wikipedia.org/wiki/LORAN https://en.m.wikipedia.org/wiki/LORAN
- squarefoot 5y ago
  They could use UWB positioning systems. Accuracy is still not the best, but power consumption is very low and range more than adequate for a small base camp like the landing area. Solar power alone should be enough to power each beacon during Mars' daytime. https://www.firaconsortium.org/discover/how-uwb-works https://www.firaconsortium.org/discover/how-uwb-works

dougmany 5y ago

After five successful flights, the Mars Helicopter had a minor incident during its sixth voyage that was fixed using advanced control systems.

iancarroll 5y ago

It’s cool to see how they compensated for the IMU inaccuracy. I bought an IMU off of Amazon once and tried to use it to measure the position of a steering wheel. This worked for one rotation, but as it mentions, it’s incredibly hard to compensate for the error margin — as you integrate more measurements, the error builds up irreconcilably to the point where it’s a useless instrument. I am sure the one on the Mars helicopter had more precision though :)

dieortin 5y ago
I might be wrong, but I think the data from the accelerometers would be enough to know the position of the steering wheel without any accumulating error. Of course, the gyro data can be incorporated to improve the system.
- ganzuul 5y ago
  The accelerometer would be fine until you take a corner. Then things would get exciting.

njoubert 5y ago

I see no mention of "advanced control techniques"? Sounds like there is just a limit on roll/pitch angles and a limit on distance applied. Saying "advanced control techniques [for multirotors]" sets an expectation for something along the lines of Tedrake's Underactuated Control approaches: http://underactuated.mit.edu/ http://underactuated.mit.edu/

idiotsecant 5y ago
Sensor fusion is part of the control system. Sensor fusion that incorporates visual information is outside of the typical classical controls sphere and is likely considered part of an advanced controls segment of the devices firmware as opposed to the bog standard deterministic controls algorithm. You need not gatekeep here, this is a real term.
- njoubert 5y ago
  The article title is still misleading. The sensor fusion was the root cause of the malfunction, it was not the advanced control technique that was deployed to survive the anomaly.
  - arbirk 5y ago
    Agree. It's interesting that they don't treat the optical and gyroscopic systems as two separate sensors with the possibility to disregard one if it disagrees too much. A quick restart of the visual system would have been the most optimal solution #2020-hindsight
    - afrodc_ 5y ago
      How would you know which one is wrong in this scenario?
      - omeze 5y ago
        This reminds me of the adage “ Never go to sea with two chronometers; take one or three” [1] to avoid this exact conundrum [1] https://en.m.wikipedia.org/wiki/Triple_modular_redundancy#Chronometers https://en.m.wikipedia.org/wiki/Triple_modular_redundancy#Ch...

isatty 5y ago

Terrible article - rehashed old news with a new title and even though it’s on “control.com” has no mentions of what said advanced control techniques are. Atleast it has pictures.

kklisura 5y ago

Here's a research paper that might provide more information: "Vision-Based Navigation for the NASA Mars Helicopter" https://sci-hub.se/10.2514/6.2019-1411 https://sci-hub.se/10.2514/6.2019-1411 > This paper provides an overview of the Mars Helicopter navigation system, architecture, sensors, vision processing and state estimation algorithms.

deleted 5y ago
[deleted]

ealloc 5y ago

The article says the problem was a dropped frame from the camera, but that just further piques my curiosity: Presumably they use some kind of Kalman Filter, but those are easy to program to account for missing frames, or frames at non-discrete timepoints, perhaps even for screwy camera images if the programmer had a reasonable prior for the likelihood of it happening. Kalman Filters by design account for measurement error.

jcims 5y ago
It seems like the issue wasn’t that there was a dropped frame, it’s that the time slot for that frame got filled by the next frame, then every subsequent frame was off by one resulting in a persistent timestamp offset of the vision data from reality for the remainder of the flight. I didn’t read into it too much so I may not have all the details right, but I think this is the gist of it.
- ealloc 5y ago
  That would be a straight-up, avoidable software/hardware bug: The incoming timestamp is incorrect, and garbage in is garbage out. That would make me curious how the timestamp error occurred: software, hardware? Camera or Navigation code? I assume they have very high standards, what was the process failure point?
  - cm2187 5y ago
    Or timezone bug! Always hard to test.
tonyarkles 5y ago
Thank you for your comment, because it triggered an interesting chain of thoughts about a semi-related problem I’m working on at work. Usually with a Kalman filter, you’re taking into account the spatial measurement error (gyro-measured roll rate error, accelerometer-measured acceleration error, etc) but I don’t think I’ve ever encountered a system that explicitly modelled sensor latency variation relative to timestamps. Based on the description of the problem they encountered here, I suspect what happened is that it lost a frame but didn’t adjust the “photo timestamps” appropriately; every frame that came along afterwards would have had an incorrect timestamp? Even if the Kalman filter was set up to handle “this photo was taken 20ms ago” when doing its forward integration, if they didn’t model “this photo was taken 50ms ago but is reporting that it was taken 20ms ago” then you’d pretty readily get the kinds of oscillation they were getting. Edit: yeah, just like the sibling comment said :)

jollybean 5y ago

Can anyone hint why they wouldn't use a gyro? When it's on land, they can make the gyro reliably point 'down'. Then at least during flight they know which way 'down' is. Would this be too fragile for Mars?

yupper32 5y ago
They have an inclinometer, so they know which way is down.
tonyarkles 5y ago
When the article refers to an IMU, they’re referring to a combination of sensors; most commonly on Earth that’d be a 3-axis gyro, a 3-axis accelerometer, and a 3-axis magnetometer (compass). The problem with just using that is accumulated error and drift. On super small aircraft like this one, we usually use MEMS parts instead of big spinning physical gyros, and they don’t have great long-term performance. MEMS Gyros measure angular rate, not absolute angle; to compute an actual angle, you’re taking the integral of the rate from t=0 to now. Any small errors in the measurements add up quickly to give you completely nonsensical results. For drones-on-Earth, we use a variation of the Kalman filter to combine short-term and long-term measurements. As an example, an accelerometer requires a double-integral to turn into position, so errors accumulate very quickly, but we can correct those errors using GPS. The accelerometer and its integrals give us really quick acceleration, velocity, and position updates (at, say 500hz), and then the GPS is used to correct the long-term position and velocity (at, say 5hz).

ncmncm 5y ago

The control system is clearly designed wrong. Navigation input should not be able to affect the closed-loop control system directly. It should affect only the calibration, incrementally. If that had been done, there would have been no flight instability, just disagreement between the IMU and navigation about how far they had flown. There are certainly people involved in the project who could have explained this to them. I hope they are learning fast.

baybal2 5y ago
> There are certainly people involved in the project who could have explained this to them. I hope they are learning fast. This is how nearly all modern high-end quadcopter drones fly - navigation using optical flow camera sensors short circuited directly into attitude/motor control loops. I guess there is not a small chance they entrusted helicopter autonomous operations programming to people with quadcopter background.
- londons_explore 5y ago
  Put tape over the downward facing camera on almost any quadcopter and you'll soon find out why... It turns out sideways drift accumulates very quickly - and so quickly that unless you are a very practiced drone operator its very hard to compensate for by hand. GPS compensates somewhat, but obviously that isn't available on mars (or indoors).
- ncmncm 5y ago
  It is one thing to feed averaged relative motion into the control system, entirely another to feed in absolute position. The flight instability in response to navigational position error is incontrovertible proof of a mistaken design. Fixing the off-by-one coding error just papers over the design error. Testing clearly failed to detect the mistake. I would not be at all surprised to learn that commercial quads share the design mistake. I was surprised to learn that NASA professionals copied it into a Mars probe. But, notably, not into the vehicle that delivered the lander.

HALtheWise 5y ago

I can't think of a polite way to say this, but as someone who professionally develops drone software, both of the software failures experienced by Ingenuity have been embarrassingly amateur at a technical level. The first failure, which delayed the initial spin test, was described as a "watchdog timeout", which for anyone not familiar with embedded development basically means the code crashed. We all write code that crashes, but I am having trouble thinking of an excuse to justify the fact that their code crashed before takeoff, on Mars, and they didn't see it coming. There is nothing about sitting on the ground on Mars that shouldn't have been tested repeatedly on earth, and testing in production is _really_ not the right way to do Aerospace development (although Boeing Starliner would beg to differ) Similarly, there are a huge number of things that can and will result in dropped frames when running Linux on a Qualcomm mobile chip, and having a software stack that infers frame timing purely from the sequence number is brittle, and would definitely not have passed code review and testing where I work (I actually checked, we do have a robust solution). If I had to guess, I suspect the root cause of the dropped frame wasn't actually anything exciting like a cosmic ray, but instead was some run-of-the-mill event that would have been caught by a couple hours of flight testing on Earth. Either way, it shouldn't have made it to Mars. I'm sure that there are a lot of great engineers working on the Ingenuity project that _don't_ write these sorts of bugs, and am glad that theae amateur fuckups (barely) haven't crashed the drone before it has been able to do some incredible technology demonstration work.

fermuch 5y ago
It might be related to the hostility of the environment. The chips aren't radiation proof, so it is expected to have some bit flops due to radiation.
- x86_64Ubuntu 5y ago
  Why on earth wouldn't they use rad-proofed chips on a planet closer to the sun, with virtually no atmosphere compared to earth?
  - dieortin 5y ago

rsp1984 5y ago

Fusing video with IMU for navigation is called VIN (visual-inertial navigation) or VIO (visual-inertial odometry) and the field has made enormous progress over the last 10-15 years. It's the same technology that the iPhone uses for all its AR features. Dropped frames are one of the easiest things to handle. Yes, the visual feature tracking depends on frames of video but even cheap phone IMUs these days are good enough to dead-reckon for a second or two, especially when embedded into a sensor fusion framework, so the prediction errors resulting from a single lost frame should be very minimal and not enough to throw off the tracking. That's why I find it hard to believe that the VIN in use by the Mars Helicopter (part of a multi billion dollar program) wouldn't be able to deal with a dropped frame. It just doesn't add up. I suspect that the situation is much more complex than what the article suggests and that more things went wrong than just a dropped camera frame.

ncmncm 5y ago
It means that the people who know most about closed-loop vehicle control were not involved in the design of the copter. Such fragility is a really elementary design mistake any experienced engineer would not make. Certainly the vehicle that delivered the lander would not suffer from the same mistake. My interpretion is that the copter, as an inessential system component, was seen as an opportunity for junior people to get some end-to-end experience. I hope they are learning the right things.
- ehsankia 5y ago
  The article makes it sound like after the missed frame, every subsequent frame had the wrong timestamp and was processed wrong, so it sounds like more of a bug. If the subsequent timestamps had the right timestamps, and the calculation was based on delta timestamps, then it would basically basically interpolate over the missing frame and then recover. The fact that every subsequent frame was assumed to be 33ms in the past was the issue here.
  - mikepurvis 5y ago
    And that being a system boundary thing, it’s easy to imagine it as an integration issue, where the controls side maybe assumed it would get a dummy frame or something.

jsrcout 5y ago

Article from the Mars Helicopter chief pilot: https://mars.nasa.gov/technology/helicopter/status/305/surviving-an-in-flight-anomaly-what-happened-on-ingenuitys-sixth-flight/ https://mars.nasa.gov/technology/helicopter/status/305/survi...

renewiltord 5y ago

NASA systems appear to have the property that they are both perfectly designed when HN commenters do not understand the code and amateurishly designed when they have errors. This bathtub style curve for perception of NASA design by HN commenters makes me question if the perception correlates with reality.

kibwen 5y ago
You're simply coming to the astute observation that any individual HN commenter who may very well be a genius WRT an extremely narrow area of expertise will be just as hopelessly clueless as the average person when it comes to any area outside of their expertise.
stevenhuang 5y ago
Perfectly designed when it's right and badly designed when it's wrong, you mean? Strange way to state a tautology to make yet another boring statement on ye olde HN commenter.
- renewiltord 5y ago
  No, actually, I do not mean that. I mean exactly what I said.

heisenbit 5y ago

> the inertial measurement unit (IMU) and the navigational camera. The IMU measures acceleration in three dimensions, using data from several sensors to estimate altitude, velocity, and position. Even though this system samples at 500 Hz, the error would accumulate over time, causing the helicopter to become lost quickly. I understand the IMU is not an ideal input and integration over time leads to positional errors. But gyros are much better and the orientation of the drone in flight is paramount. What I wonder is what ‚advanced‘ control law allowed the drone to become unstable wrt. orientation when there was noisy positional input.

jamiedamien 5y ago
I suspect what happened is that the position and velocity estimates from the VIO system (cameras) were wrong (and perhaps wildly jumping) and since position and velocity errors are inputs to the attitude controller, the tilt oscillated as well. It's not that the IMU did a poor job estimating attitude (tilt), but rather that the attitude controller was asked to tilt in erratic directions to compensate for the erratic and incorrect position and velocity readings. It's quite hard to detect when position and velocities are "obviously incorrect", especially when they come from VIO, where the optimization result can jump around in non ideal conditions, so I'm not surprised there was not a more graceful anomaly detection.