13 ms·

Scott Manly has a video on why he thinks the last one failed. It sounds like their landing software didn't have "oh crap, we're way off course, just land wherev

by jvm___ 3y ago

Scott Manly has a video on why he thinks the last one failed. It sounds like their landing software didn't have "oh crap, we're way off course, just land wherever". It only had the happy path of "fly to here and land", so when it switched to the landing phase it tried valiently (including flying upside down) to try to fly back to the landing zone, but the landing zone was much further than fuel supplies allowed. Hopefully they have upgraded software to just gracefully attempt a landing, and hopefully they won't be off course.

raverbashing 3y ago
This applies to general software development but is especially true in this case While we think "this cannot ever happen" in a lot of cases it can, in ways you did not consider. Both for good and bad
- oaktowner 3y ago
  While I worked at Google I once searched for "//this should never happen" in the code base. It was in there. A lot.
  - gora_mohanty 3y ago
    That might still be OK if that path literally should never happen in normal operation, but us used, e.g., to handle an error, or at least log it, if the impossible happens because the software is deployed in some unexpected configuration
    - oaktowner 3y ago
      Yes! Totally agreed.
- rob74 3y ago
  In general software development it's usually not a problem if your program crashes, then you can fix the bug and run it again. If the thing that crashes is a lunar lander however, you should put a bit more effort into covering all the eventualities...
- pjmorris 3y ago
  I really ought to dig up a reference for this, but there are strong echoes from the past here. Margaret Hamilton (who coined the term 'software engineering' and can be seen standing next to a tall pile of green bar printouts of the Apollo software) brought her daughter to work one weekend during the Apollo program and she (daughter) fiddled with the buttons and caused an error condition. Hamilton, based on this, argued that the software should account for the possibility of mistakes. Management's view was that the highly-trained astronauts wouldn't make mistakes. In time, Hamilton prevailed, and was proven correct.
  - stuaxo 3y ago
    Having seen what happens when you let a toddler start randomly pressing stuff on an Android tablet she was spot on, and if anything every environment should be out through this.
  - monitron 3y ago
    If you find a reference (or anyone does), please share. This is too good a story not to be told widely. Edit: here, at least, is a mention of the thing with her daughter in a Google Blog article: https://blog.google/products/maps/margaret-hamilton-apollo-11-tribute/ https://blog.google/products/maps/margaret-hamilton-apollo-1...
- extraduder_ire 3y ago
  The recent japanese lander had something like this happen a while back. The altitude radar noticed a sudden 2km drop in ground-level, and the system assumed it was broken and stopped using that data. Turns out it just flew over a cliff edge that actually does that. Completely by accident.
  - jvm___ 3y ago
    They picked a new landing site late in the program and didn't get the topography near that site, or run enough combinations through the software in test.
The_Colonel 3y ago
Not considering such a basic error condition seems like a gross omission. This can't even come from the software engineering, but must be some kind of managerial failure (e.g. we're short on time, but have to report great progress to my boss, so skip this scenario).
- xwdv 3y ago
  It probably wasn’t even the error. It could have been an accumulation of error % on some unbounded input.
- delta_p_delta_x 3y ago
  > seems like such a gross omission. Almost all space mission code only ever has the so-called 'happy path'. We rely on extremely tight mechanical and aerospace engineering tolerances to achieve that happy path. The Hubble Space Telescope's primary mirror grinding was off by a matter of micrometres, and resulted in blurry images. Consider all the Mars rovers. Imagine some wind gust threw the descending module off course, or a retro-rocket failed because of vibrations. Writing code for space missions isn't like writing a CRUD app. Developers can't just teleport to a space probe millions to billions of kilometres away to rectify errors and debug running code on the fly. For the record, the 'failure path' for Apollo 11 was to get the US President to announce to the world that the two astronauts would likely be marooned on the Moon. Apollo 13 very nearly failed, too.
  - astrange 3y ago
    If you write the failure path, then you also have to ensure it doesn't fire when there wasn't really a failure and do unnecessary heroics. Same reason you can't program a self driving car to save a person by sacrificing a squirrel. It's just going to run over the squirrel when it didn't need to.
  - ianburrell 3y ago
    Apollo 1 landed on the "failure path". Neil Armstrong noticed that the target was a boulder field, took manual control, discovered his new spot had a crater, and finally found level ground. They had low fuel but not dangerously low. If Apollo 11 had followed the "happy path", they would have crashed and died. Hubble was also the "failure path". The main mirror was flawed and had to be corrected.
deleted 3y ago
[deleted]
jnsaff2 3y ago
[flagged]
- wayFuLtH 3y ago
  [flagged]
hoseja 3y ago
Incredibly Kerbal vibe. I think something like that happened to me when using MechJeb. (it even has a "land whenever" feature, which it was too late to use at that point)
- dotancohen 3y ago
  That is just a testament to the incredible work of the Squad team. The Indian landing is not similar to Kerbal, rather, Kerbal is very very similar to the real experience. Amazing.
franky47 3y ago
So in Scott's words, a "fly safe" subroutine?
m0llusk 3y ago
If anything they seem to have overcorrected for this. This landing path stopped with a near hover at around 800m, then a prolonged hover at 150m while the lander scoped the situation, then an extremely slow descent that allowed for corrections the whole way down. Very impressive.
- morepork 3y ago
  If they have the fuel available, may as well be very conservative. Either burn the fuel on descent or have it sit in the tank forever on the lunar surface.
- 8bitsrule 3y ago
  Possibly the lander had a choice to be more careful/picky if the remaining fuel permitted. (Just as Armstrong did.)
- hoseja 3y ago
  Also incredibly inefficient.
  - user_7832 3y ago
    A lot of things in the world trade inefficiencies for safety.
    - hoseja 3y ago
      I was surprised they had the propulsion budget for it.
      - ansible 3y ago
        It helps a lot that the Moon's gravity is about 1/6th of Earth's.
      - deleted 3y ago
        [deleted]
      - 3y ago
seatac76 3y ago
They also had an issue with the thrust gradient it could only do it increments in 20% which was too much of a change this one was finer which allowed for better control authority.
ansible 3y ago
Link to the Scott Manley video - The Real Reason Why Chandrayaan 2 Crashed on the Moon: https://www.youtube.com/watch?v=2ngVl6iO94c https://www.youtube.com/watch?v=2ngVl6iO94c
Rygian 3y ago
That's an example of failing to follow the "Aviate, Navigate, Communicate" rule of thumb.
- goku12 3y ago
  Could you elaborate how that applies here? Communication is out of the question - there's nothing the people on the ground can correct. And the lander apparently failed because it tried to navigate its way back to the designated landing site.
  - vikingerik 3y ago
    Communicate wasn't the important part, but you got it in your last sentence. It was trying to navigate over aviate - trying to get to its designated landing spot too hard so that it neglected/failed to stay airborne (spaceborne).
    - goku12 3y ago
      That's much clearer. Aviate > Navigate. Thanks!
  - dadadad100 3y ago
    If you crashed trying to ensure your antenna was optimally oriented then you chose “communicate” over “aviate”. It’s a stretch, but the point is to clearly define your priorities and stick to them, even in a panic
sumodm 3y ago
Yes, this seems to be one of the issues. Here is a talk by ISRO Chairman S Somanath at Indian Institute of Science (IISc) about the same and what they added. He goes into details of what went wrong. These are based on my limited understanding. One of the thrusters had an issue and got activated for longer (or may be activation profile of thrusters at its extreme's were different from modeled). They had a narrow landing region selected as final position (even one possible point). So now their control system tried to correct this but the algorithm had a bug and that caused it to be further delayed. At this point the correction required, i.e; thrusters to be activated, was outside the tolerance levels. So finally ended up with 50m/s vertical speed. https://www.youtube.com/watch?v=fZ2sNRP1opY&t=1440s https://www.youtube.com/watch?v=fZ2sNRP1opY&t=1440s With new one, they did couple of things. Larger area to be selected for landing based on camera input. Escape sequence to more achievable points, if something like this happens again. They increased the tolerance from 10 degrees to 25 degrees and guessing fixed the bug in code. They also did some smoothening of the trajectory for different phases to make it more continuous. I think they also made other changes in engines among other things and a whole host of testing.
publicola1990 3y ago
Also did they use image processing to guide the landing this time? American and Chinese probes seems to use it to do the final phase of soft landing.
- perryizgr8 3y ago
  They have to, don't they? There's no other way.

by jvm___ 3y ago

raverbashing 3y ago

This applies to general software development but is especially true in this case While we think "this cannot ever happen" in a lot of cases it can, in ways you did not consider. Both for good and bad

oaktowner 3y ago
While I worked at Google I once searched for "//this should never happen" in the code base. It was in there. A lot.
- gora_mohanty 3y ago
  That might still be OK if that path literally should never happen in normal operation, but us used, e.g., to handle an error, or at least log it, if the impossible happens because the software is deployed in some unexpected configuration
  - oaktowner 3y ago
    Yes! Totally agreed.
rob74 3y ago
In general software development it's usually not a problem if your program crashes, then you can fix the bug and run it again. If the thing that crashes is a lunar lander however, you should put a bit more effort into covering all the eventualities...
pjmorris 3y ago
I really ought to dig up a reference for this, but there are strong echoes from the past here. Margaret Hamilton (who coined the term 'software engineering' and can be seen standing next to a tall pile of green bar printouts of the Apollo software) brought her daughter to work one weekend during the Apollo program and she (daughter) fiddled with the buttons and caused an error condition. Hamilton, based on this, argued that the software should account for the possibility of mistakes. Management's view was that the highly-trained astronauts wouldn't make mistakes. In time, Hamilton prevailed, and was proven correct.
- stuaxo 3y ago
  Having seen what happens when you let a toddler start randomly pressing stuff on an Android tablet she was spot on, and if anything every environment should be out through this.
- monitron 3y ago
  If you find a reference (or anyone does), please share. This is too good a story not to be told widely. Edit: here, at least, is a mention of the thing with her daughter in a Google Blog article: https://blog.google/products/maps/margaret-hamilton-apollo-11-tribute/ https://blog.google/products/maps/margaret-hamilton-apollo-1...
extraduder_ire 3y ago
The recent japanese lander had something like this happen a while back. The altitude radar noticed a sudden 2km drop in ground-level, and the system assumed it was broken and stopped using that data. Turns out it just flew over a cliff edge that actually does that. Completely by accident.
- jvm___ 3y ago
  They picked a new landing site late in the program and didn't get the topography near that site, or run enough combinations through the software in test.

The_Colonel 3y ago

Not considering such a basic error condition seems like a gross omission. This can't even come from the software engineering, but must be some kind of managerial failure (e.g. we're short on time, but have to report great progress to my boss, so skip this scenario).

xwdv 3y ago
It probably wasn’t even the error. It could have been an accumulation of error % on some unbounded input.
delta_p_delta_x 3y ago
> seems like such a gross omission. Almost all space mission code only ever has the so-called 'happy path'. We rely on extremely tight mechanical and aerospace engineering tolerances to achieve that happy path. The Hubble Space Telescope's primary mirror grinding was off by a matter of micrometres, and resulted in blurry images. Consider all the Mars rovers. Imagine some wind gust threw the descending module off course, or a retro-rocket failed because of vibrations. Writing code for space missions isn't like writing a CRUD app. Developers can't just teleport to a space probe millions to billions of kilometres away to rectify errors and debug running code on the fly. For the record, the 'failure path' for Apollo 11 was to get the US President to announce to the world that the two astronauts would likely be marooned on the Moon. Apollo 13 very nearly failed, too.
- astrange 3y ago
  If you write the failure path, then you also have to ensure it doesn't fire when there wasn't really a failure and do unnecessary heroics. Same reason you can't program a self driving car to save a person by sacrificing a squirrel. It's just going to run over the squirrel when it didn't need to.
- ianburrell 3y ago
  Apollo 1 landed on the "failure path". Neil Armstrong noticed that the target was a boulder field, took manual control, discovered his new spot had a crater, and finally found level ground. They had low fuel but not dangerously low. If Apollo 11 had followed the "happy path", they would have crashed and died. Hubble was also the "failure path". The main mirror was flawed and had to be corrected.

deleted 3y ago

[deleted]

jnsaff2 3y ago

[flagged]

wayFuLtH 3y ago
[flagged]

hoseja 3y ago

Incredibly Kerbal vibe. I think something like that happened to me when using MechJeb. (it even has a "land whenever" feature, which it was too late to use at that point)

dotancohen 3y ago
That is just a testament to the incredible work of the Squad team. The Indian landing is not similar to Kerbal, rather, Kerbal is very very similar to the real experience. Amazing.

franky47 3y ago

So in Scott's words, a "fly safe" subroutine?

m0llusk 3y ago

If anything they seem to have overcorrected for this. This landing path stopped with a near hover at around 800m, then a prolonged hover at 150m while the lander scoped the situation, then an extremely slow descent that allowed for corrections the whole way down. Very impressive.

morepork 3y ago
If they have the fuel available, may as well be very conservative. Either burn the fuel on descent or have it sit in the tank forever on the lunar surface.
8bitsrule 3y ago
Possibly the lander had a choice to be more careful/picky if the remaining fuel permitted. (Just as Armstrong did.)
hoseja 3y ago
Also incredibly inefficient.
- user_7832 3y ago
  A lot of things in the world trade inefficiencies for safety.
  - hoseja 3y ago
    I was surprised they had the propulsion budget for it.
    - ansible 3y ago
      It helps a lot that the Moon's gravity is about 1/6th of Earth's.
    - deleted 3y ago
      [deleted]
    - 3y ago

seatac76 3y ago

They also had an issue with the thrust gradient it could only do it increments in 20% which was too much of a change this one was finer which allowed for better control authority.

ansible 3y ago

Link to the Scott Manley video - The Real Reason Why Chandrayaan 2 Crashed on the Moon: https://www.youtube.com/watch?v=2ngVl6iO94c https://www.youtube.com/watch?v=2ngVl6iO94c

Rygian 3y ago

That's an example of failing to follow the "Aviate, Navigate, Communicate" rule of thumb.

goku12 3y ago
Could you elaborate how that applies here? Communication is out of the question - there's nothing the people on the ground can correct. And the lander apparently failed because it tried to navigate its way back to the designated landing site.
- vikingerik 3y ago
  Communicate wasn't the important part, but you got it in your last sentence. It was trying to navigate over aviate - trying to get to its designated landing spot too hard so that it neglected/failed to stay airborne (spaceborne).
  - goku12 3y ago
    That's much clearer. Aviate > Navigate. Thanks!
- dadadad100 3y ago
  If you crashed trying to ensure your antenna was optimally oriented then you chose “communicate” over “aviate”. It’s a stretch, but the point is to clearly define your priorities and stick to them, even in a panic

sumodm 3y ago

Yes, this seems to be one of the issues. Here is a talk by ISRO Chairman S Somanath at Indian Institute of Science (IISc) about the same and what they added. He goes into details of what went wrong. These are based on my limited understanding. One of the thrusters had an issue and got activated for longer (or may be activation profile of thrusters at its extreme's were different from modeled). They had a narrow landing region selected as final position (even one possible point). So now their control system tried to correct this but the algorithm had a bug and that caused it to be further delayed. At this point the correction required, i.e; thrusters to be activated, was outside the tolerance levels. So finally ended up with 50m/s vertical speed. https://www.youtube.com/watch?v=fZ2sNRP1opY&t=1440s https://www.youtube.com/watch?v=fZ2sNRP1opY&t=1440s With new one, they did couple of things. Larger area to be selected for landing based on camera input. Escape sequence to more achievable points, if something like this happens again. They increased the tolerance from 10 degrees to 25 degrees and guessing fixed the bug in code. They also did some smoothening of the trajectory for different phases to make it more continuous. I think they also made other changes in engines among other things and a whole host of testing.

publicola1990 3y ago

Also did they use image processing to guide the landing this time? American and Chinese probes seems to use it to do the final phase of soft landing.

perryizgr8 3y ago
They have to, don't they? There's no other way.