6 ms·

How NASA’s Mission to Pluto Was Nearly Lost

101 points by dnetesn 9y ago

koverstreet 8y ago
They waited until 10 days before flyby to upload a shit ton of code? @_@
- SiempreViernes 8y ago
  You need a good idea of the actual orbit to plan your pointings accurately, and how are you gonna get that if you don’t know were the thing actually ended up?
  - tqkxzugoaupvwqr 8y ago
    Arm-chair thoughts of an outsider: Since they knew their objectives in advance, couldn’t they have uploaded the code/program/sequence in advance and only sent the configuration/parameters on final approach? Waiting until the last 10 days of a decade long mission to upload mission critical code strikes me as risky. Even more so when there is a nine hour latency.
    - therealjumbo 8y ago
      Reading this story sent shivers down my spine. I'm very glad my stuff can't go this wrong this fast with that many consequences and then have to work that hard to clean it back up. However, I'm guessing the reason they didn't upload said mission code earlier (or before launch) is because it wasn't done yet, and that's mainly because they didn't know what needed to be coded up yet or it's so incredibly unique to their mission its never been written before especially within their constraints. So they launch, and figure out some of the requirements and code up the solution while its in flight. If they did it all beforehand, the mission would've been 10 years longer assuming that it did in fact take 10 years to figure out and code the solution. Furthermore I'm guessing of the requirements may not have been clear until they were in flight (or past Jupiter or whatever) and then they had to write the solution and software development is unpredictable so... At any rate, hats off to them for such an incredible job.
    - gmueckl 8y ago
      While I am not completely sure, I believe that the commands that get sent to flying missions are mostly high level scripts comtrolling the sequences of events that need to happen. And these need to be determined during the mission for various reasons (new science, engineering mistakes, error correction for trajectory deviations...).
testplzignore 8y ago
Why did they need to wait for confirmation before moving on in each 9 hour cycle? Why not just send everything as fast as possible all at once and wait for confirmation at the end? Seems like they would be wasting lots of bandwidth waiting for the ack.
- vardump 8y ago
  Maybe those had interdependencies that would have caused a total failure if the previous phase failed?
- greglindahl 8y ago
  Wasted bandwidth? The DSN supports all of our inter-planetary spacecraft, not just this one.
  - gizmo686 8y ago
    How many other time critical communications did they have going on at the time. I assume the new horizon team was given all the DSN time they wanted during this.
    - greglindahl 8y ago
      So you think the DSN was idle for 9 hours while they waited for the known speed of light delay? That's not how the DSN is scheduled. The only way that the DSN wastes bandwidth is when they retransmit something that was actually received correctly. New Horizons would never ask for time that was unnecessary.
SiempreViernes 8y ago
I wonder why the hero of the story so rarely go to speak with her own voice.
- perl4ever 8y ago
  Which person are you referring to, and why is she "the hero of the story"?
  - deleted 8y ago
    [deleted]
  - perl4ever 8y ago
    Since it must not have been clear, I was asking whether Alice or Leslie was the "hero of the story" and why.
- Jaruzel 8y ago
  The clue here is in the moniker they assigned her: 'MOM' I know it's acronym of her job title, but in Europe you'd never get away with such a traditional role-based label assigned to a female professional like that. Women in the US still have such a long way to go before the patriarchal system even starts to recognise them as equals.
  - entityDev 8y ago
    Perhaps some introspection is required on your part, as to determine how you have become so quickly and ignorantly triggered. "MOM" has nothing to do with her gender, for if you had read the article that you are commenting on, you would have realized that it stands for "Missions Operations Manager". Hmm, women in the US have such a long way to go before they can apply logic and reasoning to their frequently triggered and incorrect outburst about the non-existent "Patriarchy", and to also recognize that women and men are NOT equal in all things, but bring a unique perspective and wonderful sparks when they work together.
  - dctoedt 8y ago
    It sounds like an inside joke among equals, not a put-down.
abecedarius 8y ago
https://en.wikipedia.org/wiki/Deep_Space_1#Remote_Agent https://en.wikipedia.org/wiki/Deep_Space_1#Remote_Agent 20 years ago was meant to start developing the tech to avoid this kind of almost-disaster -- an early proposal for the project highlighted this scenario of falling into safe mode during a flyby. The aim was to go from preplanning each detailed motion to an adaptable AI working from a representation of mission goals, a plan to achieve them, and a model of the hardware. When things go wrong, it tries to diagnose the fault and repair the plan. When every mission costs a zillion dollars and much of many people's careers, of course you're going to stick with the hardcoded scripts until you're really sure of the above tech, sadly.
billforsternz 8y ago
I imagine that one semi-psychological problem with working on this crisis would be that it would be hard to put aside the feeling that the system had crashed under load once, maybe it would do it again. Presumably NASA's famous software quality regime imbues a machine with a "feeling" that this thing is rock solid - you'd trust it not to just crash on you because it was doing two things at once. That trust would be obliterated by an incident like this.
celsoazevedo 8y ago
Deep Space Network: https://eyes.nasa.gov/dsn/dsn.html https://eyes.nasa.gov/dsn/dsn.html
mannykannot 8y ago
"Something key they discovered very quickly was that just before the spacecraft’s signal stopped, the main computer had been doing two things at once, both of which were computationally demanding. One of these tasks was compressing 63 Pluto images taken previously, in order to free up memory space for the close flyby imaging soon to begin. At the same time, the computer was also receiving the Core load from Earth and storing it in its memory." Concurrency is a powerful multiplier of risk.

101 points by dnetesn 9y ago

koverstreet 8y ago

They waited until 10 days before flyby to upload a shit ton of code? @_@

SiempreViernes 8y ago
You need a good idea of the actual orbit to plan your pointings accurately, and how are you gonna get that if you don’t know were the thing actually ended up?
- tqkxzugoaupvwqr 8y ago
  Arm-chair thoughts of an outsider: Since they knew their objectives in advance, couldn’t they have uploaded the code/program/sequence in advance and only sent the configuration/parameters on final approach? Waiting until the last 10 days of a decade long mission to upload mission critical code strikes me as risky. Even more so when there is a nine hour latency.
  - therealjumbo 8y ago
    Reading this story sent shivers down my spine. I'm very glad my stuff can't go this wrong this fast with that many consequences and then have to work that hard to clean it back up. However, I'm guessing the reason they didn't upload said mission code earlier (or before launch) is because it wasn't done yet, and that's mainly because they didn't know what needed to be coded up yet or it's so incredibly unique to their mission its never been written before especially within their constraints. So they launch, and figure out some of the requirements and code up the solution while its in flight. If they did it all beforehand, the mission would've been 10 years longer assuming that it did in fact take 10 years to figure out and code the solution. Furthermore I'm guessing of the requirements may not have been clear until they were in flight (or past Jupiter or whatever) and then they had to write the solution and software development is unpredictable so... At any rate, hats off to them for such an incredible job.
  - gmueckl 8y ago
    While I am not completely sure, I believe that the commands that get sent to flying missions are mostly high level scripts comtrolling the sequences of events that need to happen. And these need to be determined during the mission for various reasons (new science, engineering mistakes, error correction for trajectory deviations...).

testplzignore 8y ago

Why did they need to wait for confirmation before moving on in each 9 hour cycle? Why not just send everything as fast as possible all at once and wait for confirmation at the end? Seems like they would be wasting lots of bandwidth waiting for the ack.

vardump 8y ago
Maybe those had interdependencies that would have caused a total failure if the previous phase failed?
greglindahl 8y ago
Wasted bandwidth? The DSN supports all of our inter-planetary spacecraft, not just this one.
- gizmo686 8y ago
  How many other time critical communications did they have going on at the time. I assume the new horizon team was given all the DSN time they wanted during this.
  - greglindahl 8y ago
    So you think the DSN was idle for 9 hours while they waited for the known speed of light delay? That's not how the DSN is scheduled. The only way that the DSN wastes bandwidth is when they retransmit something that was actually received correctly. New Horizons would never ask for time that was unnecessary.

SiempreViernes 8y ago

I wonder why the hero of the story so rarely go to speak with her own voice.

perl4ever 8y ago
Which person are you referring to, and why is she "the hero of the story"?
- deleted 8y ago
  [deleted]
- perl4ever 8y ago
  Since it must not have been clear, I was asking whether Alice or Leslie was the "hero of the story" and why.
Jaruzel 8y ago
The clue here is in the moniker they assigned her: 'MOM' I know it's acronym of her job title, but in Europe you'd never get away with such a traditional role-based label assigned to a female professional like that. Women in the US still have such a long way to go before the patriarchal system even starts to recognise them as equals.
- entityDev 8y ago
  Perhaps some introspection is required on your part, as to determine how you have become so quickly and ignorantly triggered. "MOM" has nothing to do with her gender, for if you had read the article that you are commenting on, you would have realized that it stands for "Missions Operations Manager". Hmm, women in the US have such a long way to go before they can apply logic and reasoning to their frequently triggered and incorrect outburst about the non-existent "Patriarchy", and to also recognize that women and men are NOT equal in all things, but bring a unique perspective and wonderful sparks when they work together.
- dctoedt 8y ago
  It sounds like an inside joke among equals, not a put-down.

abecedarius 8y ago

https://en.wikipedia.org/wiki/Deep_Space_1#Remote_Agent https://en.wikipedia.org/wiki/Deep_Space_1#Remote_Agent 20 years ago was meant to start developing the tech to avoid this kind of almost-disaster -- an early proposal for the project highlighted this scenario of falling into safe mode during a flyby. The aim was to go from preplanning each detailed motion to an adaptable AI working from a representation of mission goals, a plan to achieve them, and a model of the hardware. When things go wrong, it tries to diagnose the fault and repair the plan. When every mission costs a zillion dollars and much of many people's careers, of course you're going to stick with the hardcoded scripts until you're really sure of the above tech, sadly.

billforsternz 8y ago

I imagine that one semi-psychological problem with working on this crisis would be that it would be hard to put aside the feeling that the system had crashed under load once, maybe it would do it again. Presumably NASA's famous software quality regime imbues a machine with a "feeling" that this thing is rock solid - you'd trust it not to just crash on you because it was doing two things at once. That trust would be obliterated by an incident like this.

celsoazevedo 8y ago

Deep Space Network: https://eyes.nasa.gov/dsn/dsn.html https://eyes.nasa.gov/dsn/dsn.html

mannykannot 8y ago

"Something key they discovered very quickly was that just before the spacecraft’s signal stopped, the main computer had been doing two things at once, both of which were computationally demanding. One of these tasks was compressing 63 Pluto images taken previously, in order to free up memory space for the close flyby imaging soon to begin. At the same time, the computer was also receiving the Core load from Earth and storing it in its memory." Concurrency is a powerful multiplier of risk.