9 ms·

NASA Pulls Off Delicate Thruster Swap, Keeping Voyager 1 Mission Alive

304 points by Stratoscope 2y ago

qingcharles 2y ago
Great to see they bought her some more time. I watched this excellent Voyager documentary recently: https://www.itsquieterfilm.com/ https://www.itsquieterfilm.com/
a_e_k 2y ago
I continue to be fascinated by how they: 1. Are able to diagnose the problems remotely at such distances and on such old hardware. How can they even measure the thruster tube apertures here? 2. Decide what actions to take. It's not like they have a local test device to experiment on is it? (Even if they did I can't imagine how they'd reproduce the conditions of the real thing.) And if they choose poorly, I'd assume the mission's over. There's no replacing Voyager 1 if they brick it. 3. Have such fine control over the hardware. For something built in the 70's when RAM was largely measured in kB, they seem to have an insane amount of flexibility to remotely reconfigure the equipment. Whatever they did, there must have been some real foresight.
- big-green-man 2y ago
  What blows my mind is the organizational knowledge needed after so many years to keep it going. You don't just hand a guy some man pages when he comes onto the project. I'm sure people have aged out, yet they still understand the complexities in the design. That is something we need to understand and prioritize in the systems we build today.
  - tommiegannert 2y ago
    Startup: "Do you remember when we inserted this quirky module running in AWS? We can use that to implement this next feature. That was a useful decision!" Voyager: "Do you remember when our parents inserted this quirky module that has since left the solar system? We can use that to turn it off and on again. That was a mission critical decision!"
    - exe34 2y ago
      that aws module isn't supported anymore though, we need to npm install half the internet for the 2 line library that replaces it.
    - HeatrayEnjoyer 2y ago
      Not parents for most of us. Grand or great-grandparents. The senior engineers were highly educated and experienced, implying ages 30-50s during development. They are in their 80-100s now.
  - tverbeure 2y ago
    When Voyager failed last year with a CMOS memory error, one of the big problems was that a bunch of low level information was gone or conflicting. For example, they sometimes had to guess assembler instructions because the code printouts were low quality photocopied pages. Or because there were handwritten comments or comments scratched out with pencil without any clue about why it was done. One saving grace was the fact that the architecture and the code space was simple enough so that they could reason through the symptoms and actions to take, something that would have been much harder with modern spacecraft. Check out this amazing talk: https://youtu.be/dF_9YcehCZo?si=W_b3NJ7vgxaYS1__ https://youtu.be/dF_9YcehCZo?si=W_b3NJ7vgxaYS1__
    - N3cr0ph4g1st 2y ago
      Thank you for this link!
  - methuselah_in 2y ago
    This thing hits me. True.
- tomooot 2y ago
  I would imagine if the design/assembly information was broadly available (internally) in the past, there's probably one or several "digital twin" emulations of the craft, or at the very least specific subsystems of it's computing resources. There must be some kind of analog/simulation of it's software just for proving "bugfixes" before upload, like the coms error and subsequent setting of the "solar system record" for "furthest distance remote code update" earlier this year.
  - onedognight 2y ago
    There isn’t a simulator or digital twin for voyager. It has a bespoke processor made with 74* style logic. One guy will puts together a command and they will have a review where the other engineers will try and independently verify it. Then they copy and paste the command somewhere to “run it”. It happened, fairly recently, that the command had a typo that was caught in review, but the “wrong” pre-review command was used and the attitude became off by so much that they lost contact. It was only by cranking up the power at Goldstone that they got a command through. This fundamentally changed their understanding of the largest angle for which they could still communicate with the spacecraft. They just hadn’t wanted to try larger angles before because it was too risky.
    - wildzzz 2y ago
      That's the great thing about such a simple design, you can actually sit down with pen and paper and verify operations.
  - tverbeure 2y ago
    Forget about a digital twin, they don’t even have an assembler for the CPUs. It’s all hex values.
- basementcat 2y ago
  In short, the Voyager spacecraft were designed (after considerable design and operational experience from Mariner and Pioneer spacecraft) to be able to operate for a long time largely automatically without too much hand-holding. All major systems are multiply redundant and may be remotely turned on and off. While there has been personnel turnover on the program, it has not been of a magnitude to jeopardize program continuation. Finally, program management has been media savvy and well politically connected ensuring that operations are still funded. (Contrast with other missions such as Magellan to Venus which was deorbited while it still had propellant reserve leaving some portions of the planet unmapped) JPL has (or had?) a "retiree badge" program that permitted retired staff to continue to access their office. Many programs benefited from highly knowledgeable personnel essentially continuing to report to the office every day without pay (not being paid comes with the luxury of not having to worry about being laid off if your charge accounts don’t have enough funds!) It was an absolute privilege to learn from these people.
  - SSLy 2y ago
    So, culture preservation is important for success of highly technical endeavours. Don't tell it to your run of the mill MBA.
    - basementcat 2y ago
      I think it is inevitable that organizational "culture" changes. The tricky part is figuring out exactly what parts need change and what parts don’t. For example JPL used to have beauty pageants ("Miss Guided Missile"). More recently, management appears to be trying to adopt policies and procedures from some venture funded commercial space companies. It is not clear how helpful these efforts will be given that these organizations are in fundamentally different businesses. https://www.jpl.nasa.gov/images/slice-of-history-70th-anniversary-of-miss-guided-missile https://www.jpl.nasa.gov/images/slice-of-history-70th-annive...
      - fnord77 2y ago
        [flagged]
- olabyne 2y ago
  And quite recently they also recovered from a faulty ram module, with a radio link speed that is maybe below the kbit/s now !
  - deleted 2y ago
    [deleted]
  - lisper 2y ago
    And you should see the ping times.
  - liamwire 2y ago
    I suspect it’s far lower than that.
  - stordoff 2y ago
    > Uplink communications is via S-band (16-bits/sec command rate) while an X-band transmitter provides downlink telemetry at 160 bits/sec normally and 1.4 kbps for playback of high-rate plasma wave data. https://science.nasa.gov/mission/voyager/spacecraft/ https://science.nasa.gov/mission/voyager/spacecraft/ The data playbacks were initially transmitted at 7200 bps, but this was dropped to 1400 bps in 2007 (https://voyager.gsfc.nasa.gov/Library/DeepCommo_Chapter3--141029.pdf https://voyager.gsfc.nasa.gov/Library/DeepCommo_Chapter3--14... - page 72). RTT ~46 hours (one-way light time 22:49:59, per https://science.nasa.gov/mission/voyager/where-are-they-now/ https://science.nasa.gov/mission/voyager/where-are-they-now/)
- avidphantasm 2y ago
  > It's not like they have a local test device to experiment on is it? I would bet they have one or more simulators (“digital twin” in the parlance of our times). I’d want one simulator to always reflect the current state of the probe (with state data assimilated periodically from the real probe). Then other simulators can be used to test management changes to see how the system might respond.
  - onedognight 2y ago
    > I would bet they have one or more simulators Sadly they do not, but they are starting to write one.
- ForHackernews 2y ago
  There's a documentary "It's Quieter in the Twilight" about the team keeping the Voyager missions alive: https://www.youtube.com/watch?v=8vJT8AW0wYw https://www.youtube.com/watch?v=8vJT8AW0wYw
  - ornornor 2y ago
    Looks very interesting , thanks for sharing
  - whycome 2y ago
    Included with Amazon Prime in Canada
- tverbeure 2y ago
  You should totally check out the talk from a few weeks ago that I link to in my other comment. It answers your 3 questions and more.
japanuspus 2y ago
The Voyager mission is such a wild achievement. Both the sublime design and craftsmanship that must have gone into the hardware, and the deep institutional knowledge required to keep it running is awe-inspiring.
litoE 2y ago
It's been operating for 47 years and it still has fuel left to make attitude corrections. I wonder how they managed that feat.
- turblety 2y ago
  I presume solar power?
  - Tor3 2y ago
    No, way too little sun out here. The sun is just a dot far away. The Voyagers both use RTGs, radioisotope thermoelectric generators. Decaying plutonium, essentially. That's for the electrically powered systems. The thrusters use liquid hydrazine, which is common for those kind of thrusters. Edit: There's more about that in the NASA link in a sibling comment.
  - krige 2y ago
    Solar has nothing to do with it. Voyager uses hydrazine, of which over 80% has been used up over the years. They simply use not that much of it as it's not for thrust, but for aiming at Earth more or less.
  - meindnoch 2y ago
    Lol. How would that work exactly?
    - JumpCrisscross 2y ago
      > How would that work exactly? Light sails. Laser propulsion. Photon rockets. None of which apply to Voyager. But none of which are laughably dismissible.
  - gpderetta 2y ago
    Even with something like an ion engine (which I'm not sure were available when Voyager was launched), you would need leftover reaction mass.
- 2y ago
NaOH 2y ago
Probably a better link: https://science.nasa.gov/missions/voyager-program/voyager-1/voyager-1-team-accomplishes-tricky-thruster-swap/ https://science.nasa.gov/missions/voyager-program/voyager-1/...
perihelions 2y ago
- "A fuel tube inside the thrusters has filled up with silicon dioxide, a side effect of age within the spacecraft’s fuel tank." Where the heck do you get SiO2 from on a spacecraft? Some kind of silicone? edit: "clogged with silicon dioxide, a byproduct that appears with age from a rubber diaphragm in the spacecraft’s fuel tank"[0] —I'm guessing that is a silicone rubber. I didn't know that rubber can decompose into sand. [0] https://science.nasa.gov/missions/voyager-program/voyager-1/voyager-1-team-accomplishes-tricky-thruster-swap/ https://science.nasa.gov/missions/voyager-program/voyager-1/...
- tecleandor 2y ago
  Seems like silicone rubber is extracted from SiO2, so maybe it's impurities? Or maybe the Si and O are doing something to leave the polymer? (I have no idea if that last thing is even possible) https://en.wikipedia.org/wiki/Silicone_rubber https://en.wikipedia.org/wiki/Silicone_rubber
- AStonesThrow 2y ago
  [flagged]
- numpad0 2y ago
  They expel the Hydrazine(N2H4) fuel out of a spherical Ti tank by inflating a rubber balloon that involve Teflon inside the tank using helium supply. I guess N2H4 was potent enough to degrade even those space age materials. 1: https://ntrs.nasa.gov/api/citations/19810001583/downloads/19810001583.pdf https://ntrs.nasa.gov/api/citations/19810001583/downloads/19...
- shmeeed 2y ago
  It doesn't have to be silicone. SiO2 is a necessary ingredient in all rubber compounds.
onewheeltom 2y ago
Just another day at the office for the Voyager 1 team. Wow.
Buttons840 2y ago
I've heard good things about some Voyager documentaries, and I've wanted to watch one with my daughter, but NASA keeps making the documentaries out-of-date and incomplete. How many amazing stories will there be to tell by the time Voyager is truly beyond our knowledge?
- Something1234 2y ago
  You mention voyager documentaries but fail to mention which one. There's lots, and its impossible to know which one is good until you finish it. Please tell us which one you wanted to watch. It's sad how much documentation is honestly unreachable now.
Twisol 2y ago
I had the extreme pleasure of seeing Bruce Waggoner of the Voyager team give a keynote at !!Con just last month. The recording landed on YouTube just a couple days ago, so this is great timing: https://www.youtube.com/watch?v=dF_9YcehCZo https://www.youtube.com/watch?v=dF_9YcehCZo
tverbeure 2y ago
2 weeks ago, Bruce Wagoner from the Voyager program gave a talk at !!Con about how they recovered from the CMOS memory issue that they had a year ago. It’s basically blind debugging with a latency of 45 hours. The talk is amazing and goes through the computer architecture of the spacecraft as well as the challenges of dealing with something that is so old, with so documentation that has conflicting information or unreadable etc. https://youtu.be/dF_9YcehCZo?si=W_b3NJ7vgxaYS1__ https://youtu.be/dF_9YcehCZo?si=W_b3NJ7vgxaYS1__
ChrisArchitect 2y ago
Official release: https://science.nasa.gov/missions/voyager-program/voyager-1/voyager-1-team-accomplishes-tricky-thruster-swap/ https://science.nasa.gov/missions/voyager-program/voyager-1/... (https://news.ycombinator.com/item?id=41505008 https://news.ycombinator.com/item?id=41505008)

304 points by Stratoscope 2y ago

Great to see they bought her some more time. I watched this excellent Voyager documentary recently: https://www.itsquieterfilm.com/ https://www.itsquieterfilm.com/

a_e_k 2y ago

I continue to be fascinated by how they: 1. Are able to diagnose the problems remotely at such distances and on such old hardware. How can they even measure the thruster tube apertures here? 2. Decide what actions to take. It's not like they have a local test device to experiment on is it? (Even if they did I can't imagine how they'd reproduce the conditions of the real thing.) And if they choose poorly, I'd assume the mission's over. There's no replacing Voyager 1 if they brick it. 3. Have such fine control over the hardware. For something built in the 70's when RAM was largely measured in kB, they seem to have an insane amount of flexibility to remotely reconfigure the equipment. Whatever they did, there must have been some real foresight.

big-green-man 2y ago
What blows my mind is the organizational knowledge needed after so many years to keep it going. You don't just hand a guy some man pages when he comes onto the project. I'm sure people have aged out, yet they still understand the complexities in the design. That is something we need to understand and prioritize in the systems we build today.
- tommiegannert 2y ago
  Startup: "Do you remember when we inserted this quirky module running in AWS? We can use that to implement this next feature. That was a useful decision!" Voyager: "Do you remember when our parents inserted this quirky module that has since left the solar system? We can use that to turn it off and on again. That was a mission critical decision!"
  - exe34 2y ago
    that aws module isn't supported anymore though, we need to npm install half the internet for the 2 line library that replaces it.
  - HeatrayEnjoyer 2y ago
    Not parents for most of us. Grand or great-grandparents. The senior engineers were highly educated and experienced, implying ages 30-50s during development. They are in their 80-100s now.
- tverbeure 2y ago
  When Voyager failed last year with a CMOS memory error, one of the big problems was that a bunch of low level information was gone or conflicting. For example, they sometimes had to guess assembler instructions because the code printouts were low quality photocopied pages. Or because there were handwritten comments or comments scratched out with pencil without any clue about why it was done. One saving grace was the fact that the architecture and the code space was simple enough so that they could reason through the symptoms and actions to take, something that would have been much harder with modern spacecraft. Check out this amazing talk: https://youtu.be/dF_9YcehCZo?si=W_b3NJ7vgxaYS1__ https://youtu.be/dF_9YcehCZo?si=W_b3NJ7vgxaYS1__
  - N3cr0ph4g1st 2y ago
    Thank you for this link!
- methuselah_in 2y ago
  This thing hits me. True.
tomooot 2y ago
I would imagine if the design/assembly information was broadly available (internally) in the past, there's probably one or several "digital twin" emulations of the craft, or at the very least specific subsystems of it's computing resources. There must be some kind of analog/simulation of it's software just for proving "bugfixes" before upload, like the coms error and subsequent setting of the "solar system record" for "furthest distance remote code update" earlier this year.
- onedognight 2y ago
  There isn’t a simulator or digital twin for voyager. It has a bespoke processor made with 74* style logic. One guy will puts together a command and they will have a review where the other engineers will try and independently verify it. Then they copy and paste the command somewhere to “run it”. It happened, fairly recently, that the command had a typo that was caught in review, but the “wrong” pre-review command was used and the attitude became off by so much that they lost contact. It was only by cranking up the power at Goldstone that they got a command through. This fundamentally changed their understanding of the largest angle for which they could still communicate with the spacecraft. They just hadn’t wanted to try larger angles before because it was too risky.
  - wildzzz 2y ago
    That's the great thing about such a simple design, you can actually sit down with pen and paper and verify operations.
- tverbeure 2y ago
  Forget about a digital twin, they don’t even have an assembler for the CPUs. It’s all hex values.
basementcat 2y ago
In short, the Voyager spacecraft were designed (after considerable design and operational experience from Mariner and Pioneer spacecraft) to be able to operate for a long time largely automatically without too much hand-holding. All major systems are multiply redundant and may be remotely turned on and off. While there has been personnel turnover on the program, it has not been of a magnitude to jeopardize program continuation. Finally, program management has been media savvy and well politically connected ensuring that operations are still funded. (Contrast with other missions such as Magellan to Venus which was deorbited while it still had propellant reserve leaving some portions of the planet unmapped) JPL has (or had?) a "retiree badge" program that permitted retired staff to continue to access their office. Many programs benefited from highly knowledgeable personnel essentially continuing to report to the office every day without pay (not being paid comes with the luxury of not having to worry about being laid off if your charge accounts don’t have enough funds!) It was an absolute privilege to learn from these people.
- SSLy 2y ago
  So, culture preservation is important for success of highly technical endeavours. Don't tell it to your run of the mill MBA.
  - basementcat 2y ago
    I think it is inevitable that organizational "culture" changes. The tricky part is figuring out exactly what parts need change and what parts don’t. For example JPL used to have beauty pageants ("Miss Guided Missile"). More recently, management appears to be trying to adopt policies and procedures from some venture funded commercial space companies. It is not clear how helpful these efforts will be given that these organizations are in fundamentally different businesses. https://www.jpl.nasa.gov/images/slice-of-history-70th-anniversary-of-miss-guided-missile https://www.jpl.nasa.gov/images/slice-of-history-70th-annive...
    - fnord77 2y ago
      [flagged]
olabyne 2y ago
And quite recently they also recovered from a faulty ram module, with a radio link speed that is maybe below the kbit/s now !
- deleted 2y ago
  [deleted]
- lisper 2y ago
  And you should see the ping times.
- liamwire 2y ago
  I suspect it’s far lower than that.
- stordoff 2y ago
  > Uplink communications is via S-band (16-bits/sec command rate) while an X-band transmitter provides downlink telemetry at 160 bits/sec normally and 1.4 kbps for playback of high-rate plasma wave data. https://science.nasa.gov/mission/voyager/spacecraft/ https://science.nasa.gov/mission/voyager/spacecraft/ The data playbacks were initially transmitted at 7200 bps, but this was dropped to 1400 bps in 2007 (https://voyager.gsfc.nasa.gov/Library/DeepCommo_Chapter3--141029.pdf https://voyager.gsfc.nasa.gov/Library/DeepCommo_Chapter3--14... - page 72). RTT ~46 hours (one-way light time 22:49:59, per https://science.nasa.gov/mission/voyager/where-are-they-now/ https://science.nasa.gov/mission/voyager/where-are-they-now/)
avidphantasm 2y ago
> It's not like they have a local test device to experiment on is it? I would bet they have one or more simulators (“digital twin” in the parlance of our times). I’d want one simulator to always reflect the current state of the probe (with state data assimilated periodically from the real probe). Then other simulators can be used to test management changes to see how the system might respond.
- onedognight 2y ago
  > I would bet they have one or more simulators Sadly they do not, but they are starting to write one.
ForHackernews 2y ago
There's a documentary "It's Quieter in the Twilight" about the team keeping the Voyager missions alive: https://www.youtube.com/watch?v=8vJT8AW0wYw https://www.youtube.com/watch?v=8vJT8AW0wYw
- ornornor 2y ago
  Looks very interesting , thanks for sharing
- whycome 2y ago
  Included with Amazon Prime in Canada
tverbeure 2y ago
You should totally check out the talk from a few weeks ago that I link to in my other comment. It answers your 3 questions and more.

japanuspus 2y ago

The Voyager mission is such a wild achievement. Both the sublime design and craftsmanship that must have gone into the hardware, and the deep institutional knowledge required to keep it running is awe-inspiring.

litoE 2y ago

It's been operating for 47 years and it still has fuel left to make attitude corrections. I wonder how they managed that feat.

turblety 2y ago
I presume solar power?
- Tor3 2y ago
  No, way too little sun out here. The sun is just a dot far away. The Voyagers both use RTGs, radioisotope thermoelectric generators. Decaying plutonium, essentially. That's for the electrically powered systems. The thrusters use liquid hydrazine, which is common for those kind of thrusters. Edit: There's more about that in the NASA link in a sibling comment.
- krige 2y ago
  Solar has nothing to do with it. Voyager uses hydrazine, of which over 80% has been used up over the years. They simply use not that much of it as it's not for thrust, but for aiming at Earth more or less.
- meindnoch 2y ago
  Lol. How would that work exactly?
  - JumpCrisscross 2y ago
    > How would that work exactly? Light sails. Laser propulsion. Photon rockets. None of which apply to Voyager. But none of which are laughably dismissible.
- gpderetta 2y ago
  Even with something like an ion engine (which I'm not sure were available when Voyager was launched), you would need leftover reaction mass.
2y ago

NaOH 2y ago

Probably a better link: https://science.nasa.gov/missions/voyager-program/voyager-1/voyager-1-team-accomplishes-tricky-thruster-swap/ https://science.nasa.gov/missions/voyager-program/voyager-1/...

perihelions 2y ago

- "A fuel tube inside the thrusters has filled up with silicon dioxide, a side effect of age within the spacecraft’s fuel tank." Where the heck do you get SiO2 from on a spacecraft? Some kind of silicone? edit: "clogged with silicon dioxide, a byproduct that appears with age from a rubber diaphragm in the spacecraft’s fuel tank"[0] —I'm guessing that is a silicone rubber. I didn't know that rubber can decompose into sand. [0] https://science.nasa.gov/missions/voyager-program/voyager-1/voyager-1-team-accomplishes-tricky-thruster-swap/ https://science.nasa.gov/missions/voyager-program/voyager-1/...

tecleandor 2y ago
Seems like silicone rubber is extracted from SiO2, so maybe it's impurities? Or maybe the Si and O are doing something to leave the polymer? (I have no idea if that last thing is even possible) https://en.wikipedia.org/wiki/Silicone_rubber https://en.wikipedia.org/wiki/Silicone_rubber
AStonesThrow 2y ago
[flagged]
numpad0 2y ago
They expel the Hydrazine(N2H4) fuel out of a spherical Ti tank by inflating a rubber balloon that involve Teflon inside the tank using helium supply. I guess N2H4 was potent enough to degrade even those space age materials. 1: https://ntrs.nasa.gov/api/citations/19810001583/downloads/19810001583.pdf https://ntrs.nasa.gov/api/citations/19810001583/downloads/19...
shmeeed 2y ago
It doesn't have to be silicone. SiO2 is a necessary ingredient in all rubber compounds.

onewheeltom 2y ago

Just another day at the office for the Voyager 1 team. Wow.

Buttons840 2y ago

I've heard good things about some Voyager documentaries, and I've wanted to watch one with my daughter, but NASA keeps making the documentaries out-of-date and incomplete. How many amazing stories will there be to tell by the time Voyager is truly beyond our knowledge?

Something1234 2y ago
You mention voyager documentaries but fail to mention which one. There's lots, and its impossible to know which one is good until you finish it. Please tell us which one you wanted to watch. It's sad how much documentation is honestly unreachable now.

Twisol 2y ago

I had the extreme pleasure of seeing Bruce Waggoner of the Voyager team give a keynote at !!Con just last month. The recording landed on YouTube just a couple days ago, so this is great timing: https://www.youtube.com/watch?v=dF_9YcehCZo https://www.youtube.com/watch?v=dF_9YcehCZo

tverbeure 2y ago

2 weeks ago, Bruce Wagoner from the Voyager program gave a talk at !!Con about how they recovered from the CMOS memory issue that they had a year ago. It’s basically blind debugging with a latency of 45 hours. The talk is amazing and goes through the computer architecture of the spacecraft as well as the challenges of dealing with something that is so old, with so documentation that has conflicting information or unreadable etc. https://youtu.be/dF_9YcehCZo?si=W_b3NJ7vgxaYS1__ https://youtu.be/dF_9YcehCZo?si=W_b3NJ7vgxaYS1__

ChrisArchitect 2y ago

Official release: https://science.nasa.gov/missions/voyager-program/voyager-1/voyager-1-team-accomplishes-tricky-thruster-swap/ https://science.nasa.gov/missions/voyager-program/voyager-1/... (https://news.ycombinator.com/item?id=41505008 https://news.ycombinator.com/item?id=41505008)