8 ms·

The Loyal Engineers Steering NASA’s Voyager Probes Across the Universe

307 points by gaius 9y ago

bitexploder 9y ago
There is so much cool stuff in this article. The engineers stuck in the 70s with modern tools to aid them, but that only gets them so far. The love and dedication to this spacecraft. These engineers believe in the mission of space so deeply they married themselves to Voyager. A marriage more substantial than many real marriages. Just thinking about this dedication and how space exploration is, in my opinion, one of the few ways we can truly advance as a species beyond our ruts here on Earth gave me chills. There is something, existential, poetic, and sad about the lonely work these engineers have done for so long. If you didn't read this article the portraits of the engineers and snapshots of the lives is moving.
- rootbear 9y ago
  I love the Voyagers and I'm very happy that there are people dedicated to keeping us in contact with them as long as they last. These Engineers are splendid examples of taking the long view, something we could use more of these days.
  - wjnc 9y ago
    For me the article has a somewhat sad component (next to technological awesomeness): "In retirement, Zottarelli told me, he would like to see Florida again. He wonders how it has changed. In his garage is a 1954 Swallow Doretti, a fixer-upper. ‘‘It probably needs new brakes,’’ he said. I asked him if there was anywhere he liked to drive for fun. ‘‘No,’’ he replied. ‘‘Not anymore.’’" Imagine that's your grandfather. With his expertise, shouldn't he be able to 'see Florida twice a year'? I would have wished for a way to have both this awesome technological specialization over long stretches of time, but also given the people on the program some more time to go their own ways. And then the toll of being in the same hierarchy for 30+ years. Ack. But perhaps these people self-selected for stoicism, who am I to judge ;)
    - secstate 9y ago
      That's a really well made point. And for what it's worth, stoicism doesn't ask you to never travel or have fun :) The most depressing line for me was him effectively waiting (albeit in a joking manner) for his second stroke. No one should be worked to the point of fatalism. On the one hand I love the team's dedication. On the other hand, I don't think it's just competitive pay that keeps young engineers out of that office. I imagine that the energy and excitement in the early 80s does not really exist anymore, and it's not fair to dump on young engineers for wanting that energy where they work.
      - mturmon 9y ago
        The younger engineers ar JPL might rather work on Mars 2020: https://www.jpl.nasa.gov/missions/mars-2020/ https://www.jpl.nasa.gov/missions/mars-2020/ One of the proposed instruments is a 1kg helicopter. * They have tried to build a pipeline with the Mars exploration missions, in which senior people can move from mission to mission, and younger people can train up on older, mature missions before they move to newer ones. This pipeline did not really exist with the Voyagers.
rodgerd 9y ago
I recommend seeing The Farthest (https://www.youtube.com/watch?v=znTdk_de_K8 https://www.youtube.com/watch?v=znTdk_de_K8) if you have any interest at all in this.
- pat_space 9y ago
  Wow, I got goosebumps from the trailer. Thanks so much for the recommendation.
  - mitchdoogle 9y ago
    Looks like it will air on PBS on August 23. http://www.pbs.org/the-farthest/home/ http://www.pbs.org/the-farthest/home/
- mtm 9y ago
  I'd also highly recommend "NASA Voyager 1 & 2 Owners' Workshop Manual" https://smile.amazon.com/gp/product/0857337750 https://smile.amazon.com/gp/product/0857337750 It goes into amazing detail on the history of the spacecraft and the events that made it possible. For example: the story of an intern figuring out the '3 body' problem and, from there, the concept of 'gravity assist' gave me goosebumps
laydn 9y ago
You know, getting these electronics systems ready, sending out to deep space and communicating with these systems is an incredible feat. But, for me, another amazing thing about these long term projects is the sheer amount of knowledge that needs to be maintained and passed on to new people, over the course of several decades. To pull it off, this need of knowledge management and transfer must be so deeply engraved in the culture of the organization. How do they do it in this day and age of "FAQs", "Forums" and "Helpdesks"? :) In all seriousness, what do you do to maintain "knowledge" in your organization?
- bitexploder 9y ago
  This is a good insight. Having read about NASA's culture some from books like "An Astronaut's Guide to Life on Earth", they tend to be very dedicated to documenting everything. Every last little detail goes into handbooks or manuals about how they fixed problems or things that could have gotten someone killed, adapting procedures to take into account even the smallest of practical details. These engineers come from a different era. You can see all the printed binders on their desks. I think in many ways the process of documentation for space missions is more expensive, pain staking, and solid than modern practices. I think also they do have the advantage of all working in the same place. You just go to Tom, the human search engine/librarian, and ask him what he knows about thing X. (The article mentioned Tom was the self styled "librarian" of the project). Most organizations need a "Tom" of their own and are poorer for not having one. Our company is building a remote team, and similar to Gitlab, we are discovering that staying in sync is not just a good value but a way of life. Trello, wiki, and code help us repeat our assessment work and ensure everyone is on the same page.
  - marsRoverDev 9y ago
    To give some kind of context on more "modern" methods, we use stupendous numbers of PDFs, DOORS for requirements traceability and tons of code commenting. At last count our project had 0.73 commenting lines per line of code (and we use Allman style bracketing). We are also putting non mission-critical information in a Wiki; basically as a tl;dr of the more dense documentation.
danesparza 9y ago
This article is pretty dang cool. Looking over the pictures of the NASA engineers I couldn't help but think "Damn, that engineer looks TIRED". Did anybody else think the same thing?
yourapostasy 9y ago
> ...[the heliosphere] blocks 75 percent of cosmic radiation... It sounds like a solar system-scale Van Allen belt [1] when described like this. So we have at least three layers of cosmic radiation protection identified (heliosphere, outer VAB, inner VAB). If the heliosphere has implications for high-precision, high-accuracy independent-of-Earth interplanetary spaceflight similar to gravimetric and magnetic readings on Earth do for submarines today, then mapping the heliosphere would be a future asset. [1] https://en.wikipedia.org/wiki/Van_Allen_radiation_belt https://en.wikipedia.org/wiki/Van_Allen_radiation_belt
- deleted 9y ago
  [deleted]
sehugg 9y ago
... its oscillator, which allows it to accept a wide range of frequencies, had quit, essentially shrinking the target for transmissions from Earth. Assuming a much narrower bandwidth, and manually subtracting the Doppler effect, they recalibrated their signal. It worked — but to this day, the same calculation must precede every command. IIRC from an old SciAm article, the temperature of the spacecraft's electronics is also taken into account when doing this calculation.
- pmcjones 9y ago
  Yes, the article later says "(On Voyager 2, because of the broken oscillator, any change in temperature also tweaks the receiver frequency.)"
rootbear 9y ago
A lot of what has been said here about software documentation and process is true of NASA flight projects, especially manned. Outside of flight, not so much. In some cases, it's a bit more like grad-school-project-that-escaped. An exception would be the modeling code used for things like weather simulation on supercomputers. The scientists need to know that their code actually matches the model they're testing and that things like numerical precision and error bounds are being handled properly. I'll add that the above is anecdotal, based on my experiences at NASA Goddard.
- mynameisnobody 9y ago
  Yup. I worked in a satellite data office of NASA and there was no formal documentation process. Much of the code running in the facility had been written by a single developer over 20 years, and all of the knowledge left with him. Masterful Perl can be frustrating stuff without documentation or comments.
odammit 9y ago
I didn't know Sammy Hagar was moon-lighting at NASA. Very impressive.
dingo_bat 9y ago
How do you retain such talented people for such a long time? These people could get jobs in the most cutting-edge company in their respective fields. Still they choose to stay at the same place for decades.
- rhombocombus 9y ago
  By having them work on one of humanity's most important exploration projects ever. I get the impression that a lot of these folks are big picture types and that it would take a lot more than a pay raise to drag them away from their life's work.
- bitexploder 9y ago
  It is the love of the mission. The knowledge that you are advancing humanity in this concrete unique way, and that due to circumstance, if you quit it has a tangible increase in the chance of failure for this mission. It is how places like the NSA and other organizations within the government can retain such talented people. It is why PhD scientists toil in their field for much less than they could make in the public sector. It is why people choose to work at a lifestyle company making less than their counterparts at tech giants. A love of the mission/work and or a belief in the organization and its overall goals. See: Daniel Pink and the whole autonomy, mastery, and purpose idea. This mission checks those three boxes quite strongly, likely captivating the engineers and folks on the mission :)
- Merad 9y ago
  It's a combination of things. Many (most?) of the people want to be there and are passionate about what they're doing. Financially it's still a great job by normal standards and once you have tenure (I think they may call it something else) it's an extremely secure job. I have a friend who works there - brilliant guy who went to school for physics but is also very talented at low level embedded systems type development. When he graduated with a physics masters he turned down a six figure offer to start at NASA at around $60k. From what he's told me though, he's essentially guaranteed to hit the six figure mark within 6-8 years. Last time we spoke he had just started his PhD (paid by NASA) and was working on the Mars 2020 rover.
rbanffy 9y ago
In the end, the engineer's prayer is probably a lot like the astronaut's. "Please, dear God, don't let me fuck up."
CocoaGeek 9y ago
Love the Sun workstation on Enrique Medina's desk 8)
- FireBeyond 9y ago
  At the tender age of 39, I vividly remember my first IT job, for a bus company of all things, where at age 20 I had a Sparc 5 on my desk. I still love those Sun keyboards. (We used it to solve for the most efficient driver/ bus scheduling).
mrunkel 9y ago
I read far too far into the article before I realized it was the mobile site. :) In the mid 1980's I went on a tour of JPL in Pasadena and actually saw the computers that were (at the time) in charge of recording and storing the telemetry data. I'm vague on the exact details, but apparently the computers were donated from the US Army and were field models (early "portable" computers) and they operated on 48V DC power. So not only were the computers themselves large fridge sized units, they had near matching transformers that were fairly unreliable. I recall reading that in the mid 90s NASA replaced all the mission control systems for the space shuttle with a single Sun Workstation, so I assume that JPL also at some point replaced the downlink computers for V'ger. My hat is off to these many fine people who stuck it out with jobs that were probably long periods of drudgery interspersed with moments of sheer terror.
nlh 9y ago
One thing I've always been amazed at when reading about spacecraft is the incredibly detailed level of control the scientists have over the craft. It's something I'm not used to here on earth -- when something breaks, I've learned you either crack it open and replace the part or buy a new device :) Can someone talk (or provide a link) about how this sort of system design works?

307 points by gaius 9y ago

There is so much cool stuff in this article. The engineers stuck in the 70s with modern tools to aid them, but that only gets them so far. The love and dedication to this spacecraft. These engineers believe in the mission of space so deeply they married themselves to Voyager. A marriage more substantial than many real marriages. Just thinking about this dedication and how space exploration is, in my opinion, one of the few ways we can truly advance as a species beyond our ruts here on Earth gave me chills. There is something, existential, poetic, and sad about the lonely work these engineers have done for so long. If you didn't read this article the portraits of the engineers and snapshots of the lives is moving.

rootbear 9y ago
I love the Voyagers and I'm very happy that there are people dedicated to keeping us in contact with them as long as they last. These Engineers are splendid examples of taking the long view, something we could use more of these days.
- wjnc 9y ago
  For me the article has a somewhat sad component (next to technological awesomeness): "In retirement, Zottarelli told me, he would like to see Florida again. He wonders how it has changed. In his garage is a 1954 Swallow Doretti, a fixer-upper. ‘‘It probably needs new brakes,’’ he said. I asked him if there was anywhere he liked to drive for fun. ‘‘No,’’ he replied. ‘‘Not anymore.’’" Imagine that's your grandfather. With his expertise, shouldn't he be able to 'see Florida twice a year'? I would have wished for a way to have both this awesome technological specialization over long stretches of time, but also given the people on the program some more time to go their own ways. And then the toll of being in the same hierarchy for 30+ years. Ack. But perhaps these people self-selected for stoicism, who am I to judge ;)
  - secstate 9y ago
    That's a really well made point. And for what it's worth, stoicism doesn't ask you to never travel or have fun :) The most depressing line for me was him effectively waiting (albeit in a joking manner) for his second stroke. No one should be worked to the point of fatalism. On the one hand I love the team's dedication. On the other hand, I don't think it's just competitive pay that keeps young engineers out of that office. I imagine that the energy and excitement in the early 80s does not really exist anymore, and it's not fair to dump on young engineers for wanting that energy where they work.
    - mturmon 9y ago
      The younger engineers ar JPL might rather work on Mars 2020: https://www.jpl.nasa.gov/missions/mars-2020/ https://www.jpl.nasa.gov/missions/mars-2020/ One of the proposed instruments is a 1kg helicopter. * They have tried to build a pipeline with the Mars exploration missions, in which senior people can move from mission to mission, and younger people can train up on older, mature missions before they move to newer ones. This pipeline did not really exist with the Voyagers.

rodgerd 9y ago

I recommend seeing The Farthest (https://www.youtube.com/watch?v=znTdk_de_K8 https://www.youtube.com/watch?v=znTdk_de_K8) if you have any interest at all in this.

pat_space 9y ago
Wow, I got goosebumps from the trailer. Thanks so much for the recommendation.
- mitchdoogle 9y ago
  Looks like it will air on PBS on August 23. http://www.pbs.org/the-farthest/home/ http://www.pbs.org/the-farthest/home/
mtm 9y ago
I'd also highly recommend "NASA Voyager 1 & 2 Owners' Workshop Manual" https://smile.amazon.com/gp/product/0857337750 https://smile.amazon.com/gp/product/0857337750 It goes into amazing detail on the history of the spacecraft and the events that made it possible. For example: the story of an intern figuring out the '3 body' problem and, from there, the concept of 'gravity assist' gave me goosebumps

laydn 9y ago

You know, getting these electronics systems ready, sending out to deep space and communicating with these systems is an incredible feat. But, for me, another amazing thing about these long term projects is the sheer amount of knowledge that needs to be maintained and passed on to new people, over the course of several decades. To pull it off, this need of knowledge management and transfer must be so deeply engraved in the culture of the organization. How do they do it in this day and age of "FAQs", "Forums" and "Helpdesks"? :) In all seriousness, what do you do to maintain "knowledge" in your organization?

bitexploder 9y ago
This is a good insight. Having read about NASA's culture some from books like "An Astronaut's Guide to Life on Earth", they tend to be very dedicated to documenting everything. Every last little detail goes into handbooks or manuals about how they fixed problems or things that could have gotten someone killed, adapting procedures to take into account even the smallest of practical details. These engineers come from a different era. You can see all the printed binders on their desks. I think in many ways the process of documentation for space missions is more expensive, pain staking, and solid than modern practices. I think also they do have the advantage of all working in the same place. You just go to Tom, the human search engine/librarian, and ask him what he knows about thing X. (The article mentioned Tom was the self styled "librarian" of the project). Most organizations need a "Tom" of their own and are poorer for not having one. Our company is building a remote team, and similar to Gitlab, we are discovering that staying in sync is not just a good value but a way of life. Trello, wiki, and code help us repeat our assessment work and ensure everyone is on the same page.
- marsRoverDev 9y ago
  To give some kind of context on more "modern" methods, we use stupendous numbers of PDFs, DOORS for requirements traceability and tons of code commenting. At last count our project had 0.73 commenting lines per line of code (and we use Allman style bracketing). We are also putting non mission-critical information in a Wiki; basically as a tl;dr of the more dense documentation.

danesparza 9y ago

This article is pretty dang cool. Looking over the pictures of the NASA engineers I couldn't help but think "Damn, that engineer looks TIRED". Did anybody else think the same thing?

yourapostasy 9y ago

> ...[the heliosphere] blocks 75 percent of cosmic radiation... It sounds like a solar system-scale Van Allen belt [1] when described like this. So we have at least three layers of cosmic radiation protection identified (heliosphere, outer VAB, inner VAB). If the heliosphere has implications for high-precision, high-accuracy independent-of-Earth interplanetary spaceflight similar to gravimetric and magnetic readings on Earth do for submarines today, then mapping the heliosphere would be a future asset. [1] https://en.wikipedia.org/wiki/Van_Allen_radiation_belt https://en.wikipedia.org/wiki/Van_Allen_radiation_belt

deleted 9y ago
[deleted]

sehugg 9y ago

... its oscillator, which allows it to accept a wide range of frequencies, had quit, essentially shrinking the target for transmissions from Earth. Assuming a much narrower bandwidth, and manually subtracting the Doppler effect, they recalibrated their signal. It worked — but to this day, the same calculation must precede every command. IIRC from an old SciAm article, the temperature of the spacecraft's electronics is also taken into account when doing this calculation.

pmcjones 9y ago
Yes, the article later says "(On Voyager 2, because of the broken oscillator, any change in temperature also tweaks the receiver frequency.)"

rootbear 9y ago

A lot of what has been said here about software documentation and process is true of NASA flight projects, especially manned. Outside of flight, not so much. In some cases, it's a bit more like grad-school-project-that-escaped. An exception would be the modeling code used for things like weather simulation on supercomputers. The scientists need to know that their code actually matches the model they're testing and that things like numerical precision and error bounds are being handled properly. I'll add that the above is anecdotal, based on my experiences at NASA Goddard.

mynameisnobody 9y ago
Yup. I worked in a satellite data office of NASA and there was no formal documentation process. Much of the code running in the facility had been written by a single developer over 20 years, and all of the knowledge left with him. Masterful Perl can be frustrating stuff without documentation or comments.

odammit 9y ago

I didn't know Sammy Hagar was moon-lighting at NASA. Very impressive.

dingo_bat 9y ago

How do you retain such talented people for such a long time? These people could get jobs in the most cutting-edge company in their respective fields. Still they choose to stay at the same place for decades.

rhombocombus 9y ago
By having them work on one of humanity's most important exploration projects ever. I get the impression that a lot of these folks are big picture types and that it would take a lot more than a pay raise to drag them away from their life's work.
bitexploder 9y ago
It is the love of the mission. The knowledge that you are advancing humanity in this concrete unique way, and that due to circumstance, if you quit it has a tangible increase in the chance of failure for this mission. It is how places like the NSA and other organizations within the government can retain such talented people. It is why PhD scientists toil in their field for much less than they could make in the public sector. It is why people choose to work at a lifestyle company making less than their counterparts at tech giants. A love of the mission/work and or a belief in the organization and its overall goals. See: Daniel Pink and the whole autonomy, mastery, and purpose idea. This mission checks those three boxes quite strongly, likely captivating the engineers and folks on the mission :)
Merad 9y ago
It's a combination of things. Many (most?) of the people want to be there and are passionate about what they're doing. Financially it's still a great job by normal standards and once you have tenure (I think they may call it something else) it's an extremely secure job. I have a friend who works there - brilliant guy who went to school for physics but is also very talented at low level embedded systems type development. When he graduated with a physics masters he turned down a six figure offer to start at NASA at around $60k. From what he's told me though, he's essentially guaranteed to hit the six figure mark within 6-8 years. Last time we spoke he had just started his PhD (paid by NASA) and was working on the Mars 2020 rover.

rbanffy 9y ago

In the end, the engineer's prayer is probably a lot like the astronaut's. "Please, dear God, don't let me fuck up."

CocoaGeek 9y ago

Love the Sun workstation on Enrique Medina's desk 8)

FireBeyond 9y ago
At the tender age of 39, I vividly remember my first IT job, for a bus company of all things, where at age 20 I had a Sparc 5 on my desk. I still love those Sun keyboards. (We used it to solve for the most efficient driver/ bus scheduling).

mrunkel 9y ago

I read far too far into the article before I realized it was the mobile site. :) In the mid 1980's I went on a tour of JPL in Pasadena and actually saw the computers that were (at the time) in charge of recording and storing the telemetry data. I'm vague on the exact details, but apparently the computers were donated from the US Army and were field models (early "portable" computers) and they operated on 48V DC power. So not only were the computers themselves large fridge sized units, they had near matching transformers that were fairly unreliable. I recall reading that in the mid 90s NASA replaced all the mission control systems for the space shuttle with a single Sun Workstation, so I assume that JPL also at some point replaced the downlink computers for V'ger. My hat is off to these many fine people who stuck it out with jobs that were probably long periods of drudgery interspersed with moments of sheer terror.

nlh 9y ago

One thing I've always been amazed at when reading about spacecraft is the incredibly detailed level of control the scientists have over the craft. It's something I'm not used to here on earth -- when something breaks, I've learned you either crack it open and replace the part or buy a new device :) Can someone talk (or provide a link) about how this sort of system design works?