12 ms·

Apollo 11 Had a Hidden Hero: Software

204 points by psim1 7y ago

spamizbad 7y ago
CuriousMarc's Apollo Guidance Computer (AGC) restoration is a must-watch: https://www.youtube.com/watch?v=2KSahAoOLdU&list=PL-_93BVApb59FWrLZfdlisi_x7-Ut_-w7 https://www.youtube.com/watch?v=2KSahAoOLdU&list=PL-_93BVApb...
- acomjean 7y ago
  I’ll second that. I particularly liked the part on home the memory worked
  - SpikeDad 7y ago
    I'm amazed at the technical skill and knowledge he and his team has. Fabricating simulations, replacement parts, unpotting modules and repairing them, etc, etc, etc. I thought the Xerox Alto restoration was amazing - this beats it handily.
johncalvinyoung 7y ago
I knew a retired couple near where I lived growing up who, I found out later, were software programmers on the Apollo Program. Quite possibly worked in that lab. The husband passed away soon afterwards, and I never got around to asking the wife about her experiences (they both worked for the program!). But it was really cool reading about how the core software was written.
criddell 7y ago
> MIT received the first major Apollo contract, the only one awarded to a university, and the only one given without competitive bidding. Do these kinds of mission-critical components generally go to the lowest bidder?
- ghaff 7y ago
  Competitive bid doesn't have to mean lowest bidder. I've put tons of (non-government) projects out for bid because you're usually expected to do so to keep everyone relatively honest. I imagine the MIT Instrumentation Lab were the only people with the inertial guidance expertise at that time.
  - benj111 7y ago
    "Competitive bid doesn't have to mean lowest bidder" No you have to wine and dine the right people, and line a job up for them when they retire.
    - Jtsummers 7y ago
      Which gets someone thrown in jail or a lot of fines these days. Not that it doesn't happen, but it's a lot harder to hide now than 20+ years ago.
      - benj111 7y ago
        I was perhaps being slightly misanthropic. But the game keepers always seem to turn poacher. High ranking military bods end up with defence contractors. Financial regulators end up with working at banks (and visa versa), and politicians seem to end up getting various well paid consulting jobs. I suppose you could argue there are good reasons for it, I cant believe on some level these public servants aren't thinking of their career after retirement.
- datalus 7y ago
  I think bidding government contracts is a lot more complicated than just picking the cheapest option. There's a lot of other dimensions of validation that bids move through other than cost. For something like this, I'm sure there is stringent engineering validation. I know some initial NN research was part of a larger government project of funding research around stabilizing missiles in flight for ICBMs that was not cheap.
- 7y ago
thomasjudge 7y ago
Love the code comments referenced in the article
playing_colours 7y ago
There is a good book on the computer systems of Apollo project: Digital Apollo https://mitpress.mit.edu/books/digital-apollo https://mitpress.mit.edu/books/digital-apollo It describes interesting questions that were faced on the project: say, how much control over a lunar module could be given to a computer system, and how much they should rely on human engagement.
- danbr 7y ago
  I read this book a couple years ago, it’s great. In fact, I picked up the book 2nd hand with a note from the author inscribed on a cover page. Neat. This book is especially interesting to anyone building systems where humans are required to interface with a computer on a “mission-critical” level. It goes in-depth about the new type of electrical component-integrated circuits used in the guidance system. And more about other aspects of the navigation systems used on Apollo. I recommend.
- bumby 7y ago
  It's interesting to me that 50+ years later this debate on system control is still very much relevant to aerospace today, like the recent 737-MAX issue.
  - bronson 7y ago
    There's not really any debate. The MAX ignored very basic controls lessons that were learned and institutionalized, as you say, over 50 years ago.
    - bumby 7y ago
      I agree that the 737-MAX wasn't a good example. My limited understanding in a broader sense (as opposed to the specific 737-MAX case) is there is still considerable debate on who should have ultimate control (human or software) of the aircraft when making critical decisions. I remember reading an article a while back that made it seem like Boeing and Airbus were on different sides of this spectrum in basic design philosophy, but I don't know how true that is.
mmoez 7y ago
Fastcompany published a similar article about the 4th crew member of Apollo 11 a few weeks ago. I shared it on HN but it did not get any traction. https://news.ycombinator.com/item?id=20173672 https://news.ycombinator.com/item?id=20173672
_eric 7y ago
Kind of off topic, but for those interested, this reminded me of the public repo on github which has the original guidance computer source code for the command and lunar modules[0]. [0] https://github.com/chrislgarry/Apollo-11 https://github.com/chrislgarry/Apollo-11
- retSava 7y ago
  Went looking for some smart-ass pull request (tongue in cheek) and was gladly surprised there were some! Alas, just proof reading changes and such.
a-wu 7y ago
I recently started listening to a fantastic podcast called 13 Minutes To The Moon[0] produced by the BBC. It’s a retelling of the last 13 minutes until the Apollo 11 lander touched down. Episode 2 talked about the software and the 1202 program code. They interviewed the Guidance controller who was only 26 at the time and talked about how the average age of the flight controllers was 27. I thought that was incredible, you don’t see that much responsibility placed in such junior people anymore. [0] https://www.bbc.co.uk/programmes/w13xttx2 https://www.bbc.co.uk/programmes/w13xttx2
- benj111 7y ago
  "you don’t see that much responsibility placed in such junior people anymore" Sure you do, typically in new industries, like internet companies today, or space in the 60s. Eventually the unicorns will be manned by aging suits, and JavaScript will be the new Cobol, and the hip young things will think vowels in names are actually kinda cool.
- Angostura 7y ago
  It's a really good podcast, but I found it a bit bloated in places, I could have done without the Hans Zimmer theme, of which they are so inordinately proud.
  - a-wu 7y ago
    That’s funny because I actually love the Hans Zimmer theme.
    - dereknewsom 7y ago
      I really loved the Hans Zimmer music too. I got pumped up every time I heard it. I felt some bloat at times when they really dug into that 1202 error. It just kept coming up again and again.
      - philjohn 7y ago
        Yes - it's bloated for a reason ... all of the things they've spent long episodes on are so you know what's going on when they do the realtime recordings of cap com.
jonshariat 7y ago
I've been listening to "13 minutes to the Moon" from the BBC recently. https://www.bbc.com/news/science-environment-48232627 https://www.bbc.com/news/science-environment-48232627 One part jumped out at me as brilliant especially for the time: when the computer was overloaded with tasks during the decent, it switched off all non-essential extras like screen displays in order to free up processing power. After that it shed even more to focus on only plotting the landing!
- zeristor 7y ago
  A few weeks back they said it was due to the delta h discrepancy, and Buzz did nothing wrong. This week they were saying that it was down to the docking rendezvous system being turned on, which had not been done before. I don't know if they came up with new information and forget to redo that episode. The delta-h discrepancy was a mismatch between the actual height above the moon, and what the navigation system expected, I believe Mission Control took over that. I feel like the programme murkied the waters.
EvanAnderson 7y ago
"The Charming Genius of the Apollo Guidance Computer" is a wonderful talk: https://www.youtube.com/watch?v=xY45YE7ggng https://www.youtube.com/watch?v=xY45YE7ggng The AGC implemented a virtual machine to run higher-level computational routines. The whole architecture is really interesting.
- rst 7y ago
  Eyles has a memoir out: info available here. (The title is "Sunburst and Luminary", which were the names for LM software builds for earth-orbit testing and lunar landings, respectively.) http://www.sunburstandluminary.com/SLhome.html http://www.sunburstandluminary.com/SLhome.html
fortran77 7y ago
It's nice to hear the Don Eyles story; I haven't heard it before, having only seen stories about Margaret Hamilton and her role.
elteto 7y ago
This contains more technical information behind the 1202 alarms [0]. It is an awesome read. [0] https://www.hq.nasa.gov/alsj/a11/a11.1201-pa.html https://www.hq.nasa.gov/alsj/a11/a11.1201-pa.html
- SmileyRedBall 7y ago
  > This contains more technical information behind the 1202 alarms .. “You also have to remember that, long before Bill Gates, we had developed a real-time multi-tasking operating system.” “When a job was to be scheduled, a call would be made to the appropriate executive routine - sort of like a DOS call today” edit: I thought this was hysterical, on so many levels, just imagine the scenario if Microsoft had got the contract for the AGC.
- sytelus 7y ago
  Relevant text: So what was happening during Apollo 11, as I recall, was that repeated jobs to process rendezvous radar data (that of course were not really there) were scheduled because a misconfiguration of the radar switches. Thus, the core sets got filled up and a 1202 alarm was generated. The 1201 that came later in the landing was because the scheduling request that caused the actual overflow was one that had requested a VAC area. Quite fascinating that these alarms automatically rebooted computer and picked up essential jobs exactly where they had left off.
mopsi 7y ago
This is the best description of AGC I have seen so far, from hardware design to mission software, layer by layer: "The Ultimate Apollo Guidance Computer Talk" https://www.youtube.com/watch?v=xx7Lfh5SKUQ https://www.youtube.com/watch?v=xx7Lfh5SKUQ
roboman 7y ago
Best quote in the excellent article: “You can’t get a degree in how to fly to the moon,” says Dana Densmore, who joined the lab in 1965 and became a control supervi-sor for the lunar-lander software. “You had to get people who know how to think, who are creative and alert. It was all in-vented on the spot.”
- dTal 7y ago
  She certainly sounds interesting. https://en.wikipedia.org/wiki/Cell_16 https://en.wikipedia.org/wiki/Cell_16
roboman 7y ago
Do yourself a favor: head to a newsstand or Starbucks and buy a copy of today’s WSJ. This special insert on the Apollo mission is well worth it.
roboman 7y ago
“You can’t get a degree in how to fly to the moon,” says Dana Densmore, who joined the lab in 1965 and became a control supervi-sor for the lunar-lander software. “You had to get people who know how to think, who are creative and alert. It was all in-vented on the spot.”
temptemptemp111 7y ago
I can't believe a bunch of professional engineers think we've traveled to the moon... It's mental. Live stream video in '69 from the moon? How can you be so daft?
vipbababd 7y ago
I knew a retired couple near where I lived growing up who, I found out later, were software programmers on the Apollo Program. Quite possibly worked in that lab. The husband passed away soon afterwards, and I never got around to asking the wife about her experiences (they both worked for the program!). But it was really cool reading about how the core software was written.
f_allwein 7y ago
> When it came to hiring women for engineering or management, NASA “had a few women, and they kept them hidden,” says Ms. Densmore. “At the lab it was very different,” and there were opportunities for women. Quite amazing what an important role women like Margaret Hamilton played in the Apollo project, given how hard it is nowadays to get more women into engineering positions. You may remember the famous picture of Hamilton next to the stack of code she and her team wrote for Apollo: see https://en.m.wikipedia.org/wiki/Margaret_Hamilton_(software_engineer) https://en.m.wikipedia.org/wiki/Margaret_Hamilton_(software_...
ozmaverick72 7y ago
The article mentions the overload on the computer was caused by "a mismatched power supply to a radar" triggering too many calculations. Any idea how or why the power supply was mismatched ? Luckily software saves the day but what was the root cause of the hardware fault ? Was that a case of a mismanaged change in configuration ?
- detaro 7y ago
  If I remember correctly, two AC power sources were required to be in phase for measurements of the antenna direction to work correctly, but weren't actually specced as such, and thus not built that way, and randomly ended up so out of sync that it caused problems on that flight. EDIT: this article has more detail https://arstechnica.com/science/2019/07/no-a-checklist-error-did-not-almost-derail-the-first-moon-landing/2/ https://arstechnica.com/science/2019/07/no-a-checklist-error...
mulmen 7y ago
Neat story but is the software really hidden? I feel like I have heard this story many times. It was one of the many feats of the Apollo program, I don't think I would consider it hidden.
- psim1 7y ago
  As a casual space program buff, I thought the same. But I think most people do not realize how sophisticated this early computer and software were, and how important to the mission. The article emphasizes the well-calculated trust that the engineers and astronauts put into this first computer-controlled spaceflight operation.
Zanneth 7y ago
If anyone wants to learn more about the Apollo Guidance Computer, this talk[1] from 34C3 will teach you everything you wanted to know and more. The AGC had some really fascinating features for the time, including a cooperative multitasking system, interrupt driven IO, and even a virtual machine for doing linear algebra! [1]https://youtu.be/xx7Lfh5SKUQ https://youtu.be/xx7Lfh5SKUQ
grandridge 7y ago
Yeah, racist software
dang 2y ago
https://archive.ph/Bcoe2 https://archive.ph/Bcoe2

204 points by psim1 7y ago

CuriousMarc's Apollo Guidance Computer (AGC) restoration is a must-watch: https://www.youtube.com/watch?v=2KSahAoOLdU&list=PL-_93BVApb59FWrLZfdlisi_x7-Ut_-w7 https://www.youtube.com/watch?v=2KSahAoOLdU&list=PL-_93BVApb...

acomjean 7y ago
I’ll second that. I particularly liked the part on home the memory worked
- SpikeDad 7y ago
  I'm amazed at the technical skill and knowledge he and his team has. Fabricating simulations, replacement parts, unpotting modules and repairing them, etc, etc, etc. I thought the Xerox Alto restoration was amazing - this beats it handily.

johncalvinyoung 7y ago

I knew a retired couple near where I lived growing up who, I found out later, were software programmers on the Apollo Program. Quite possibly worked in that lab. The husband passed away soon afterwards, and I never got around to asking the wife about her experiences (they both worked for the program!). But it was really cool reading about how the core software was written.

criddell 7y ago

> MIT received the first major Apollo contract, the only one awarded to a university, and the only one given without competitive bidding. Do these kinds of mission-critical components generally go to the lowest bidder?

ghaff 7y ago
Competitive bid doesn't have to mean lowest bidder. I've put tons of (non-government) projects out for bid because you're usually expected to do so to keep everyone relatively honest. I imagine the MIT Instrumentation Lab were the only people with the inertial guidance expertise at that time.
- benj111 7y ago
  "Competitive bid doesn't have to mean lowest bidder" No you have to wine and dine the right people, and line a job up for them when they retire.
  - Jtsummers 7y ago
    Which gets someone thrown in jail or a lot of fines these days. Not that it doesn't happen, but it's a lot harder to hide now than 20+ years ago.
    - benj111 7y ago
      I was perhaps being slightly misanthropic. But the game keepers always seem to turn poacher. High ranking military bods end up with defence contractors. Financial regulators end up with working at banks (and visa versa), and politicians seem to end up getting various well paid consulting jobs. I suppose you could argue there are good reasons for it, I cant believe on some level these public servants aren't thinking of their career after retirement.
datalus 7y ago
I think bidding government contracts is a lot more complicated than just picking the cheapest option. There's a lot of other dimensions of validation that bids move through other than cost. For something like this, I'm sure there is stringent engineering validation. I know some initial NN research was part of a larger government project of funding research around stabilizing missiles in flight for ICBMs that was not cheap.
7y ago

thomasjudge 7y ago

Love the code comments referenced in the article

playing_colours 7y ago

There is a good book on the computer systems of Apollo project: Digital Apollo https://mitpress.mit.edu/books/digital-apollo https://mitpress.mit.edu/books/digital-apollo It describes interesting questions that were faced on the project: say, how much control over a lunar module could be given to a computer system, and how much they should rely on human engagement.

danbr 7y ago
I read this book a couple years ago, it’s great. In fact, I picked up the book 2nd hand with a note from the author inscribed on a cover page. Neat. This book is especially interesting to anyone building systems where humans are required to interface with a computer on a “mission-critical” level. It goes in-depth about the new type of electrical component-integrated circuits used in the guidance system. And more about other aspects of the navigation systems used on Apollo. I recommend.
bumby 7y ago
It's interesting to me that 50+ years later this debate on system control is still very much relevant to aerospace today, like the recent 737-MAX issue.
- bronson 7y ago
  There's not really any debate. The MAX ignored very basic controls lessons that were learned and institutionalized, as you say, over 50 years ago.
  - bumby 7y ago
    I agree that the 737-MAX wasn't a good example. My limited understanding in a broader sense (as opposed to the specific 737-MAX case) is there is still considerable debate on who should have ultimate control (human or software) of the aircraft when making critical decisions. I remember reading an article a while back that made it seem like Boeing and Airbus were on different sides of this spectrum in basic design philosophy, but I don't know how true that is.

mmoez 7y ago

Fastcompany published a similar article about the 4th crew member of Apollo 11 a few weeks ago. I shared it on HN but it did not get any traction. https://news.ycombinator.com/item?id=20173672 https://news.ycombinator.com/item?id=20173672

_eric 7y ago

Kind of off topic, but for those interested, this reminded me of the public repo on github which has the original guidance computer source code for the command and lunar modules[0]. [0] https://github.com/chrislgarry/Apollo-11 https://github.com/chrislgarry/Apollo-11

retSava 7y ago
Went looking for some smart-ass pull request (tongue in cheek) and was gladly surprised there were some! Alas, just proof reading changes and such.

a-wu 7y ago

I recently started listening to a fantastic podcast called 13 Minutes To The Moon[0] produced by the BBC. It’s a retelling of the last 13 minutes until the Apollo 11 lander touched down. Episode 2 talked about the software and the 1202 program code. They interviewed the Guidance controller who was only 26 at the time and talked about how the average age of the flight controllers was 27. I thought that was incredible, you don’t see that much responsibility placed in such junior people anymore. [0] https://www.bbc.co.uk/programmes/w13xttx2 https://www.bbc.co.uk/programmes/w13xttx2

benj111 7y ago
"you don’t see that much responsibility placed in such junior people anymore" Sure you do, typically in new industries, like internet companies today, or space in the 60s. Eventually the unicorns will be manned by aging suits, and JavaScript will be the new Cobol, and the hip young things will think vowels in names are actually kinda cool.
Angostura 7y ago
It's a really good podcast, but I found it a bit bloated in places, I could have done without the Hans Zimmer theme, of which they are so inordinately proud.
- a-wu 7y ago
  That’s funny because I actually love the Hans Zimmer theme.
  - dereknewsom 7y ago
    I really loved the Hans Zimmer music too. I got pumped up every time I heard it. I felt some bloat at times when they really dug into that 1202 error. It just kept coming up again and again.
    - philjohn 7y ago
      Yes - it's bloated for a reason ... all of the things they've spent long episodes on are so you know what's going on when they do the realtime recordings of cap com.

jonshariat 7y ago

I've been listening to "13 minutes to the Moon" from the BBC recently. https://www.bbc.com/news/science-environment-48232627 https://www.bbc.com/news/science-environment-48232627 One part jumped out at me as brilliant especially for the time: when the computer was overloaded with tasks during the decent, it switched off all non-essential extras like screen displays in order to free up processing power. After that it shed even more to focus on only plotting the landing!

zeristor 7y ago
A few weeks back they said it was due to the delta h discrepancy, and Buzz did nothing wrong. This week they were saying that it was down to the docking rendezvous system being turned on, which had not been done before. I don't know if they came up with new information and forget to redo that episode. The delta-h discrepancy was a mismatch between the actual height above the moon, and what the navigation system expected, I believe Mission Control took over that. I feel like the programme murkied the waters.

EvanAnderson 7y ago

"The Charming Genius of the Apollo Guidance Computer" is a wonderful talk: https://www.youtube.com/watch?v=xY45YE7ggng https://www.youtube.com/watch?v=xY45YE7ggng The AGC implemented a virtual machine to run higher-level computational routines. The whole architecture is really interesting.

rst 7y ago
Eyles has a memoir out: info available here. (The title is "Sunburst and Luminary", which were the names for LM software builds for earth-orbit testing and lunar landings, respectively.) http://www.sunburstandluminary.com/SLhome.html http://www.sunburstandluminary.com/SLhome.html

fortran77 7y ago

It's nice to hear the Don Eyles story; I haven't heard it before, having only seen stories about Margaret Hamilton and her role.

elteto 7y ago

This contains more technical information behind the 1202 alarms [0]. It is an awesome read. [0] https://www.hq.nasa.gov/alsj/a11/a11.1201-pa.html https://www.hq.nasa.gov/alsj/a11/a11.1201-pa.html

SmileyRedBall 7y ago
> This contains more technical information behind the 1202 alarms .. “You also have to remember that, long before Bill Gates, we had developed a real-time multi-tasking operating system.” “When a job was to be scheduled, a call would be made to the appropriate executive routine - sort of like a DOS call today” edit: I thought this was hysterical, on so many levels, just imagine the scenario if Microsoft had got the contract for the AGC.
sytelus 7y ago
Relevant text: So what was happening during Apollo 11, as I recall, was that repeated jobs to process rendezvous radar data (that of course were not really there) were scheduled because a misconfiguration of the radar switches. Thus, the core sets got filled up and a 1202 alarm was generated. The 1201 that came later in the landing was because the scheduling request that caused the actual overflow was one that had requested a VAC area. Quite fascinating that these alarms automatically rebooted computer and picked up essential jobs exactly where they had left off.

mopsi 7y ago

This is the best description of AGC I have seen so far, from hardware design to mission software, layer by layer: "The Ultimate Apollo Guidance Computer Talk" https://www.youtube.com/watch?v=xx7Lfh5SKUQ https://www.youtube.com/watch?v=xx7Lfh5SKUQ

roboman 7y ago

Best quote in the excellent article: “You can’t get a degree in how to fly to the moon,” says Dana Densmore, who joined the lab in 1965 and became a control supervi-sor for the lunar-lander software. “You had to get people who know how to think, who are creative and alert. It was all in-vented on the spot.”

dTal 7y ago
She certainly sounds interesting. https://en.wikipedia.org/wiki/Cell_16 https://en.wikipedia.org/wiki/Cell_16

roboman 7y ago

Do yourself a favor: head to a newsstand or Starbucks and buy a copy of today’s WSJ. This special insert on the Apollo mission is well worth it.

roboman 7y ago

“You can’t get a degree in how to fly to the moon,” says Dana Densmore, who joined the lab in 1965 and became a control supervi-sor for the lunar-lander software. “You had to get people who know how to think, who are creative and alert. It was all in-vented on the spot.”

temptemptemp111 7y ago

I can't believe a bunch of professional engineers think we've traveled to the moon... It's mental. Live stream video in '69 from the moon? How can you be so daft?

vipbababd 7y ago

f_allwein 7y ago

> When it came to hiring women for engineering or management, NASA “had a few women, and they kept them hidden,” says Ms. Densmore. “At the lab it was very different,” and there were opportunities for women. Quite amazing what an important role women like Margaret Hamilton played in the Apollo project, given how hard it is nowadays to get more women into engineering positions. You may remember the famous picture of Hamilton next to the stack of code she and her team wrote for Apollo: see https://en.m.wikipedia.org/wiki/Margaret_Hamilton_(software_engineer) https://en.m.wikipedia.org/wiki/Margaret_Hamilton_(software_...

ozmaverick72 7y ago

The article mentions the overload on the computer was caused by "a mismatched power supply to a radar" triggering too many calculations. Any idea how or why the power supply was mismatched ? Luckily software saves the day but what was the root cause of the hardware fault ? Was that a case of a mismanaged change in configuration ?

detaro 7y ago
If I remember correctly, two AC power sources were required to be in phase for measurements of the antenna direction to work correctly, but weren't actually specced as such, and thus not built that way, and randomly ended up so out of sync that it caused problems on that flight. EDIT: this article has more detail https://arstechnica.com/science/2019/07/no-a-checklist-error-did-not-almost-derail-the-first-moon-landing/2/ https://arstechnica.com/science/2019/07/no-a-checklist-error...

mulmen 7y ago

Neat story but is the software really hidden? I feel like I have heard this story many times. It was one of the many feats of the Apollo program, I don't think I would consider it hidden.

psim1 7y ago
As a casual space program buff, I thought the same. But I think most people do not realize how sophisticated this early computer and software were, and how important to the mission. The article emphasizes the well-calculated trust that the engineers and astronauts put into this first computer-controlled spaceflight operation.

Zanneth 7y ago

If anyone wants to learn more about the Apollo Guidance Computer, this talk[1] from 34C3 will teach you everything you wanted to know and more. The AGC had some really fascinating features for the time, including a cooperative multitasking system, interrupt driven IO, and even a virtual machine for doing linear algebra! [1]https://youtu.be/xx7Lfh5SKUQ https://youtu.be/xx7Lfh5SKUQ

grandridge 7y ago

Yeah, racist software

dang 2y ago

https://archive.ph/Bcoe2 https://archive.ph/Bcoe2