2 ms·

Does anyone know how they now debug things in Space? If they are using compiled C programs it seems like it would be very hard to make changes. Is the key just

Does anyone know how they now debug things in Space? If they are using compiled C programs it seems like it would be very hard to make changes. Is the key just to not need to make any changes? Or is this a case of Greenspun's tenth rule, where they've built REPL like functionality on-top of C?

monocasa 5y ago
The cases I've seen deploy relatively full binaries, not patches. And they don't typically even have a REPL, as doing brain surgery on a satellite (that's already in a bad way if you're debugging something) is rarely the best option. So lots of telemetry logging, with 'feature flags' to turn on different logs. Then a change and code push once root cause is found, or a new build with new telemetry gathering if it's still not clear. I'm sure just about every model for code structure has someone doing it though.
- adamgordonbell 5y ago
  Ok cool, I guess you can send enough data with earth satellites to resend a full binary.
ncmncm 5y ago
Probably the best "debugging in space" story is Don Eyles hacking the Apollo moon lander program that was in "core rope ROM" to work around a hardware failure on Apollo 14. It is findable online, but he published a book "Sunburst and Luminary" (Fort Point Press, 2019) about the whole process of getting the moon landing code ready in time to use, and the hack. As I understand it (I just got the book, haven't read it yet) the Apollo Guidance Computer, one each in the Command Module and the LEM, was programmed by Margaret Hamilton (inventor of Software Engineering as we know it today) with a real-time executive and interpreter emulating a saner machine, and Eyles coded the landing to the interpreter. Because it was interpreted code, it was patchable, and he came up with a patch on the fly that the astronauts punched into the AGC by hand, and saved the mission. Margaret Hamilton's real-time executive itself saved the day when Apollo 11 crew left some extra stuff turned on, by accident, that the system had not been tested with and that burned excess CPU cycles during the landing. When it trapped a scheduling failure, it checkpointed important state and was able to resume the important tasks where they left off. That happened several times during the landing.
- adamgordonbell 5y ago
  Looks good. Ordering. https://www.goodreads.com/book/show/39294745-sunburst-and-luminary---an-apollo-memoir?from_search=true&from_srp=true&qid=ahDP1l1b58&rank=1 https://www.goodreads.com/book/show/39294745-sunburst-and-lu...
  - ncmncm 5y ago
    I usually order my books from Bookshop.org (never Amazon). I ordered this one directly from https://www.sunburstandluminary.com/SLhome.html https://www.sunburstandluminary.com/SLhome.html and it showed up very quickly with a minimum of fuss.
- ncmncm 5y ago
  Correction: the interpreter was coded by Charlie Muntz.

monocasa 5y ago

The cases I've seen deploy relatively full binaries, not patches. And they don't typically even have a REPL, as doing brain surgery on a satellite (that's already in a bad way if you're debugging something) is rarely the best option. So lots of telemetry logging, with 'feature flags' to turn on different logs. Then a change and code push once root cause is found, or a new build with new telemetry gathering if it's still not clear. I'm sure just about every model for code structure has someone doing it though.

adamgordonbell 5y ago
Ok cool, I guess you can send enough data with earth satellites to resend a full binary.

ncmncm 5y ago

Probably the best "debugging in space" story is Don Eyles hacking the Apollo moon lander program that was in "core rope ROM" to work around a hardware failure on Apollo 14. It is findable online, but he published a book "Sunburst and Luminary" (Fort Point Press, 2019) about the whole process of getting the moon landing code ready in time to use, and the hack. As I understand it (I just got the book, haven't read it yet) the Apollo Guidance Computer, one each in the Command Module and the LEM, was programmed by Margaret Hamilton (inventor of Software Engineering as we know it today) with a real-time executive and interpreter emulating a saner machine, and Eyles coded the landing to the interpreter. Because it was interpreted code, it was patchable, and he came up with a patch on the fly that the astronauts punched into the AGC by hand, and saved the mission. Margaret Hamilton's real-time executive itself saved the day when Apollo 11 crew left some extra stuff turned on, by accident, that the system had not been tested with and that burned excess CPU cycles during the landing. When it trapped a scheduling failure, it checkpointed important state and was able to resume the important tasks where they left off. That happened several times during the landing.

adamgordonbell 5y ago
Looks good. Ordering. https://www.goodreads.com/book/show/39294745-sunburst-and-luminary---an-apollo-memoir?from_search=true&from_srp=true&qid=ahDP1l1b58&rank=1 https://www.goodreads.com/book/show/39294745-sunburst-and-lu...
- ncmncm 5y ago
  I usually order my books from Bookshop.org (never Amazon). I ordered this one directly from https://www.sunburstandluminary.com/SLhome.html https://www.sunburstandluminary.com/SLhome.html and it showed up very quickly with a minimum of fuss.
ncmncm 5y ago
Correction: the interpreter was coded by Charlie Muntz.