6 ms·

Lessons Learned Developing Software for Space Vehicles

117 points by someperson 14y ago

rurounijones 14y ago
I would love to see a more in depth article about what kind of programming you need to do for a system controlling a rocket. I remember reading that they cannot use exceptions and have to handle every possible scenario. Part of me wonders how much of the source code is just error handling.
- w3pm 14y ago
  No dynamic memory allocation, no recursion, no longjmp, etc. Program execution must be entirely predictable, every time. SpaceX's programming rules are probably very similar to those used at NASA JPL: http://lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf http://lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf
  - paulrademacher 14y ago
    That's just the tiniest tip of the iceberg. Doesn't get into formal specs, verification, validation, etc. This article is often referenced for NASA's software practices, although it's now almost 20 years old: http://www.fastcompany.com/28121/they-write-right-stuff http://www.fastcompany.com/28121/they-write-right-stuff Is there anything more comprehensive and current than that?
- loeg 14y ago
  "Just" error handling? That's the hard part!
  - rurounijones 14y ago
    I didn't mean "just" in that sense, I mean "just" as in "solely dedicated to". Most apps have error handling code taking up what? 20-30% of the code at the most? For some reason I imagine this stuff being almost 70% error handling but I am not sure if that is realistic. So, assuming there are shed loads of error handling that knocks on to everything else about the system. How do they test it? How do they spec out all the possible conditions? What do they do differently to someone writing a SaaS, Desktop app or IOS app? It looks like it would be a very different world which is not really written about (Apart from the old articles posted in one of the above comments)
    - Swannie 14y ago
      20 - 30? What are you writing?! I work mostly on integration code. I'd say I see more like 60% code to handle errors, exceptions, etc.
    - loeg 14y ago
      > Most apps have error handling code taking up what? 20-30% of the code at the most? For some reason I imagine this stuff being almost 70% error handling but I am not sure if that is realistic. I work in a FreeBSD-derived kernel in areas related to the buf / caching subsystem, which probably has a lot of similarities to pre-2.6 linux (they have diverged more since the 90s). I don't know that I can quantify it, but 20-30% feels like a low number. > So, assuming there are shed loads of error handling that knocks on to everything else about the system. How do they test it? One thing we use to test error-case handling is "fail points" — sysctls usable from unit tests that trigger some fault condition. These aren't perfect — they are manually added to code, and because of this, coverage is fairly low — but they're a start. > How do they spec out all the possible conditions? What do they do differently to someone writing a SaaS, Desktop app or IOS app? Good questions :-).
cpeterso 14y ago
I'm curious for more details about the software stack used on their rockets. If they are using gcc and gdb, it sounds like everything is C. Which is a scary thought. :\
- manaskarekar 14y ago
  From their Reddit AMA [1] "The rocket is all C++ The rocket and spacecraft are all C++. On the ground, we use National Instruments LabVIEW extensively." - spacexdevtty [2] [1]http://www.reddit.com/r/IAmA/comments/1853ap/we_are_spacex_software_engineers_we_launch/ http://www.reddit.com/r/IAmA/comments/1853ap/we_are_spacex_s... [2]http://www.reddit.com/r/IAmA/comments/1853ap/we_are_spacex_software_engineers_we_launch/c8bpr00 http://www.reddit.com/r/IAmA/comments/1853ap/we_are_spacex_s...
  - cpeterso 14y ago
    Thanks for the links! That was exactly what I was looking for. :)
- mbell 14y ago
  > it sounds like everything is C. Which is a scary thought. :\ Why is that scary?
  - jacques_chester 14y ago
    C has a number of safety-compromising properties that require positive human effort to counteract. You can go a long way to counteracting some of these with automated toolchains. You can probably go further by using languages which simply eliminate certain classes of error and then having the same level of toolchain elaboration. That's not to say that C code can't be safe. It can absolutely be made safe. It just costs more, and any system that requires positive human effort is more likely to fail than one that doesn't.
    - furyofantares 14y ago
      Can you name a potentially appropriate language and a class of error it eliminates that cannot be eliminated with automated toolchains in C or C++?
speeder 14y ago
Very interesting to see how game development affects space research... Granted, this might explain Richard Garriot fiddling with space companies, and John Carmack manufacturing space engines! If I was rich I would do space research too! Also, I must be a software engineer, since I don't like Justin Bieber either.
- spc476 14y ago
  Well, Richard Garriot's father was an astronaut, so that may have had something to do with his decision to fiddle with space companies.
contingencies 14y ago
Triply-redundant hardware must be a power consumption headache in such environments. Anyone know what kind of power source these things use?
- w3pm 14y ago
  The clockspeeds on the processors they're using are probably relatively slow. Plus they don't have to worry about retina displays or WiFi or other major power-consumers in today's devices... I bet the overall consumption isn't that bad.
  - avmich 14y ago
    You bet. The problem of getting to orbit was solved "somehow" in 1950's, with emphasis on reliability, then terminal control optimized the result some decades later, when the hardware became available... But the basic task is still way below capabilities of even modest modern computers. Space X may afford to have quite a bit of redundancy.
- whatshisface 14y ago
  I imagine that if you are going into space, finding energy to run the computers should not be that hard. I am not sure about the falcon, but I know many jet planes simply attach a generator to the turbines. According to http://airandspace.si.edu/exhibitions/attm/a11.jo.fc.1.html http://airandspace.si.edu/exhibitions/attm/a11.jo.fc.1.html , both Apollo and Gemini got power from fuel cells.
  - javert 14y ago
    I have been told by aerospace engineers that when designing airplanes, you really don't have any extra anything, including energy. The avionics people have to lobby for as much energy as they can get, and try not to use too much. I don't know if that's still true for space stuff... probably.
    - venomsnake 14y ago
tgflynn 14y ago
They do not have hard realtime requirements If spacecraft avionics doesn't have hard realtime requirements what does ?
- Volpe 14y ago
  Manufacturing (I believe) (i.e moving production line)
- whatshisface 14y ago
  I recall hard realtime as being defined by an answer being worthless if it is late. If you are rocketing through the air, you can fire your rockets a few msec late, and as long as you compensate later nothing will break.
- biotech 14y ago
  Parts of the spacecraft will have hard realtime requirements, but those parts can be modular and controlled by soft-realtime systems
  - tgflynn 14y ago
    Wouldn't those modules be considered part of avionics though ? I got the impression the article was talking about the totality of avionics software run on the spacecraft, perhaps that impression is inaccurate.
- javert 14y ago
  Regular avionics, and stuff in cars
martinced 14y ago
"The Byzantine generals' algorithm is used to handle situations where the computers do not agree. That situation could come about because of a radiation event changing memory or register values, for example." This always got me wondering: what happens if the algorithm used for handling that is itself affected by a radiation event? Is that just too unlikely because such few code is executed? Or what if the computer in charge of verifying that the other computers do come up with a correct answer is itself dying, isn't it a SPOF? (curious mind wants to know)
- whatshisface 14y ago
  I think the point is that all the redundant computers check on eachother, and it is very unlikely for the majority of computers to have the exact same fault at the exact same time.
  - chickopozo 14y ago
    The same hardware, in the same location, running the same software, same power source, next to each other. It's quite likely actually. Do you do your backups to an identical* computer right next to your main one? (* factor in wear)
    - shabble 14y ago
      Multi-version programming[1] (independent implementations of the same specification) is one of the classic solutions to this problem. Likewise for power, location, etc. If you really care about these failure modes, you'll have N different designs of PSU & hardware fed via redundantly pathed links, etc. Aside from the (huge) cost/dev time, the biggest issue is that you still can't protect against logic errors in the specification, and the difficulty in testing every sequence of failure modes across implementations. [1] https://en.wikipedia.org/wiki/N-version_programming https://en.wikipedia.org/wiki/N-version_programming
      - atsaloli 14y ago
        You can protect against design errors through formal logic verification of the model. Www.spinroot.com
jacques_chester 14y ago
> They have tried various techniques for estimating time requirements, from wideband delphi to evidence-based scheduling and found that no technique by itself works well for the group. Since they are software engineers, "we wrote our own tool", he said with a chuckle, that is a hybrid of several different techniques. There is "no silver bullet" for scheduling, and it is "unlikely you could pick up our method and apply it" to your domain. It so happens that I'm working on a tool that amongst other things would allow combined PERT 3-point estimates and Delphi Wideband estimates. However the most important lesson in estimating is: do not rely on one estimation method. Ever. Use as many as you can. For high stakes estimates, try to have at least one of each of the 4 main classes of estimate: judgement, analogical, parametric and decomposition/recomposition. My tool fits in the 4th category, which is IMO currently underserved. Those of you in the machine learning world will recognise the vibe of ensemble learning.
vaidabhishek 14y ago
Normally the biggest challenge for such software testing is formal testing methods. You cannot just keep on randomly testing every bit of input. Hence, you need to put formal proofs to get rid of that last shred of doubt about whether the software can fail.
avar 14y ago
> Linux runs everywhere at SpaceX, > he said, on everything from desktops > to spacecraft. Yet every time I've seen them advertise for sysadmin sysadmin or programmer positions the indications are that at least all their internal network and website run heavily on Microsoft software, but their actual spacecraft use Linux/embedded. It would be interesting to get an actual overview of what they use Linux for, and what they don't.
- devbug 14y ago
  They use Linux for embedded systems, i.e., the Dragon and Falcon; and Windows on the ground. [1] http://www.reddit.com/r/IAmA/comments/1853ap/we_are_spacex_software_engineers_we_launch/c8bodrj http://www.reddit.com/r/IAmA/comments/1853ap/we_are_spacex_s...
- peripetylabs 14y ago
  If I understand correctly, the "Ground Software team" uses LabVIEW extensively, which has only limited support on OS X and Linux: http://www.ni.com/labview/os-support/ http://www.ni.com/labview/os-support/
cju 14y ago
Here are some papers about software/µP in space for those who want more examples/details: * Space Software Validation using Abstract Interpretation (2009) http://www.di.ens.fr/~rival/dasia09.pdf http://www.di.ens.fr/~rival/dasia09.pdf From the conclusion: This study has shown that embedded space software are difficult to analyze due to non linearity (mainly in quaternion computations) and complex control command algorithms involved (e.g., Kalman filter). It should be noticed that the software architecture which suits static analysis by abstract interpretat ion best is also the more readable one and maintainable one. This technique can thus be a metrics of good architectures. In spite of these difficulties, abstract interpretation techniques can greatly improve the quality of space embedded software. * Qualification of the Atomated Transfer Vehicule (ATV) Flight Control ftp://ftp.elet.polimi.it/outgoing/Marco.Lovera/ESAGNC08/S01/01_Clerc.pdf A lot of information about the test process, the usage of Hadware-In-the-Loop and Monte Carlo testing. * Requirements review, Needs for launchers, space vehicles & orbital infrastructures (2006) http://microelectronics.esa.int/conferences/ngmp2006/D2-1200-ESA_RT.pdf http://microelectronics.esa.int/conferences/ngmp2006/D2-1200... High-level requirements for On-Board Computer (OBC).
kabdib 14y ago
"Detecting the use of Emacs..." Is that for real, or tongue-in-cheek?
pootch 14y ago
That article was one of the larger loads of crap Ive read in quite a while-x

117 points by someperson 14y ago

I would love to see a more in depth article about what kind of programming you need to do for a system controlling a rocket. I remember reading that they cannot use exceptions and have to handle every possible scenario. Part of me wonders how much of the source code is just error handling.

w3pm 14y ago
No dynamic memory allocation, no recursion, no longjmp, etc. Program execution must be entirely predictable, every time. SpaceX's programming rules are probably very similar to those used at NASA JPL: http://lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf http://lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf
- paulrademacher 14y ago
  That's just the tiniest tip of the iceberg. Doesn't get into formal specs, verification, validation, etc. This article is often referenced for NASA's software practices, although it's now almost 20 years old: http://www.fastcompany.com/28121/they-write-right-stuff http://www.fastcompany.com/28121/they-write-right-stuff Is there anything more comprehensive and current than that?
loeg 14y ago
"Just" error handling? That's the hard part!
- rurounijones 14y ago
  I didn't mean "just" in that sense, I mean "just" as in "solely dedicated to". Most apps have error handling code taking up what? 20-30% of the code at the most? For some reason I imagine this stuff being almost 70% error handling but I am not sure if that is realistic. So, assuming there are shed loads of error handling that knocks on to everything else about the system. How do they test it? How do they spec out all the possible conditions? What do they do differently to someone writing a SaaS, Desktop app or IOS app? It looks like it would be a very different world which is not really written about (Apart from the old articles posted in one of the above comments)
  - Swannie 14y ago
    20 - 30? What are you writing?! I work mostly on integration code. I'd say I see more like 60% code to handle errors, exceptions, etc.
  - loeg 14y ago
    > Most apps have error handling code taking up what? 20-30% of the code at the most? For some reason I imagine this stuff being almost 70% error handling but I am not sure if that is realistic. I work in a FreeBSD-derived kernel in areas related to the buf / caching subsystem, which probably has a lot of similarities to pre-2.6 linux (they have diverged more since the 90s). I don't know that I can quantify it, but 20-30% feels like a low number. > So, assuming there are shed loads of error handling that knocks on to everything else about the system. How do they test it? One thing we use to test error-case handling is "fail points" — sysctls usable from unit tests that trigger some fault condition. These aren't perfect — they are manually added to code, and because of this, coverage is fairly low — but they're a start. > How do they spec out all the possible conditions? What do they do differently to someone writing a SaaS, Desktop app or IOS app? Good questions :-).

cpeterso 14y ago

I'm curious for more details about the software stack used on their rockets. If they are using gcc and gdb, it sounds like everything is C. Which is a scary thought. :\

manaskarekar 14y ago
From their Reddit AMA [1] "The rocket is all C++ The rocket and spacecraft are all C++. On the ground, we use National Instruments LabVIEW extensively." - spacexdevtty [2] [1]http://www.reddit.com/r/IAmA/comments/1853ap/we_are_spacex_software_engineers_we_launch/ http://www.reddit.com/r/IAmA/comments/1853ap/we_are_spacex_s... [2]http://www.reddit.com/r/IAmA/comments/1853ap/we_are_spacex_software_engineers_we_launch/c8bpr00 http://www.reddit.com/r/IAmA/comments/1853ap/we_are_spacex_s...
- cpeterso 14y ago
  Thanks for the links! That was exactly what I was looking for. :)
mbell 14y ago
> it sounds like everything is C. Which is a scary thought. :\ Why is that scary?
- jacques_chester 14y ago
  C has a number of safety-compromising properties that require positive human effort to counteract. You can go a long way to counteracting some of these with automated toolchains. You can probably go further by using languages which simply eliminate certain classes of error and then having the same level of toolchain elaboration. That's not to say that C code can't be safe. It can absolutely be made safe. It just costs more, and any system that requires positive human effort is more likely to fail than one that doesn't.
  - furyofantares 14y ago
    Can you name a potentially appropriate language and a class of error it eliminates that cannot be eliminated with automated toolchains in C or C++?

speeder 14y ago

Very interesting to see how game development affects space research... Granted, this might explain Richard Garriot fiddling with space companies, and John Carmack manufacturing space engines! If I was rich I would do space research too! Also, I must be a software engineer, since I don't like Justin Bieber either.

spc476 14y ago
Well, Richard Garriot's father was an astronaut, so that may have had something to do with his decision to fiddle with space companies.

contingencies 14y ago

Triply-redundant hardware must be a power consumption headache in such environments. Anyone know what kind of power source these things use?

w3pm 14y ago
The clockspeeds on the processors they're using are probably relatively slow. Plus they don't have to worry about retina displays or WiFi or other major power-consumers in today's devices... I bet the overall consumption isn't that bad.
- avmich 14y ago
  You bet. The problem of getting to orbit was solved "somehow" in 1950's, with emphasis on reliability, then terminal control optimized the result some decades later, when the hardware became available... But the basic task is still way below capabilities of even modest modern computers. Space X may afford to have quite a bit of redundancy.
whatshisface 14y ago
I imagine that if you are going into space, finding energy to run the computers should not be that hard. I am not sure about the falcon, but I know many jet planes simply attach a generator to the turbines. According to http://airandspace.si.edu/exhibitions/attm/a11.jo.fc.1.html http://airandspace.si.edu/exhibitions/attm/a11.jo.fc.1.html , both Apollo and Gemini got power from fuel cells.
- javert 14y ago
  I have been told by aerospace engineers that when designing airplanes, you really don't have any extra anything, including energy. The avionics people have to lobby for as much energy as they can get, and try not to use too much. I don't know if that's still true for space stuff... probably.
  - venomsnake 14y ago

tgflynn 14y ago

They do not have hard realtime requirements If spacecraft avionics doesn't have hard realtime requirements what does ?

Volpe 14y ago
Manufacturing (I believe) (i.e moving production line)
whatshisface 14y ago
I recall hard realtime as being defined by an answer being worthless if it is late. If you are rocketing through the air, you can fire your rockets a few msec late, and as long as you compensate later nothing will break.
biotech 14y ago
Parts of the spacecraft will have hard realtime requirements, but those parts can be modular and controlled by soft-realtime systems
- tgflynn 14y ago
  Wouldn't those modules be considered part of avionics though ? I got the impression the article was talking about the totality of avionics software run on the spacecraft, perhaps that impression is inaccurate.
javert 14y ago
Regular avionics, and stuff in cars

martinced 14y ago

"The Byzantine generals' algorithm is used to handle situations where the computers do not agree. That situation could come about because of a radiation event changing memory or register values, for example." This always got me wondering: what happens if the algorithm used for handling that is itself affected by a radiation event? Is that just too unlikely because such few code is executed? Or what if the computer in charge of verifying that the other computers do come up with a correct answer is itself dying, isn't it a SPOF? (curious mind wants to know)

whatshisface 14y ago
I think the point is that all the redundant computers check on eachother, and it is very unlikely for the majority of computers to have the exact same fault at the exact same time.
- chickopozo 14y ago
  The same hardware, in the same location, running the same software, same power source, next to each other. It's quite likely actually. Do you do your backups to an identical* computer right next to your main one? (* factor in wear)
  - shabble 14y ago
    Multi-version programming[1] (independent implementations of the same specification) is one of the classic solutions to this problem. Likewise for power, location, etc. If you really care about these failure modes, you'll have N different designs of PSU & hardware fed via redundantly pathed links, etc. Aside from the (huge) cost/dev time, the biggest issue is that you still can't protect against logic errors in the specification, and the difficulty in testing every sequence of failure modes across implementations. [1] https://en.wikipedia.org/wiki/N-version_programming https://en.wikipedia.org/wiki/N-version_programming
    - atsaloli 14y ago
      You can protect against design errors through formal logic verification of the model. Www.spinroot.com

jacques_chester 14y ago

> They have tried various techniques for estimating time requirements, from wideband delphi to evidence-based scheduling and found that no technique by itself works well for the group. Since they are software engineers, "we wrote our own tool", he said with a chuckle, that is a hybrid of several different techniques. There is "no silver bullet" for scheduling, and it is "unlikely you could pick up our method and apply it" to your domain. It so happens that I'm working on a tool that amongst other things would allow combined PERT 3-point estimates and Delphi Wideband estimates. However the most important lesson in estimating is: do not rely on one estimation method. Ever. Use as many as you can. For high stakes estimates, try to have at least one of each of the 4 main classes of estimate: judgement, analogical, parametric and decomposition/recomposition. My tool fits in the 4th category, which is IMO currently underserved. Those of you in the machine learning world will recognise the vibe of ensemble learning.

vaidabhishek 14y ago

Normally the biggest challenge for such software testing is formal testing methods. You cannot just keep on randomly testing every bit of input. Hence, you need to put formal proofs to get rid of that last shred of doubt about whether the software can fail.

avar 14y ago

> Linux runs everywhere at SpaceX, > he said, on everything from desktops > to spacecraft. Yet every time I've seen them advertise for sysadmin sysadmin or programmer positions the indications are that at least all their internal network and website run heavily on Microsoft software, but their actual spacecraft use Linux/embedded. It would be interesting to get an actual overview of what they use Linux for, and what they don't.

devbug 14y ago
They use Linux for embedded systems, i.e., the Dragon and Falcon; and Windows on the ground. [1] http://www.reddit.com/r/IAmA/comments/1853ap/we_are_spacex_software_engineers_we_launch/c8bodrj http://www.reddit.com/r/IAmA/comments/1853ap/we_are_spacex_s...
peripetylabs 14y ago
If I understand correctly, the "Ground Software team" uses LabVIEW extensively, which has only limited support on OS X and Linux: http://www.ni.com/labview/os-support/ http://www.ni.com/labview/os-support/

cju 14y ago

Here are some papers about software/µP in space for those who want more examples/details: * Space Software Validation using Abstract Interpretation (2009) http://www.di.ens.fr/~rival/dasia09.pdf http://www.di.ens.fr/~rival/dasia09.pdf From the conclusion: This study has shown that embedded space software are difficult to analyze due to non linearity (mainly in quaternion computations) and complex control command algorithms involved (e.g., Kalman filter). It should be noticed that the software architecture which suits static analysis by abstract interpretat ion best is also the more readable one and maintainable one. This technique can thus be a metrics of good architectures. In spite of these difficulties, abstract interpretation techniques can greatly improve the quality of space embedded software. * Qualification of the Atomated Transfer Vehicule (ATV) Flight Control ftp://ftp.elet.polimi.it/outgoing/Marco.Lovera/ESAGNC08/S01/01_Clerc.pdf A lot of information about the test process, the usage of Hadware-In-the-Loop and Monte Carlo testing. * Requirements review, Needs for launchers, space vehicles & orbital infrastructures (2006) http://microelectronics.esa.int/conferences/ngmp2006/D2-1200-ESA_RT.pdf http://microelectronics.esa.int/conferences/ngmp2006/D2-1200... High-level requirements for On-Board Computer (OBC).

kabdib 14y ago

"Detecting the use of Emacs..." Is that for real, or tongue-in-cheek?

pootch 14y ago

That article was one of the larger loads of crap Ive read in quite a while-x