7 ms·

I am an engineer who builds with a computer. And when I see such things as this, I sometimes spend a minute or so marveling and thinking about the complexity of

I am an engineer who builds with a computer. And when I see such things as this, I sometimes spend a minute or so marveling and thinking about the complexity of building and orchestrating such a launch. In fact, I really cannot comprehend how something like this can be successfully done, given how so many things could go wrong. Yes, I understand the idea of redundancy, I understand gravity, I understand that physics and chemistry goes into this, but still, the engineer in me wonders how on earth can we make all of this work together. Really, kudos to spaceX, kudos to NASA, and and kudos to those who believe in science.

mandeepj 8y ago
To help you with not feeling overwhelmed - they are doing it since May 6, 2002. It has taken a lot of time, of a lot of people to get here. There was a lot of failures also, along the way.
- bsaul 8y ago
  I remember reading here a document about C programming guidelines for military / space software ( basically no dynamic memory allocation ever), but that was set a long time ago. I wonder if spaceX reinvented some parts of the software stack, or if they went along the traditional way. I don’t expect seing rust code in the shuttle, but maybe formal proof with custom functionnal languages...
  - danpalmer 8y ago
    From what I understand, they are pushing the boundaries on the hardware front, i.e. relatively recent, typical Linux computers onboard, and just relying on having a few redundant ones and short flight times, rather than very radiation hardened systems like those used in satellites or real-time operating systems. However on the SpaceX software side I think they’re fairly conservative and “boring”, so I’d guess C and similar guidelines to what you’ve mentioned.
    - biesnecker 8y ago
      I'd be surprised if they were able to be human flight certified without all the redundancy and hardening that any other manned spacecraft has, though.
      - gpm 8y ago
        For example, my understanding is that the redundancy approach is basically "do everything on 3 computers, vote on the outcome".
        creatornator 8y ago
        Sometimes you even have different firmware/software teams write code for each of the three different configurations (often targeting different processors/architectures, etc). Pretty cool
      - JshWright 8y ago
        You can "skimp" on hardening if you use more redundancy (which is generally cheaper, and often lighter (which is the real cost savings)).
        danpalmer 8y ago
        You can also skimp on hardening if the operational lifetime in space is minutes to days, instead of months to years like most satellites.
    - mandeepj 8y ago
      They build everything in-house - software and hardware. You can read about it in detail in Musk's biography - https://www.amazon.com/Elon-Musk-SpaceX-Fantastic-Future-ebook/dp/B00KVI76ZS https://www.amazon.com/Elon-Musk-SpaceX-Fantastic-Future-ebo...
  - elteto 8y ago
    The flight software team did a reddit AMA a while back [0]. They use C++ for writing the flight software (what flies falcon, dragon, etc) running on a customized RT Linux. [0] https://www.reddit.com/r/IAmA/comments/1853ap/we_are_spacex_software_engineers_we_launch/ https://www.reddit.com/r/IAmA/comments/1853ap/we_are_spacex_...
    - monocasa 8y ago
      And FreeRTOS. RTLinux is only barely soft realtime.
      - avmich 8y ago
        > RTLinux is only barely soft realtime. Linux used to run as a non-privileged process in hard real-time kernel, which taken together was guaranteeing responses in nanoseconds range. If this is still the case (along with what Wikipedia states) , the whole system is by definition hard real-time.
        solarkraft 8y ago
        But only the underlying kernel part is actually real-time, the Linux process wouldn't be, right? That leads me to the question: Shouldn't all things in a rocket be real time? Are they "post-poning" stuff like telemetry and other communications?
        avmich 8y ago
        All critical processes can run outside of Linux, in hard real-time fashion. Linux is used for non-critical things, like networking... The whole RTLinux includes Linux (non-RT part) and a hard RT part, for which you can write your code.
        monocasa 8y ago
        Wow, that's really silly that someone went out, made a hard realtime system using Linux as the lowest priority level, and then called it RTLinux. RTLinux for years had meant Linux with the PREEMPT_RT patches. In any case, SpaceX doesn't use Linux for hard realtime systems. > They do not have hard realtime requirements https://lwn.net/Articles/540368/ https://lwn.net/Articles/540368/ And I probably wouldn't use the RTLinux for a real hard realtime system if I were making the choice. Putting all of the realtime code in the same memory domain as the rest of the Linux kernel is a huge surface area for corruption. I'd probably use sel4 with a virtualized Linux to get the same semantics.
- JshWright 8y ago
  A lot of what they are doing dates back a lot further than that. The innovations SpaceX has brought to the industry are more process related than technical. SpaceX has done a great job of setting a streamlined vertical integration for their production processes, which has allowed them to iterate a lot faster than others in the industry. That isn't to say they haven't been doing interesting things from a technical perspective (they're landing rockets on barges, after all), but the real secret to their success has been taking proven technology and doing it better/more efficiently (for instance, the fundamental design of their rocket engine, the Merlin, dates back to 50's).
  - joezydeco 8y ago
    So, out of curiosity, was there one person responsible for SpaceX operating in this manner? Was it Musk, or someone else? Unless they accomplished a miracle by building a world-class engineering team from zero, there must have been a catalyst.
    - JshWright 8y ago
      Elon Musk definitely deserves a huge share of the credit for SpaceX's success, but there have definitely been other folks who have made (and continue to make) critical contributions. Just a couple examples would include Tom Mueller, who was responsible for most of the design of the propulsion systems SpaceX currently uses (including the Merlin and Draco engines), and Gwynne Shotwell, SpaceX's President and COO. SpaceX absolutely owes its existence to Shotwell, both its commercial success, and the fact that it didn't die (a number of times) are directly due to her. From an outsider's perspective (_very_ outside), it seems like it's really the "synergy" of Elon and Gwynne that have made SpaceX what it is today.
    - mandeepj 8y ago
      > there must have been a catalyst There is - having visionary, charismatic founder(s). This pattern has been successful at Apple, Microsoft, Amazon, Facebook, Netflix, Tesla\SpaceX, Google.
  - 420codebro 8y ago
sidcool 8y ago
Totally understand what you mean. The complexity is mind boggling. And as more and more processes become software driven, the programmer's responsibility increases manifold. It's a wonder of engineering and team work.
buboard 8y ago
if we want to be snarky about it, it's basically a computer with rockets and doors attached
boris 8y ago
This comes down to the cost of failure. If your app crashes or hangs sometimes, some of us might get annoyed but in the end we are all used to this now. It's a different story if your engine blows up from time to time. I am sure if every failure meant a 6 month and a million dollars setback, we would be designing software differently.
laliberte 8y ago
Agree. Amazing accomplishment by SpaceX, cannot fathom how many things could have gone wrong, and yet they persevered anyway.
Gravityloss 8y ago
Computer engineering is new. We're still at the phase where bridges collapse and ships sink regularly.
- twtw 8y ago
  Computer engineering is not substantially newer than aerospace engineering.
  - mLuby 8y ago
    It's true, they just have different focuses. Aerospace (in general) has traded huge costs for safety and reliability. Computer engineering (in general) has traded correctness/reliability for low cost and rapid development.
- the8472 8y ago
  It's a matter of tradeoffs. If your webapp is unusable 2 days a year people will just do something else in the meantime. If your flight control software fails for two days a year planes might come crashing down. The former is tolerable even if unpleasant from a customer perspective, thus they don't pay for flight control levels of reliability. And it's not like this is unique to software. It's the difference between a disposable plastic fork and surgical instruments.
  - Gravityloss 8y ago
    Yes, I agree. I was wildly exaggerating. Life sustaining equipment is regulated and has a lot of resources spent on it to make it reliable. And toy software doesn't matter if it falls down. I guess medium level services fall in between. Still causing inconvenience and monetary losses when they fail.
- oconnor663 8y ago
  To be fair, a big portion of high profile computer engineering failures involve attackers actively searching for vulnerabilities, which bridges and ships don't need to deal with as much.
deleted 8y ago
[deleted]
felipemnoa 8y ago
>>In fact, I really cannot comprehend how something like this can be successfully done, given how so many things could go wrong. I have no idea how it is actually done. But my guess is that lots and lots of automated testing. It is the only way to make sure that any changes you make do no break some other critical parts.
- aerophilic 8y ago
  Yes, lots of testing. Sadly, not as automated as you would hope/think. The industry is working toward fuller automated testing, but it is much harder to do with “real hardware” in the loop. Part of the reason why space has [historically] been expensive is exactly because you have to have specialized humans doing a ton of tests. One reason Elon has been able to lower costs by reusable rockets is exactly because you don’t have to do as much testing per launch.
bisby 8y ago
Last night watching the launch I said to my friend: If I worked at SpaceX I would be second-guessing everything I'd ever done every time there was a launch. "Did I finish that weld good enough?" "Yes, we checked it 1000 times" "But what if I missed something?" And clearly, those kinds of mistakes have happened. And that's why we check 1000 times instead of 500.
utdiscant 8y ago
I was a summer intern at CERN a few years back. I had the same feeling of watching thousands of people working on projects together including tons of code, wiring, machinery, and complex physics to measure extremely hard things at ridiculous precision. I could never really shake the feeling that it should have been practically impossible to make such projects work. It is hard enough to not make bugs in personal projects, but when you are hundreds of people writing code and plugging in wires over time, some mistakes should appear that breaks the results.

mandeepj 8y ago

To help you with not feeling overwhelmed - they are doing it since May 6, 2002. It has taken a lot of time, of a lot of people to get here. There was a lot of failures also, along the way.

bsaul 8y ago
I remember reading here a document about C programming guidelines for military / space software ( basically no dynamic memory allocation ever), but that was set a long time ago. I wonder if spaceX reinvented some parts of the software stack, or if they went along the traditional way. I don’t expect seing rust code in the shuttle, but maybe formal proof with custom functionnal languages...
- danpalmer 8y ago
  From what I understand, they are pushing the boundaries on the hardware front, i.e. relatively recent, typical Linux computers onboard, and just relying on having a few redundant ones and short flight times, rather than very radiation hardened systems like those used in satellites or real-time operating systems. However on the SpaceX software side I think they’re fairly conservative and “boring”, so I’d guess C and similar guidelines to what you’ve mentioned.
  - biesnecker 8y ago
    I'd be surprised if they were able to be human flight certified without all the redundancy and hardening that any other manned spacecraft has, though.
    - gpm 8y ago
      For example, my understanding is that the redundancy approach is basically "do everything on 3 computers, vote on the outcome".
      - creatornator 8y ago
        Sometimes you even have different firmware/software teams write code for each of the three different configurations (often targeting different processors/architectures, etc). Pretty cool
    - JshWright 8y ago
      You can "skimp" on hardening if you use more redundancy (which is generally cheaper, and often lighter (which is the real cost savings)).
      - danpalmer 8y ago
        You can also skimp on hardening if the operational lifetime in space is minutes to days, instead of months to years like most satellites.
  - mandeepj 8y ago
    They build everything in-house - software and hardware. You can read about it in detail in Musk's biography - https://www.amazon.com/Elon-Musk-SpaceX-Fantastic-Future-ebook/dp/B00KVI76ZS https://www.amazon.com/Elon-Musk-SpaceX-Fantastic-Future-ebo...
- elteto 8y ago
  The flight software team did a reddit AMA a while back [0]. They use C++ for writing the flight software (what flies falcon, dragon, etc) running on a customized RT Linux. [0] https://www.reddit.com/r/IAmA/comments/1853ap/we_are_spacex_software_engineers_we_launch/ https://www.reddit.com/r/IAmA/comments/1853ap/we_are_spacex_...
  - monocasa 8y ago
    And FreeRTOS. RTLinux is only barely soft realtime.
    - avmich 8y ago
      > RTLinux is only barely soft realtime. Linux used to run as a non-privileged process in hard real-time kernel, which taken together was guaranteeing responses in nanoseconds range. If this is still the case (along with what Wikipedia states) , the whole system is by definition hard real-time.
      - solarkraft 8y ago
        But only the underlying kernel part is actually real-time, the Linux process wouldn't be, right? That leads me to the question: Shouldn't all things in a rocket be real time? Are they "post-poning" stuff like telemetry and other communications?
        avmich 8y ago
        All critical processes can run outside of Linux, in hard real-time fashion. Linux is used for non-critical things, like networking... The whole RTLinux includes Linux (non-RT part) and a hard RT part, for which you can write your code.
      - monocasa 8y ago
        Wow, that's really silly that someone went out, made a hard realtime system using Linux as the lowest priority level, and then called it RTLinux. RTLinux for years had meant Linux with the PREEMPT_RT patches. In any case, SpaceX doesn't use Linux for hard realtime systems. > They do not have hard realtime requirements https://lwn.net/Articles/540368/ https://lwn.net/Articles/540368/ And I probably wouldn't use the RTLinux for a real hard realtime system if I were making the choice. Putting all of the realtime code in the same memory domain as the rest of the Linux kernel is a huge surface area for corruption. I'd probably use sel4 with a virtualized Linux to get the same semantics.
JshWright 8y ago
A lot of what they are doing dates back a lot further than that. The innovations SpaceX has brought to the industry are more process related than technical. SpaceX has done a great job of setting a streamlined vertical integration for their production processes, which has allowed them to iterate a lot faster than others in the industry. That isn't to say they haven't been doing interesting things from a technical perspective (they're landing rockets on barges, after all), but the real secret to their success has been taking proven technology and doing it better/more efficiently (for instance, the fundamental design of their rocket engine, the Merlin, dates back to 50's).
- joezydeco 8y ago
  So, out of curiosity, was there one person responsible for SpaceX operating in this manner? Was it Musk, or someone else? Unless they accomplished a miracle by building a world-class engineering team from zero, there must have been a catalyst.
  - JshWright 8y ago
    Elon Musk definitely deserves a huge share of the credit for SpaceX's success, but there have definitely been other folks who have made (and continue to make) critical contributions. Just a couple examples would include Tom Mueller, who was responsible for most of the design of the propulsion systems SpaceX currently uses (including the Merlin and Draco engines), and Gwynne Shotwell, SpaceX's President and COO. SpaceX absolutely owes its existence to Shotwell, both its commercial success, and the fact that it didn't die (a number of times) are directly due to her. From an outsider's perspective (_very_ outside), it seems like it's really the "synergy" of Elon and Gwynne that have made SpaceX what it is today.
  - mandeepj 8y ago
    > there must have been a catalyst There is - having visionary, charismatic founder(s). This pattern has been successful at Apple, Microsoft, Amazon, Facebook, Netflix, Tesla\SpaceX, Google.
- 420codebro 8y ago

sidcool 8y ago

Totally understand what you mean. The complexity is mind boggling. And as more and more processes become software driven, the programmer's responsibility increases manifold. It's a wonder of engineering and team work.

buboard 8y ago

if we want to be snarky about it, it's basically a computer with rockets and doors attached

boris 8y ago

This comes down to the cost of failure. If your app crashes or hangs sometimes, some of us might get annoyed but in the end we are all used to this now. It's a different story if your engine blows up from time to time. I am sure if every failure meant a 6 month and a million dollars setback, we would be designing software differently.

laliberte 8y ago

Agree. Amazing accomplishment by SpaceX, cannot fathom how many things could have gone wrong, and yet they persevered anyway.

Gravityloss 8y ago

Computer engineering is new. We're still at the phase where bridges collapse and ships sink regularly.

twtw 8y ago
Computer engineering is not substantially newer than aerospace engineering.
- mLuby 8y ago
  It's true, they just have different focuses. Aerospace (in general) has traded huge costs for safety and reliability. Computer engineering (in general) has traded correctness/reliability for low cost and rapid development.
the8472 8y ago
It's a matter of tradeoffs. If your webapp is unusable 2 days a year people will just do something else in the meantime. If your flight control software fails for two days a year planes might come crashing down. The former is tolerable even if unpleasant from a customer perspective, thus they don't pay for flight control levels of reliability. And it's not like this is unique to software. It's the difference between a disposable plastic fork and surgical instruments.
- Gravityloss 8y ago
  Yes, I agree. I was wildly exaggerating. Life sustaining equipment is regulated and has a lot of resources spent on it to make it reliable. And toy software doesn't matter if it falls down. I guess medium level services fall in between. Still causing inconvenience and monetary losses when they fail.
oconnor663 8y ago
To be fair, a big portion of high profile computer engineering failures involve attackers actively searching for vulnerabilities, which bridges and ships don't need to deal with as much.

deleted 8y ago

[deleted]

felipemnoa 8y ago

>>In fact, I really cannot comprehend how something like this can be successfully done, given how so many things could go wrong. I have no idea how it is actually done. But my guess is that lots and lots of automated testing. It is the only way to make sure that any changes you make do no break some other critical parts.

aerophilic 8y ago
Yes, lots of testing. Sadly, not as automated as you would hope/think. The industry is working toward fuller automated testing, but it is much harder to do with “real hardware” in the loop. Part of the reason why space has [historically] been expensive is exactly because you have to have specialized humans doing a ton of tests. One reason Elon has been able to lower costs by reusable rockets is exactly because you don’t have to do as much testing per launch.

bisby 8y ago

Last night watching the launch I said to my friend: If I worked at SpaceX I would be second-guessing everything I'd ever done every time there was a launch. "Did I finish that weld good enough?" "Yes, we checked it 1000 times" "But what if I missed something?" And clearly, those kinds of mistakes have happened. And that's why we check 1000 times instead of 500.

utdiscant 8y ago

I was a summer intern at CERN a few years back. I had the same feeling of watching thousands of people working on projects together including tons of code, wiring, machinery, and complex physics to measure extremely hard things at ridiculous precision. I could never really shake the feeling that it should have been practically impossible to make such projects work. It is hard enough to not make bugs in personal projects, but when you are hundreds of people writing code and plugging in wires over time, some mistakes should appear that breaks the results.