12 ms·
The software that flies SpaceX rockets and starships
- sgt 5y agoThey're not running a dependency-ridden NodeJS backend with MongoDB and random cloud services? How are they even able to get to space?
- chinathrow 5y agoWell... https://twitter.com/tobi/status/1266867215325974528 https://twitter.com/tobi/status/1266867215325974528
- awestroke 5y agoThat's the frontend. It's fed data by the main computer, but if it crashes nothing else is affected.
- chinathrow 5y agoIf it crashes and Dragon is in manual mode, then what?
- awestroke 5y agoHow would I know? However, if you could think of that scenario, you can be 100% sure spacex thought of it over a decade ago and have measures to handle that situation
- jve 5y agoHow about a response from SpaceX software guy that works on Dragon software, including these displays: > Dragon is a fully autonomous vehicle, so it is capable of completing the trip to and from the ISS without any interaction from the crew. But the displays and button panel do provide the crew with capabilities should they need to take action due an unexpected scenario or emergency. As for the tablets and displays, the tablets themselves act as a sort of backup, and include copies of important data such as procedures. The displays themselves are designed to be fully redundant, so if a single display failed, the other 2 could fully take its place. Even if we were to experience a failure of all 3 displays, the crew has a button panel that can be used to initiate emergency responses, and ground commanding is also possible. > -Jarrett https://old.reddit.com/r/spacex/comments/ncj4vz/we_are_the_spacex_software_team_ask_us_anything/gy8wlkv/ https://old.reddit.com/r/spacex/comments/ncj4vz/we_are_the_s...
- extropy 5y agoThe manual controls are a separate system and would still work. And you can peer out of the window for navigation, Apollo style.
- horstmeyer 5y agoMongoDB is web scale, not universe scale.
- Stevvo 5y agoYou turn it on and it scales right up. Right up to space.
- freemint 5y agoWould have been so more funny if you wrote "cloud scale".
- api 5y agoIt's interesting how much contempt people have around here for one of the most successful open source oriented companies in recent memory. Most of the bad things about MongoDB are actually bad things about the NoSQL fad, and that fad was huge around here a few years ago.
- freemint 5y agoI used NoSQL databases that are ACID.
- indymike 5y agoI used acid and selected a nosql.
- andrewxdiamond 5y agoMongo’s “return before write” behavior is what earned it this title. It makes it look better on benchmarks, convincing unwitting devs that it is a better choice.
- 908B64B197 5y agoMongo and NoSQL, to me, are really symptoms of the Bootcamp craze that happened a few years ago. It's way easier to understand than relational databases, can be simply browsed and retrieved as json (usefull when your only language is JS) and had good adapters for the cookie cutter frameworks used at the time. And you don't need to bother with a schema or joins. There are use-cases for NoSQL but it's not a once size fits all solution.
- qwertyuiop_ 5y agoNo Kubernetes and Blockchain ?
- magwa101 5y ago
- ironmagma 5y agoThey're only using C++ which has 1,853 pages worth of complexity[1] along with all the toolchains that go on top of it. [1] https://www.iso.org/standard/79358.html https://www.iso.org/standard/79358.html
- yellow_lead 5y agoI'm sure they're using a strict subset of C++ but it is still complex. I wonder if they'll move to Rust some day.
- jcelerier 5y agoMost of the standard is taken by the description of the standard library ; the language specs ends at page 457 in C++20 (last draft). How long would be your language spec if every single function's documentation was part of it ? To give a reference with other languages: - This is a bit less than C# (whose standard language spec ends at page 462 in ECMA-364 2nd ed.). - The python spec for 3.9.9 is 151 pages. Okay, but let's compare equivalent things in the C++ spec and Python spec: awaiting a coroutine. * In Python (the whole "await" spec): Suspend the execution of coroutine on an awaitable object. Can only be used inside a coroutine function. await_expr ::= "await" primary New in version 3.5. * In C++ (5% of the "co_await" spec): The co_await expression is used to suspend evaluation of a coroutine (9.5.4) while awaiting completion of the computation represented by the operand expression. await-expression : co_await cast-expression But, for C++ this is only the first sentence (which is not written in an abbreviated style but as a proper sentence). Then a whole page explains the semantics in detail, and gives a complete example of how to use it. The python spec is just insufficient in comparison.
- ironmagma 5y agoI'm sure it would be long. The point was more that pretty much every (productive) development environment is complex.
- spyremeown 5y agoIf someone from SpaceX is reading (after 5pm, of course), can we get a more nitty-gritty of what's going on software-wise on your end? I'd love to see what control and embedded wizardry you guys are up to.
- tangoed 5y agoThere is a Reddit AmA by SpaceX software team that has some important details!
- jagged-chisel 5y agoA direct link would be very much appreciated
- yreg 5y agohttps://old.reddit.com/r/spacex/comments/ncj4vz/we_are_the_spacex_software_team_ask_us_anything/ https://old.reddit.com/r/spacex/comments/ncj4vz/we_are_the_s...
- i67vw3 5y agohttps://teddit.net/r/spacex/comments/ncj4vz/we_are_the_spacex_software_team_ask_us_anything/ https://teddit.net/r/spacex/comments/ncj4vz/we_are_the_space...
- 7thaccount 5y agoYep, this and there is a Quora question somewhere I think iirc. It's mostly LabView I think, which is an interesting choice given all the hate it gets. For those that don't know, LabView is a graphical (drag and drop) language for the most part where you interconnect various hardware components together. The advantage is that a lot of labs need to plug a lot of devices and sensors to run an experiment and LabView has support for a large amount of hardware. So you connect it all and can graphically view what the outputs are doing in chart form. The downside is that I don't think it's always very easy to specify exactly what you want to do, and code can pretty easily become spaghetti. There are a lot of horror stories about engineers having to maintain someone else's code. Edit: -Flight software is C++ -Launch is mostly LabView and yes, they have LabView running in mission control
- c7DJTLrn 5y agoI appreciate that they might not be able to share but I was hoping for a bit more source code and technical details given where this is posted.
- mattlondon 5y agoInteresting that they are doing updates at only 10 and 50Hz. I had kinda assumed they'd be looking at much higher frequencies for telemetry especially for things like the rocket motor metrics
- tomxor 5y agoI thought that too initially, but if you think about the physics it makes sense.. the larger the mass the more stable the object, the lower the frequency of acceleration needed to adjust it's trajectory. Also even though it's dynamic, it's not articulated or anything like dynamic robots. Take the opposite, a little quad copter, very low mass, will need very high frequencies. I used to manually fly miniature remote control helicopters before all of the automatic control stuff came in, they were extremely, inherently unstable, very tricky to control.
- chmod775 5y ago> the larger the mass the more stable the object Not really. Only when scaling mass, margin of error, and forgetting thrust/gravity. And even if you scaled mass, thrust, and margin of error proportionally, you'd still have an object exactly as stable as before (force X applied to the small object gives you the same acceleration as proportionally scaled force Xs applied to a larger object). Gravity accelerates all things the same, and so do proportionally stronger thrusters. But being off by one meter can destroy a small drone just as well as a large spacecraft, so you don't have the luxury of scaling your margin of error by significant amounts. The only reason larger objects appear more stable to humans is because we, and the environment[1], tend to apply relatively weaker forces to them, and also you don't tend to notice they're already moving at 5m/s because of their size. [1]Larger objects will better compensate for "truly random" forces, if their size causes them to encounter more of them, canceling them out and/or they're generally small.
- 7952 5y agoA spacecraft may need to be super accurate at docking, but outside of that precision is far less important.
- whatever1 5y agoIt is fascinating that they solve robustly 50 times per second a control problem and handle gracefully numerical issues. Granted they have dedicated hardware to do so that fast, but it never stops to amaze me.
- tonyarkles 5y agoIn some ways, a quadrotor's control system is a more challenging controls problem! And I'm not at all suggesting that the rocket control loop isn't hard, but a few things to consider: - Quadrotors control their attitude solely by adjusting the thrust generated by the four motors. Unlike rockets, or fixed-wing aircraft, or cars even, the moment there is any kind of disturbance (whether software, hardware, or environmental), the aircraft immediately starts to fall out of the sky. - For the work I do, we typically fly around 5-7m above ground. If the motors completely quit, we have around 1.2 sec before we hit the ground at 42km/h. - The aircraft is also capable of roll rates >50deg/sec. Even if you're high enough that you don't immediately hit the ground, motor problems can also result in you pointing upside down and pulling yourself towards the Earth instead of pushing away. - Most of the parts have notoriously bad repeatability, plus small variations would wreck any repeatability anyway. You can't just set all of your motor throttles to 60% and expect them to have the same thrust, it's nested-upon-nested control loops to keep everything going - The sensors are generally crap too with non-trivial amounts of bias and drift. And the wildest part of all of this? You can get a top notch flight controller to handle all of this for you (and open source!) for $150 USD https://shop.holybro.com/pixhawk-5x_p1279.html https://shop.holybro.com/pixhawk-5x_p1279.html?
- PaulHoule 5y agoThe data rate for flight control systems (air or space) is not that great but the requirement for reliability is. That’s not just an absence of software defects but also reliability in the face of hardware failures. (Compare the A320 flight envelope protection to the 737 MAX MCAS.)
- dotancohen 5y ago> ...when you plan to reuse booster rockets and shuttles on future missions. Shuttles? I'm aware of the general term shuttle as a vehicle for going back and forth, but here I'm certain that the better term would be craft, capsule, or Dragon. Certainly not shuttle.
- HPsquared 5y agoDoes that include the Starship?
- happy-go-lucky 5y agoA space shuttle is by definition a reusable spacecraft that carries people and cargo between Earth and space. You mean the shuttle has been replaced by something newer and hence is an obsolete word?
- dotancohen 5y agoNo, I mean that they are talking about "boosters and ___" therefore ___ refers to a component, not the system as a whole. The term shuttle might refer to the whole system that is needed to shuttle people up and down, but the terms that fit the structure "boosters and ___" are craft, capsule, Dragon, maybe a few others, but not shuttle. Doubly so when that is not a term that SpaceX uses to refer to any component or system.
- alcappuccino 5y ago
- hughrr 5y agoSo basically it’s a large PLC :)
- fallingmeat 5y agoInteresting they seem to ad-hoc the lifecycle processes, or at least no mention of any industry standards mandatory (RTCA/DO-178C, ISO26262 or even sys eng like SAE ARP4754, etc). Also, no talk of formal methods (formal modeling or formal verification, a la DO-333 with something like Simulink Design Verf or TLA+). Works for them obviously!
- procarch2019 5y agoI’m not sure that I took this away from the article. Looked like they just didn’t go into detail. They mentioned storing data for keeping track of equipment health. To me that sounds like a super broad stroke that would be equivalent to historized time series data as well as contextualized asset data. I’m not familiar with the standards you called out (not an aerospace engineer), but I would imagine it’s similar to what we do in the controls engineering and automation space. I’m actually really curious about what software/methods they use to contextualize assets and what those data points are. How do They digitize the equipment data that comes from visual inspections/manual testing. I wish it was a little more detailed in general.
- icegreentea2 5y agoI am quite interested to see how they deal with regulations and standards (I'm in medical device space). I scanned through their job postings and I can't find a single standard listed (this is up to the Sr Engineer/Developer positions), as well as a lack of dedicated software QA/test positions. Obviously they are doing some amazing, and quite high quality work, so I am quite interested in what their internal setup is like.
- procarch2019 5y agoYes, considering how fast they move I’m wondering if it’s closer to self regulated/peer quality control. Some QA/QC can make work come to a screeching halt in biopharma space (makes sense when a “batch” in the cell and gene therapy space is a single person and their wellbeing). It’s not like regulation and quality have not fallen by the wayside or been too relaxed for other aerospace companies, especially in recent history. Quality should be meaningful and not just a rubber stamp. It’s a fine line
- RSHEPP 5y agoI know on the testing/V&V side, they use a mix of LabVIEW and Python, at least for some specific teams. Lots of automated (physical) tests are being written in both.
- hughrr 5y agoA minute of silence for the poor LabVIEW developers please.
- qaq 5y ago
- numpad0 5y agoHow are distributed realtime systems like a rocket is synchronized? Are there central timeservers and allocated transmission slots and timestamped query-response action messaging or is it just small individual realtime nodes making their own decisions? I’ve been wondering about this for a year or so because I can’t seem to find one that matches my description, other than concept drawings on NIST 4D/RCS paper, but there got to be one in the world.
- Beached 5y agoiirc all flight systems are in triplicate and all data is checked against the others, then a quarum is needed from the 3 to make a decision. other rockets use rad hardened computers that make decisions on their own, but for cost and time, I believe SpaceX just triplicated all necessary systems and votes out and reboots any system that gets bit flips.
- bodhiandpysics1 5y agoThey actually use more than three now. The basic architecture is that computers are organized in "strings". Each string is doubled. If the two don't agree, the result is chucked, and the system reboots. Each of these "strings" then participates in a quorum. The result is extreme reliability, with relatively fast convergence.
- inamberclad 5y agoThere's typically a central time source sending a 1Hz pulse into the main computer(s). The sensors will use something like Time Triggered Ethernet, RS-485, or a MIL-1553 bus.
- lokimedes 5y agoThis article provides an excellent summary of why the defense and space industry is different from “tech”. What SpaceX is doing is normal practice in Defense, but it reads like “1975” when you’re used to TypeScript and cloud instances. I really love my role of spanning the gulf between these worlds. There’s so much to learn from each side to the other.
- pbourke 5y agoThere are some echos of trading systems (control cycles, publishing telemetry streams, small-world distributed systems). I was also reminded of James Gosling’s VJUG talk about the Liquid Robotics onboard software for autonomous ocean vehicles: https://youtu.be/EgUctRxaiLs https://youtu.be/EgUctRxaiLs
- CodeGlitch 5y agoInteresting they didn't go with MISRA C and used C++ instead.
- johnisgood 5y agoInteresting they didn't go with Ada/SPARK, to be honest.
- pyjarrett 5y agoAda/SPARK is pretty close to unheard of in the US. People tend to solve problems with the languages they know and can find people to hire who know it. C++ is fast, and there's a good number of people to hire who know it. Actual comments I've heard from senior developers when I tell them I write Ada in my spare time: - "That language is still around?" - "Other than you, I've never heard or anyone who has ever used Ada."
- bdavis__ 5y agowhen i was first exposed to ada, early 1990's, the language was unusable. compile times where excessive (like hours). machines of that era were slow and had small memories (4MB was a common size, and more than 1 CPU was a luxury). i must also admit, ada was hard. you had to do some mojo just to print out a value ("package int_io is new integer_io blah, blah). it had a standard of a couple of hundred pages that you kept on your desk. and you looked at it on a regular basis. and the compiler was picky (strongly typed). companies dropped ada because they couldn't hire. C++ people were more available. my opinion today is ada is relatively small, makes very fast code, and supports all the programming paradigms of C++ in a very sane way. if the first ada compiler was "ada95" instead of "ada83", it would be very popular. it came out a decade before the world was ready.
- pyjarrett 5y agoAda 2012 is what I learned, but I probably would have moved on if had learned Ada 95. They've done a fantastic job modernizing it, and the next version looks even better (Ada 2022). > my opinion today is ada is relatively small, makes very fast code, and supports all the programming paradigms of C++ in a very sane way. This is why I've been sticking with it. I also get pre/post conditions, a real module system (yes, I know C++20), and bounds checked arrays/strings. Easy built-in multitasking has me using concurrency a lot more often as well. > companies dropped ada because they couldn't hire. C++ people were more available. I work professionally in C++, and honestly think you could convert a C++ programmer into an Ada one in about a month or so, due to the conceptual similarities.
- mlindner 5y agoThis is a re-upload from earlier in the year.
- dang 5y agoYup: Exploring the software that flies SpaceX rockets and starships - https://news.ycombinator.com/item?id=27115372 https://news.ycombinator.com/item?id=27115372 - May 2021 (112 comments) Also related: SpaceX Software Team AMA - https://news.ycombinator.com/item?id=23444738 https://news.ycombinator.com/item?id=23444738 - June 2020 (6 comments) We are the SpaceX software team, ask us anything - https://news.ycombinator.com/item?id=23432996 https://news.ycombinator.com/item?id=23432996 - June 2020 (28 comments) Software Engineering Within SpaceX - https://news.ycombinator.com/item?id=23403800 https://news.ycombinator.com/item?id=23403800 - June 2020 (313 comments) We Are SpaceX Software Engineers - https://news.ycombinator.com/item?id=13705762 https://news.ycombinator.com/item?id=13705762 - Feb 2017 (6 comments)
- solarkraft 5y agoA bit of a disappointing read. All I got out of it is that the individual control systems run at 10 or 50 Hz (less than I expected). From my own background knowledge I remember that the rockets actually run on Intel chips and the Dragon touchscreens are Chromium-based (less certain about that one). I was hoping for a lot more details like that.
- cwt137 5y agoThe control loop reminds me of how you would create a game ie. get inputs, update states, based on the current state of the environment and inputs, render (write output). No wonder John Carmack got into rockets!