14 ms·
What computer and software is used by the Falcon 9? (2015)
- jamespetercook 6y ago“They use Chromium and JavaScript for the Dragon 2 flight interface.“ This is the most interesting part for me being a web dev! How likely do you think this is?
- aetherspawn 6y agoJust because they do, doesn’t mean you should. I’m running an automotive firmware project at the moment and there are such things as software integrity levels, and each time you add something to the stack you create quite a large amount of testing and validation and process that needs to be followed. And it’s pretty much impossible to take something existing like Chromium and certify it to what you need. The architecture is wrong. The process is wrong. There is a lack of formal requirement traceability. Many other issues.
- wadim 6y agoBut mixed-criticality systems are quite normal in many deployments nowadays, especially with the advent of virtualization and hypervisors you should be able to run safety critical and non-safety critical functions at the same time on the same SoC.
- jki275 6y agoHeresy.
- aetherspawn 6y agoYes, but you have to prove that if one display considered to be "non critical" shows misleading data, it will not cause harm to the crew. For example, if the Chromium crashed and showed the heading or fuel level incorrectly, would they fly the rocket ship into the ocean or run out of fuel and drop out of the sky. Yes, it's called software integrity decomposition, and it's complicated and requires much analysis.
- fsh 6y agoI believe the choice of Chromium and JS for the UI simply shows that the astronauts are merely inconvenient payloads. Gagarin's first flight was already fully automated 59 years ago.
- TeMPOraL 6y agoShould've paired them with a dog. As the old joke goes, the astronauts are there to feed the dog; the dog is there to bite the astronauts if either of them tries to touch any of the controls.
- rurban 6y agoAs we saw in a recent Space Force episode on Netflix, the ape was much better than the dog. But you need pair them, as with Doug and Bob. Doug being the ape of course.
- usrusr 6y agoThe UI design consisting exclusively of elegant but low contrast shades of blue certainly gives a similar vibe. But that might still be misleading, perhaps they deliberately keep the routine visualizations in a low key "style over substance" passenger entertainment mode so that they have plenty of visual headroom for when something actually important combs up?
- CapricornNoble 6y agoThat's an interesting thought. If the UI is always blue-ish...and suddenly a bright red triangle appears with an Exclamation Point inside....you might be about to die and should act accordingly.
- racingmars 6y agoThis reminds me of the Airbus concept. They follow what's called a "lights out" design for all of those buttons on the overhead panel. When things are set and working the way they should be for normal flight, there is no indication. Only if something is non-normal does the indication or button light up. For example, buttons that should be "on" for flight only light up when the system they control is "off", and things like emergency overrides or backup systems that should be "off" normally have buttons that light up when they're "on". It's an interesting concept. Obvious benefit is you can glance at the overhead panel and instead of having to consider the state of, say, 50 or more buttons, you can instantly see if everything is "normal" for flight. But the downside is that it's not as intuitive to answer the question "is system X turned on (or off)"...you have to look at the button, then know what's considered "normal" for that system, on or off, and determine if the lack or presence of indicator light means on or off. That said...the pilots know the plane well enough where it's not a problem.
- je42 6y agoI think it is not helpful to call out that their architecture is "wrong" and their process is "wrong". The attribute "wrong" is very opaque. As somebody who doesn't have your background, i cannot follow you. I have no idea about automotive and space software requirements. Since there are many aspects that lead to choices of a system architecture it requires more words to explain why a choice is not a good choice for the job.
- mrmonkeyman 6y ago"Wrong" for certification. Something like Chromium is so enormously humongously complex and integrated it will never pass any formal test.
- aetherspawn 6y agoI was referring to the architecture of Chromium being wrong to achieve certification for safety critical applications. And look, I haven't said impossible, but rough estimate I'd say removing 75% of the features and 5+ years to rewrite it to be compliant. And the subset of javascript that it could run would be very limited - it might not even have loops or functions.
- api 6y agoThat was the most surprising part to me too, but not that surprising. Lots of automotive and newer industrial UIs use Qt Quick with QML which is not all that different from HTML5 and JS with something like React. It's used in the presentation layer, not the actually control systems.
- TomMarius 6y agoIt is very different from the audit/certification viewpoint, and Qt specifically caters to the automotive market
- skunkworker 6y agoI’m now curious about their memory structure, are they just using off the shelf multibit ECC memory?
- vardump 6y agoYou might need multibit ECC also in CPU caches. Pretty much all CPU caches have ECC, but no idea how large of an error they can fix. Then again, probably control logic is running on something radiation hardened and GUI layer (with some redundant hardware) can be easily rebooted as necessary.
- brandmeyer 6y agoI don't think that's necessarily true. I've worked with a couple of cacheful CPUs that only used parity checks on caches, not ECC. To get safety, the cache has to be set up as write-through instead of write-back, so that it is almost always safe to throw away a line that suffers a parity error.
- vardump 6y agoAt least I've seen Machine Check Exceptions on x86 CPUs about L3 ECC errors on non-Xeon CPUs. I'd presume also other levels are protected.
- gumby 6y agoInteresting that it’s three identical processors running he same software. I remember just the automated Vienna central train station used a pair of machines, x86 and SPARC, one running the logic in a procedural language (Chill) and the other running in a logic language (Prolog). They didn’t want both processors to share any code, and therefore potentially the same bug.
- hliyan 6y agoLooks like the goal is radiation tolerance, not n-version programming.
- fit2rule 6y agoThat hasn't been the case in Vienna for some years, so you must have seen that setup a decade or so, ago. These days its THALES-built systems running SIL-4 rated code in a best 2-of-3 or 3-of-5 voting system, and each of the nodes can be either PPC, Arm or x86 - but not all the same, for architecture reasons. These systems produce a vote that has to match, or else the system shuts itself down. Chill and Prolog haven't been shipped in that capacity in decades... (Disclaimer: worked for Thales on these systems, had to help remove a ton of Chill code, fix bugs in its compiler, etc.)
- gumby 6y agoYeah, I was involved in the Alcatel system in the 90s. Still, the basic point is interesting: the Thales system also uses a separation of architecture.
- hoorayimhelping 6y ago>They didn’t want both processors to share any code, and therefore potentially the same bug. They're not worried about bugs, they're worried about electromagnetic radiation (which is way more present when you get outside the atmosphere) flipping bits.
- fit2rule 6y agoThere have been cases of homogenous CPU architectures in safety-critical where all CPU's failed at once. Bitflippinrads are one of a multi-variate set of problems that these systems are designed to detect and avoid. Nodes can degrade otherwise: SRAM failure, battery/cap depletion, all sorts of things. The idea is to detect when a voting node has had a catastrophic system failure, and then remove offending nodes from the decision as rapidly as possible. A multi-architecture system assists with managing (and certifying) things such as compiler bugs and impact from system-level homogeneity, where critical components all exhibit the same bug/failure. An x86 hardware bug is fatal if all 5 of the nodes in a voting system are x86 and making a critical decision; it is less of an issue if the other 2 or 3 nodes are PPC, ARM, MIPS, etc., or a mixture there-of, and - as variant hardware architectures - are democratically still voting during the period .. the idea is to avoid homogeneity and pursue heterogeneity.
- irrational 6y agoI’ve read a bit about how strict the space shuttle coders were. I wonder what they think about the code running this spacecraft. https://www.fastcompany.com/28121/they-write-right-stuff https://www.fastcompany.com/28121/they-write-right-stuff https://history.nasa.gov/computers/Ch4-5.html https://history.nasa.gov/computers/Ch4-5.html https://www.nasa.gov/mission_pages/shuttle/flyout/flyfeature_shuttlecomputers.html https://www.nasa.gov/mission_pages/shuttle/flyout/flyfeature...
- ncmncm 6y agoThere are reports that the procedure for fixing bugs on the STS was so heavy-weight that they often redefined non-fatal bugs as correct operation. The example I read of was an error event that dumped thousands of lines of text to the flight console screen, obscuring useful information.
- OnlyOneCannolo 6y agoBasically every change, whether software or hardware, had a lengthy acceptance process. Reviews, analysis, verification and validation, documentation, operation and maintenance procedures, etc.
- ashleyn 6y agoCOTS software is nowhere near what I'd call "spaceworthy". They either determined the UI was not life-critical or ignored that fact.
- ncmncm 6y agoHeaven help them if the Javascript they run depends on npm components. When I read about these multiply redundant systems I always want to know what provisions are made for restarting or shutting down a part that is producing wrong answers. Can the Power device reset the machine whose instructions lost the vote? If the two cores disagree, do they both reset? Also, what is done for procedures that must not be interrupted? E.g., once de-orbit is initiated, if the microcontrollers get no new instructions, do they have enough to re-enter safely? Or do they need updates whenever something is supposed to happen?
- api 6y agoI highly doubt JS is used outside the UI layer.
- tonetheman 6y agoWhy is this comment grayed out... This is so true. Assuming they dumb enough to put chromium on a spaceship for a UI then there a good chance that yes they did use bower/npm. Unless they designed it in pure HTML/JS... again this seems unlikely ... given the blunt stupid of the initial choice.
- cowmix 6y agoJavascript is what will be used to program the James Web Telescope is/when it ever goes into orbit.
- nsajko 6y agohttps://en.wikipedia.org/wiki/Triple_modular_redundancy https://en.wikipedia.org/wiki/Triple_modular_redundancy
- nullifidian 6y agoSounds kinda disappointing/irresponsible. Chromium is an overly complex, relatively buggy piece of software. For example I recently experienced a bug where a text I selected was significantly offset from what should have been selected given the pointer position (on linux), a misclicky kind of bug. Considering that the majority of Crew Dragon's controls are through touch screen, maybe even manual docking(not sure, but I don't see a joystick anywhere), my personal experience with chromium does not instill confidence for such critical applications. But I guess they tested it very thoroughly. C++ and linux do not strike me as mission critical piece of software / language either. My heart desires for something with verified properties, advanced languages with integrated proof checkers, verified OSs, or at least smaller purpose built OSs that were carefully reviewed. But I guess SpaceX is not to blame for this. It's likely such approaches are simply not ready yet. And I know that C++ and linux are common in mission critical applications. JSF avionics is written in C++ for example. (Not to mention JavaScript in space, in a mission critical role. That shouldn't have happened.)
- jchw 6y agoOnce you decide that absolute reliability is unattainable, you can instead aim for sufficient redundancy. It’s not perfect, but with enough safe guards in place it may very well be good enough.
- microcolonel 6y agoSoftware is not always like hardware though: the same software failure can cascade across redundant instances rapidly just as easily as it affects the primary.
- jchw 6y agoThe same hardware failure can cascade too; redundancy alone is not a panacea, you need some safeguards. Ideally, nothing should fully trust anything else.
- microcolonel 6y ago
- Torkel 6y ago”Source: Discussion with various SpaceX engineers at GDC 2015/2016” I wonder if they had to change any of this for the human rated mission? Would be super exciting to hear tech talks from Spacex!
- 7leafer 6y agoThe whole Dragon is a simulation written in Python and running inside a VM running inside an OS written in JS and running inside an Electron app running in a container somewhere on Amazon AWS.
- woile 6y agoFrom the latest video NASA uploaded, the astronauts explain that the UI sends "commands", and if it fails, they have physical buttons below the screen to send these commands. https://mobile.twitter.com/NASA/status/1266885097359388672 https://mobile.twitter.com/NASA/status/1266885097359388672
- smarx007 6y agoThe UI at 2:22 is very similar to the one discussed in https://news.ycombinator.com/item?id=23162820 https://news.ycombinator.com/item?id=23162820 as seems to use three.js and GSAP. Also, the open-source Lato font seems to be used throughout the interface.
- throworangeaway 6y agoopen-source Lato font That'd be a really nice touch, one of my favourite fonts for effects. Released after client who commissioned it, a chain of retail bookstores, rejected the design. Largest deployment till now, although just dumped in redesign, was for headings at classifieds website OLX. Caught eyes for a while I guess. No need then for an especially legible font for cockpit screens like Airbus required https://b612-font.com/ https://b612-font.com/ following decades of practices in avionics. Of course the use-case is different and I'm sure everything is adequate for SpaceX goals. Like with the several layers of spontaneously rebooting virtual machines in Tesla cars, in the end branding looks gorgeous.
- elteto 6y agoA slight inaccuracy: the flight strings are not x86, but ARM, running on a custom board. Also, only the actual graphical display application uses Chromium/JS. The rest of the system is all C++. The display code has 100% test coverage, down to validation of graphical output (for example if you have a progress bar and you set it to X% the tests verify that it is actually drawn correctly).
- Game_Ender 6y agoHow stateless and independent is each display?
- elteto 6y agoCan't say exactly re statelessness, but I would not be surprised if you could power-cycle all displays if needed in the middle of a mission. As far as independent, completely independent from each other.
- yellowapple 6y ago> but I would not be surprised if you could power-cycle all displays if needed in the middle of a mission. Indeed you can, judging by the instruction to power off and clean all displays (then power them back on) prior to ISS rendezvous/docking in the live stream.
- zeroc8 6y agoHow is that done? By pixelwise comparison with a test image?
- warrentr 6y agoI'm curious about this as well. From what I remember chromium does what you suggest to verify what is rendered down to the pixel: https://developers.google.com/web/updates/2020/02/chromium-chronicle https://developers.google.com/web/updates/2020/02/chromium-c...
- neurostimulant 6y agoNext time someone complaining about electron I'm going to tell them it's technically chromium and thus space worthy :)
- erikbye 6y agoSpaceX has had a booth at GDC at least two times, they attend in order to recruit game developers (programmers), as they feel these programmers work with a lot of the same constraints they do in terms of resources and experience with real-time, performant systems, etc.
- cbanek 6y agoIt's true, I was there! (And I was a gamedev previously). Although you forgot crunch time which I think is the real required experience. ;)
- throwaway31338 6y agoUsing a gigantic overly complex software written in a non-memory-safe language for a system that gives commands to the flight control system? This is how you get the Cylons pwning your spaceships. >sigh<
- s1k3s 6y agoWe get it, you like Go. Meanwhile they are in space tho..
- Kye 6y agoAdama wouldn't have worried so much about networked ships if the people who programmed this ship programmed Battlestars. They still used COBOL.
- goatinaboat 6y agoThey still used COBOL. But they were experts in using it, some even say Lords
- MattGaiser 6y agoIsn't mission critical software overwhelmingly C++?
- fnordfnordfnord 6y agoA lot of Ada as well.
- techboyg5 6y agoJust like a PC?
- phkahler 6y agoFor triple redundancy, is there a way to re-sync if one of them disagrees? Control systems tend to have internal states that may be disrupted even if an input has glitched for a single cycle. Will it eventually converge back to nominal? Or is that processor rejected for the duration of the mission?
- elteto 6y agoIf a flight string desyncs (i.e. it has state at the end of a cycle different from its two siblings) then it is automatically rebooted. Then state is automatically resynced from any other sibling after boot. There is no converging between the strings, they are either in complete sync or they aren’t.
- sansnomme 6y agoWelp, I called it two weeks ago - https://news.ycombinator.com/item?id=23146976 https://news.ycombinator.com/item?id=23146976
- historyremade 6y agoIt would have taken them century if they written whole thing in memory safe langs.
- ipython 6y agoFascinating. The increased complexity in the flight control software and user interface probably makes the job of being an astronaut easier in the “happy path” - less cognitive load interpreting metrics and raw telemetry when you have the computer doing that for you. My wonder then is how to effectively manage when things go wrong. The Apollo spacecraft were able to be essentially hot patched directly in the cockpit via the dsky. Can you do that with a fancy touch screen interface now? Perhaps up/down links to the capsule from earth are good/reliable enough to send patches from ground control, if need be? I wonder how the astronauts feel about a potential “loss” of control in case something goes wrong?
- person_of_color 6y agoWhat about radiation hardening? Or do we just rely on ECC now? What about ECC for flops?
- Simulacra 6y agoI’m really curious what language the telemetry data is written in. Does anyone know what the historically preferred language is?
- cwebster2 6y agoNo idea for SpaceX, but a lot of commercial avionics run code written in Ada.
- sneeuwpopsneeuw 6y agoDepending on the platforms and customers C and Java are also heavily used next to Ada. The history behind Ada is mostly that the military required components to be made for that language. That requirement shifted around 10 years ago to a preference.
- crocal 6y agoBytecode, it seems :) https://mars.nasa.gov/MPF/rovercom/telem.html https://mars.nasa.gov/MPF/rovercom/telem.html
- zo1 6y agoProbably running Node Javascript (or Typescript), transmitted in JSON over a restful API, hosted on a K8s cluster using Docker containers. Also, the JSON payload has a formal schema in the form of JSonSchema V5. /s On a more serious note. Some of the amazing stuff that the SpaceX team are accomplishing really puts into perspective the day to day work I and others here do.
- varjag 6y agoThe Flight Software team is about 35 people. We write all the code for Falcon 9, Grasshopper, and Dragon applications; and do the core platform work, also on those vehicles; we also write simulation software; test the flight code; write the communications and analysis software, deployed in our ground stations. We also work in Mission Control to support active missions.
- jonplackett 6y agoImagine if everyone on earth was as productive as those 35 people.
- Cyph0n 6y agoI’m imagining the insane work hours.
- varjag 6y agoThere is probably quite some overtime involved. That said, you don't get reliable flight software simply by putting long work hours.
- BiteCode_dev 6y agoIt reminds me of this olognion piece: https://www.theolognion.com/unreal-engine-5-is-meant-to-ridicule-web-developers/#click=https://t.co/sVuOZVdpWA https://www.theolognion.com/unreal-engine-5-is-meant-to-ridi... That's not just hilarious, that's also pretty close to how I feel most of the time. I mean, some people have built the Airbus A380. On the other hand, last week, I created a test DB with the user "test" and the password "test". I couldn't connect to it, and after a long debugging session, noticed I misspelled "test". So, yeah...
- ralston3 6y agoThis is amazing XD
- jonplackett 6y ago
- tosh 6y agoseems like while most of us worried about the singularity; leftpad managed to expand from planet earth to the ISS. well played! #universalpaperclips
- BiteCode_dev 6y agoThey should have said "Visual Basic and some Excel macros", just to see the Twitter reactions.
- peter303 6y agoOther references say parts of Falcon9 and Dragon a version of Unix calLed VxWorks from Charles River. It is realtime (not virtual) and predates Linux. NASA uses it in a lot of its space probes because its been debugged for 33 years. That doesnt means its perfect. The two 2003 Mars rovers experienced a bug in the then-newfangled flash memory driver. It was successfully fixed and reloaded from Earth. https://en.wikipedia.org/wiki/VxWorks https://en.wikipedia.org/wiki/VxWorks
- BiteCode_dev 6y agoAnd they added Python 3.8 support last year: https://blogs.windriver.com/wind_river_blog/2019/09/vxworks-7-now-has-a-pet-snake/ https://blogs.windriver.com/wind_river_blog/2019/09/vxworks-... This damn snake can be compiled on anything.
- dilyevsky 6y agoI’ve worked with vxworks and I don’t think it can be considered Unix unless something changed in last 10 years
- BiteCode_dev 6y agoCan you give some details about the experience? I'm curious about this system, but not curious enough to spend a week playing with it.
- dilyevsky 6y agoWe used it for “hard realtime” embedded system didn’t even use virtual memory. Used to have pretty archaic toolchain (diab and some clunky eclipse setup). The os itself seemed pretty meh too (we had license and hence all the sources). Not sure how you would even get to try it without some trial agreement with their rep? Maybe try Qnx instead? It implements Posix so at least you’ll have easier time moving to linux. Also hard realtime and actually is a micro kernel arch which is nice
- tonyedgecombe 6y ago
- qchris 6y agoKnowing that the ground control software is written in LabView is one of the more disappointing things I've read recently. I did a bunch of test engineering during co-ops during college, including multiple "codebases" of LabView, plus the normal class coursework and research lab instrumentation. I hate LabView, to the degree that I made a commitment never to apply to a job that has it's use as one of the requirements or as part of the desired qualifications. I remember working on a getting a brushless motor test stand up and running using a LabView VI, which was written by one of the senior test engineers a couple years before. After spending days working on it, I eventually went to him for help. He sat down with me and it for another two hours, and even then, he needed to take the code (that he wrote!) and go sit down with it alone the next day for several hours to understand what was going on. If you're not working it essentially full-time, on the same codebase, it's insanely difficult to maintain any non-trivial code. What makes the situation even a bigger shame is that National Instruments makes Labwindows/CVI, an ANSI C equivalent to LabView. The few times I've used it (in a graduate-level experimental physics class), it was so much easier to use and maintain.
- jccooper 6y agoThat AMA is from 2013, when LabView would have seemed like a more reasonable choice. Dunno if they've changed since then, but it's possible they've come to their senses.
- bmitc 6y agoHow well do you actually know LabVIEW? Have you ever used its object-oriented system, actor framework, network streams, or get/set control value by indices (a feature actually implemented for SpaceX) functionalities? These are all things a professional LabVIEW engineer will be familiar with and use. The fact that a system didn't use these means it's either seriously outdated or the person writing it wasn't aware. It sounds like you hate LabVIEW because you worked on a few poorly architected systems, as I don't see anything in your hate for it that actually applies to LabVIEW the language/environment. > If you're not working it essentially full-time, on the same codebase, it's insanely difficult to maintain any non-trivial code. It's hard to maintain any codebase if you're not giving it attention. I would argue that LabVIEW, when properly architected, can be maintained just fine and maybe even easier than other languages. I've done it. LabVIEW is highly performant and excels at throwing data around and processing it and interacting with hardware. Its dataflow nature actually makes it easy to modify and maintain, when done properly. There aren't many languages that make it that easy to make user interfaces and distributed systems. Another language/ecosystem that I would consider for such a system is Elixir/Erlang, but you'll be searching for a user interface solution and lots of hardware drivers. However, with Elixir/Erlang you gain a system even more built for distribution but you lose out a lot on synchronous/sequential performance. C#/F#/.NET are other options to use since NI makes quite a lot of .NET drivers.
- sm4rk0 6y ago(2015)
- dang 6y agoUrl changed from https://space.stackexchange.com/a/9446 https://space.stackexchange.com/a/9446 to the main page.
- rurban 6y agoLabView? Lol. That's the cheapest SW for this kind, usually used only in countries which cannot afford or is not allowed to use the industry standard simulation SW. Like Iran or North Korea. The rest looks good, similar to the architecture used in the shuttle program (3 redundant dSpace realtime Linux boxes).
- AndreyKarpov 6y ago> The flight software is written in C/C++ I wish we could check the source code using the PVS-Studio analyzer to see what we can find out there :). It would be more interesting than these standard articles on project checks.