55 ms·
Software Engineering Within SpaceX
- theanirudh 6y agoI wonder how they manage not to have accidental taps on the touch screen during liftoff and or re-entry. As I understand there are a lot of G's and violent vibrations and I would assume it's hard to keep a steady hand? (Atleast this is my understanding from watching Apollo documentaries/movies etc.)
- tericho 6y agoWatching the launch, I noticed they seem to have finger rests below the screens. I wonder if this is the reason.
- NikolaeVarius 6y agoThe screen compensates, and they train with vibrations in mind.
- rrss 6y agoand they also don't have anything to do during launch except maybe abort with the physical handle.
- theanirudh 6y agoImpressive, especially considering some of the touch targets are pretty small.
- NikolaeVarius 6y agoThey probably aren't bothering with using non critical buttons on a launch that is automated anyway
- karatestomp 6y agoI doubt anything on screen can affect flight during lift-off or re-entry. Anything it might had damn well better be locked-out anyway. Lift off's automated and I'd think any significant failure of that automation would be a cue to abort (physical control for that); as for re-entry, by the time you're deep enough in atmosphere that things are vibrating you're really just falling with style, and there's no "flying" to do nor anything to control.
- bkanber 6y agoLiftoff and re-entry are completely automated. The astronauts are passengers and don't interact with the craft during these maneuvers. This was the case even for Apollo. The Crew Dragon seats also have physical arm-rest controls. Those arm-rest controls are what the astronauts use under acceleration, and IIRC they're primarily comms controls.
- wlesieutre 6y agoInteresting read. I've wondered about their use of big touchscreen interfaces having heard a friend's experience with the similar setup in a Model 3. On multiple occasions they've had to pull off the highway to turn their car off and on again to get the screen working. Not really an option on your way to space.
- theanirudh 6y agoThey have physical buttons Incase the touch functionality misbehaves.
- remarkEon 6y agoBut there’s only a handful of buttons, no? Unless I’m missing something. Might be a dumb question but the Apollo control panels had hundreds. How would you manually fly the Dragon in the event of a screen failure? I’m assuming this question was answered somewhere, since the thing was certified to fly, but I must’ve missed it. Edit: Is the answer simply "it doesn't, it gets flown remotely from Mission Control"?
- NikolaeVarius 6y agoYou don't manually fly it? The physical buttons correspond to pre-programmed procedures/abort modes. They use a touchscreen so they don't need to have hundreds of buttons for things that aren't necessary to have around all the time.
- theanirudh 6y agoIm not sure, even I would like to know the answer to this.
- SiempreViernes 6y agoWell, Apollo did a lot of complicated stuff in space and on the moon, while the spacex capsule just goes up and then down. Only questionable bit is how they do the docking without the screens.
- ChrisMarshallNY 6y agoThis was cool! Thanks for sharing that with us. I'd be interested in finding out how they iterate. I'm absolutely positive that they do.
- thoughtpalette 6y agoOff topic but we share the same name. Except I'd be ChrisMarshallCHI.
- ChrisMarshallNY 6y agoI met another Chris Marshall that worked for Kodak, back in the 1990s. Have you ever worked for Kodak?
- thoughtpalette 6y agoI have not. Though, I have been eyeballing chrismarshall.com for years, setting calendar reminders on expiration's and such. Have you been doing the same? Was curious if all the CM's are competing for that domain lol
- ChrisMarshallNY 6y agoI've had cmarshall.[com|org|net] for a long time. I recently also fetched chrismarshallny.[com|org|net|dev], along with a bunch of various social media handles (see my HN ID). Right now, they point to my LLC site, but I'll probably set up a personal site; sooner or later. I did notice that a squatter had grabbed the Pinterest name. I don't care. I've decided that ChrisMarshallNY is my "Google Me" name. It's working fairly well.
- thoughtpalette 6y agoAhhh nice. Fair enough. I settled on thoughtpalette on everything but I wish whomever owned our namesake domain would do something with it \_O.o_/
- aphextron 6y ago>SpaceX also made use of Chromium and JavaScript for Dragon 2 flight interface. I am not sure how that passed the certification. I assume it was allowed because for every mission-critical input on the display, there was a physical button underneath the display as well I think that's all the validation we need for HTML/CSS/JS as the best tool for UI development nowadays. I wonder if there was actual shared code from the Dragon UI used in their online docking simulator. How neat.
- ShinTakuya 6y agoDepends on the environment. If RAM or CPU time is at a premium then no. But that's so rarely an issue these days, especially if you separate UI from the main processing thread.
- deleted 6y ago[deleted]
- Aperocky 6y agoI'm surprised you're allowed mission-critical or even non-critical inputs on that touchscreen. It might be OK if the entire screen was READ-ONLY. Anything else sounds like a disaster waiting to happen.. I also don't understand why companies race to have their laptop display touch, but then again I'm just a random software engineer.
- ecmascript 6y agoWhy is it a disaster waiting to happen, according to you?
- thoraway1010 6y agoI'm also curious about this disaster waiting to happen. They've emphasized an escape capability all the way into orbit. They emphasized various types of automation (also used by other spacecraft).
- sdwedq 6y ago
- naringas 6y agoI would have expected them to use formal verification tools in the vein of TLA+ and such... or maybe use ADA for mission critical systems? But they only mention Astree[1] which seems to be a propietary analyzer for C code [1] https://en.wikipedia.org/wiki/Astr%C3%A9e_(static_analysis) https://en.wikipedia.org/wiki/Astr%C3%A9e_(static_analysis)
- Denvercoder9 6y agoIt's just a bad article. The whole section about Astree is based on a quote on HN about the ATV, a European resupply craft developed 15 years ago that has absolutely nothing to do with SpaceX.
- dahfizz 6y agoTLA+ is basically an academic tool. It allows you to verify your specification, but it is useless to detect bugs in your implementation. In my experience, the process of writing your specification in TLA is a buggy enough process to make it all a waste of time.
- superqd 6y agoNo, not just for academia. Apparently, Amazon uses it to check their designs for correctness. https://lamport.azurewebsites.net/tla/formal-methods-amazon.pdf https://lamport.azurewebsites.net/tla/formal-methods-amazon....
- im_down_w_otp 6y agoI would have expected them to use SCADE, but given their choice of stacks that are entirely unverifiable, due to a combination of their complexity and their semantics, I can only assume that "control panel" was basically decorative, and that actual critical functions and alerts were handled by something with much, much higher safety assurances.
- chasd00 6y agoat the bottom of the article they mention model rockets and the three levels of certification. Each level grants you access to more powerful motors and therefore higher or larger flights. The hobby is self-goverend by NAR and Tripoli who manage level certification. It's a fun hobby, although large motors get pricey. The largest can be 4-5 figures per launch. However, you can get very advanced and do things you wouldn't typically expect in a hobby. Here's a two stage ( 4" diameter booster, 3" diameter sustainer ) reaching over 200k feet in altitude. The Karman Line is about 330k feet. https://www.youtube.com/watch?v=g0imcpdLdB8 https://www.youtube.com/watch?v=g0imcpdLdB8
- 0xDEEPFAC 6y agoNo mention of Ada or their methods of writing the important software. I wonder what they use. "Avionics Test team ...The main objective is to write very comprehensive and robust software to be able to automate finding issues with the hardware at high volume...."
- lavezza 6y agoThe "important" software is the Flight Software that runs on Falcon and Dragon. That is written in C++.
- stevofolife 6y agoTaken from the article: "We leverage C#/MVC4/EF/SQL; Javascript/Knockout/Handlebars/LESS/etc and a super sexy REST API." Knockout.js, good times.
- nightowl_games 6y agoI'm sitting over here believing Knockout.js was better than today's JS systems.
- hu3 6y agoI believe Trello uses Knockout.js And at least before Atlassian bought it (I haven't used since then), it was a responsive and plasant piece of software.
- benjamind 6y agoThe Crew Displays do not use Knockout.js. As far as I am aware this is possibly used in some other asset management software developed in-house for manufacturing teams but definitely not onboard Dragon.
- theanirudh 6y agoThe astronauts show parts of the touchscreen and physical controls here: https://youtu.be/llbIzbOStt4?t=150 https://youtu.be/llbIzbOStt4?t=150
- tectonic 6y agoGood writeup! In general, the direction in modern aerospace is to use COTS (commercial off-the-shelf) parts with redundancy and failback for radiation hardening. If you’re into this sort of thing, I co-write a weekly newsletter about the space industry and often talk about software. https://orbitalindex.com https://orbitalindex.com
- sq_ 6y agoThank you so much for the Orbital Index! I've been subscribed for a while now, and I look forward to it every week. Great content!
- junon 6y agoSounds neat, first newsletter I've voluntarily subscribed to since I was a child. Looking forward to the next issue.
- yasoob 6y agoI just went through the last issue and enjoyed it quite a lot. Good stuff :)
- ThisIsTheWay 6y agoJust FYI but OI has become a great resource for me and my team to maintain situational awareness on what's going on in the industry. Thanks for all you do!
- Symmetry 6y agoI'd worry that the COTS approach might be fine for dealing with cosmic rays within the Van Allen belt but it might not be sufficient to deal with the solar wind if you're going to GEO or Mars.
- f00_ 6y agotouch screens are a bad choice to me I want the buttons and knobs. Love the old soviet control rooms posted awhile ago: https://designyoutrust.com/2018/01/vintage-beauty-soviet-control-rooms/ https://designyoutrust.com/2018/01/vintage-beauty-soviet-con... Need John Carmack's opinion of SpaceX
- josefresco 6y agoOh so beautiful: https://main-designyoutrust.netdna-ssl.com/wp-content/uploads/2018/01/18-12.jpg?iv=146 https://main-designyoutrust.netdna-ssl.com/wp-content/upload...
- f00_ 6y agoi love the windows xp screen saver ):<
- sammycdubs 6y agoThis seems like a remarkably small team for the scale of what they're building! For some reason I imagined they'd have legions of engineers.
- bdcravens 6y agoI think we’ve been conditioned to believe that any application of medium complexity requires hundreds or thousands of developers. Snarkiness aside, they have a very concrete set of functional constraints and total control over the hardware, network, and software environment. Issues like resource management, dependency conflicts, security, scale, etc are taken out of the equation, things that are often the biggest time and resource sinks.
- junon 6y agoIME, more bodies usually always equates to more bugs and slower release cycles.
- adambyrtek 6y agoAccording to the post these quotes are from 7 years ago... I'm sure a lot has changed since then.
- bkanber 6y agoNASA's software teams are also remarkably small, and manage to produce code with one of the lowest defect rates of all time. (Space shuttle code had 0 defects in 500k SLOC!) Just goes to show: quality over quantity!
- 0xffff2 6y agoShuttle's code quality was outstanding, but 0 defects is a myth. There are at least 3 publicly documented bugs in the Shuttle code. https://space.stackexchange.com/a/37116 https://space.stackexchange.com/a/37116
- yasoob 6y agoHi guys! I am the author of this article. Excited to see it on the first page :)
- chrisfinazzo 6y agoHearing about the Flight Software and Avionics teams reminds me of this, although they don't seem to be on that level quite yet. https://www.fastcompany.com/28121/they-write-right-stuff https://www.fastcompany.com/28121/they-write-right-stuff
- randtrain34 6y agoNote that the data within the post is from an AMA 7 years ago
- oxguy3 6y agoI'm so relieved to hear all the redundancy and testing in place. I had heard that the touchscreens were built in Chromium/JS and was rather alarmed. Don't get me wrong – I do a lot of web stuff and I love that environment, but I've never seen a web app I would trust two human lives to. This, however, sounds like they really thought it through and made it safe.
- paxys 6y agoA contrary opinion - a tool/framework used and tested by billions of people every day is a lot less prone to crashing when used for its intended purpose than something custom-built. There are tons of developers out there building complex apps by following beginner JavaScript tutorials, but SpaceX is obviously going to enforce better standards. HTML/CSS/JavaScript/V8 are all extremely solid technologies that have stood the test of time, and there is nothing better to build a user interface with today.
- drummer 6y agoIt would be awesome if some SpaceX engineers would give a few presentations at events like CppCon and talk about their software development process including some code examples and demos.
- danans 6y agoYears ago, astronaut Chris Hadfield told an audience of software engineers (including me) that the moment the space shuttle was in stable orbit, the crew would pull out laptops and set up an ethernet network for all the scientific work of their expedition, as the space shuttle's own computers, though limited in raw computing power, ran software that was so thoroughly tested that there was every reason not to "upgrade" them in any way to support the scientific work.
- WalterBright 6y agoIt makes perfect sense to use computers for research on the shuttle that were not connected in any way to the shuttle systems.
- thechao 6y agoAs a child, a friend's mother was a programmer for the Shuttle (mid-80s to early 90s). Her job sounded awful: a 'science person' (I was a child, remember!) designed a piece of math; an engineer (software?) would take that piece of math and reduce it to an algorithm; my friend's mother would program that algorithm in machine code, in parallel to an assembly listing. Each instruction (and all the data) would be reviewed line-by-line. She had to provide reasoning (roughly a proof) that each instruction correctly implemented the algorithm. My guess is the algorithm went through the same equivalence checking to the 'math'. I don't know how much code she wrote, but it was only a few programs (functions) per year. That was my view of 'programming', and I wasn't disabused of that until very late 90s.
- mips_avatar 6y agoI'd love to work with physical software (software that interacts with the real world through sensors and actuators), as a C developer, how should I move into this space? Every time I try intro to ARM kits I feel like I'm in over my head.
- Symmetry 6y agoI've gotten recruiter email from SpaceX but I've worked in embedded sensor systems, big radars for the government, and robotics so those might be stepping stones. Be warned you might have be a US citizen.
- zachlite 6y agoI worked with AVR microcontrollers, and the avr-gcc toolchain when I was in college. It's basically an abstraction layer below what an Arduino provides you. https://en.wikipedia.org/wiki/AVR_microcontrollers https://en.wikipedia.org/wiki/AVR_microcontrollers
- lnsru 6y agoStart reading data sheets of these ARM parts. It’s straightforward when you know what peripherals do you need. Get familiar with FreeRTOS or other embedded operating system. It’s really not rocket science. You only need to know, what do you want to build. Otherwise it gets boring very quickly.
- davedx 6y agoI’m having a lot of fun with Arduino at the moment. You can do a lot with them.
- chasd00 6y agoI know you said you're a C developer by the MicroPython boards get you up and running pretty quickly. It's straightforward to get a blinking LED ( the Hello World of embedded programming ). Otherwise, I would use a STM32 dev board and their IDEs and tools. It will get you off the ground and blinking LEDs way faster than working out the toolchain, loaders, ASM, setting the clocks and peripherals and all that work that has to be done before you even get to your application.
- dang 6y agoA thread from a few days ago: https://news.ycombinator.com/item?id=23368109 https://news.ycombinator.com/item?id=23368109. The current article quotes from it, in fact. This article looks like a fine overview but when it comes to follow-up posts, the test is: does the new submission contain enough SNI (significant new information) to support a substantially different discussion? In this case it looks like not, but I can't really tell. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=by%3Adang%20%22significant%20new%20information%22&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20follow-up&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
- fallingmeat 6y agoDoes this article imply that RTCA/DO-178B is used as a means of demonstrating compliance in some way, or otherwise is used to define lifecycle processes for their development/verification/systems teams? Anyone know where this was mentioned by SpaceX?
- extrapickles 6y agoWhen you think about it from a first principals perspective, having multiple touchscreens is better than only having physical switches. When a switch is damaged/fails, you are out of luck. When a touchscreen is damaged/fails, you use the one next to it. On a rocket you do not have the mass or room to have more than 1 of all but the most critical of switches. There have been quite a few missions that nearly caused death or mission failure directly due to a switch getting broken (Apollo 11, lander return engine-arm switch) or going faulty (Apollo 14 abort switch). What really matters is that they have no single point of failure (touch screens can do everything switches can, an individual touch screen is not important, and switches can cover abort/return scenarios to protect the crew). For the software, it only matters that its been fully tested, including random bit flips and hardware failure. From a cost savings perspective, its vastly cheaper to verify that 3 touchscreens are working correctly than the 600 switches they replace.
- colllectorof 6y ago>When a switch is damaged/fails, you are out of luck. When a touchscreen is damaged/fails, you use the one next to it. This is a trivial problem to solve on a physical interface. One solution could be what is commonly used on hardware synthesizers. A shift button or switch. You engage it and all controls begin to perform their secondary functions. You get redundancy for the price of one extra control and a secondary set of labels in a different color. Also, use of displays to virtually label buttonss is common. In such case you can reassign a control if one fails. In any case Dragon capsule had physical buttons for important functions as a backup.
- extrapickles 6y agoThe touchscreen frees you from the complexity that comes with giving switches alternative modes, and gives you the mass to have multiple copies of critical switches. Also multimode switches greatly increase the complexity and failure modes, so they need to be done so that if a switch is triggered in the wrong mode its recoverable (eg: the switch for aux radio power isn’t also the undock switch). When you get to the point of having displays for the switches why not go full touchscreen and eliminate all of that cost and complexity of a bunch of tiny displays?
- Animats 6y agoIt's interesting that the mission control console systems are written in LabView.
- themeiguoren 6y agoI can see how it’d be a decent tool for plotting up a bunch of raw telemetry streams, but as someone who had to write a moderately complicate program in LabView once, I’m astonished they’ve scaled it up that far and very glad it’s not me who had to do it. It’s a real PITA to work with.
- b20000 6y agodoesn't sound like a great idea to involve 10 different technology stacks
- scep12 6y ago> The Flight Software team is about 35 people. I'm shocked the discussion is about UI tech and not that there was only 35 people on the team that built the software to land Falcon 9. Surely it's changed in the last 7 years. Anyone know the size now?
- jonpurdy 6y ago> The secondary ports go into the primary ports, which are heavy-duty actuators that connect to what’s called a “summing bar,” which is no more than a massive steel rod. "In Rod We Trust"
- MrSaints 6y agoThey will be doing an AMA on Reddit again soon according to https://youtu.be/y4xBFHjkUvw?t=674 https://youtu.be/y4xBFHjkUvw?t=674