19 ms·
Starliner faced “catastrophic” failure before software bug found
- ceejayoz 7y ago> According to the source, Boeing patched a software code error just two hours before the vehicle reentered Earth's atmosphere. Had the error not been caught, the source said, proper thrusters would not open during the reentry process, and the vehicle would have been lost. Uh, that's extremely concerning for a CREWED capsule.
- jpollock 7y agoDoesn't that depend on the testing schedule? If the schedule called for simulations to be run in parallel with the live test, then it's an expected outcome. It should be _expected_ that every test will find a problem. Since this was uncrewed, there was no risk (other than to the uncompleted tests possibly requiring a second flight) to running them in parallel and porting across fixes for any problems that were found. It was a schedule compression attempt with a cost of second test flight risk if it failed.
- JshWright 7y agoBoeing took the "we'll do very rigorous engineering up front and prove everything on paper" approach, where SpaceX took the "we'll prove it works by actually launching it" approach (which isn't to say SpaceX isn't operating with engineering rigor). In theory Boeing ran all their simulations over the past couple years, and this flight should have just been a formality. As it turns out, Boeing is running into a lot of issues when they actually test their hardware.
- mechhacker 7y agoThe problem with simulations and paperwork with high tech engineering is you need enough competent, independent reviewers that understand how the whole system works together. That sort of thing is rarely organized. So we test it instead.
- V_Terranova_Jr 7y agoThis is likely an oversimplification. There are multiple organizations, even within Boeing Defense & Space, that write their own flight software. All stovepiped and largely working in parallel. This doesn't even include the commercial folks, infamous for the 737 Max. My understanding is that the St. Louis teams are better regarded, and the folks that worked on DARPA HACMS deserve some credit, but they seem to be outliers. Boeing's culture doesn't seem to prioritize modern software development methods or software rigor on the whole. Functional testing should be the last layer of bug-hunting techniques, not the first or primary. The issue seen on their capsule didn't surprise me at all. Other BDS software groups use utterly outdated software development methods and we should all be a little bit worried.
- JshWright 7y agoObviously it's a simplification; comparing and contrasting the approaches of SpaceX and Boeing would require several walls of text... At the core of the issue though, the process that SpaceX pitched to NASA (and NASA approved) involved quite a bit of actual hardware testing. Boeing's plan (also approved by NASA) relied much more heavily on simulation, modelling, and other sorts of process validation. For instance, Boeing did not perform an in-flight abort.
- torpfactory 7y agoIt kind-of depends on what sort of issue you're finding in your test. Complex systems like spacecraft often have unexpected interaction effects which only testing can reveal. I would call these the 'good' kind of test learnings - ones that an army of great engineers wouldn't be able to predict. This is editorializing but it looks like Boeing didn't uncover very complicated interactions - they failed at a more basic level of competency - timer synchronization for the launch issue and then a major software bug for an important orbital maneuver. Those sorts of issues really should be sorted out on the ground using hardware simulators. Furthermore, the timer synchronization failure prevented testing of the docking hardware, further delaying the overall program. For a human rated vehicle, personally I think you should have at least one full-up, fully nominal test before you send anyone along for the ride.
- moftz 7y agoIt depends on if the crew would have been able to control those thrusters themselves. Obviously you want the entire system to be autonomous enough that no crew interaction is required but things do happen and the crew needs to be able to act and fix the problem if capsule can't do it nor the ground. When the Starliner started a burn at the wrong time, a crew would have been able to stop it and prevent the loss of fuel. I wonder if this re-entry thruster issue was a result of the earlier thruster issue (or a result of the troubleshooting of it). There are uncrewed test flights for a reason. You can't always simulate every possible failure mode. Things fail on the ground that wouldn't be possible during normal operation and vice versa.
- ceejayoz 7y ago> You can't always simulate every possible failure mode. Things fail on the ground that wouldn't be possible during normal operation and vice versa. This should be concerning, then: https://spaceflightnow.com/2019/11/04/boeing-starliner-pad-abort/ https://spaceflightnow.com/2019/11/04/boeing-starliner-pad-a... > “Boeing is not going to do an in-flight abort test,” said Jon Cowart, deputy manager of the mission management office for NASA’s commercial crew program, before the pad abort test. “They’re just going to do the ground one. They think that they can get enough data and then extrapolate that out, with good analytical techniques that we’ve endorsed. They will go and do it in that particular way, versus SpaceX, which is going to do both.
- ClumsyPilot 7y agoYou'd think that after two air plane disasters, they'd tread carefully
- xvf22 7y ago> Finally, before the meeting ended, the chair of the safety panel, Patricia Sanders, noted yet another ongoing evaluation of Boeing. "Given the potential for systemic issues at Boeing, I would also note that NASA has decided to proceed with an organizational safety assessment with Boeing as they previously conducted with SpaceX," she said. This is a welcome development.
- sjg007 7y agoMan I hope the QA person who found that bug was rewarded..
- galaxyLogic 7y agoMakes me wonder why put all this effort to make space-travel safe for crews. Why not focus on remotely controlled or AI autonomous vehicles instead?
- jve 7y ago> Makes me wonder why put all this effort to make space-travel safe for crews Would you want to board that capsule yourself, otherwise? > Why not focus on remotely controlled or AI autonomous vehicles instead? Buzzwords doesn't make something more reliable/less error prone. You really think that by throwing in something like AI, which can fail in unexpected ways, be a good idea?
- galaxyLogic 7y agoI think it's worth studying and developing further, for instance study how to make AI which will not "fail in unexpected ways"
- ben_w 7y agoI think we already do — there have been roughly 15 satellites etc. launched for every human who has ever reached earth orbit over the entire history of human spaceflight.
- galaxyLogic 7y agoGood. I was just wondering why this project in particular needed to have a live crew.
- jonplackett 7y agoIt’s hard enough to get an AI to work well in a known situation line driving, never mind an unknown planet. But more importantly - where is the fun in that?
- galaxyLogic 7y agoYou are right some people find it fun to go to dangerous places, I assume. Think about the crew of StarTrek and of course Buzz Lightyear. Personally I would rather send someone else to space than go there myself :-)
- jonplackett 7y agoWouldn’t like to be the guy pushing an update to a crewed capsule just before re-entry. I stress out enough about pushing code as it is. But then I wouldn’t like to be the guy who left an error in the code to begin with.
- smoyer 7y agoSo we know there were catastrophic bugs in the 737 Max, they've found additional bugs that haven't been catastrophic yet and now we hear that the Starliner also has software bugs. I'm going to order a copy of "The Mythical Man-Month" for Boeing ... they need to get way back to the basics. (Where's Margaret Hamilton when you need here?
- trhway 7y ago> I'm going to order a copy of "The Mythical Man-Month" for Boeing too late. Boeing is already deep in Agile, a methodology which promises that a child can be delivered by the way of 9 incremental monthly deliveries. https://www.infosysconsultinginsights.com/2019/06/12/the-risks-of-moving-to-mature-agile-too-fast-a-cautionary-tale/ https://www.infosysconsultinginsights.com/2019/06/12/the-ris... "Boeing was an early Agile adopter in 2008 surpassing its rival, Airbus, in 2012 by deploying a newly renovated 737 Max 8 faster to market. [...] The 2008 article Boeing Frontiers- Goin’ Agile by Doug Cantwell from Boeing describes how Boeing, in partnership with Lockheed Martin, created an Agile lab to move changes to the aircraft to market it faster, cutting down flight test times from months to days. "
- tracerbulletx 7y agoAgile is great when the maximum cost of system failure is momentary decreased sales.
- deevolution 7y ago"Individuals and Interaction" over "Process and Tools" doesn't seem like a great idea when developing software that handles life or death scenarios. Boeing, being the finatialized zombie company that they are now though have probably run some economic analysis and determined the optomal time to test / cost of life ratio. They must have determined that the cost of adequately testing their software is waaaaay more expensive than losing an astronaut here and there or some plebian passengers.
- erikpukinskis 7y agoThe baby analogy is pithy, but I don’t really see the comparison. You don’t do your software development in increments smaller than a month? What does that mean, you spend two months just writing specs?
- mzs 7y ago"Given the potential for systemic issues at Boeing, I would also note that NASA has decided to proceed with an organizational safety assessment with Boeing as they previously conducted with SpaceX"
- geerlingguy 7y agoAh so just the fact that Boeing has been around for half a century doesn't absolve them from being subject to safety reviews? Nice to see some sense.
- pstuart 7y agoIt's McDonnell-Douglas with a Boeing name -- beancounters run the show now.
- webpaymentsguy 7y agoYou put a very disappointing feeling into words. Such a shame.
- rflrob 7y agoWell, their CEO also doesn’t go on podcasts and smoke pot. Not that I think that’s necessarily a better or worse sign that SpaceX is or is not safe than a CEO drinking moderately in public, but we are prudish about these things in the States.
- deleted 7y ago[deleted]
- m4rtink 7y ago""While this anomaly was corrected in flight, if it had gone uncorrected it would have led to erroneous thruster firing and uncontrolled motion during SM separation for deorbit, with the potential for catastrophic spacecraft failure," Hill said during the meeting." I guess some things just never get old, citing from https://en.wikipedia.org/wiki/Soyuz_5 https://en.wikipedia.org/wiki/Soyuz_5 : "The flight was also memorable for its dramatic re-entry. The craft's service module did not separate, so it entered the atmosphere nose-first, leaving cosmonaut Boris Volynov hanging by his restraining straps. As the craft aerobraked, the atmosphere burned through the module. But the craft righted itself before the escape hatch was burned through." This actually happened three times so far with the Soyuz (in all cases without the loss of crew): "An incomplete separation between the Service and Reentry Modules led to emergency situations during Soyuz 5, Soyuz TMA-10 and Soyuz TMA-11, which led to an incorrect reentry orientation (crew ingress hatch first)." (from https://en.wikipedia.org/wiki/Soyuz_(spacecraft)#Service_module https://en.wikipedia.org/wiki/Soyuz_(spacecraft)#Service_mod...) One would kinda expect that past crewed vehicle emergencies would be studied in detail when designing a new one & that the developers would make extra sure they can't reasonably happen with their design.
- worik 7y agoGiven commercial pressures would a Boeing reentry vehicle be over designed to such a extent that on such a failure (failure to separate, entering with the wrong attitude ?correct term?) result in "...the craft righted itself before the escape hatch was burned through"?
- m4rtink 7y agoI would assume it would work the same as with the soyuz, provided that the service module separates/explodes before the capsule reentering the wrong way is irrecoverably damaged. Basically, all space capsules have their of gravity placed in such a way that they will automatically orient themselves heat shield forward once they encounter the atmosphere. So once the service module is gone, it should flip into the correct orientation just by physics alone. (BTW, this is the same reason why the Crew Dragon spacecraft keeps it's aft section "ring" attached during a launch escape, where it's super draco thrusters drag it to a safe distance from a failing launcher. The aft ring prevents the capsule from trying to flip over during the abort. Then once in safe distance from the vapor & debris cloud that used to be the launcher, the aft section is jettisoned and the capsule again automatically re-orients itself heat shield forward.)
- classicsnoot 7y agoOn a certain blog that is completely outside acceptable standards for wrongthink and political correctness, a very popular topic of late is why NASA seems to have it in for Elon Musk Personally and SpaceX generally. The commonly stated reasons, and I will be paraphrasing and transliterating freely, are: HR culture defining administration wide objectives and methods and reasoning, professional embarrassment over languishing reputation, gross incompetence, turf defense of budget and status, and a desire to stay firmly planted on Terra while being lauded for dreaming of the stars. I am always skeptical of any argument that is unfamiliar, but more and more it does appear that NASA has lost its way. The shuttle was an obvious mistake in retrospect; there may even be some credibility to the obscure theory that NASA only did it to further separate themselves from DoD. I think NASA has become a political creature that is less concerned with science and more concerned with SCIENCE™. If this is the case, they will fight tooth and nail against any expansion of manned space exploration (because it will be both private and military in nature), the will fight against innovation that doesn't spring from their own workshop(s), and they will use Cape Canaveral (and their heritage facilities/infrastructure) as a way to bully "adversaries" into submission. I hope this isn't the case, and if it is, I hope they can reverse whatever practices and policies that have led us to where we are. As it stands though, it appears NASA is more like OSHA then it is like its historical instance.
- mdocherty 7y agoWhich blog?
- jshevek 7y agoThe author says elsewhere that he has (paraphrased) gotten in trouble with hacker news for linking to it in the past.
- jbay808 7y agoI'm confused by what you mean. NASA has provided SpaceX with lots of expertise and assistance. And you are aware that Starliner is a Boeing vehicle, not SpaceX?
- m0zg 7y agoI wonder if Starliner software too was outsourced to India like 737 Max. In contrast, all of SpaceX R&D is in the US, and the hiring bar is pretty stratospheric (pun intended).
- axilmar 7y agoWhat I would like to know is what the software bug was. Unfortunately the article does not say it, and I am sure no information online exists about this.