4 ms·
I was curious about how reliable we should predict a Falcon 9 launch to be based on the stated design parameters (survives the loss of any 2 engines) and curren
by btilly 14y ago
I was curious about how reliable we should predict a Falcon 9 launch to be based on the stated design parameters (survives the loss of any 2 engines) and current launch data.
I found that under a simple Bayesian model, the estimated probability of catastrophic failure during the next launch is 3.6%. See http://bentilly.blogspot.com/2012/10/how-reliable-will-falcon-9-be.html http://bentilly.blogspot.com/2012/10/how-reliable-will-falco... for details.
Update: I made a careless calculation error. Change 3.6% to 2%. Oops.
- ghostfish 14y agoTwo thoughts here. 1) You're working from an awfully small sample, especially for a component that is supposed to be of such a high reliability. Then again, look at the probabilistic assessment of Shuttle success vs. actual. 2) The Merlin 1C engine is only going to make a few more flights afaik. Starting some time in 2013 they'll be switching to Merlin 1D.
- btilly 14y agoIndeed, predictions are extremely hard. Especially about the future. At least I am explicit in the assumptions behind that model. They are: 1. A priori, all possible reliability numbers for a single engine are assumed equally likely. (This can be debated endlessly, but you need SOME prior for Bayesian analysis. If we had more data, then the prior would matter a lot less, but we don't so it does.) 2. Failure of engines is independent. 3. The rocket actually operates according to design parameters. That is it will survive the loss of any 2 engines, but not the loss of 3. 4. Past performance is a predictor of future performance. Every one of those assumptions is questionable.
- pilom 14y agoUnfortunately #2 is extremely unlikely to be true. Engine failures are very related in very unexpected ways. The SpaceX guys have worked very hard to make them as independent as possible but with so many failure modes it is impossible to say that an engine failure (i.e. explosion) will have no impact on its neighbors.
- btilly 14y agoYup. I invite you to create a toy model that takes that into account in some way and see what answers you come up with.
- neurotech1 14y agoActually, in the NAS Oceana F/A-18 crash, two different engines failed for different reasons. The first one was a right engine compressor failure, the second was an apparent afterburner blowout. A twin engine jet crashed as a result.
- pilom 14y agoDo you have a primary source? I'm genuinely interested in the report if it is available. Google just gives news stories saying "OMG a plane crashed into an apartment building!" which isn't helpful to the engineer in me.
- neurotech1 14y agoHere is the mishap report http://goo.gl/GuHG5 http://goo.gl/GuHG5 I mentioned this because double engine failures due to separate causes do actually occur, although rarely. Short version: The right engine compressor failed due to apparent fuel ingestion, causing a major over-temp. The noise was mistakenly attributed to a blown tire, so the pilot left the gear down. This required MAX Afterburner on the remaining engine to recover, except the engine had a afterburner blowout, and didn't provide MAX power and the jet departed controlled flight due to low speed.
- pilom 14y agoThanks for the report. A really interesting read for the engineer in me. Under "primary cause analysis" on page 18 of the report (pg 24 of the pdf) it says that they may have actually been related: In summary, after the right engine failed due to fuel ingestion, the left engine had to push some air over to the non-functioning engine (for cooling I assume but it isn't stated). When the left engine afterburner did not light, it's "relight logic" did not trigger possibly because of the lower air amount. So the engineer who wrote the relight logic, assumed that the temperature would drop at a certain rate when the afterburner failed to light. Because the engine was working to assist the failed engine, that temperature drop did not happen and thus the afterburner did not attempt to automatically relight itself. Sounds like it may be dependent after all.
- maaku 14y ago> 4. Past performance is a predictor of future performance. This is your real error. SpaceX is effectively acting in perpetual “test” mode. New lessons are learned with each flight and each rocket test, and that greatly informs all follow-on operations. The type of analysis you've done is reasonable for a fleet of deployed 747's. It is (forgive me) entirely useless for this sort of endeavour at it's current stage. I wonder what your analysis of SpaceX's future performance would have been in Oct 2008, right after their first successful Falcon 1 launch and three failed priors?
- hyperbovine 14y agoWhat is the variance of your estimate?
- btilly 14y agoSorry, your question does not make sense. In Bayesian analysis you have some set of prior theories with associated probabilities, you observe data, you alter your estimate of the likelihood of those theories, and then your next prediction becomes a weighted average of those predictions. That number has no attached variance.
- wisty 14y agoYou could talk about credible intervals, which is a similar thing.
- btilly 14y agoI could. But that would be inappropriate here. The lower bound of any such interval would be astonishingly close to a failure rate of 0, while the maximum likelihood estimator is itself fairly close to 0. Therefore any reasonable set of priors usually puts you in the upper tail. For instance one such interval has a MLE of an engine failure rate of 2.78%, a lower bound of 0.1% and an upper bound of around 15%. (Corresponding rocket failure rates range from 1 in 10 million to about 9.9%.)
- jfoutz 14y agoNot really, if you'd built your model from a billion launches and 20 million failures, a dozen failures in a row won't change your priors much at all. You've built your model off of far less data. How many failures are required to change your model from 2% to, say, 4%?
- btilly 14y agoIt is easy enough to modify the linked script for any scenario that you want. If 2 engines failed on the next launch, the model would predict a 4.8% chance of failure on the following launch. As for your "build your model from a billion launches and 20 million failures" comment, if I had that much data, then it wouldn't much matter what reasonable set of priors that I started with, I'd wind up convinced that the true failure rate was very close to 2%. Note that the prior that I am talking about is the distribution of possible theories before I saw ANY data.
- throwaway54-762 14y agoAre failures independent?
- btilly 14y agoI am assuming so, yes. In fact the failure of your neighbor likely affects the odds of your failure. So I'm being too nice to SpaceX.
- robryan 14y agoOne factor that I don't think has been answered, if this engine shutdown was going to be the 3rd and would cause a mission loss, could if have been persisted with. It isn't clear whether it was on the way to complete engine failure or it was shut down to play it safe.
- gus_massa 14y agoTo make comparisons, IIRC the failure ratio of the unmanned missions is 10-15% and in the manned missions is about 2%. An important detail is what happens when 3 or more engines fail simultaneously. Can they transform the mission in a soft landing or the "payload" is totally doomed? (For example, the Apollo XIII mission was a failure, but no one died.)
- benn 14y agoThe dragon module is meant to be able to save the crew at any stage of the launch.
- cdash 14y agoFor a crew mission, yes the mission would be aborted and the Dragon launch abort thrusters would fire getting them away from the rocket as fast as possible and deploy a parachute where they would land in the ocean.
- uvdiv 14y agoI tried this independently, and have the same result (2.0%). https://gist.github.com/3855660 https://gist.github.com/3855660 And the probability of any rocket engine failing, per launch, is 1 in 3!
- jules 14y agoHere is a simulated posterior on the number of engines that fail: https://dl.dropbox.com/u/388822/rocketfailuredistr.png https://dl.dropbox.com/u/388822/rocketfailuredistr.png Horizontal axis is how many engines failed, and vertical axis is how many times that number of engines failed in the simulation. The code that generated this is: from scipy.stats import * from pylab import * failures = [binom(9,p).rvs() for p in beta(1,1,1e5) if binom(9,p).rvs() == 1] hist(failures, bins=9, range=(0,9)) show() The way this code works is it first picks a `p` from the prior. This `p` represents the failure probability for a single engine. Then we simulate the number of engines that fail when you have 9 of them, and filter out just the worlds where that number is equal to 1 as in the SpaceX launch. Or equivalently, we annihilate all the worlds that have a different observation than ours -- this is a central tenet in Bayesian statistics in contrast to frequentist statistics: we only base our inferences on the things that happened and not on things that hypothetically could have happened but didn't. In a slogan you could say "our fantasies are irrelevant". Then in the worlds that remain, where the same observation was made as in our world, we simulate the number of engines that fail on a new rocket launch and collect the results in a histogram. So the posterior probability on the entire rocket failing for uniform prior is around 25%. Uniform prior means that you believe that all single engine failure probabilities are equally likely: you think it's equally likely that engines fail with 10% probability as with 63% probability. Tweak the first two parameters to beta to change your prior belief. If you've seen n engines fail in your life and k engines succeed, then setting the first parameter to n+1 and the second to k+1 is a reasonable choice (so the current setting corresponds to not having seen any rocket launches prior to this one). For example if you've seen 2 engines fail and 99 engines succeed you use `beta(3,100,1e5)`: https://dl.dropbox.com/u/388822/rocketfailuredistr_for_optimists.png https://dl.dropbox.com/u/388822/rocketfailuredistr_for_optim... Hardly any probability mass left for entire rocket failure :)
- wamatt 14y agoGood stuff OP, nice to see Bayesian getting more attention recently. However, after reading your blog post, I can't help get the feeling this is how a natural frequentist would approach the problem. This method is effectively relying on known, statistical record of past events to form the priors. Often a more useful and appropriate construct in the Bayesian world, is the use of a belief network or Bayesian Network. This is a probabilistic directed acyclic graph (DAG) that encodes priors, often in the form of subjective beliefs (yes subjectivity can be useful), including specific domain knowledge. Common example: Consider a naive Bayesian classifier (a specialized form of belief network) that identifies individual pieces of spam. Do we arrive at the spam score by entering the probability of past events into a simple model based of the Bayes theorem formula? No, it's trained using the vast amount of domain knowledge and pattern recognition (through our experience and own estimation of what 'spam' is) encoded in our minds, that provide the priors. Thus, even though there is a large amount of subjectivity involved, the overall result can objectively be measured, within a given utility function. Incidentally, this is often what makes many hardcore empiricists 'nervous', and hence avoid belief networks altogether. Coming back to the Falcon 9: A piece of prior information outside the scope of historic safety records, for example, one of the lead engineers having a nagging doubt about a particular technical risk based on some observed phenomenon, could have an impact on the real world probability of the next event being a failure. (Which is a pretty useful thing to know!) In fact, this exact scenario happened in 2003 with the disastrous destruction of the Space Shuttle Columbia. [1] An engineer spotted something wrong on previous flights, but management failed to heed the warning[2]. This could quite possibly have been averted, if a risk mitigation model were in place to account for such evidence. Looking forward, it's quite possible to imagine a future where this decision making has been outsourced to a sophisticated AI based off a Bayes net, with far more accurate real world modeling of risk and failure probabilities, outclassing the amount of evidence and a human or committee could possibly hope to compete with. While I've nothing against frequentist approaches (albeit Bayesian naturally makes more intuitive sense to me), a minor drawback is the reliance on the past to predict the future. For example if you had safety records on 1 million previous flights, then one might be tempted to say, "well that's that then, we now know objectively the probability of failures in the future -- end of story". But, the 1 000 001 flight may have been designed to fly on a completely different type of technology, that will change significantly change the safety record of space flight going forward for the next "x" years. Thus using a Bayesian approach account for all relevant priors, it would in theory be possible to reflect a more accurate probability for the 1 000 001 flight, before it took place. Lastly Bayes nets are not the best tool for every job, and do have drawbacks in certain situations. They are vulnerable to things like Bayesian poisoning or confirmation bias. A Bayesian approach is only as useful as the ongoing real world relevancy and accuracy of the priors. As the old adage goes, GIGO - garbage in, garbage out. [1] http://en.wikipedia.org/wiki/Space_Shuttle_Columbia_disaster http://en.wikipedia.org/wiki/Space_Shuttle_Columbia_disaster [2] http://www.guardian.co.uk/science/2003/jun/22/spaceexploration.columbia http://www.guardian.co.uk/science/2003/jun/22/spaceexplorati...