13 ms·

SpaceX concludes anomaly investigation, sets return to flight target date

309 points by dtparr 10y ago

the_duke 10y ago
To save you from the endless bla bla bla: ...concluded that one of the three composite overwrapped pressure vessels (COPVs) inside the second stage liquid oxygen (LOX) tank failed ...failure was likely due to the accumulation of oxygen between the COPV liner and overwrap in a void or a buckle in the liner, leading to ignition and the subsequent failure of the COPV. .... corrective actions .. short term ... changing the COPV configuration to allow warmer temperature helium to be loaded ... long term, SpaceX will implement design changes to the COPVs to prevent buckles altogether Next launch on January 8th.
brianshaler 10y ago
It's surprising how much of a disruption this caused to their launch schedule, but it's great that they got to the bottom of the issue before repeating the process on any further rockets. The worst kind of bug is the intermittent one that you can't reproduce or see in logs. With only 35-55 milliseconds of metrics leading up to the event and nearly obliterated evidence, it seemed nearly impossible they would be able to nail down the exact cause. Critics have pointed out risks in SpaceX's method of loading supercooled fuel. Accidentally forming solid oxygen around carbon fibers from a buckling overwrap is an example of this. There are benefits, but they will need to find and prevent all these possible edge cases exposed by the new method.
- phkahler 10y ago
  I find the whole idea of carbon fiber inside a pressurized LOX contain a bit risky. Also, what about all those reports that we're running out of helium? Is that credible threat to ongoing space flight?
  - Pharylon 10y ago
    We're not really running out, it's just getting more expensive because the government doesn't regulate it's availability anymore since the strategic helium reserve was done away with. It won't be a big deal for (already expensive) rocketry. Buying helium balloons for your birthday will be pricey, though.
    - kobeya 10y ago
      Um, no. We are in fact running out of helium. There's only a certain mount and nobody not even natural processes is making anymore in any significant quantity. And what we do have is rapidly being released into the atmosphere and escaping into space. And given that the nearest natural source of helium is Jupiter, that is a little concerning.
      - greglindahl 10y ago
        New helium is generated by alpha decay, and it comes up in natural gas wells that happen to be located in areas with natural radioactivity.
        mikeash 10y ago
        Not much, though. A quick perusal of Wikipedia (buyer beware, etc.) indicates that about 3,000 tons of helium are generated per year through radioactive decay (that's all happening in assorted rocks throughout the Earth's crust, so much of it wouldn't be easy to recover, either) while current production rates are around 9,000 tons per year.
        greglindahl 10y ago
        A natural gas well has 10s of millions of years of helium accumulation from its local area.
        mikeash 10y ago
        Right, which means that while it's renewable over geological timescales, it's more or less a fixed reserve in human timescales. Much like oil.
  - digerata 10y ago
    I believe that it's very likely that SpaceX can recover the helium used during operation. The helium isn't expelled from the fuel tanks. Since the 1st stage returns, it could be recovered.
  - extrapickles 10y ago
    Its more of we are running out of cheap helium as the US has been selling off its massive stockpile and its nearly depleted.
  - gpm 10y ago
    The cost of the physical helium in rocketry is going to remain negligible for a long time. The total cost of materials in a modern rocket is something like 1/2 of a percent of the cost of a launch, and I'm confident saying Helium is a tiny fraction of that. It doesn't matter if it goes up in price by a few orders of magnitudes. We are also still producing it in large quantities (natural gas often contains a lot of helium, which we extract). Rocket's also use a tiny amount of helium percentage wise, simply because there are so few rocket launches. However even if we did somehow run out of Helium entirely, once SpaceX switches to Methane fuel (as they plan to do for the ITS) they won't need it anymore. Currently it's used to pressurize the tanks, but Methane does that for itself. It would mean speeding up development of 'small' methane launch vehicles presumably, but that's it.
    - mikeash 10y ago
      I imagine that there are other ways to accomplish the same goal, as well, such as using other noble gasses. Helium provides the best cost/benefit tradeoff but surely other approaches would be practical if helium becomes unavailable.
    - wildcat55 10y ago
      How would you pressurize the liquid oxygen needed for the methane to burn? Per the other comment, no other noble gas has a low enough boiling point to remain a gas at LOX temperatures...
      - Symmetry 10y ago
        If you need to you can heat up the oxygen to increase it's volume until the pressure is correct. I forget the reason that it's easier to do with methane than oxygen but it's possible to do with oxygen too if you really need to. ULA's new ACES upper stage is planned to use autogenous pressurization for both it's oxygen and hydrogen tanks.
- ghostly_s 10y ago
  How does the gas get into this void between liner and overwrap? I presume the liner is meant to be impermeable? Or is it only impermeable when properly backed with the overwrap?
  - tsuraan 10y ago
    The helium COPV is submerged in LO2, so it's the oxygen that's soaking into the wrapping. I think the failure is that the oxygen is soaking into the wrapping before the helium is fully pressurized, and then the metal container expands slightly and crushes the oxygen that soaked in between the metal vessel and the carbon wrapping.
- mikeash 10y ago
  This "anomaly" caused hundreds of millions of dollars in losses, and was an unprecedented failure mode. It doesn't surprise me at all that it disrupted the launch schedule for the better part of a year (edit: actually the lesser part of a year, about four months total if they launch soon). If they manage to return to flight this month, it strikes me as rather quick. I wonder if supercooled propellants are really worth it. It seems like a huge risk in exchange for a small performance increase. Hopefully this will be the last vehicle loss they suffer from it.
  - andars 10y ago
    "small performance increase" FT can get 30% more mass to LEO. That is a pretty big performance increase, especially considering that it enables SpaceX to carry the maximum payload of v1.1 in expendable configuration and still land the FT rocket. Even if we assume that half of that performance boost came from the other improvements, 15% is a significant increase.
    - ansible 10y ago
      That's the seductive side of design tradeoffs with rockets. Shaving just a little weight (especially on the 2nd stage) or squeezing out just a little more performance can have dramatic differences in mass to LEO. The question is how much this will increase overall operational cost. Up until this year or so, it has seemed that SpaceX has been willing to compromise on performance to get lower overall operational cost and reliability. I'm a bit worried that they've started to go the other way with some of the improvements they've made this year.
  - krastanov 10y ago
    It disrupted the schedule for 3 months, not for "the better part of a year". Otherwise you are making a fair point (but SpaceX seem to believe that it is worth it and that they will iteratively fix all these issues without losing customer trust).
    - mikeash 10y ago
      Huh, I was sure it was in June. Actually September 1st. I guess it just feels longer to me. Thanks for pointing that out.
- asmithmd1 10y ago
  To really be sure they got to the bottom of the issue and are not seeing an intermittent issue they should duplicate the failure, but it may not be possible to do this safely. I detect a little bit of hand waving around the source of ignition: "breaking fibers or friction can ignite the oxygen in the overwrap" I would like to see at least this ignition source duplicated.
  - ghshephard 10y ago
    I also found it a little surprising with so much involved that they didn't take the effort to reproduce the failure case, perhaps doing so on a smaller scale than a full vehicle launch if possible.
    - snrplfth 10y ago
      The problem is that accurately reproducing this failure case is very difficult. You need to replicate the thermal properties of the loading process and the tanks, at the same speed with the same amounts of propellant and helium, effectively meaning that you have to do it with a full-size tank, and with a large amount of pad equipment. This is all extremely dangerous and almost certainly will result, if successful, in the loss of the testing stand.
  - lutorm 10y ago
    It doesn't specify whether the mechanism proposed to have caused this particular anomaly has been experimentally verified to be able to cause ignition but, even if it had, it would still not prove it was the mechanism that was at work in this particular instance.
    - deleted 10y ago
      [deleted]
    - deleted 10y ago
      [deleted]
    - asmithmd1 10y ago
      Yes, proposing a theory and reproducing a result experimentally doesn't prove anything - but it is the basis of all science.
hoorayimhelping 10y ago
The accident investigation team concluded that one of the three composite overwrapped pressure vessels (COPVs) inside the second stage liquid oxygen (LOX) tank failed. Specifically, the failure was likely due to the accumulation of oxygen between the COPV liner and overwrap in a void or a buckle in the liner, leading to ignition and the subsequent failure of the COPV. The investigation team identified several credible causes for the COPV failure, all of which involve accumulation of super chilled LOX or SOX in buckles under the overwrap. The corrective actions address all credible causes and focus on changes which avoid the conditions that led to these credible causes. In the short term, this entails changing the COPV configuration to allow warmer temperature helium to be loaded, as well as returning helium loading operations to a prior flight proven configuration based on operations used in over 700 successful COPV loads. In the long term, SpaceX will implement design changes to the COPVs to prevent buckles altogether, which will allow for faster loading operations. SpaceX is targeting return to flight from Vandenberg's Space Launch Complex 4E (SLC-4E) with the Iridium NEXT launch on January 8. Are COPVs a universal thing in rocketry? Or has SpaceX actually loaded fuel into its rockets over 700 times?
- schiffern 10y ago
  It's true that COPVs are rather common in aerospace (being the lightest pressure tank technology available). > Or has SpaceX actually loaded fuel into its rockets over 700 times? This. SpaceX has done development test cycles, and for each launch performs stage acceptance tests at McGregor Texas and both "wet dress rehearsals" (now generally skipped) and static fires at their launch pads. All of these operations involve loading and unloading helium and LOX.
  - deleted 10y ago
    [deleted]
  - astrodust 10y ago
    Given the nature of carbon-fiber composite manufacturing where it's usually applied in layers, is the risk of a tiny bubble a lot higher when it's subjected to repeated heating/cooling cycles?
ashnyc 10y ago
Does anyone know if the public can watch when they launch at Vandenberg AFB Space Launch Complex. I have always wanted to do this but never sure if it is open to the public
- wolf550e 10y ago
  Kinda. Read the details at r/spacex FAQ and the links there https://www.reddit.com/r/spacex/wiki/faq/watching#wiki_i.27m_at_vandenberg._where.27s_the_best_place_to_watch_the_launch.3F https://www.reddit.com/r/spacex/wiki/faq/watching#wiki_i.27m...
cnnsucks 10y ago
>> This "anomaly" caused hundreds of millions... It also produced a large quantity of nonsense. WAPO kicked that off with speculation of "sabotage." [1] By the time that stuff had finished ricocheting around the echo chamber it was SPETSNAZ snipers. Today WAPO has finally grounded out it's "Russian's hacked the US power grid" fiction. [2] [1] https://news.ycombinator.com/item?id=12617990 https://news.ycombinator.com/item?id=12617990 [2] https://yro.slashdot.org/story/16/12/31/1533245/washington-post-retracts-story-about-russian-hackers-penetrating-us-electricity-grid https://yro.slashdot.org/story/16/12/31/1533245/washington-p... It would be nice if Elon could take a few minutes and install some grown-up editors at his paper. It's turning into a tabloid. * Oh dear, I've mixed up my celebrity billionaires... Yes, Elon would have to get Bezos to fix the paper.
- aristus 10y ago
  Are you perhaps confusing Jeff Bezos with Elon Musk? https://en.wikipedia.org/wiki/The_Washington_Post#Jeff_Bezos_era_.282013.E2.80.93present.29 https://en.wikipedia.org/wiki/The_Washington_Post#Jeff_Bezos...
  - duskwuff 10y ago
    Mind that Bezos also has his own aerospace company -- Blue Origin -- so it'd in fact be in his interest to report negatively about SpaceX. (If you subscribe to the theory that he's personally interfering with WaPo's editorial choices, which I do not.)
- deleted 10y ago
  [deleted]
amorphid 10y ago
What is the primary purpose of using liquid helium in a Falcon 9? A quick google search for `liquid helium rocket` introduced me to pressure-fed engines [1]. A pressure-fed engine uses liquid helium, and is a replacement for a turbopumps. Since the Falcon 9 uses turbopumps, I don't understand what the liquid helium is for. If I had to guess, it sounds like a Falcon 9 has a pressure-fed engine AND a turbopump. On a car, that'd be kind of like using both a turbocharger and a supercharger, which sounds complicated. [1]https://en.wikipedia.org/wiki/Pressure-fed_engine https://en.wikipedia.org/wiki/Pressure-fed_engine
- snrplfth 10y ago
  It's not liquid helium, it's gaseous helium, which is why it is at such high pressures. It's primarily used as a pressurant for the liquid oxygen and rocket fuel (RP-1) tanks. Since liquid oxygen and RP-1 are being sucked out of the tanks at high speed by the engine turbopumps, it needs some low-density gas to fill the tanks so that they don't crumple from negative pressure. Falcon-9 has no pressure-fed-only engines, all 10 engines are fed by turbopumps.
  - amorphid 10y ago
    Gotcha. Thanks!
- motoboi 10y ago
  Not only is pressurized gas used to avoid the tanks from crumpling from negative pressure, they cannot sustain themselves without internal pressure. This gas need to be as inert as possible, hence helium. Another noble gas could in theory be used, but I think price is the reason.
  - snrplfth 10y ago
    Not just price, but density is the reason. The next noble gas up is neon, which is quite scarce and more expensive than helium, and denser anyways. After that it's argon, which is cheap and plentiful, but about ten times denser, so it's not nearly as good as helium.
hackuser 10y ago
Could we link to something less intrinsically biased, such as a news report, and not SpaceX's press release? Here's one: http://www.nytimes.com/2017/01/02/science/spacex-launch-rockets-explosion.html http://www.nytimes.com/2017/01/02/science/spacex-launch-rock...
- apendleton 10y ago
  The release is SpaceX reporting on an investigation they themselves conducted. There probably aren't other sources besides SpaceX that are better equipped to report on an investigation conducted by SpaceX or to independently corroborate or dispute its findings; all we can do is take their word for it, biased or not. Indeed, that NYT article is mostly just a regurgitation of the release.
  - hackuser 10y ago
    > There probably aren't other sources besides SpaceX that are better equipped to report on an investigation conducted by SpaceX I disagree; SpaceX has a clear interest (and a well-established track record) of presenting things in a way that bolsters their image. Outside experts could provide critiques of the report: What represents bad luck, what is trivial, and what is a serious error; where is SpaceX being straightforward and where are they spinning things; where are the gaps, where are there alternative analyses of the same data, and what seems solid. That is a big reason why journalism is important: It provides context and other voices.
    - andars 10y ago
      That's all fine and good, but the only source NYTimes article you suggested is SpaceX, so you're not getting commentary from any outside experts. Instead, you are just getting quotes and paraphrases from this and previous SpaceX press releases.
      - hackuser 10y ago
        It's just one article of many possibilities. Also, they still can contextualize info and filter out spin. Incidentally, any time you post something that isn't fully saturated with the SpaceX Kool-Aid, you get a strong reaction. Look at all the comments and the downvoting over a pretty mundane detail and a common concern: Don't link to a press release. I didn't even criticize the Great and Powerful Wizard or his rocket ships. To be clear, I don't care about the votes, but I think the overall response says more than the comments.
GizaDog 10y ago
I've been trying to dig up the launch time. Does anyone know when that will be? Thanks.
- mabbo 10y ago
  https://www.reddit.com/r/spacex https://www.reddit.com/r/spacex usually has the most up to date information. They are saying 10:28/18:28 (PST/GMT).
CydeWeys 10y ago
That's multiple complete losses now that have been caused by issues with COPVs. I remember reading some discussion on /r/spacex about SpaceX going with really aggressive COPV technologies to shave as much weight as possible, but they're definitely paying for it in reliability. Hopefully they can iron it out, and maybe they'll switch to less aggressive COPV designs soon?
- snrplfth 10y ago
  The previous loss was, as far as they can tell, related to a strut that broke under strain and released an otherwise-fine COPV into the oxygen tank. As far as they can tell it didn't have anything to do with aggressive design, just a part failing under its rated strength.
  - JamesUtah07 10y ago
    I don't think it even failed under it's rated strength, it was that their supplier for the struts provided sub-par/faulty struts and they couldn't handle the pressure they were supposed to be rated for
    - snrplfth 10y ago
      That's what I meant. It was rated for a certain load, but there was a faulty strut in the batch, and it failed well under that rated load.
    - ambicapter 10y ago
      Isn't that what 'failed under its rated strength' means?
- the_duke 10y ago
  "Multiple complete losses" is quite a misleading statement, in my opinion. It's been exactly two.
  - CydeWeys 10y ago
nrjdhsbsid 10y ago
Does anyone else think the timing is kinda funny? Today, at least in the US, is probably the slowest news day of the year. Great time to unload anything unfavorable :)
- arijun 10y ago
  Wouldn't you want to unload something unfavorable on a fast news day, where it would get buried by other, bigger news?
- JshWright 10y ago
  They are announcing that they are returning to flight. Hardly 'unfavorable'.
throwawasiudy 10y ago
This NASA article on COPV failure is enlightening. https://www.nasa.gov/offices/nesc/home/Feature_COPVs_Jan-2012.html https://www.nasa.gov/offices/nesc/home/Feature_COPVs_Jan-201... Turns out COPV's have a tendency to explode semi randomly, with failure modes that are not fully understood
- manicdee 10y ago
  The failure modes are slightly better understood now, as are the boundaries between success and failure!
dmix 10y ago
Ouch it looks like the Isreali company lost their previous satellite too after 4 yrs of being in space: http://www.haaretz.com/israel-news/business/1.687543 http://www.haaretz.com/israel-news/business/1.687543 There is still no cause listed on Wikipedia. > Industry experts described the total loss of contact with the satellite as a highly uncommon event. This one was built by Russians for the first time. It looks like they went back to a domestic built one with SpaceX and it blew up. That's some bad luck :/ https://en.wikipedia.org/wiki/Amos_(satellite) https://en.wikipedia.org/wiki/Amos_(satellite)
- dispose13432 10y ago
  That's why insurance isn't the end-all/be-all. They've got enough money out of insurance to build themselves a new satellite, the problem is that they need that satellite yesterday, not today. On the other hand, SpaceX isn't living off poor blokes who were hoodwinked by a starry-eyed agent. Their customers are professionals with lawyers and accountants taking care of things, so I don't feel that SpaceX is particularly unethical.
ChuckMcM 10y ago
Two things I learned from the report, one was that the helium tanks are actually inside the oxygen tanks. All the rockets that I can recall seeing the insides of, had the helium tanks as a sort of 'ring of pearls' around the end of the tank. Clearly from a volume perspective it doesn't matter if they are inside or outside, and from a thermal perspective it means the LOX can keep the helium cool (or vice versa I guess). And the second is that their wrapping method allows for the creation of voids between the inner aluminum tank and the carbon fiber wrap. I can certainly imagine how that could happen with carbon fiber prepreg (it's a prewoven fabric which maximizes strength by pre-laying out the various ways the fiber goes through it) but if you were to spool on straight up fiber that would make it easier to be conformal. I wonder if other people do the 'tanks within tanks' architecture and if so how they avoid voids and buckles in their overwrap.
- cloudwalking 10y ago
  In general it seems the voids and buckles problem is only seen at very very low temperatures. The AMOS-6 failure occured because the LOX was loaded too quickly after loading the helium tanks, without allowing the He to come up in temperature. SpaceX was pushing the loading speed to expand the flight window. The update (and Ms. Shotwell a few months ago) indicate the short term fix for this is to slow down the propellant loading to allow the He to warm up a bit before LOX arrives... Which is a long way to say that it's likely other companies have not pushed the envelope far enough to encounter this voids and buckles problem. The update also says they will eventually adjust their COPV construction to eliminate this problem and return to rapid propellant loading, but I doubt we'll hear anything about HOW they improve it. A not insignificant amount of SpaceX's "secret sauce" is in their composits manufacturing (case in point the enormous ITS LOX tank recently tested.)
- electric0 10y ago
  I'm not aware of any modern US rockets (other than SpaceX) that install COPV tanks inside LO2 fuel tanks. SpaceX is certainly getting a performance gain from the design at the risk of these kind of thermal related failure modes. I hope they figure out how mitigate these failure modes as a COPV exploding inside a LO2 tank is scary! Especially once they start flying humans on-board in 2018 or 2019.
dnautics 10y ago
My suspicion was that SpaceX was rushing to validate their carbon fiber tank plans, which is why they did it on stage 2 (process is lower scale and easier to experiment). Well, at least they failed fast. I made this prediction in a private twitter DM (yes, weaksauce, I should have been bolder with a public post) the day that Musk announced their plans for giant carbon fiber tanks, with no knowledge that the stage 2 tank was made of carbon fiber. https://drive.google.com/open?id=0B3uVSPZl-I22UjE4aElRcDFaaEk https://drive.google.com/open?id=0B3uVSPZl-I22UjE4aElRcDFaaE...
- greglindahl 10y ago
  Are you confusing the small COPV helium tanks with the much larger metal LOX tank?
- InclinedPlane 10y ago
  100% wrong. COPVs and carbon fiber tanks are entirely different beasts. In fact, if carbon fiber He tanks had been used instead of the COPV bottles this failure wouldn't have happened.
- snrplfth 10y ago
  The Carbon-Overwrapped Pressure Vessels are quite different from the large-scale carbon fiber tanks they're building now. The new large tanks are designed to be able to contain large quantities of propellant at moderate pressures, possibly without a liner, while the COPVs are meant to contain pressurant gas at extremely high pressure, with a metal liner. SpaceX has been using COPVs to hold helium pressurant since the first Falcon 9 back in 2010. This is not a new technology. What is new is the large propellant tanks, which have not been done before at this scale, and have not been used in Falcon 9s at any point. So no, you shouldn't have been bolder with a public post.
dispose13432 10y ago
My question is: 1. How often does this happen? 2. Hoe many tests did SpaceX do before the first FT launch? Considering that most rockets don't use supercooled fuels, I would expect insane tests (they can't rely on NASA's historical data).
- ben-schaaf 10y ago
  All the well known rockets use cryogenic fuel at least in their ascent stages: Space Shuttle, Saturn V, Soyuz. They certainly can't rely solely on NASA's or Russia's data, but it will/has undoutably helped.
  - xorxornop 10y ago
    Cryogenic fuel is not the same is supercooled; supercooled is closer to freezing than to boiling

309 points by dtparr 10y ago

To save you from the endless bla bla bla: ...concluded that one of the three composite overwrapped pressure vessels (COPVs) inside the second stage liquid oxygen (LOX) tank failed ...failure was likely due to the accumulation of oxygen between the COPV liner and overwrap in a void or a buckle in the liner, leading to ignition and the subsequent failure of the COPV. .... corrective actions .. short term ... changing the COPV configuration to allow warmer temperature helium to be loaded ... long term, SpaceX will implement design changes to the COPVs to prevent buckles altogether Next launch on January 8th.

brianshaler 10y ago

It's surprising how much of a disruption this caused to their launch schedule, but it's great that they got to the bottom of the issue before repeating the process on any further rockets. The worst kind of bug is the intermittent one that you can't reproduce or see in logs. With only 35-55 milliseconds of metrics leading up to the event and nearly obliterated evidence, it seemed nearly impossible they would be able to nail down the exact cause. Critics have pointed out risks in SpaceX's method of loading supercooled fuel. Accidentally forming solid oxygen around carbon fibers from a buckling overwrap is an example of this. There are benefits, but they will need to find and prevent all these possible edge cases exposed by the new method.

phkahler 10y ago
I find the whole idea of carbon fiber inside a pressurized LOX contain a bit risky. Also, what about all those reports that we're running out of helium? Is that credible threat to ongoing space flight?
- Pharylon 10y ago
  We're not really running out, it's just getting more expensive because the government doesn't regulate it's availability anymore since the strategic helium reserve was done away with. It won't be a big deal for (already expensive) rocketry. Buying helium balloons for your birthday will be pricey, though.
  - kobeya 10y ago
    Um, no. We are in fact running out of helium. There's only a certain mount and nobody not even natural processes is making anymore in any significant quantity. And what we do have is rapidly being released into the atmosphere and escaping into space. And given that the nearest natural source of helium is Jupiter, that is a little concerning.
    - greglindahl 10y ago
      New helium is generated by alpha decay, and it comes up in natural gas wells that happen to be located in areas with natural radioactivity.
      - mikeash 10y ago
        Not much, though. A quick perusal of Wikipedia (buyer beware, etc.) indicates that about 3,000 tons of helium are generated per year through radioactive decay (that's all happening in assorted rocks throughout the Earth's crust, so much of it wouldn't be easy to recover, either) while current production rates are around 9,000 tons per year.
        greglindahl 10y ago
        A natural gas well has 10s of millions of years of helium accumulation from its local area.
        mikeash 10y ago
        Right, which means that while it's renewable over geological timescales, it's more or less a fixed reserve in human timescales. Much like oil.
- digerata 10y ago
  I believe that it's very likely that SpaceX can recover the helium used during operation. The helium isn't expelled from the fuel tanks. Since the 1st stage returns, it could be recovered.
- extrapickles 10y ago
  Its more of we are running out of cheap helium as the US has been selling off its massive stockpile and its nearly depleted.
- gpm 10y ago
  The cost of the physical helium in rocketry is going to remain negligible for a long time. The total cost of materials in a modern rocket is something like 1/2 of a percent of the cost of a launch, and I'm confident saying Helium is a tiny fraction of that. It doesn't matter if it goes up in price by a few orders of magnitudes. We are also still producing it in large quantities (natural gas often contains a lot of helium, which we extract). Rocket's also use a tiny amount of helium percentage wise, simply because there are so few rocket launches. However even if we did somehow run out of Helium entirely, once SpaceX switches to Methane fuel (as they plan to do for the ITS) they won't need it anymore. Currently it's used to pressurize the tanks, but Methane does that for itself. It would mean speeding up development of 'small' methane launch vehicles presumably, but that's it.
  - mikeash 10y ago
    I imagine that there are other ways to accomplish the same goal, as well, such as using other noble gasses. Helium provides the best cost/benefit tradeoff but surely other approaches would be practical if helium becomes unavailable.
  - wildcat55 10y ago
    How would you pressurize the liquid oxygen needed for the methane to burn? Per the other comment, no other noble gas has a low enough boiling point to remain a gas at LOX temperatures...
    - Symmetry 10y ago
      If you need to you can heat up the oxygen to increase it's volume until the pressure is correct. I forget the reason that it's easier to do with methane than oxygen but it's possible to do with oxygen too if you really need to. ULA's new ACES upper stage is planned to use autogenous pressurization for both it's oxygen and hydrogen tanks.
ghostly_s 10y ago
How does the gas get into this void between liner and overwrap? I presume the liner is meant to be impermeable? Or is it only impermeable when properly backed with the overwrap?
- tsuraan 10y ago
  The helium COPV is submerged in LO2, so it's the oxygen that's soaking into the wrapping. I think the failure is that the oxygen is soaking into the wrapping before the helium is fully pressurized, and then the metal container expands slightly and crushes the oxygen that soaked in between the metal vessel and the carbon wrapping.
mikeash 10y ago
This "anomaly" caused hundreds of millions of dollars in losses, and was an unprecedented failure mode. It doesn't surprise me at all that it disrupted the launch schedule for the better part of a year (edit: actually the lesser part of a year, about four months total if they launch soon). If they manage to return to flight this month, it strikes me as rather quick. I wonder if supercooled propellants are really worth it. It seems like a huge risk in exchange for a small performance increase. Hopefully this will be the last vehicle loss they suffer from it.
- andars 10y ago
  "small performance increase" FT can get 30% more mass to LEO. That is a pretty big performance increase, especially considering that it enables SpaceX to carry the maximum payload of v1.1 in expendable configuration and still land the FT rocket. Even if we assume that half of that performance boost came from the other improvements, 15% is a significant increase.
  - ansible 10y ago
    That's the seductive side of design tradeoffs with rockets. Shaving just a little weight (especially on the 2nd stage) or squeezing out just a little more performance can have dramatic differences in mass to LEO. The question is how much this will increase overall operational cost. Up until this year or so, it has seemed that SpaceX has been willing to compromise on performance to get lower overall operational cost and reliability. I'm a bit worried that they've started to go the other way with some of the improvements they've made this year.
- krastanov 10y ago
  It disrupted the schedule for 3 months, not for "the better part of a year". Otherwise you are making a fair point (but SpaceX seem to believe that it is worth it and that they will iteratively fix all these issues without losing customer trust).
  - mikeash 10y ago
    Huh, I was sure it was in June. Actually September 1st. I guess it just feels longer to me. Thanks for pointing that out.
asmithmd1 10y ago
To really be sure they got to the bottom of the issue and are not seeing an intermittent issue they should duplicate the failure, but it may not be possible to do this safely. I detect a little bit of hand waving around the source of ignition: "breaking fibers or friction can ignite the oxygen in the overwrap" I would like to see at least this ignition source duplicated.
- ghshephard 10y ago
  I also found it a little surprising with so much involved that they didn't take the effort to reproduce the failure case, perhaps doing so on a smaller scale than a full vehicle launch if possible.
  - snrplfth 10y ago
    The problem is that accurately reproducing this failure case is very difficult. You need to replicate the thermal properties of the loading process and the tanks, at the same speed with the same amounts of propellant and helium, effectively meaning that you have to do it with a full-size tank, and with a large amount of pad equipment. This is all extremely dangerous and almost certainly will result, if successful, in the loss of the testing stand.
- lutorm 10y ago
  It doesn't specify whether the mechanism proposed to have caused this particular anomaly has been experimentally verified to be able to cause ignition but, even if it had, it would still not prove it was the mechanism that was at work in this particular instance.
  - deleted 10y ago
    [deleted]
  - deleted 10y ago
    [deleted]
  - asmithmd1 10y ago
    Yes, proposing a theory and reproducing a result experimentally doesn't prove anything - but it is the basis of all science.

hoorayimhelping 10y ago

The accident investigation team concluded that one of the three composite overwrapped pressure vessels (COPVs) inside the second stage liquid oxygen (LOX) tank failed. Specifically, the failure was likely due to the accumulation of oxygen between the COPV liner and overwrap in a void or a buckle in the liner, leading to ignition and the subsequent failure of the COPV. The investigation team identified several credible causes for the COPV failure, all of which involve accumulation of super chilled LOX or SOX in buckles under the overwrap. The corrective actions address all credible causes and focus on changes which avoid the conditions that led to these credible causes. In the short term, this entails changing the COPV configuration to allow warmer temperature helium to be loaded, as well as returning helium loading operations to a prior flight proven configuration based on operations used in over 700 successful COPV loads. In the long term, SpaceX will implement design changes to the COPVs to prevent buckles altogether, which will allow for faster loading operations. SpaceX is targeting return to flight from Vandenberg's Space Launch Complex 4E (SLC-4E) with the Iridium NEXT launch on January 8. Are COPVs a universal thing in rocketry? Or has SpaceX actually loaded fuel into its rockets over 700 times?

schiffern 10y ago
It's true that COPVs are rather common in aerospace (being the lightest pressure tank technology available). > Or has SpaceX actually loaded fuel into its rockets over 700 times? This. SpaceX has done development test cycles, and for each launch performs stage acceptance tests at McGregor Texas and both "wet dress rehearsals" (now generally skipped) and static fires at their launch pads. All of these operations involve loading and unloading helium and LOX.
- deleted 10y ago
  [deleted]
- astrodust 10y ago
  Given the nature of carbon-fiber composite manufacturing where it's usually applied in layers, is the risk of a tiny bubble a lot higher when it's subjected to repeated heating/cooling cycles?

ashnyc 10y ago

Does anyone know if the public can watch when they launch at Vandenberg AFB Space Launch Complex. I have always wanted to do this but never sure if it is open to the public

wolf550e 10y ago
Kinda. Read the details at r/spacex FAQ and the links there https://www.reddit.com/r/spacex/wiki/faq/watching#wiki_i.27m_at_vandenberg._where.27s_the_best_place_to_watch_the_launch.3F https://www.reddit.com/r/spacex/wiki/faq/watching#wiki_i.27m...

cnnsucks 10y ago

>> This "anomaly" caused hundreds of millions... It also produced a large quantity of nonsense. WAPO kicked that off with speculation of "sabotage." [1] By the time that stuff had finished ricocheting around the echo chamber it was SPETSNAZ snipers. Today WAPO has finally grounded out it's "Russian's hacked the US power grid" fiction. [2] [1] https://news.ycombinator.com/item?id=12617990 https://news.ycombinator.com/item?id=12617990 [2] https://yro.slashdot.org/story/16/12/31/1533245/washington-post-retracts-story-about-russian-hackers-penetrating-us-electricity-grid https://yro.slashdot.org/story/16/12/31/1533245/washington-p... It would be nice if Elon could take a few minutes and install some grown-up editors at his paper. It's turning into a tabloid. * Oh dear, I've mixed up my celebrity billionaires... Yes, Elon would have to get Bezos to fix the paper.

aristus 10y ago
Are you perhaps confusing Jeff Bezos with Elon Musk? https://en.wikipedia.org/wiki/The_Washington_Post#Jeff_Bezos_era_.282013.E2.80.93present.29 https://en.wikipedia.org/wiki/The_Washington_Post#Jeff_Bezos...
- duskwuff 10y ago
  Mind that Bezos also has his own aerospace company -- Blue Origin -- so it'd in fact be in his interest to report negatively about SpaceX. (If you subscribe to the theory that he's personally interfering with WaPo's editorial choices, which I do not.)
deleted 10y ago
[deleted]

amorphid 10y ago

What is the primary purpose of using liquid helium in a Falcon 9? A quick google search for `liquid helium rocket` introduced me to pressure-fed engines [1]. A pressure-fed engine uses liquid helium, and is a replacement for a turbopumps. Since the Falcon 9 uses turbopumps, I don't understand what the liquid helium is for. If I had to guess, it sounds like a Falcon 9 has a pressure-fed engine AND a turbopump. On a car, that'd be kind of like using both a turbocharger and a supercharger, which sounds complicated. [1]https://en.wikipedia.org/wiki/Pressure-fed_engine https://en.wikipedia.org/wiki/Pressure-fed_engine

snrplfth 10y ago
It's not liquid helium, it's gaseous helium, which is why it is at such high pressures. It's primarily used as a pressurant for the liquid oxygen and rocket fuel (RP-1) tanks. Since liquid oxygen and RP-1 are being sucked out of the tanks at high speed by the engine turbopumps, it needs some low-density gas to fill the tanks so that they don't crumple from negative pressure. Falcon-9 has no pressure-fed-only engines, all 10 engines are fed by turbopumps.
- amorphid 10y ago
  Gotcha. Thanks!
motoboi 10y ago
Not only is pressurized gas used to avoid the tanks from crumpling from negative pressure, they cannot sustain themselves without internal pressure. This gas need to be as inert as possible, hence helium. Another noble gas could in theory be used, but I think price is the reason.
- snrplfth 10y ago
  Not just price, but density is the reason. The next noble gas up is neon, which is quite scarce and more expensive than helium, and denser anyways. After that it's argon, which is cheap and plentiful, but about ten times denser, so it's not nearly as good as helium.

hackuser 10y ago

Could we link to something less intrinsically biased, such as a news report, and not SpaceX's press release? Here's one: http://www.nytimes.com/2017/01/02/science/spacex-launch-rockets-explosion.html http://www.nytimes.com/2017/01/02/science/spacex-launch-rock...

apendleton 10y ago
The release is SpaceX reporting on an investigation they themselves conducted. There probably aren't other sources besides SpaceX that are better equipped to report on an investigation conducted by SpaceX or to independently corroborate or dispute its findings; all we can do is take their word for it, biased or not. Indeed, that NYT article is mostly just a regurgitation of the release.
- hackuser 10y ago
  > There probably aren't other sources besides SpaceX that are better equipped to report on an investigation conducted by SpaceX I disagree; SpaceX has a clear interest (and a well-established track record) of presenting things in a way that bolsters their image. Outside experts could provide critiques of the report: What represents bad luck, what is trivial, and what is a serious error; where is SpaceX being straightforward and where are they spinning things; where are the gaps, where are there alternative analyses of the same data, and what seems solid. That is a big reason why journalism is important: It provides context and other voices.
  - andars 10y ago
    That's all fine and good, but the only source NYTimes article you suggested is SpaceX, so you're not getting commentary from any outside experts. Instead, you are just getting quotes and paraphrases from this and previous SpaceX press releases.
    - hackuser 10y ago
      It's just one article of many possibilities. Also, they still can contextualize info and filter out spin. Incidentally, any time you post something that isn't fully saturated with the SpaceX Kool-Aid, you get a strong reaction. Look at all the comments and the downvoting over a pretty mundane detail and a common concern: Don't link to a press release. I didn't even criticize the Great and Powerful Wizard or his rocket ships. To be clear, I don't care about the votes, but I think the overall response says more than the comments.

GizaDog 10y ago

I've been trying to dig up the launch time. Does anyone know when that will be? Thanks.

mabbo 10y ago
https://www.reddit.com/r/spacex https://www.reddit.com/r/spacex usually has the most up to date information. They are saying 10:28/18:28 (PST/GMT).

CydeWeys 10y ago

That's multiple complete losses now that have been caused by issues with COPVs. I remember reading some discussion on /r/spacex about SpaceX going with really aggressive COPV technologies to shave as much weight as possible, but they're definitely paying for it in reliability. Hopefully they can iron it out, and maybe they'll switch to less aggressive COPV designs soon?

snrplfth 10y ago
The previous loss was, as far as they can tell, related to a strut that broke under strain and released an otherwise-fine COPV into the oxygen tank. As far as they can tell it didn't have anything to do with aggressive design, just a part failing under its rated strength.
- JamesUtah07 10y ago
  I don't think it even failed under it's rated strength, it was that their supplier for the struts provided sub-par/faulty struts and they couldn't handle the pressure they were supposed to be rated for
  - snrplfth 10y ago
    That's what I meant. It was rated for a certain load, but there was a faulty strut in the batch, and it failed well under that rated load.
  - ambicapter 10y ago
    Isn't that what 'failed under its rated strength' means?
the_duke 10y ago
"Multiple complete losses" is quite a misleading statement, in my opinion. It's been exactly two.
- CydeWeys 10y ago

nrjdhsbsid 10y ago

Does anyone else think the timing is kinda funny? Today, at least in the US, is probably the slowest news day of the year. Great time to unload anything unfavorable :)

arijun 10y ago
Wouldn't you want to unload something unfavorable on a fast news day, where it would get buried by other, bigger news?
JshWright 10y ago
They are announcing that they are returning to flight. Hardly 'unfavorable'.

throwawasiudy 10y ago

This NASA article on COPV failure is enlightening. https://www.nasa.gov/offices/nesc/home/Feature_COPVs_Jan-2012.html https://www.nasa.gov/offices/nesc/home/Feature_COPVs_Jan-201... Turns out COPV's have a tendency to explode semi randomly, with failure modes that are not fully understood

manicdee 10y ago
The failure modes are slightly better understood now, as are the boundaries between success and failure!

dmix 10y ago

Ouch it looks like the Isreali company lost their previous satellite too after 4 yrs of being in space: http://www.haaretz.com/israel-news/business/1.687543 http://www.haaretz.com/israel-news/business/1.687543 There is still no cause listed on Wikipedia. > Industry experts described the total loss of contact with the satellite as a highly uncommon event. This one was built by Russians for the first time. It looks like they went back to a domestic built one with SpaceX and it blew up. That's some bad luck :/ https://en.wikipedia.org/wiki/Amos_(satellite) https://en.wikipedia.org/wiki/Amos_(satellite)

dispose13432 10y ago
That's why insurance isn't the end-all/be-all. They've got enough money out of insurance to build themselves a new satellite, the problem is that they need that satellite yesterday, not today. On the other hand, SpaceX isn't living off poor blokes who were hoodwinked by a starry-eyed agent. Their customers are professionals with lawyers and accountants taking care of things, so I don't feel that SpaceX is particularly unethical.

ChuckMcM 10y ago

Two things I learned from the report, one was that the helium tanks are actually inside the oxygen tanks. All the rockets that I can recall seeing the insides of, had the helium tanks as a sort of 'ring of pearls' around the end of the tank. Clearly from a volume perspective it doesn't matter if they are inside or outside, and from a thermal perspective it means the LOX can keep the helium cool (or vice versa I guess). And the second is that their wrapping method allows for the creation of voids between the inner aluminum tank and the carbon fiber wrap. I can certainly imagine how that could happen with carbon fiber prepreg (it's a prewoven fabric which maximizes strength by pre-laying out the various ways the fiber goes through it) but if you were to spool on straight up fiber that would make it easier to be conformal. I wonder if other people do the 'tanks within tanks' architecture and if so how they avoid voids and buckles in their overwrap.

cloudwalking 10y ago
In general it seems the voids and buckles problem is only seen at very very low temperatures. The AMOS-6 failure occured because the LOX was loaded too quickly after loading the helium tanks, without allowing the He to come up in temperature. SpaceX was pushing the loading speed to expand the flight window. The update (and Ms. Shotwell a few months ago) indicate the short term fix for this is to slow down the propellant loading to allow the He to warm up a bit before LOX arrives... Which is a long way to say that it's likely other companies have not pushed the envelope far enough to encounter this voids and buckles problem. The update also says they will eventually adjust their COPV construction to eliminate this problem and return to rapid propellant loading, but I doubt we'll hear anything about HOW they improve it. A not insignificant amount of SpaceX's "secret sauce" is in their composits manufacturing (case in point the enormous ITS LOX tank recently tested.)
electric0 10y ago
I'm not aware of any modern US rockets (other than SpaceX) that install COPV tanks inside LO2 fuel tanks. SpaceX is certainly getting a performance gain from the design at the risk of these kind of thermal related failure modes. I hope they figure out how mitigate these failure modes as a COPV exploding inside a LO2 tank is scary! Especially once they start flying humans on-board in 2018 or 2019.

dnautics 10y ago

My suspicion was that SpaceX was rushing to validate their carbon fiber tank plans, which is why they did it on stage 2 (process is lower scale and easier to experiment). Well, at least they failed fast. I made this prediction in a private twitter DM (yes, weaksauce, I should have been bolder with a public post) the day that Musk announced their plans for giant carbon fiber tanks, with no knowledge that the stage 2 tank was made of carbon fiber. https://drive.google.com/open?id=0B3uVSPZl-I22UjE4aElRcDFaaEk https://drive.google.com/open?id=0B3uVSPZl-I22UjE4aElRcDFaaE...

greglindahl 10y ago
Are you confusing the small COPV helium tanks with the much larger metal LOX tank?
InclinedPlane 10y ago
100% wrong. COPVs and carbon fiber tanks are entirely different beasts. In fact, if carbon fiber He tanks had been used instead of the COPV bottles this failure wouldn't have happened.
snrplfth 10y ago
The Carbon-Overwrapped Pressure Vessels are quite different from the large-scale carbon fiber tanks they're building now. The new large tanks are designed to be able to contain large quantities of propellant at moderate pressures, possibly without a liner, while the COPVs are meant to contain pressurant gas at extremely high pressure, with a metal liner. SpaceX has been using COPVs to hold helium pressurant since the first Falcon 9 back in 2010. This is not a new technology. What is new is the large propellant tanks, which have not been done before at this scale, and have not been used in Falcon 9s at any point. So no, you shouldn't have been bolder with a public post.

dispose13432 10y ago

My question is: 1. How often does this happen? 2. Hoe many tests did SpaceX do before the first FT launch? Considering that most rockets don't use supercooled fuels, I would expect insane tests (they can't rely on NASA's historical data).

ben-schaaf 10y ago
All the well known rockets use cryogenic fuel at least in their ascent stages: Space Shuttle, Saturn V, Soyuz. They certainly can't rely solely on NASA's or Russia's data, but it will/has undoutably helped.
- xorxornop 10y ago
  Cryogenic fuel is not the same is supercooled; supercooled is closer to freezing than to boiling