4 ms·

Poop and Soda Bottles Threaten Air Force Rocket Program, Investigators Say

33 points by QAPereo 9y ago

pvg 9y ago
Original source at http://gazette.com/poop-pop-threaten-air-force-rocket-program-investigators-say/article/1618540 http://gazette.com/poop-pop-threaten-air-force-rocket-progra...
- sctb 9y ago
  Thanks, we've updated the link from http://taskandpurpose.com/poop-soda-threaten-air-force-rocket-program-investigators-say/ http://taskandpurpose.com/poop-soda-threaten-air-force-rocke....
Floegipoky 9y ago
> At SpaceX, investigators say they spotted workers using tools that weren’t specified in a manual. Could you imagine being so micromanaged that you'd literally need the Federal Government to change a manual to approve a tool you need?
- QAPereo 9y ago
  I have a cousin in law enforcement, so yes I can, but it’s still maddening. The formula is essentially: a problem arises. Everyone covers their asses with red tape, and new rules and regulations are born, and never die. It’s a classic, yet very difficult-to-manage bureaupathology. The alternative is just that a friend of someone in power makes the tools.
  - ashark 9y ago
    A friend of mine works at a state facility where, as he describes it, it's impossible to be a manager without constantly breaking rules, because many contradict one another or violate the laws of reality. Result is that management is a nasty snake-pit where everyone knows which rules everyone else is breaking and they'll collectively point their fingers at anyone who tries to rock the boat (including, and perhaps especially, improving things), quickly getting them fired (or at least transferred somewhere they'd rather not be). Perhaps significantly, management positions aren't represented by the state workers' union, which is the only thing (barely) insulating the lower level workers from that insanity.
- empath75 9y ago
  yes: https://aws.amazon.com/compliance/fedramp/ https://aws.amazon.com/compliance/fedramp/
- andykee 9y ago
  Using the wrong wrench almost caused a nuclear catastrophe: https://en.wikipedia.org/wiki/1980_Damascus_Titan_missile_explosion https://en.wikipedia.org/wiki/1980_Damascus_Titan_missile_ex... It may seem like micromanagement, but small seemingly meaningless deviations from procedure can add up to big problems, particularly when dealing with space.
  - paulmd 9y ago
    Doesn't sound like the tool in question had any real impact on that accident (so to speak). I'm guessing if you drop a 3-foot torque wrench through a fuel tank, it does just as much damage as a 3-foot socket wrench. The fact that they grabbed the wrong tool was entirely incidental to damaging the missile. Also, it takes an extremely precise sequence of events to cause a nuclear detonation. A nearby explosion is not enough to do it, as you can obviously see here.
    - andykee 9y ago
      I'd recommend listening to the This American Life episode (https://www.thisamericanlife.org/radio-archives/episode/634/human-error-in-volatile-situations https://www.thisamericanlife.org/radio-archives/episode/634/...) where they interview the two airmen involved. tldr; The socket didn't properly fit the wrench and they had to manually hold the socket on the end of the wrench. One man dropped the socket, which fell through a small gap between the walkway and missile. The torque wrench was large enough that it would not have slipped through this gap if/when dropped.
      - ethbro 9y ago
        And the larger problem is one of knowledge asymmetry. If you're dealing with technology so complicated that one person cannot simultaneously design and repair / maintain it, then whoever performs the latter doesn't possess all the knowledge of the former. Maybe they used an flammable alloy under certain conditions for engineering reasons. Most metal doesn't burn. But do you want repair / maintenance techs making that assumption? Multiple by 100,000 parts. PS: A more realistic example was one of the Skunk Works projects. I forget the details, but either tools of special metallic composition or wooden tools had to be used to avoid damaging the flight characteristics of the aircraft.
- klenwell 9y ago
  On the other hand, I recommend this tale if you haven't heard it: https://www.thisamericanlife.org/radio-archives/episode/634/human-error-in-volatile-situations https://www.thisamericanlife.org/radio-archives/episode/634/... > In 1980, deep in a nuclear missile silo in Arkansas, a simple human error nearly caused the destruction of a giant portion of the Midwest. Spoiler: at a critical point, it involves a technician improvising with the wrong tool.
  - deleted 9y ago
    [deleted]
- cjsuk 9y ago
  Indeed. And it doesn't work. Look at the sheer number of near world destroying accidents our military have managed to muster over the years. I was actually working for a company that was a subcontractor to the USAF in the distant past. There is little room for individual creativity and ingenious problem solving in such a process heavy environment. It's actually like trying to work on an engineering project with Siri. The outcome of this is that only the problems allowed to be solved by the process are solved and none of the risks not identified by the process are reported, because it's too much paperwork already if you find a new one and you're not going to want your job to get any harder. Ergo, all unidentified risks get taken because they are attractive solutions to many problems. Looking at the spanner incident pointed out by two sibling comments, the "dont carry heavy tools or anchor then when working on the big explodey thing" rule was not enforced and double-checked but hundreds of minor rules with precisely no benefit were being applied. Someone knew of that risk but didn't put the mitigation in place, probably because it was too hard. Now military rigour and all of that doesn't like to be seen to be a turd sieve of a process so what happens is the answer is to add more processes, not look at the fundamental problem in the approach. Overall, you need to focus on the key places where risk is considered high, but for most things, meh. Listening to people and making sure risk is managed effectively is a win.
  - pavel_lishin 9y ago
    > Indeed. And it doesn't work. Look at the sheer number of near world destroying accidents our military have managed to muster over the years. What is that number? And do we know that the number would be less without these types of regulations? > Looking at the spanner incident pointed out by two sibling comments, the "dont carry heavy tools or anchor then when working on the big explodey thing" rule was not enforced and double-checked My understanding from listening to the podcast episode (this morning) is that the tool-anchoring rules were only added after that particular disaster.
    - IdeaHamstir 9y ago
      > What is that number? And do we know that the number would be less without these types of regulations? https://en.wikipedia.org/wiki/List_of_military_nuclear_accidents https://en.wikipedia.org/wiki/List_of_military_nuclear_accid... I guess it depends on your definition of "near world destroying accidents".
- mikestew 9y ago
  Could you imagine betting your multi-million dollar satellite on whether the contractor used the approved torque wrench, or something they found at Harbor Freight? Hell, who needs a torque wrench? I've been doing this long enough I can do it by feel. Using the right tool gets us to the moon. Complaining about how micromanagement stifles one's creative vision is how we get the shit show we call "software" today. Somewhere in the middle I'm sure there's a happy medium.
- gumby 9y ago
  Not only can I imagine it I’ve lived it and I think it was great. First: you are misstating the constraint: a procedure was defined by the manufacturer (in this case SpaceX) and they committed to the customer that they would use that procedure. Second: in my case it was pharmaceutical manufacture and we were giving these drugs to patients for an approval study. We had defined a reproducible procedure that gave reproducible results. Deciding the change the procedure would require proving that the result of the change was the same as before said change. As a person who takes medication I am glad for GMP (the equivalent of ISO9000 in pharma manufacture). In the case of an expensive rocket that launches something likely unrepairable after launch (and perhaps irreplaceable) I want to know that the mfr has figured out as many of the risk factors as possible and how to mitigate them.
- torpfactory 9y ago
  When working on large complicated systems the best engineering approach is often to define a process (tools included) and then always repeat the same way. You’d be surprised how many things can go wrong when you stop acting repeatably. It will definitely seems like a waste of time until the first time you blow up a half billion dollar payload because someone unknowingly trashed an important component using the wrong wrench.
yipopov 9y ago
Don't they typically test rockets in outdoors test ranges? And aren't rockets normally supposed to be able to operate outdoors? Of course there is going to be dirt, rocks and yes, even fecal matter. What are they supposed to do, blow their hot exaust gases into a climate-controlled cleanroom?
- russdill 9y ago
  FOD is a big deal and can cause a lot of damage. Flight areas are very carefully kept clear of any debris. https://en.wikipedia.org/wiki/Foreign_object_damage https://en.wikipedia.org/wiki/Foreign_object_damage
- trhway 9y ago
  such government inspectors reports are judged by the thickness. 180 pages is just too thin, even after they padded it with poop "findings".
ComputerGuru 9y ago
Sidebar: I read the title as “(poop and soda) bottles” instead of “poop and (soda bottles),” and was extremely confused. I prefer to swap words around in cases like this to make it unambiguous, for example, “soda bottles and poop...”
- pvg 9y ago
  The original title has the much punchier "poop and pop"
dustinmoorenet 9y ago
I worked on AWACS back in the early 2000s and we had an adversarial dynamic with our QA team. We would see them as 'just getting in the way', but I can now say, after working as a software developer, that without a strong QA (backed up by military laws), our planes would be flying with a lot of 'solutions' that would lead to unnecessary deaths.
russdill 9y ago
Here's a detailed discussion on the SpaceX side: https://www.reddit.com/r/spacex/comments/7ll89f/pentagon_watchdog_cites_top_us_space_contractors/ https://www.reddit.com/r/spacex/comments/7ll89f/pentagon_wat...

33 points by QAPereo 9y ago

pvg 9y ago

Original source at http://gazette.com/poop-pop-threaten-air-force-rocket-program-investigators-say/article/1618540 http://gazette.com/poop-pop-threaten-air-force-rocket-progra...

sctb 9y ago
Thanks, we've updated the link from http://taskandpurpose.com/poop-soda-threaten-air-force-rocket-program-investigators-say/ http://taskandpurpose.com/poop-soda-threaten-air-force-rocke....

Floegipoky 9y ago

> At SpaceX, investigators say they spotted workers using tools that weren’t specified in a manual. Could you imagine being so micromanaged that you'd literally need the Federal Government to change a manual to approve a tool you need?

QAPereo 9y ago
I have a cousin in law enforcement, so yes I can, but it’s still maddening. The formula is essentially: a problem arises. Everyone covers their asses with red tape, and new rules and regulations are born, and never die. It’s a classic, yet very difficult-to-manage bureaupathology. The alternative is just that a friend of someone in power makes the tools.
- ashark 9y ago
  A friend of mine works at a state facility where, as he describes it, it's impossible to be a manager without constantly breaking rules, because many contradict one another or violate the laws of reality. Result is that management is a nasty snake-pit where everyone knows which rules everyone else is breaking and they'll collectively point their fingers at anyone who tries to rock the boat (including, and perhaps especially, improving things), quickly getting them fired (or at least transferred somewhere they'd rather not be). Perhaps significantly, management positions aren't represented by the state workers' union, which is the only thing (barely) insulating the lower level workers from that insanity.
empath75 9y ago
yes: https://aws.amazon.com/compliance/fedramp/ https://aws.amazon.com/compliance/fedramp/
andykee 9y ago
Using the wrong wrench almost caused a nuclear catastrophe: https://en.wikipedia.org/wiki/1980_Damascus_Titan_missile_explosion https://en.wikipedia.org/wiki/1980_Damascus_Titan_missile_ex... It may seem like micromanagement, but small seemingly meaningless deviations from procedure can add up to big problems, particularly when dealing with space.
- paulmd 9y ago
  Doesn't sound like the tool in question had any real impact on that accident (so to speak). I'm guessing if you drop a 3-foot torque wrench through a fuel tank, it does just as much damage as a 3-foot socket wrench. The fact that they grabbed the wrong tool was entirely incidental to damaging the missile. Also, it takes an extremely precise sequence of events to cause a nuclear detonation. A nearby explosion is not enough to do it, as you can obviously see here.
  - andykee 9y ago
    I'd recommend listening to the This American Life episode (https://www.thisamericanlife.org/radio-archives/episode/634/human-error-in-volatile-situations https://www.thisamericanlife.org/radio-archives/episode/634/...) where they interview the two airmen involved. tldr; The socket didn't properly fit the wrench and they had to manually hold the socket on the end of the wrench. One man dropped the socket, which fell through a small gap between the walkway and missile. The torque wrench was large enough that it would not have slipped through this gap if/when dropped.
    - ethbro 9y ago
      And the larger problem is one of knowledge asymmetry. If you're dealing with technology so complicated that one person cannot simultaneously design and repair / maintain it, then whoever performs the latter doesn't possess all the knowledge of the former. Maybe they used an flammable alloy under certain conditions for engineering reasons. Most metal doesn't burn. But do you want repair / maintenance techs making that assumption? Multiple by 100,000 parts. PS: A more realistic example was one of the Skunk Works projects. I forget the details, but either tools of special metallic composition or wooden tools had to be used to avoid damaging the flight characteristics of the aircraft.
klenwell 9y ago
On the other hand, I recommend this tale if you haven't heard it: https://www.thisamericanlife.org/radio-archives/episode/634/human-error-in-volatile-situations https://www.thisamericanlife.org/radio-archives/episode/634/... > In 1980, deep in a nuclear missile silo in Arkansas, a simple human error nearly caused the destruction of a giant portion of the Midwest. Spoiler: at a critical point, it involves a technician improvising with the wrong tool.
- deleted 9y ago
  [deleted]
cjsuk 9y ago
Indeed. And it doesn't work. Look at the sheer number of near world destroying accidents our military have managed to muster over the years. I was actually working for a company that was a subcontractor to the USAF in the distant past. There is little room for individual creativity and ingenious problem solving in such a process heavy environment. It's actually like trying to work on an engineering project with Siri. The outcome of this is that only the problems allowed to be solved by the process are solved and none of the risks not identified by the process are reported, because it's too much paperwork already if you find a new one and you're not going to want your job to get any harder. Ergo, all unidentified risks get taken because they are attractive solutions to many problems. Looking at the spanner incident pointed out by two sibling comments, the "dont carry heavy tools or anchor then when working on the big explodey thing" rule was not enforced and double-checked but hundreds of minor rules with precisely no benefit were being applied. Someone knew of that risk but didn't put the mitigation in place, probably because it was too hard. Now military rigour and all of that doesn't like to be seen to be a turd sieve of a process so what happens is the answer is to add more processes, not look at the fundamental problem in the approach. Overall, you need to focus on the key places where risk is considered high, but for most things, meh. Listening to people and making sure risk is managed effectively is a win.
- pavel_lishin 9y ago
  > Indeed. And it doesn't work. Look at the sheer number of near world destroying accidents our military have managed to muster over the years. What is that number? And do we know that the number would be less without these types of regulations? > Looking at the spanner incident pointed out by two sibling comments, the "dont carry heavy tools or anchor then when working on the big explodey thing" rule was not enforced and double-checked My understanding from listening to the podcast episode (this morning) is that the tool-anchoring rules were only added after that particular disaster.
  - IdeaHamstir 9y ago
    > What is that number? And do we know that the number would be less without these types of regulations? https://en.wikipedia.org/wiki/List_of_military_nuclear_accidents https://en.wikipedia.org/wiki/List_of_military_nuclear_accid... I guess it depends on your definition of "near world destroying accidents".
mikestew 9y ago
Could you imagine betting your multi-million dollar satellite on whether the contractor used the approved torque wrench, or something they found at Harbor Freight? Hell, who needs a torque wrench? I've been doing this long enough I can do it by feel. Using the right tool gets us to the moon. Complaining about how micromanagement stifles one's creative vision is how we get the shit show we call "software" today. Somewhere in the middle I'm sure there's a happy medium.
gumby 9y ago
Not only can I imagine it I’ve lived it and I think it was great. First: you are misstating the constraint: a procedure was defined by the manufacturer (in this case SpaceX) and they committed to the customer that they would use that procedure. Second: in my case it was pharmaceutical manufacture and we were giving these drugs to patients for an approval study. We had defined a reproducible procedure that gave reproducible results. Deciding the change the procedure would require proving that the result of the change was the same as before said change. As a person who takes medication I am glad for GMP (the equivalent of ISO9000 in pharma manufacture). In the case of an expensive rocket that launches something likely unrepairable after launch (and perhaps irreplaceable) I want to know that the mfr has figured out as many of the risk factors as possible and how to mitigate them.
torpfactory 9y ago
When working on large complicated systems the best engineering approach is often to define a process (tools included) and then always repeat the same way. You’d be surprised how many things can go wrong when you stop acting repeatably. It will definitely seems like a waste of time until the first time you blow up a half billion dollar payload because someone unknowingly trashed an important component using the wrong wrench.

yipopov 9y ago

Don't they typically test rockets in outdoors test ranges? And aren't rockets normally supposed to be able to operate outdoors? Of course there is going to be dirt, rocks and yes, even fecal matter. What are they supposed to do, blow their hot exaust gases into a climate-controlled cleanroom?

russdill 9y ago
FOD is a big deal and can cause a lot of damage. Flight areas are very carefully kept clear of any debris. https://en.wikipedia.org/wiki/Foreign_object_damage https://en.wikipedia.org/wiki/Foreign_object_damage
trhway 9y ago
such government inspectors reports are judged by the thickness. 180 pages is just too thin, even after they padded it with poop "findings".

ComputerGuru 9y ago

Sidebar: I read the title as “(poop and soda) bottles” instead of “poop and (soda bottles),” and was extremely confused. I prefer to swap words around in cases like this to make it unambiguous, for example, “soda bottles and poop...”

pvg 9y ago
The original title has the much punchier "poop and pop"

dustinmoorenet 9y ago

I worked on AWACS back in the early 2000s and we had an adversarial dynamic with our QA team. We would see them as 'just getting in the way', but I can now say, after working as a software developer, that without a strong QA (backed up by military laws), our planes would be flying with a lot of 'solutions' that would lead to unnecessary deaths.

russdill 9y ago

Here's a detailed discussion on the SpaceX side: https://www.reddit.com/r/spacex/comments/7ll89f/pentagon_watchdog_cites_top_us_space_contractors/ https://www.reddit.com/r/spacex/comments/7ll89f/pentagon_wat...