3 ms·

Power failed at SpaceX mission control before September spacewalk

27 points by carabiner 2y ago

carabiner 2y ago
"The outage also hit servers that host procedures meant to overcome such an outage and hindered SpaceX's ability to transfer mission control to a backup facility in Florida, the people said. Company officials had no paper copies of backup procedures, one of the people added, leaving them unable to respond until power was restored." Also had colleague at NASA who visited USSR back in the day to work on space operations. He said they had power outages on the reg, and they'd just bring out candles and continue like normal.
taylodl 2y ago
WTF?!?!?! This is Operations 101 stuff. Even crazier is the fact Tesla has energy management solutions such as AMOS that could be used to handle such an event. Wouldn't that have been great advertisement? The power went out, but SpaceX was unaffected, thanks in part to Tesla energy management solutions. Hopefully this is being rectified.
- inemesitaffia 2y ago
  Wouldn't have helped
saturn8601 2y ago
[flagged]
exe34 2y ago
I'm actually impressed they haven't killed anyone yet - move fast and break things has been very successful so far in getting to space.
- rohan_ 2y ago
  you should read "The Right Stuff" - the 1950's and 60's definitely had the "daring" and "risky" culture around their test-pilot program. we would not have gotten to space as quickly without that culture.
- caseyy 2y ago
  Many things worth doing involve personal risk, sometimes even to life and limb. But companies like SpaceX, and many other successful ones, have mastered the distinction between necessary risks and unnecessary ones. And they do a lot to eliminate the latter. Risks from unknowable unknowns are acceptable, but those caused by negligence (especially outsized risks in this category) are not. You can move fast, break things, and still not be negligent. Consider big failures in business, finance and engineering – Enron, WeWork, collateralized debt obligations and the Challenger disaster perhaps. All had elements of reckless avoidable risks – known problems that were known to be outsized risks/exposures, that go against good known good practices. Known, known, known – that is negligence and dereliction of common sense. One does not move fast and break things for long before all the things are broken this way. But by not being negligent, I think you can go quite far, even into space. And the opposite of negligent is diligent – sweating the details. SpaceX really does sweat the details a lot. They will not have a power outage like this again. And that's what separates them from the examples of companies and endeavors I've given above. Enron, WeWork, CDOs, the Challenger – all had the same problems, week after week, month after month, quarter after quarter, even year after year that people knew about and chose to ignore.
perihelions 2y ago
- "The outage also hit servers that host procedures meant to overcome such an outage and hindered SpaceX's ability to transfer mission control to a backup facility in Florida, the people said. Company officials had no paper copies of backup procedures, one of the people added, leaving them unable to respond until power was restored." "Yes, we have backups."

27 points by carabiner 2y ago

"The outage also hit servers that host procedures meant to overcome such an outage and hindered SpaceX's ability to transfer mission control to a backup facility in Florida, the people said. Company officials had no paper copies of backup procedures, one of the people added, leaving them unable to respond until power was restored." Also had colleague at NASA who visited USSR back in the day to work on space operations. He said they had power outages on the reg, and they'd just bring out candles and continue like normal.

taylodl 2y ago

WTF?!?!?! This is Operations 101 stuff. Even crazier is the fact Tesla has energy management solutions such as AMOS that could be used to handle such an event. Wouldn't that have been great advertisement? The power went out, but SpaceX was unaffected, thanks in part to Tesla energy management solutions. Hopefully this is being rectified.

inemesitaffia 2y ago
Wouldn't have helped

saturn8601 2y ago

[flagged]

exe34 2y ago

I'm actually impressed they haven't killed anyone yet - move fast and break things has been very successful so far in getting to space.

rohan_ 2y ago
you should read "The Right Stuff" - the 1950's and 60's definitely had the "daring" and "risky" culture around their test-pilot program. we would not have gotten to space as quickly without that culture.
caseyy 2y ago
Many things worth doing involve personal risk, sometimes even to life and limb. But companies like SpaceX, and many other successful ones, have mastered the distinction between necessary risks and unnecessary ones. And they do a lot to eliminate the latter. Risks from unknowable unknowns are acceptable, but those caused by negligence (especially outsized risks in this category) are not. You can move fast, break things, and still not be negligent. Consider big failures in business, finance and engineering – Enron, WeWork, collateralized debt obligations and the Challenger disaster perhaps. All had elements of reckless avoidable risks – known problems that were known to be outsized risks/exposures, that go against good known good practices. Known, known, known – that is negligence and dereliction of common sense. One does not move fast and break things for long before all the things are broken this way. But by not being negligent, I think you can go quite far, even into space. And the opposite of negligent is diligent – sweating the details. SpaceX really does sweat the details a lot. They will not have a power outage like this again. And that's what separates them from the examples of companies and endeavors I've given above. Enron, WeWork, CDOs, the Challenger – all had the same problems, week after week, month after month, quarter after quarter, even year after year that people knew about and chose to ignore.

perihelions 2y ago

- "The outage also hit servers that host procedures meant to overcome such an outage and hindered SpaceX's ability to transfer mission control to a backup facility in Florida, the people said. Company officials had no paper copies of backup procedures, one of the people added, leaving them unable to respond until power was restored." "Yes, we have backups."