6 ms·
Southwest Airlines pauses departures after reporting technical problems
- bastardoperator 3y agoI love how they immediately place blame on the vendor. You would think an operation of this scale and importance would have some redundancy built in or after the previous disaster would have attacked single points of failure.
- Paul-Craft 3y agoWhere do they "place blame on the vendor?" What SPoF should they have addressed? They blame "a vendor-supplied firewall," which is different from the scheduling software that caused the incident over the holidays, and different from blaming the vendor itself.
- bastardoperator 3y agoAre we reading the same article? “Early this morning, a vendor-supplied firewall went down and connection to some operational data was unexpectedly lost,” They're blaming the vendor and absolving themselves of any responsibility. If losing a connection causes catastrophic failure, who's really at fault?
- more_corn 3y agoIf you have a critical bottleneck and you don’t have a spare you made a mistake. A couple.
- unethical_ban 3y agoTrue, but being technically accurate isn't bad. If a vendor owns it, a vendor owns it. Not having HA on a network segment is bad, though.
- bastardoperator 3y agoCan we validate the accuracy? Probably not, but I can say in my own postmortem or RCA analysis taking zero responsibility or deflecting is a sign of a greater problem. There is always room for learning and growing.
- Paul-Craft 3y agoYou can't claim they aren't taking responsibility unless you've seen the internal postmortem. External messaging is PR. It's designed to not affect the share price. You can't extrapolate anything about internal process from it.
- bastardoperator 3y agoYou seem to be pretty invested here, everything okay? It's shit PR, devoid of any responsibility, and they're already in the dog house when it comes to stock value. For me personally, I just won't fly this airline, I can't trust them on the ground why would I put my life or valuable time in their hands or reward them with business? We don't need to a postmortem, the blame has been solely laid on someone else's shoulders. I think the only thing we can extrapolate from this conversation is that you possibly work for SW, or that you're upside down on SW shares. Have a great day and welcome to HN.
- unethical_ban 3y agoSaying a fact isn't deflection. I have been in post-mortems too, and accuracy is important. If I work at a call center and there is an outage on a partner's PBX, then the summary will say "vendor PBX caused an outage". Part of that conversation will be "how to we mitigate the risks of a vendor PBX outage in the future", but the technical fact stands: Someone else's equipment failed.
- Paul-Craft 3y agoThat's a description of what happened. It doesn't say why it happened. Nobody is being blamed for anything. For all we know, somebody at Southwest tripped on a power cord and shut down the machine running the firewall. Facetious, yes, but I have once had someone literally trip on a power cord for a machine running test software I cared about, shutting it down. (Yes, we fixed the issue and got UPS's for these machines after that lol)
- AnimalMuppet 3y agoAt a minimum, that firewall appears to have been a single point of failure.
- jrockway 3y agoThat's how security stuff intrinsically works. It sits in the critical path and fails closed. If it failed open, then it would be "malware installed on Southwest's production network due to unpatched node.js module" or something.
- m348e912 3y ago>> [security stuff] sits in the critical path and fails closed. Not entirely accurate. Some inline IPS appliances fail open, and networks can be architected to fail around a single firewall in a high availability pair.
- bink 3y agoThat's a dramatic over-simplification. You have redundant / clustered firewalls that allow for testing configuration changes before they're deployed everywhere so you never end up in a situation where you have to "fail closed". You have redundant links into critical data centers, either hot/hot or hot/warm. You have redundant power supplies in the racks running the firewalls and routers. You have redundant data centers (where necessary).
- dilyevsky 3y agoNothing stopping you from running redundant links with redundant appliances, except beancounters ofc
- ano-ther 3y agoInteresting approach. What does it even mean? Wouldn’t most of an airline run on “vendor supplied $thing”? “Today, a vendor-supplied plane went down […]”
- deleted 3y ago[deleted]
- javawizard 3y agoIt reminds me of the saying: "when you owe the bank $100 that's your problem; when you owe the bank $100 million that's the bank's problem" It doesn't matter if fault lies with an external vendor, people can't fly and Southwest is still going to take reputational damage as a result.
- TylerE 3y agoThe way I’ve heard it, which I think is a bit punchier is “If you owe the bank $10,000 dollars, the bank owns you. If you owe the bank $100 million dollars, you own the bank.”
- nothingneko 3y agoThis is their second time if I'm not mistaken. It's sad that there's no changes to prevent this.
- Paul-Craft 3y ago> Southwest had a meltdown over the year-end holidays, which drove thousands of flight cancellations and resulted in an $800 million pretax hit for the fourth quarter. The airline upgraded scheduling software that couldn’t handle numerous changes to crew assignments during severe winter weather late last year. > The issues Tuesday did not appear to be related to that software. From the article, emphasis added.
- bobbylarrybobby 3y agoUnrelated except that the same ethos and processes probably led to both.
- borner791 3y agohttps://news.ycombinator.com/item?id=34149340 https://news.ycombinator.com/item?id=34149340
- 0zemp1c 3y agothis is why I walk everywhere - no tech debt!
- xeromal 3y agoAn earthquake is a form of tech debt. I think.