4 ms·

Are we reading the same article? “Early this morning, a vendor-supplied firewall went down and connection to some operational data was unexpectedly lost,” The

Are we reading the same article? “Early this morning, a vendor-supplied firewall went down and connection to some operational data was unexpectedly lost,” They're blaming the vendor and absolving themselves of any responsibility. If losing a connection causes catastrophic failure, who's really at fault?

more_corn 3y ago
If you have a critical bottleneck and you don’t have a spare you made a mistake. A couple.
unethical_ban 4y ago
True, but being technically accurate isn't bad. If a vendor owns it, a vendor owns it. Not having HA on a network segment is bad, though.
- bastardoperator 4y ago
  Can we validate the accuracy? Probably not, but I can say in my own postmortem or RCA analysis taking zero responsibility or deflecting is a sign of a greater problem. There is always room for learning and growing.
  - Paul-Craft 4y ago
    You can't claim they aren't taking responsibility unless you've seen the internal postmortem. External messaging is PR. It's designed to not affect the share price. You can't extrapolate anything about internal process from it.
    - bastardoperator 4y ago
      You seem to be pretty invested here, everything okay? It's shit PR, devoid of any responsibility, and they're already in the dog house when it comes to stock value. For me personally, I just won't fly this airline, I can't trust them on the ground why would I put my life or valuable time in their hands or reward them with business? We don't need to a postmortem, the blame has been solely laid on someone else's shoulders. I think the only thing we can extrapolate from this conversation is that you possibly work for SW, or that you're upside down on SW shares. Have a great day and welcome to HN.
  - unethical_ban 4y ago
    Saying a fact isn't deflection. I have been in post-mortems too, and accuracy is important. If I work at a call center and there is an outage on a partner's PBX, then the summary will say "vendor PBX caused an outage". Part of that conversation will be "how to we mitigate the risks of a vendor PBX outage in the future", but the technical fact stands: Someone else's equipment failed.
Paul-Craft 4y ago
That's a description of what happened. It doesn't say why it happened. Nobody is being blamed for anything. For all we know, somebody at Southwest tripped on a power cord and shut down the machine running the firewall. Facetious, yes, but I have once had someone literally trip on a power cord for a machine running test software I cared about, shutting it down. (Yes, we fixed the issue and got UPS's for these machines after that lol)
- AnimalMuppet 4y ago
  At a minimum, that firewall appears to have been a single point of failure.
  - jrockway 4y ago
    That's how security stuff intrinsically works. It sits in the critical path and fails closed. If it failed open, then it would be "malware installed on Southwest's production network due to unpatched node.js module" or something.
    - m348e912 4y ago
      >> [security stuff] sits in the critical path and fails closed. Not entirely accurate. Some inline IPS appliances fail open, and networks can be architected to fail around a single firewall in a high availability pair.
    - bink 4y ago
      That's a dramatic over-simplification. You have redundant / clustered firewalls that allow for testing configuration changes before they're deployed everywhere so you never end up in a situation where you have to "fail closed". You have redundant links into critical data centers, either hot/hot or hot/warm. You have redundant power supplies in the racks running the firewalls and routers. You have redundant data centers (where necessary).
    - dilyevsky 4y ago
      Nothing stopping you from running redundant links with redundant appliances, except beancounters ofc
ano-ther 4y ago
Interesting approach. What does it even mean? Wouldn’t most of an airline run on “vendor supplied $thing”? “Today, a vendor-supplied plane went down […]”

more_corn 3y ago

If you have a critical bottleneck and you don’t have a spare you made a mistake. A couple.

unethical_ban 4y ago

True, but being technically accurate isn't bad. If a vendor owns it, a vendor owns it. Not having HA on a network segment is bad, though.

bastardoperator 4y ago
Can we validate the accuracy? Probably not, but I can say in my own postmortem or RCA analysis taking zero responsibility or deflecting is a sign of a greater problem. There is always room for learning and growing.
- Paul-Craft 4y ago
  You can't claim they aren't taking responsibility unless you've seen the internal postmortem. External messaging is PR. It's designed to not affect the share price. You can't extrapolate anything about internal process from it.
  - bastardoperator 4y ago
    You seem to be pretty invested here, everything okay? It's shit PR, devoid of any responsibility, and they're already in the dog house when it comes to stock value. For me personally, I just won't fly this airline, I can't trust them on the ground why would I put my life or valuable time in their hands or reward them with business? We don't need to a postmortem, the blame has been solely laid on someone else's shoulders. I think the only thing we can extrapolate from this conversation is that you possibly work for SW, or that you're upside down on SW shares. Have a great day and welcome to HN.
- unethical_ban 4y ago
  Saying a fact isn't deflection. I have been in post-mortems too, and accuracy is important. If I work at a call center and there is an outage on a partner's PBX, then the summary will say "vendor PBX caused an outage". Part of that conversation will be "how to we mitigate the risks of a vendor PBX outage in the future", but the technical fact stands: Someone else's equipment failed.

Paul-Craft 4y ago

That's a description of what happened. It doesn't say why it happened. Nobody is being blamed for anything. For all we know, somebody at Southwest tripped on a power cord and shut down the machine running the firewall. Facetious, yes, but I have once had someone literally trip on a power cord for a machine running test software I cared about, shutting it down. (Yes, we fixed the issue and got UPS's for these machines after that lol)

AnimalMuppet 4y ago
At a minimum, that firewall appears to have been a single point of failure.
- jrockway 4y ago
  That's how security stuff intrinsically works. It sits in the critical path and fails closed. If it failed open, then it would be "malware installed on Southwest's production network due to unpatched node.js module" or something.
  - m348e912 4y ago
    >> [security stuff] sits in the critical path and fails closed. Not entirely accurate. Some inline IPS appliances fail open, and networks can be architected to fail around a single firewall in a high availability pair.
  - bink 4y ago
    That's a dramatic over-simplification. You have redundant / clustered firewalls that allow for testing configuration changes before they're deployed everywhere so you never end up in a situation where you have to "fail closed". You have redundant links into critical data centers, either hot/hot or hot/warm. You have redundant power supplies in the racks running the firewalls and routers. You have redundant data centers (where necessary).
  - dilyevsky 4y ago
    Nothing stopping you from running redundant links with redundant appliances, except beancounters ofc

ano-ther 4y ago

Interesting approach. What does it even mean? Wouldn’t most of an airline run on “vendor supplied $thing”? “Today, a vendor-supplied plane went down […]”