4 ms·

That's a description of what happened. It doesn't say why it happened. Nobody is being blamed for anything. For all we know, somebody at Southwest tripped on

That's a description of what happened. It doesn't say why it happened. Nobody is being blamed for anything. For all we know, somebody at Southwest tripped on a power cord and shut down the machine running the firewall. Facetious, yes, but I have once had someone literally trip on a power cord for a machine running test software I cared about, shutting it down. (Yes, we fixed the issue and got UPS's for these machines after that lol)

AnimalMuppet 4y ago
At a minimum, that firewall appears to have been a single point of failure.
- jrockway 4y ago
  That's how security stuff intrinsically works. It sits in the critical path and fails closed. If it failed open, then it would be "malware installed on Southwest's production network due to unpatched node.js module" or something.
  - m348e912 4y ago
    >> [security stuff] sits in the critical path and fails closed. Not entirely accurate. Some inline IPS appliances fail open, and networks can be architected to fail around a single firewall in a high availability pair.
  - bink 4y ago
    That's a dramatic over-simplification. You have redundant / clustered firewalls that allow for testing configuration changes before they're deployed everywhere so you never end up in a situation where you have to "fail closed". You have redundant links into critical data centers, either hot/hot or hot/warm. You have redundant power supplies in the racks running the firewalls and routers. You have redundant data centers (where necessary).
  - dilyevsky 4y ago
    Nothing stopping you from running redundant links with redundant appliances, except beancounters ofc
    - jrockway 4y ago
      I guess I've accepted that it's something intrinsic to the software that's going to make it fail. Bad config push, bad packet segfaults the packet inspection software, etc. (Yup, people still write parsers for untrusted user input in C, and don't fuzz it. That's the state of the industry.) In those cases, all of your replicas fail identically at the same time. For mere hardware failure, we seem to have moved past that as an industry. Cloud providers will "transparently" migrate your app to a new computer if the one you're running on blows up. (Transparently is in quotes because you'll see the P99.9 latency increase when it happens.)
  - deleted 4y ago
    [deleted]
  - unethical_ban 4y ago
    Network security devices are often deployed with high availability baked in. If one firewall appliance dies on a production system, it should have a peer to failover to. We don't know what happened, but at least for hardware, HA is the standard.

AnimalMuppet 4y ago

At a minimum, that firewall appears to have been a single point of failure.

jrockway 4y ago
That's how security stuff intrinsically works. It sits in the critical path and fails closed. If it failed open, then it would be "malware installed on Southwest's production network due to unpatched node.js module" or something.
- m348e912 4y ago
  >> [security stuff] sits in the critical path and fails closed. Not entirely accurate. Some inline IPS appliances fail open, and networks can be architected to fail around a single firewall in a high availability pair.
- bink 4y ago
  That's a dramatic over-simplification. You have redundant / clustered firewalls that allow for testing configuration changes before they're deployed everywhere so you never end up in a situation where you have to "fail closed". You have redundant links into critical data centers, either hot/hot or hot/warm. You have redundant power supplies in the racks running the firewalls and routers. You have redundant data centers (where necessary).
- dilyevsky 4y ago
  Nothing stopping you from running redundant links with redundant appliances, except beancounters ofc
  - jrockway 4y ago
    I guess I've accepted that it's something intrinsic to the software that's going to make it fail. Bad config push, bad packet segfaults the packet inspection software, etc. (Yup, people still write parsers for untrusted user input in C, and don't fuzz it. That's the state of the industry.) In those cases, all of your replicas fail identically at the same time. For mere hardware failure, we seem to have moved past that as an industry. Cloud providers will "transparently" migrate your app to a new computer if the one you're running on blows up. (Transparently is in quotes because you'll see the P99.9 latency increase when it happens.)
- deleted 4y ago
  [deleted]
- unethical_ban 4y ago
  Network security devices are often deployed with high availability baked in. If one firewall appliance dies on a production system, it should have a peer to failover to. We don't know what happened, but at least for hardware, HA is the standard.