4 ms·

And Southwest had two crew-management outages in 2022[0], so let's not sing their praises for escaping the CrowdStrike disruption. Southwest has been widely cri

And Southwest had two crew-management outages in 2022[0], so let's not sing their praises for escaping the CrowdStrike disruption. Southwest has been widely critized for under-investment in technology, Delta on the other hand purchased one of the best security products on the market and that backfired. [0] https://en.wikipedia.org/wiki/2022_Southwest_Airlines_scheduling_crisis https://en.wikipedia.org/wiki/2022_Southwest_Airlines_schedu...

chgs 2y ago
Delta put all their eggs in one basket and had no DR capability
- Someone1234 2y ago
  What basis do you have for saying that? It is likely their DR was running on a mirror of their production systems, and was similarly impacted by the Crowdstrike outage. So they fell back to Windows Servers similarly stuck in a boot-loop. Keep in mind there was no way to opt out or delay CS Channel updates.
  - chgs 2y ago
    If your DR system is susceptible to the same faults as your main system it’s not a DR system. It would be like claiming raid1 is a backup.
    - TheDong 2y ago
      Or it would be like claiming my backup isn’t a backup because both systems run openssh, so a remote code execution vuln there could take down both systems. Any DR system will have to accept some risks, and those don’t necessarily invalidate it in general, just make it insufficient for some scenarios. Conversely, if they ran the main system on windows with crowdstrike and the DR one on poorly configured linux with no security software, they probably would have needed more sysadmins, had more trouble maintaining software for both, and been vulnerable to risk from both linux and windows bugs, so I feel like they made the right tradeoff in general. I’m sure you, who can deride this DR system, have devised your own system such that it is resilient to a meteor destroying the earth.
      - shagie 2y ago
        > I’m sure you, who can deride this DR system, have devised your own system such that it is resilient to a meteor destroying the earth. That reminds me one of Corey Quinn's comfortable AWS truths. https://x.com/QuinnyPig/status/1173371749808783360 https://x.com/QuinnyPig/status/1173371749808783360 > If your DR plan assumes us-east-1 dies unrecoverably, what you're really planning for is 100 square miles of Northern Virginia no longer existing. Good luck with that ad farm in a nuclear wasteland, buddy!
    - dredmorbius 2y ago
      As HN itself discovered a couple of years ago when a set of same-manufacturer, same-batch disks within both RAID arrays and backup server failed within a few hours of one another: <https://news.ycombinator.com/item?id=32048148 https://news.ycombinator.com/item?id=32048148> <https://news.ycombinator.com/item?id=32031243 https://news.ycombinator.com/item?id=32031243>
  - freeopinion 2y ago
    > Keep in mind there was no way to opt out or delay CS Channel updates. Do CS updates somehow work over airgaps? You know, the kind that production systems have to prevent any access to or from external networks? Well... some production systems anyway.
    - nradov 2y ago
      What's your point? An air gapped disaster recovery system would be useless. An airline operations application has to connect to a bunch of other external systems to be of any use.
  - amluto 2y ago
    One idea: build a DR system and turn it off. Ideally it would be cloneable, but even without that ability, one could test it every few months to make sure it boots adequately quickly and then turn it back off. The attack surface of a bunch of computers or instances that are powered down is pretty low.
    - compiler-guy 2y ago
      Better yet, alternate between them every month or two.
shiroiushi 2y ago
>Delta on the other hand purchased one of the best security products on the market and that backfired. It looks like it wasn't a good security product after all...

chgs 2y ago

Delta put all their eggs in one basket and had no DR capability

Someone1234 2y ago
What basis do you have for saying that? It is likely their DR was running on a mirror of their production systems, and was similarly impacted by the Crowdstrike outage. So they fell back to Windows Servers similarly stuck in a boot-loop. Keep in mind there was no way to opt out or delay CS Channel updates.
- chgs 2y ago
  If your DR system is susceptible to the same faults as your main system it’s not a DR system. It would be like claiming raid1 is a backup.
  - TheDong 2y ago
    Or it would be like claiming my backup isn’t a backup because both systems run openssh, so a remote code execution vuln there could take down both systems. Any DR system will have to accept some risks, and those don’t necessarily invalidate it in general, just make it insufficient for some scenarios. Conversely, if they ran the main system on windows with crowdstrike and the DR one on poorly configured linux with no security software, they probably would have needed more sysadmins, had more trouble maintaining software for both, and been vulnerable to risk from both linux and windows bugs, so I feel like they made the right tradeoff in general. I’m sure you, who can deride this DR system, have devised your own system such that it is resilient to a meteor destroying the earth.
    - shagie 2y ago
      > I’m sure you, who can deride this DR system, have devised your own system such that it is resilient to a meteor destroying the earth. That reminds me one of Corey Quinn's comfortable AWS truths. https://x.com/QuinnyPig/status/1173371749808783360 https://x.com/QuinnyPig/status/1173371749808783360 > If your DR plan assumes us-east-1 dies unrecoverably, what you're really planning for is 100 square miles of Northern Virginia no longer existing. Good luck with that ad farm in a nuclear wasteland, buddy!
  - dredmorbius 2y ago
    As HN itself discovered a couple of years ago when a set of same-manufacturer, same-batch disks within both RAID arrays and backup server failed within a few hours of one another: <https://news.ycombinator.com/item?id=32048148 https://news.ycombinator.com/item?id=32048148> <https://news.ycombinator.com/item?id=32031243 https://news.ycombinator.com/item?id=32031243>
- freeopinion 2y ago
  > Keep in mind there was no way to opt out or delay CS Channel updates. Do CS updates somehow work over airgaps? You know, the kind that production systems have to prevent any access to or from external networks? Well... some production systems anyway.
  - nradov 2y ago
    What's your point? An air gapped disaster recovery system would be useless. An airline operations application has to connect to a bunch of other external systems to be of any use.
- amluto 2y ago
  One idea: build a DR system and turn it off. Ideally it would be cloneable, but even without that ability, one could test it every few months to make sure it boots adequately quickly and then turn it back off. The attack surface of a bunch of computers or instances that are powered down is pretty low.
  - compiler-guy 2y ago
    Better yet, alternate between them every month or two.

shiroiushi 2y ago

>Delta on the other hand purchased one of the best security products on the market and that backfired. It looks like it wasn't a good security product after all...