7 ms·

>> Why were other airlines able to get back to normal so much faster than Delta? I read somewhere that their crew tracking software was hit hard and took time

>> Why were other airlines able to get back to normal so much faster than Delta? I read somewhere that their crew tracking software was hit hard and took time to recover. Will look for source on that. (Edited) source: https://news.delta.com/update-delta-customers-ceo-ed-bastian https://news.delta.com/update-delta-customers-ceo-ed-bastian “… and in particular one of our crew tracking-related tools was affected and unable to effectively process the unprecedented number of changes triggered by the system shutdown…”

Onavo 2y ago
Because they used Windows 3.1
- ZeWaka 2y ago
  In the article it says Southwest used 3.1, not Delta (though, that's apparently incorrect according to other posters).
  - Someone1234 2y ago
    And Southwest had two crew-management outages in 2022[0], so let's not sing their praises for escaping the CrowdStrike disruption. Southwest has been widely critized for under-investment in technology, Delta on the other hand purchased one of the best security products on the market and that backfired. [0] https://en.wikipedia.org/wiki/2022_Southwest_Airlines_scheduling_crisis https://en.wikipedia.org/wiki/2022_Southwest_Airlines_schedu...
    - chgs 2y ago
      Delta put all their eggs in one basket and had no DR capability
      - Someone1234 2y ago
        What basis do you have for saying that? It is likely their DR was running on a mirror of their production systems, and was similarly impacted by the Crowdstrike outage. So they fell back to Windows Servers similarly stuck in a boot-loop. Keep in mind there was no way to opt out or delay CS Channel updates.
        chgs 2y ago
        If your DR system is susceptible to the same faults as your main system it’s not a DR system. It would be like claiming raid1 is a backup.
        TheDong 2y ago
        Or it would be like claiming my backup isn’t a backup because both systems run openssh, so a remote code execution vuln there could take down both systems. Any DR system will have to accept some risks, and those don’t necessarily invalidate it in general, just make it insufficient for some scenarios. Conversely, if they ran the main system on windows with crowdstrike and the DR one on poorly configured linux with no security software, they probably would have needed more sysadmins, had more trouble maintaining software for both, and been vulnerable to risk from both linux and windows bugs, so I feel like they made the right tradeoff in general. I’m sure you, who can deride this DR system, have devised your own system such that it is resilient to a meteor destroying the earth.
        shagie 2y ago
        > I’m sure you, who can deride this DR system, have devised your own system such that it is resilient to a meteor destroying the earth. That reminds me one of Corey Quinn's comfortable AWS truths. https://x.com/QuinnyPig/status/1173371749808783360 https://x.com/QuinnyPig/status/1173371749808783360 > If your DR plan assumes us-east-1 dies unrecoverably, what you're really planning for is 100 square miles of Northern Virginia no longer existing. Good luck with that ad farm in a nuclear wasteland, buddy!
        dredmorbius 2y ago
    - shiroiushi 2y ago
      >Delta on the other hand purchased one of the best security products on the market and that backfired. It looks like it wasn't a good security product after all...
- shagie 2y ago
  I chased through this chain the other day... https://www.tomshardware.com/software/windows/windows-31-saves-the-day-during-crowdstrike-outage https://www.tomshardware.com/software/windows/windows-31-sav... https://www.forbes.com/sites/tedreed/2024/07/20/meltdown-what-meltdown-southwest-flew-on-time-all-day/ https://www.forbes.com/sites/tedreed/2024/07/20/meltdown-wha... > A story on the website govtech.com on Friday asked the question, “Why isn’t Southwest affected by the CrowdStrike/Microsoft outage? > “That’s because major portions of the airline’s computer systems are still using Windows 3.1, a 32-year-old version of Microsoft’s computer operating software,” the website said. “It’s so old that the CrowdStrike issue doesn’t affect it so Southwest is still operating as normal. It’s typically not a good idea to wait so long to update, but in this one instance Southwest has done itself a favor.” The govetech.com article is https://www.govtech.com/question-of-the-day/why-isnt-southwest-affected-by-the-crowdstrike-microsoft-outage https://www.govtech.com/question-of-the-day/why-isnt-southwe... which linked to https://www.digitaltrends.com/computing/southwest-cloudstrike-windows-3-1/ https://www.digitaltrends.com/computing/southwest-cloudstrik... which linked to an earlier Forbes article - https://www.forbes.com/sites/hershshefrin/2022/12/31/can-southwest-airlines-fix-its-systemic-weaknesses-in-the-new-year/ https://www.forbes.com/sites/hershshefrin/2022/12/31/can-sou... > The December 2022 scheduling fiasco was the result of skimping on information technology. I am old enough to remember when Microsoft introduced a new operating system called Windows 95, to replace its predecessor operating system Windows 3.1. The 95 in Windows 95 refers to the year of its introduction: 1995. By some accounts, major portions of Southwest’s scheduling system for pilots and flight attendants is built on the Windows 95 platform. That platform is now more than 25 years old.
  - JumpCrisscross 2y ago
    Southwest does not run Windows 3.1: “That’s it. That’s where all these stories can trace their origin to. These few paragraphs do not say that Southwest is still using ancient Windows versions; it just states that the systems they developed internally, SkySolver and Crew Web Access, look ‘historic like they were designed on Windows 95’.” https://www.osnews.com/story/140301/no-southwest-airlines-is-not-still-using-windows-3-1/ https://www.osnews.com/story/140301/no-southwest-airlines-is...
smileysteve 2y ago
Re Delta It's not so much a severity as "hard"; but with the hub and spoke model that Delta uses, scheduling being down (at all on Friday), combined with FAA hour limits. It becomes exponentially difficult to reschedule flights. Put more plainly, on Friday, your scheduling software is down for 4 hours in the morning, so you "borrow" any replacements you need for employees that are late or sick. This ruins the availability for the next flights, at which time you hope the system is up again; but if it's not, you borrow from the evening flights. Combine this with each flight that was late/cancelled as you were hoping to fill now affects the hours available for the employees that were available. Finally, as you've cascaded this, you head into a weekend trying to catalog how many hours each crew member did or did not log, and you're not sure how to get them back in time.
- inferiorhuman 2y ago
  Except for Southwest the other legacy airlines (United, American) also use a hub and spoke model. So does jetBlue.
  - crazytony 2y ago
    Funny you should mention WN. Delta's meltdown is the exact same scenario as Southwest. Crew scheduling is messed up, they don't have a way of tracking where employees are, if the employee is legal, etc and so the operation grinds to a halt
    - rconti 2y ago
      To clarify, Southwest's meltdown last year, which was all about the difficulties of crew scheduling and the knock-on effects of same.
- sidewndr46 2y ago
  wouldn't this imply either an upper bound on down time (airline simply folds as it never catches up) or an upper bound on the duration of the impact ?
reaperducer 2y ago
>> Why were other airlines able to get back to normal so much faster than Delta? I read somewhere that their crew tracking software was hit hard and took time to recover. Will look for source on that. I heard on the radio (maybe NPR, not sure) it wasn't about the computers, it was about Delta's response. According to the report, the other airlines delayed flights, while Delta cancelled them outright. That left Delta with more people and planes in the wrong places, making it harder to recover.
crazytony 2y ago
One other compounding problem is that Delta's headquarters and main traffic patterns are on the east coast. Crowdstrike affected all the airlines at roughly the same time. This gave them roughly one to two fewer hours to respond before they hit their morning peak flights. As someone else pointed out, they probably weren't ready by the time they needed their systems for the morning rush so they went to their business continuity strategy (manual). This has a throughput and recovery time penalty and obviously it compounds the longer they are in that mode. I think what we're finding with the Southwest meltdown and now the Delta meltdown is that the big airlines just don't have the manpower or scheduling slack to accommodate going into business continuity. I do think this should be investigated. Hopefully financial penalties incentivize action but time will tell.
- throwaway2037 2y ago
  > big airlines just don't have the manpower or scheduling slack to accommodate going into business continuity Do small airlines have it? And, how much higher are you willing to pay in ticket prices to have this ability?
- katbyte 2y ago
  They prioritized stock buy backs instead of investing in a robust it operation
  - shiroiushi 2y ago
    As well they should! Which one profits the CEO more? Stock buy-backs or robust IT? Robust IT is only good for the company in the long term; however, with stock buy-backs or other skimping on IT, if disaster like this happens, the CEO just takes his golden parachute and leaves, but if no disaster happens, he gets a huge bonus to buy another private yacht.
- WalterBright 2y ago
  > Hopefully financial penalties incentivize action Delta already took a huge financial hit for this.

Onavo 2y ago

Because they used Windows 3.1

ZeWaka 2y ago
In the article it says Southwest used 3.1, not Delta (though, that's apparently incorrect according to other posters).
- Someone1234 2y ago
  And Southwest had two crew-management outages in 2022[0], so let's not sing their praises for escaping the CrowdStrike disruption. Southwest has been widely critized for under-investment in technology, Delta on the other hand purchased one of the best security products on the market and that backfired. [0] https://en.wikipedia.org/wiki/2022_Southwest_Airlines_scheduling_crisis https://en.wikipedia.org/wiki/2022_Southwest_Airlines_schedu...
  - chgs 2y ago
    Delta put all their eggs in one basket and had no DR capability
    - Someone1234 2y ago
      What basis do you have for saying that? It is likely their DR was running on a mirror of their production systems, and was similarly impacted by the Crowdstrike outage. So they fell back to Windows Servers similarly stuck in a boot-loop. Keep in mind there was no way to opt out or delay CS Channel updates.
      - chgs 2y ago
        If your DR system is susceptible to the same faults as your main system it’s not a DR system. It would be like claiming raid1 is a backup.
        TheDong 2y ago
        Or it would be like claiming my backup isn’t a backup because both systems run openssh, so a remote code execution vuln there could take down both systems. Any DR system will have to accept some risks, and those don’t necessarily invalidate it in general, just make it insufficient for some scenarios. Conversely, if they ran the main system on windows with crowdstrike and the DR one on poorly configured linux with no security software, they probably would have needed more sysadmins, had more trouble maintaining software for both, and been vulnerable to risk from both linux and windows bugs, so I feel like they made the right tradeoff in general. I’m sure you, who can deride this DR system, have devised your own system such that it is resilient to a meteor destroying the earth.
        shagie 2y ago
        > I’m sure you, who can deride this DR system, have devised your own system such that it is resilient to a meteor destroying the earth. That reminds me one of Corey Quinn's comfortable AWS truths. https://x.com/QuinnyPig/status/1173371749808783360 https://x.com/QuinnyPig/status/1173371749808783360 > If your DR plan assumes us-east-1 dies unrecoverably, what you're really planning for is 100 square miles of Northern Virginia no longer existing. Good luck with that ad farm in a nuclear wasteland, buddy!
        dredmorbius 2y ago
  - shiroiushi 2y ago
    >Delta on the other hand purchased one of the best security products on the market and that backfired. It looks like it wasn't a good security product after all...
shagie 2y ago
I chased through this chain the other day... https://www.tomshardware.com/software/windows/windows-31-saves-the-day-during-crowdstrike-outage https://www.tomshardware.com/software/windows/windows-31-sav... https://www.forbes.com/sites/tedreed/2024/07/20/meltdown-what-meltdown-southwest-flew-on-time-all-day/ https://www.forbes.com/sites/tedreed/2024/07/20/meltdown-wha... > A story on the website govtech.com on Friday asked the question, “Why isn’t Southwest affected by the CrowdStrike/Microsoft outage? > “That’s because major portions of the airline’s computer systems are still using Windows 3.1, a 32-year-old version of Microsoft’s computer operating software,” the website said. “It’s so old that the CrowdStrike issue doesn’t affect it so Southwest is still operating as normal. It’s typically not a good idea to wait so long to update, but in this one instance Southwest has done itself a favor.” The govetech.com article is https://www.govtech.com/question-of-the-day/why-isnt-southwest-affected-by-the-crowdstrike-microsoft-outage https://www.govtech.com/question-of-the-day/why-isnt-southwe... which linked to https://www.digitaltrends.com/computing/southwest-cloudstrike-windows-3-1/ https://www.digitaltrends.com/computing/southwest-cloudstrik... which linked to an earlier Forbes article - https://www.forbes.com/sites/hershshefrin/2022/12/31/can-southwest-airlines-fix-its-systemic-weaknesses-in-the-new-year/ https://www.forbes.com/sites/hershshefrin/2022/12/31/can-sou... > The December 2022 scheduling fiasco was the result of skimping on information technology. I am old enough to remember when Microsoft introduced a new operating system called Windows 95, to replace its predecessor operating system Windows 3.1. The 95 in Windows 95 refers to the year of its introduction: 1995. By some accounts, major portions of Southwest’s scheduling system for pilots and flight attendants is built on the Windows 95 platform. That platform is now more than 25 years old.
- JumpCrisscross 2y ago
  Southwest does not run Windows 3.1: “That’s it. That’s where all these stories can trace their origin to. These few paragraphs do not say that Southwest is still using ancient Windows versions; it just states that the systems they developed internally, SkySolver and Crew Web Access, look ‘historic like they were designed on Windows 95’.” https://www.osnews.com/story/140301/no-southwest-airlines-is-not-still-using-windows-3-1/ https://www.osnews.com/story/140301/no-southwest-airlines-is...

smileysteve 2y ago

Re Delta It's not so much a severity as "hard"; but with the hub and spoke model that Delta uses, scheduling being down (at all on Friday), combined with FAA hour limits. It becomes exponentially difficult to reschedule flights. Put more plainly, on Friday, your scheduling software is down for 4 hours in the morning, so you "borrow" any replacements you need for employees that are late or sick. This ruins the availability for the next flights, at which time you hope the system is up again; but if it's not, you borrow from the evening flights. Combine this with each flight that was late/cancelled as you were hoping to fill now affects the hours available for the employees that were available. Finally, as you've cascaded this, you head into a weekend trying to catalog how many hours each crew member did or did not log, and you're not sure how to get them back in time.

inferiorhuman 2y ago
Except for Southwest the other legacy airlines (United, American) also use a hub and spoke model. So does jetBlue.
- crazytony 2y ago
  Funny you should mention WN. Delta's meltdown is the exact same scenario as Southwest. Crew scheduling is messed up, they don't have a way of tracking where employees are, if the employee is legal, etc and so the operation grinds to a halt
  - rconti 2y ago
    To clarify, Southwest's meltdown last year, which was all about the difficulties of crew scheduling and the knock-on effects of same.
sidewndr46 2y ago
wouldn't this imply either an upper bound on down time (airline simply folds as it never catches up) or an upper bound on the duration of the impact ?

reaperducer 2y ago

>> Why were other airlines able to get back to normal so much faster than Delta? I read somewhere that their crew tracking software was hit hard and took time to recover. Will look for source on that. I heard on the radio (maybe NPR, not sure) it wasn't about the computers, it was about Delta's response. According to the report, the other airlines delayed flights, while Delta cancelled them outright. That left Delta with more people and planes in the wrong places, making it harder to recover.

crazytony 2y ago

One other compounding problem is that Delta's headquarters and main traffic patterns are on the east coast. Crowdstrike affected all the airlines at roughly the same time. This gave them roughly one to two fewer hours to respond before they hit their morning peak flights. As someone else pointed out, they probably weren't ready by the time they needed their systems for the morning rush so they went to their business continuity strategy (manual). This has a throughput and recovery time penalty and obviously it compounds the longer they are in that mode. I think what we're finding with the Southwest meltdown and now the Delta meltdown is that the big airlines just don't have the manpower or scheduling slack to accommodate going into business continuity. I do think this should be investigated. Hopefully financial penalties incentivize action but time will tell.

throwaway2037 2y ago
> big airlines just don't have the manpower or scheduling slack to accommodate going into business continuity Do small airlines have it? And, how much higher are you willing to pay in ticket prices to have this ability?
katbyte 2y ago
They prioritized stock buy backs instead of investing in a robust it operation
- shiroiushi 2y ago
  As well they should! Which one profits the CEO more? Stock buy-backs or robust IT? Robust IT is only good for the company in the long term; however, with stock buy-backs or other skimping on IT, if disaster like this happens, the CEO just takes his golden parachute and leaves, but if no disaster happens, he gets a huge bonus to buy another private yacht.
WalterBright 2y ago
> Hopefully financial penalties incentivize action Delta already took a huge financial hit for this.