15 ms·

CrowdStrike's impact on aviation

414 points by jjwiseman 2y ago

lofaszvanitt 2y ago
[flagged]
firtoz 2y ago
Is there a similar global analysis?
- fullspectrumdev 2y ago
  I’d love to have some solid numbers of “global cancellations due to” - I heard a bunch of varying figures so far.
- jjwiseman 2y ago
  Maybe I'll do a Part 2: The World.
  - opdahl 2y ago
    As a non-American that would be very interesting.
- jijji 2y ago
  love to see the airlines using linux and what kind of problems, if any, they experienced that day
feyman_r 2y ago
>> Why were other airlines able to get back to normal so much faster than Delta? I read somewhere that their crew tracking software was hit hard and took time to recover. Will look for source on that. (Edited) source: https://news.delta.com/update-delta-customers-ceo-ed-bastian https://news.delta.com/update-delta-customers-ceo-ed-bastian “… and in particular one of our crew tracking-related tools was affected and unable to effectively process the unprecedented number of changes triggered by the system shutdown…”
- Onavo 2y ago
  Because they used Windows 3.1
  - ZeWaka 2y ago
    In the article it says Southwest used 3.1, not Delta (though, that's apparently incorrect according to other posters).
    - Someone1234 2y ago
      And Southwest had two crew-management outages in 2022[0], so let's not sing their praises for escaping the CrowdStrike disruption. Southwest has been widely critized for under-investment in technology, Delta on the other hand purchased one of the best security products on the market and that backfired. [0] https://en.wikipedia.org/wiki/2022_Southwest_Airlines_scheduling_crisis https://en.wikipedia.org/wiki/2022_Southwest_Airlines_schedu...
      - chgs 2y ago
        Delta put all their eggs in one basket and had no DR capability
        Someone1234 2y ago
        What basis do you have for saying that? It is likely their DR was running on a mirror of their production systems, and was similarly impacted by the Crowdstrike outage. So they fell back to Windows Servers similarly stuck in a boot-loop. Keep in mind there was no way to opt out or delay CS Channel updates.
        chgs 2y ago
        If your DR system is susceptible to the same faults as your main system it’s not a DR system. It would be like claiming raid1 is a backup.
ssivark 2y ago
> Apparently Southwest Airlines’ ingenious strategy of never upgrading from Windows 3.1 allowed it to remain unscathed. OMFG, does this mean we need to be prepared for a (juicy) “IT failure” that brings down Southwest at some point?
- ryanmcbride 2y ago
  You don't even have to wait it's been happening
- Someone1234 2y ago
  Southwest had two of these recently. It was widely reported: https://en.wikipedia.org/wiki/2022_Southwest_Airlines_scheduling_crisis https://en.wikipedia.org/wiki/2022_Southwest_Airlines_schedu...
- deleted 2y ago
  [deleted]
- toast0 2y ago
  Southwest experienced this kind of scheduling issue in 2021 [1], and again in 2022 [2]. Honestly, if they're running win 3.1 or win 95 as suggested, I think that puts them in a better place tech wise than keeping up with the Joneses on the upgrade treadmill --- although they should consider updating to windows 3.11, because they have a workgroup :P and the microsoft hearts network is pretty cool; but they have historically done poorly on scheduling after a significant disruption. An article from last year [3] says they updated their crew assignment software as well as increased staffing in colder airports and in general and got more deicing equipment. We won't really be able to tell if it works, until they experience another disruption. [1] https://www.cnbc.com/2021/10/12/southwest-airlines-reduces-cancellations-after-mass-cancellations-from-staffing-shortage.html https://www.cnbc.com/2021/10/12/southwest-airlines-reduces-c... [2] https://www.npr.org/2022/12/26/1145536902/southwest-flight-cancellations-winter-storm https://www.npr.org/2022/12/26/1145536902/southwest-flight-c... [3] https://www.npr.org/2023/11/09/1211064462/southwest-airlines-flight-cancellations-holiday-travel https://www.npr.org/2023/11/09/1211064462/southwest-airlines...
- tracerbulletx 2y ago
bedobi 2y ago
> Apparently Southwest Airlines’ ingenious strategy of never upgrading from Windows 3.1 allowed it to remain unscathed. this is pretty damning both ways on the one hand, it's insane, unfathomable and inconceivable that anyone can run anything critical on windows 3.1 (!!!) on the other hand, it's equally insane, unfathomable and inconceivable that those who do are actually better off - 30 years of "progress" is actually just bs? what are we as an industry "even doing here"???? is computing actually a solved problem and we're really just mostly reinventing the wheel and enshittifying perfectly already working systems?
- TonyTrapp 2y ago
  From my memory, this wildly circulating Windows 3.1 quote is inaccurate. The software they were running was compared to running something like Windows 3.1, but it wasn't actually running on Windows 3.1, as far as I understand. Edit: https://kotaku.com/southwest-airlines-windows-3-1-blue-screen-crowdstrike-1851603013 https://kotaku.com/southwest-airlines-windows-3-1-blue-scree...
- Swizec 2y ago
  > is computing actually a solved problem and we're really just mostly reinventing the wheel and enshittifying perfectly already working systems? 80% of the work is json bureaucracy The other 80% is adapting to new requirements And if you’re lucky maybe 0.1% of the time you get to build something new. Fear not, a lot of this stuff was perfectly solved with pen and paper long before us computer nerds came to play in the big boy sandbox
- deleted 2y ago
  [deleted]
- nightpool 2y ago
  Obviously Southwest is not using Windows 3.1, and you should probably be thinking hard about the trustworthiness of any outlet or article that repeats that claim: https://www.osnews.com/story/140301/no-southwest-airlines-is-not-still-using-windows-3-1/ https://www.osnews.com/story/140301/no-southwest-airlines-is...
- m3kw9 2y ago
jujube3 2y ago
Sounds like we saved a lot of tons of CO2.
- more_corn 2y ago
  Always look on the bright side!
- aflag 2y ago
  Hard to say, it could actually increased emissions. As when the timings of things don't align correctly it's common to cause an increase of resource usage. Eg. people travelling less optimal route, extra commutes back and to the airport. People having to physically travel to datacentres in order to fix things, just rebooting the machine without need will use more CPU.
  - mbreese 2y ago
    Or planes taking less efficient routes or flying at faster speeds to “make up time”.
  - deleted 2y ago
    [deleted]
- systemtest 2y ago
  In my country, companies are required by law to keep track of their CO2 emissions. In the case of CrowdStrike, they would have been able to deduct this event from their emissions for decades.
aftbit 2y ago
One interesting feature of this outage was that "PROD" was generally fine, on account of mostly running on Linux and/or ancient proprietary software, while "CORP" was generally wrecked, on account of mostly running Windows. In other words, the bank systems responsible for moving money mostly worked, while the systems responsible for allowing humans to interact with them (to issue approvals, change configuration, or other ops things) often did not.
- brazzy 2y ago
  In the original thread there were some reports of people having their Linux systems taken down by Crowdstrike as well. At separate times, of course, and I supposed the greater heterogeneity of Linux distros prevents events of this magnitude. But that would be little consolation when it takes down your systems.
  - foobarchu 2y ago
    Those should be considered coincidence until proven otherwise. Crowdstrike is intended to bring down systems when it believes there was an intrusion, after all.
    - brazzy 2y ago
      Not by crashing the kernel...
- 7thaccount 2y ago
  Same thing for a lot of industries actually. PROD runs on Linux and probably has some delay to prevent this. Corp gets hosed.
  - LeifCarrotson 2y ago
    Yep, here in manufacturing production/OT PLCs run on Wind River VxWorks from Rockwell, Siemens, and others. The HMI (human-machine interface, basically a touchscreen used to display status and enter setpoints and other data) and SCADA/ERP systems run on Windows. Sometimes, this is an industrial fanless PC with eg. Ignition (Java+Python) software, other times it's a Rockwell Panelview which actually still run Windows CE 6.0. This gets to be a problem when IT wants to get their hooks into OT networks. The PLC is meant to be left alone, and will happily send its Ethernet packet to that servo drive or digital IO card every 10ms for literal decades. There is no reason to update its firmware ever, just don't expose it to the Internet. But corporate wants everything on the Internet. The PLC will reliably run its sequence when you close the contacts on the physical "Cycle Start" pushbutton. But if corporate is down, you can't know what part number you're supposed to make or how many of them, or get a serial number from and report test results to the traceability database.
ks1723 2y ago
I found it quite interesting, that crowdstrike actually exclude a bunch of services explicitly. They also basically say, don’t use, if it needs to be reliable. I don’t know if this is standard for software, but for me this was quite surprising. From crowdstrike terms and services [1]: […] THERE IS NO WARRANTY THAT THE OFFERINGS OR CROWDSTRIKE TOOLS WILL BE ERROR FREE, OR THAT THEY WILL OPERATE WITHOUT INTERRUPTION OR WILL FULFILL ANY OF CUSTOMER’S PARTICULAR PURPOSES OR NEEDS. THE OFFERINGS AND CROWDSTRIKE TOOLS ARE NOT FAULT-TOLERANT AND ARE NOT DESIGNED OR INTENDED FOR USE IN ANY HAZARDOUS ENVIRONMENT REQUIRING FAIL-SAFE PERFORMANCE OR OPERATION. NEITHER THE OFFERINGS NOR CROWDSTRIKE TOOLS ARE FOR USE IN THE OPERATION OF AIRCRAFT NAVIGATION, NUCLEAR FACILITIES, COMMUNICATION SYSTEMS, WEAPONS SYSTEMS, DIRECT OR INDIRECT LIFE-SUPPORT SYSTEMS, AIR TRAFFIC CONTROL, OR ANY APPLICATION OR INSTALLATION WHERE FAILURE COULD RESULT IN DEATH, SEVERE PHYSICAL INJURY, OR PROPERTY DAMAGE. Customer agrees that it is Customer’s responsibility to ensure safe use of an Offering and the CrowdStrike Tools in such applications and installations. CROWDSTRIKE DOES NOT WARRANT ANY THIRD PARTY PRODUCTS OR SERVICES. [1] section 8.6 of https://www.crowdstrike.com/terms-conditions/ https://www.crowdstrike.com/terms-conditions/
- objclxt 2y ago
  > I don’t know if this is standard for software This is pretty standard. There is almost identical language in the Windows and macOS EULAs, for example.
  - ale42 2y ago
    Same for datasheets of most electronic components. The manufacturers don't want the responsibility to avoid possible multi-million lawsuits.
  - SoftTalker 2y ago
    So how does it get installed on all the endpoints in 911 dispatch centers?
    - wrs 2y ago
      Because no endpoint protection software exists that doesn’t have the same disclaimer clause. So you install this one and accept the lack of vendor liability. (If such a thing did exist, it would cost a lot more!)
mjevans 2y ago
Outsourcing a core business competency and surely also cutting the contracts to the bone as well to pocket the savings embrittled Delta and I seriously hope the compensation to customers costs more than any savings or profits they made in the interim. It MUST be painful enough that they do not repeat this mistake again. The article quotes https://www.reddit.com/r/delta/comments/1edtfbh/why_did_delta_take_days_to_restore_normal_service/lf9konn/ https://www.reddit.com/r/delta/comments/1edtfbh/why_did_delt... (with improper attribution) topgun966Platinum wrote on Reddit """ These "experts" are completely wrong. The core issue was Delta did NOT have a proper DR plan ready and did NOT have a proper IT business continuity plan ready. UA, AA, and F9 recovered so fast because they had plans on stand-by and engaged them immediately. After the SWA IT problem, UA and AA put in robust DR plans staged everywhere from the server farms, to cloud solutions, to end-user stations at airports. They had plans on how to recover systems. DL outsources a lot of their IT. UA and AA engaged those plans quickly. They did not hold back paying OT for staff. UA and AA have just as much reliance on Windows as Delta. AA was recovered by end of data Friday and resumed normal operations Saturday. UA was about 12 hours behind them having it resolved by Saturday morning resuming normal schedules Saturday afternoon. The ONUS is 100% on DL C+ level in their IT decisions. The problem is that the lower level IT staff is going to get the brunt of the blame and the consequences. """
- deleted 2y ago
  [deleted]
- tiahura 2y ago
  That’s why I think the suit against crowdstrike and Ms is mostly a dud. First you have to get around the waiver (much harder for business than a consumer) and then you have to deal with comparative fault - ie delta’s disaster recovery system sucked.
rdtsc 2y ago
From the included link: https://www.techradar.com/pro/security/southwest-airlines-avoided-crowdstrike-microsoft-outage-because-its-still-running-windows-31-fourth-largest-us-airline-remained-free-of-bsod-errors-because-its-os-hasnt-been-updated-in-decades https://www.techradar.com/pro/security/southwest-airlines-av... > To give you an idea of just how outdated this operating system is, Windows 3.1 was originally launched in 1992, and Microsoft ended support for it on December 31, 2001, except for the embedded version, which was officially retired in 2008. I keep hearing the Windows 3.1 story repeated. I mean here it comes from TechRadar and even has the "Pro" in the name, they can't possibly make stuff up, right? But still don't quite believe it. Can anyone working at Southwest confirm that their main scheduling system is running on Windows 3.1?
- JumpCrisscross 2y ago
  > keep hearing the Windows 3.1 story repeated It’s wrong [1] and serves as a litmus test for whether an outlet independently verifies its claims. (“The systems [Southwest] developed internally, SkySolver and Crew Web Access, look ‘historic like they were designed on Windows 95’.” That got mangled into they run 3.1.) [1] https://www.osnews.com/story/140301/no-southwest-airlines-is-not-still-using-windows-3-1/ https://www.osnews.com/story/140301/no-southwest-airlines-is...
  - jjwiseman 2y ago
    Thanks, I updated the post.
    - kragen 2y ago
      i miss the lemonodor blog
  - xp84 2y ago
    Wow, that’s even more frustrating considering it’s conflating an unfashionable UI (which I’d argue is a good thing, since all modern UI trends are towards slick, minimalism-worshiping messes which hide everything from users) and old, provably-flawed technological foundations (like a 16-bit system without things like filesystem access control or memory protection). I knew this story was false immediately though because no company ever even in 1993 had production server systems which ran a desktop OS like Win 3.1. It just wasn’t up to the task. They would have used NT if anything.
Zigurd 2y ago
I would like to know if a solid, up to date, well-rehearsed disaster recovery plan saved anyone's butt, or if we're all just raw dogging our machines whether IT is paying for backup and recovery or not?
- Zigurd 2y ago
  I see just moments after I posted, someone posted this: https://news.ycombinator.com/item?id=41103486 https://news.ycombinator.com/item?id=41103486 So, yeah, lack of DR is why Delta was so screwed.
- ta1243 2y ago
  Our systems worked fine, we expect things to fail - including software like sentinal one, crowdstrike, etc, and have DR systems which can keep us limping along. We have DR systems which will work should other things happen - say the Thames barrier fails (i.e. no docklands) Unfortunately some of our outsourced suppliers didn't have such attitudes.
- paulddraper 2y ago
  I've never seen it. Obviously some selection bias there, but I'd love to hear some success stories.
- berniedurfee 2y ago
  They certainly have DR infrastructure primed and ready to go… with Crowdstrike pre-installed on every DR server.
- red-iron-pine 2y ago
  Sure has. HSRP and VRRP plus other SD-WAN features definitely made a difference when one of our sites had the fiber pulled by accident. data center tech screwed up bigly and took us plus at least one other of their customers down. definitely saw a blip and stuff had issues for 10 minutes, e.g. pages timed out or had to restart a process, but generally sites failed over and were able to keep limping on while we did triage. got something like a $19 ($21?) service credit and an apology from the data center. our CEO shouted a lot and threatened lawsuits but it never went anywhere. Director of IT Infra quietly thanked all of us for having failover that mostly worked.
xyst 2y ago
> Apparently Southwest Airlines’ ingenious strategy of never upgrading from Windows 3.1 allowed it to remain unscathed. The “ingenious” strategy saved them from a weeks worth of downtime this year. But that same “ingenious” strategy was the primary reason for their meltdown in 2022 [1] https://www.npr.org/2022/12/30/1146377342/5-things-to-know-about-southwests-disastrous-meltdown https://www.npr.org/2022/12/30/1146377342/5-things-to-know-a... [2] https://www.nytimes.com/2022/12/28/travel/southwest-airlines-flight-cancellations.html https://www.nytimes.com/2022/12/28/travel/southwest-airlines...
- skrebbel 2y ago
  It’s also not true
linotype 2y ago
[flagged]
- deleted 2y ago
  [deleted]
skrebbel 2y ago
I love that “CrowdStrike” is now a synonym for “global outage”. Not some cute hihi name like “heartbleed”, just the name of the company that did the screwup. Seems fair.
- jraph 2y ago
  Not sure it's fair, but I am certainly waiting for it to become a verb or a noun. crowdstrike. n. 1. A set of major disruptions caused by an update that was not tested enough, pushed to many devices across the globe. 2. The name of such an update. 3. (by extension) a joke so bad it causes major disruptions. For instance: - Congrats for your crowdstrike! Now my weekend is ruined as I'll be the one who'll be asked to fix this mess. crowdstrike. v. (simple past crowdstruck or crowdstriked¹, past participle crowdstricken, or crowdstruck, or (obsolete, regionalism) crowdstroke²) 1. Action of pushing an update to many devices that causes a global outage or major disruptions in various sectors. For instance: - We've been crowdstruck. Again. crowdstrike. adj. 1. Qualifies an update that, when pushed to many devices across the world, causes major disruptions across the globe. 2. Qualifies such a (set of) event(s). For instance: - We are sorry for the crowdstrike event we caused. We gently remind our kind customers and their end users that per our ToS, we will issue no refund, and that no liability can be held against us. Customers who don't try to contact us in the following month will get a discount for their next contract renewal. You will hear us speak before the Congress, who nicely invited us for some comedy in the hope it will appease you all. Make sure you like the related videos on the various online platforms. We wish you a nice end of the week and nice, relaxing summer holidays. ¹ people have differing but strong opinions on which simple past form is correct, mainly due to regional differences. Some avoid saying crowdstrike and say crowdhit instead. ² some people have tried to push crowdstricken, which first caught on in some areas or particular contexts. The idea that this form likens the qualified subject to the bearer of some sickness has eventually seduced a critical mass of people after some initial push back. Please also see the usage notes for strike for other, rarer, alternative forms [*]. [*] https://en.wiktionary.org/wiki/strike#Usage%20notes https://en.wiktionary.org/wiki/strike#Usage%20notes (Thanks to the contributors in this thread)
  - 2y ago
knappe 2y ago
For everyone flabbergasted by Southwest running ~Windows 3.1~ old software, I have bad news about the telecom industry. I worked at Ericsson at an R&D branch and one of the projects in the works was to move one of the main pieces of routing equipment that handled millions of telephony operatorations a day away from an ancient version of Windows. A lot of code lives on much longer than you think. The general attitude we took was that most of the code we were writing would be running for at least 30 years. And that was the attitude at an R&D branch, arguably a side of that industry where we were working on the new tech. Edit: Win 3.1 or something else, the point still stands. There is a lot of old software running out there that will continue to run our core services. Legacy software doesn't just mean v1 versus v2, it can mean v1 versus v41.
- llm_nerd 2y ago
  >For everyone flabbergasted by Southwest running Windows 3.1 Southwest isn't running Windows 3.1, though. That's some rather lame, but predictable, truth-through-repeated-assertion thing on social media. Not everyone uses CrowdStrike, and in this case SW was the lucky one that didn't.
- wrboyce 2y ago
  I find this more surprising, even if the Southwest & Win3.1 claims were true, I would expect most Ericsson systems to be Erlang based and thus happily chugging along on a (perhaps ancient) Linux box.
  - knappe 2y ago
    I did too. But I knew of only 1 project that was using Erlang. I have always wanted to use it. Instead I saw a lot of MML and (happily) TCL. https://en.m.wikipedia.org/wiki/MML_(programming_language) https://en.m.wikipedia.org/wiki/MML_(programming_language)
- ketchupdebugger 2y ago
  Good thing a lot of our banking still runs on mainframes, will never be taken out by crowdstrike
deleted 2y ago
[deleted]
otterley 2y ago
It blows my mind how many people actually believed the claim -- clearly in the obvious-joke category -- that SWA is running their mission critical flight systems on Windows 3.1. (Yes, Southwest runs a lot of old tech in their stack, but that claim is patently hyperbolic.) People need to stop believing everything they read on the Internet and have a little bit of skepticism.
- deleted 2y ago
  [deleted]
beambot 2y ago
Lawsuits inbound. Delta appears to be gearing up for one already: https://finance.yahoo.com/news/delta-air-lines-seek-compensation-211908080.html https://finance.yahoo.com/news/delta-air-lines-seek-compensa...
- hypeatei 2y ago
  > has hired a law firm and will seek compensation from Microsoft and CrowdStrike Going after Microsoft seems like a misguided move here. What does Microsoft have to do with a third party driver installed by your own IT department?
  - travoc 2y ago
    It was too tempting to include damages from accidentally enabling New New Outlook.
  - bruce511 2y ago
    I suspect the lawsuit is created by lawyers, not techies. Equally reporting on this whole issue seems to be by journalists, not techies. It's been framed (a lot) as a Windows issue not a Crowd Strike issue. (8.5 million machines were affected, out of 1.4+ billion windows machines [1]) I have one affected customer (10k machines) who assumed I'd suffered like he did, and was surprised when I said we weren't affected. The reporting was consistently that it was a Windows issue, caused by an MS update. Even this article leans into this narrative... "as the New York Times put it, “It is more apt to ask what was not affected.” The answer is Linux, Macs, and phones." Let me add "not to mention 99.4% of windows". So the journalists don't know what happened, or who was affected, and felt "some computers have a problem" was a weak headline. The lawyers get that narrative and run with it. And yes, it's easy to squint and claim the "OS should cope with this", but there's realistically limits on what an OS can do once you install a kernel-level driver on the machine. Should we go after Intel for making the chips? [1] https://www.pcworld.com/article/608447/microsoft-delighted-by-windows-11-usage-but-what-is-it.html https://www.pcworld.com/article/608447/microsoft-delighted-b...
    - 2y ago
azinman2 2y ago
What this also tells me is there are a lot of computers connected to the internet that probably shouldn’t be.
- 0cf8612b2e1e 2y ago
  I suspect it is incredibly challenging to keep a non trivial number of computers available, but somewhat airgapped. Too many ways to unintentionally bridge the networks without extreme diligence. Which is slightly incompatible with something like airlines which employ enormous numbers of people.
  - azinman2 2y ago
    These are not small time operations (most of them). They are multi billion dollar companies with complex technical needs. This is doable and the bread and butter of good networking engineers. Having not done this just cost them billions. Now imagine the US at war or another nation state that just wants to cause havoc.
    - unethical_ban 2y ago
      Crowdstrike is a trusted application on every computer in these people's farms. It is not uncommon to have specific rules for these packages to be downloaded directly from the Internet. You're suggesting either that Crowdstrike itself will get used as a vessel for an attack, or that banks and airlines have firewall rules open for enemies.
    - nradov 2y ago
      That's ridiculous. The cost of building air gapped operations systems, and including the resulting loss of productivity and efficiency, would be worse than just accepting an occasional outage. People who lack technical competence and operational experience always tend to over react to software failures. The major airlines may not be "small time operations" but none of them are even in the top 100 US corporations by market cap. They simply don't have the level of IT resources or competence that we see at major tech companies.
nostromo 2y ago
It's insane to me that CrowdStrike's stock is still up 66% year-over-year. With all of the angry customers, lots of incoming lawsuits, and the fact that their "protection" is provably more costly than no protection at all now - I can't imagine why investors aren't dumping it like mad.
- parmenidean 2y ago
  Good news then! You can short it and make a ton of money if you're confident this share price increase is a mistake.
  - concerndc1tizen 2y ago
    [dead]
- johndhi 2y ago
  My guesses: -no one really cancels their security vendors since security budgets don't shrink -they have a big moat so their customers won't be able to leave them
  - kd913 2y ago
    I am confused why they are around to begin with. Companies already trust Microsoft, they buy Windows, Office, Azure. Why would they bother with a 3rd party here when the low effort low risk solution is to pick the tool made by the OS vendor. I.e. windows defender It should be a nobody gets fired for picking IBM situation. How did this random place get so much credibility that people trust them over the manufacturer?
    - htrp 2y ago
      crowdstrike has oracle enterprise sales model. have you ever been to one of their events?
    - kccqzy 2y ago
      Because they provide far more protection than Windows defender. You can write your own custom never-before-seen malware, and CrowdStrike will detect it purely based on behavioral signals. Windows Defender is still largely an antivirus solution.
bandyaboot 2y ago
Anyone know why Minneapolis-St Paul began experiencing cancellations much earlier than other US airports?
jijji 2y ago
basically any airline using linux is not on that list
- bruce511 2y ago
  It's more accurate to say "any airline not using Crowd Strike is not on that list." Blaming Windows for this outage is like blaming Linux for Apache bugs. The two systems are distinct. It just so happens that Crowd Strike was very successful at selling to large corporates. That includes some airlines. 99.4% of Windows machines were unaffected. Including those of airlines using Windows, but not Crowd Strike.
  - jijji 2y ago
    ahhh yes you are correct
- misja111 2y ago
  Can you name one?
- namdnay 2y ago
  every airline in the world "uses linux", the core reservation and distribution systems were migrated from TPF to Linux over the past 20 years
bustling-noose 2y ago
> The outage highlighted a different kind of digital divide. On one side, gmail, Facebook, and Twitter kept running, letting us post photos of blue screens located on the other side: the Windows machines responsible for actually doing things in the world like making appointments, opening accounts, and dispatching police. At this point using windows for these tasks seems like using legacy software because training people to use an iPad or a web browser seems too complicated or because no one wants to move their age old systems to a more modern web based system because of costs. Native apps work great, but I think the world is moving to the cloud and that means web based everything should be the norm. Yes AWS AZURE outages can still happen but those can be fixed by spinning up a VM in different clouds. This is also why software jobs aren’t going anywhere thanks for a while. Many systems need to be changed to more modern and robust clouds. It might take decades for this transformation across the globe.
- amluto 2y ago
  Your “modern and robust cloud” is my “why on Earth doesn’t this thing work offline”. The world is absolutely full of things that have worked for decades to centuries without the Internet, are eventually more or less consistent (remember carbon paper credit card machines?), and did an amazing job of keeping the world running despite, wars, network partitions (the “network” would basically always be partitioned), mistakes, entire branches offline, etc. Sure, a lot of things are easier when centralized, and “the cloud” is incredibly powerful. But it’s not necessarily more robust. Also, depending on any sort of cloud means you’re also depending on the network, and networks are far from infallable. There’s a reason that a lot of stored-value transit systems still track balances on the card and will let people in even if a fare gate cannot connect to a cloud service. And CrowdStrike took out plenty of cloud instances, and recovering them can be worse than recovering physical hardware, as the “robust cloud” has an absolutely terrible ability to do anything outside the happy path of booting an instance normally.
  - dailykoder 2y ago
    Okay this sounds all very reasonable, but how do you know when your washing machine is finished, when it's not connected to the cloud and you won't get notified in your app? It sure is not an easy thing and the cloud helps very much here
pimlottc 2y ago
One thing I don't understand from these graphs - why was there a relative uptick in takeoffs starting a short time /before/ the CrowdStrike update was pushed? It's in the overall graph, as well as the graphs for United, American, and especially Delta. I can't think of any reason for this, maybe it's just random noise, or maybe there was something unusual about the previous week at the same time?
- mike_hearn 2y ago
  It was widely reported to be the busiest travel day for quite a long time, which compounded problems.
- account42 2y ago
  Yeah, shouldn't have been too hard to add a couple more weeks so you at least get an idea about variance.
- jjwiseman 2y ago
  One reason I put charts for both absolute numbers of flights and percentage change is to help understand the larger context. Those relative upticks happened just before CrowdStrike hit, so around midnight Eastern time, when traffic was already very low. So a 25% increase might be 12 extra flights taking off in the U.S. in an hour. There's plenty of noise, for sure, and looking at absolute numbers and percent change together can help give you a sense of what was going on. Looking at two days worth of data is probably enough to give you the main themes of the CrowdStrike impact, but not enough to explain every variation.
deleted 2y ago
[deleted]
camillomiller 2y ago
Berlin Brandenburg got hit hard. As a disgruntled BER user, I am NOT surprised they had one of the worse repercussion.
- nicbou 2y ago
  German IT is often hit hard by such things. Unless of course they're still running on paper. At least they immediately mentioned it on their website, as a banner right at the top. The immigration office's appointment system has been down for over a month, and it took them 3 weeks to just acknowledge it.
  - camillomiller 2y ago
    Thanks for your website btw. My partner’s currently renewing her work visa (she’s from Australia) and it’s insane how bad the situation is. To the point where it feels like it should just be illegal for Berlin to operate services like this. Appalling.
- ta1243 2y ago
  I'm still shocked that Brandenburg is actually open!
roshankhan28 2y ago
i really dont understand, how can my social media have better backup and infrastructure as compared to an OS which is being used by worldwide?
- pjc50 2y ago
  Windows has always been terrible for reliability. Adding a "security" system which is invasive and always-updated makes the reliability worse.
- hiddencost 2y ago
  Because one is in a data center that can be controlled, and the other is deployed to user owned hardware that cannot.
- Ekaros 2y ago
  Because they don't do mass rollouts on the servers. Then again those companies could fail if they had single point of failure with automatic mass deployments... This could happen for anything that supports this type of automatic mass deployment. Just in this case that thing was popular enough and happened on one of the most popular platforms.
- goodcanadian 2y ago
  Because IT is your social media's business. They know IT inside and out. They understand what can go wrong and how to mitigate it. The business for airlines (for example) is to fly planes. They are pretty damn good at it. IT, however, is just a tool to them that they buy elsewhere. They don't understand it in the same way as social media. They rely on outside contractors to do it right: outside contractors who get the job based on being the cheapest or convincing the buyers their service is "industry best practice."
- rr808 2y ago
  Meta is one of the most valuable companies in the world with the most resources to buy the best of everything. At 1,280 Billion dollars of market cap it is 30x bigger than American, Delta and United put together. It made $39 Billion last year compared to $7.8 Billion for all US airlines together. Of course it has better systems.
miohtama 2y ago
How to avoid getting rekt > Southwest wasn’t affected because they don’t use CrowdStrike
jandrusk 2y ago
Will be most interesting how this lawsuit by Delta plays out against Microsoft & Crowdstrike: https://www.marketwatch.com/story/delta-hires-law-firm-seeking-damages-from-crowdstrike-microsoft-after-massive-tech-outage-report-a882328a https://www.marketwatch.com/story/delta-hires-law-firm-seeki...
fagrobot 2y ago
[dead]
adrianwest66 2y ago
[dead]

414 points by jjwiseman 2y ago

lofaszvanitt 2y ago

[flagged]

firtoz 2y ago

Is there a similar global analysis?

fullspectrumdev 2y ago
I’d love to have some solid numbers of “global cancellations due to” - I heard a bunch of varying figures so far.
jjwiseman 2y ago
Maybe I'll do a Part 2: The World.
- opdahl 2y ago
  As a non-American that would be very interesting.
jijji 2y ago
love to see the airlines using linux and what kind of problems, if any, they experienced that day

feyman_r 2y ago

>> Why were other airlines able to get back to normal so much faster than Delta? I read somewhere that their crew tracking software was hit hard and took time to recover. Will look for source on that. (Edited) source: https://news.delta.com/update-delta-customers-ceo-ed-bastian https://news.delta.com/update-delta-customers-ceo-ed-bastian “… and in particular one of our crew tracking-related tools was affected and unable to effectively process the unprecedented number of changes triggered by the system shutdown…”

Onavo 2y ago
Because they used Windows 3.1
- ZeWaka 2y ago
  In the article it says Southwest used 3.1, not Delta (though, that's apparently incorrect according to other posters).
  - Someone1234 2y ago
    And Southwest had two crew-management outages in 2022[0], so let's not sing their praises for escaping the CrowdStrike disruption. Southwest has been widely critized for under-investment in technology, Delta on the other hand purchased one of the best security products on the market and that backfired. [0] https://en.wikipedia.org/wiki/2022_Southwest_Airlines_scheduling_crisis https://en.wikipedia.org/wiki/2022_Southwest_Airlines_schedu...
    - chgs 2y ago
      Delta put all their eggs in one basket and had no DR capability
      - Someone1234 2y ago
        What basis do you have for saying that? It is likely their DR was running on a mirror of their production systems, and was similarly impacted by the Crowdstrike outage. So they fell back to Windows Servers similarly stuck in a boot-loop. Keep in mind there was no way to opt out or delay CS Channel updates.
        chgs 2y ago
        If your DR system is susceptible to the same faults as your main system it’s not a DR system. It would be like claiming raid1 is a backup.

ssivark 2y ago

> Apparently Southwest Airlines’ ingenious strategy of never upgrading from Windows 3.1 allowed it to remain unscathed. OMFG, does this mean we need to be prepared for a (juicy) “IT failure” that brings down Southwest at some point?

ryanmcbride 2y ago
You don't even have to wait it's been happening
Someone1234 2y ago
Southwest had two of these recently. It was widely reported: https://en.wikipedia.org/wiki/2022_Southwest_Airlines_scheduling_crisis https://en.wikipedia.org/wiki/2022_Southwest_Airlines_schedu...
deleted 2y ago
[deleted]
toast0 2y ago
Southwest experienced this kind of scheduling issue in 2021 [1], and again in 2022 [2]. Honestly, if they're running win 3.1 or win 95 as suggested, I think that puts them in a better place tech wise than keeping up with the Joneses on the upgrade treadmill --- although they should consider updating to windows 3.11, because they have a workgroup :P and the microsoft hearts network is pretty cool; but they have historically done poorly on scheduling after a significant disruption. An article from last year [3] says they updated their crew assignment software as well as increased staffing in colder airports and in general and got more deicing equipment. We won't really be able to tell if it works, until they experience another disruption. [1] https://www.cnbc.com/2021/10/12/southwest-airlines-reduces-cancellations-after-mass-cancellations-from-staffing-shortage.html https://www.cnbc.com/2021/10/12/southwest-airlines-reduces-c... [2] https://www.npr.org/2022/12/26/1145536902/southwest-flight-cancellations-winter-storm https://www.npr.org/2022/12/26/1145536902/southwest-flight-c... [3] https://www.npr.org/2023/11/09/1211064462/southwest-airlines-flight-cancellations-holiday-travel https://www.npr.org/2023/11/09/1211064462/southwest-airlines...
tracerbulletx 2y ago

bedobi 2y ago

> Apparently Southwest Airlines’ ingenious strategy of never upgrading from Windows 3.1 allowed it to remain unscathed. this is pretty damning both ways on the one hand, it's insane, unfathomable and inconceivable that anyone can run anything critical on windows 3.1 (!!!) on the other hand, it's equally insane, unfathomable and inconceivable that those who do are actually better off - 30 years of "progress" is actually just bs? what are we as an industry "even doing here"???? is computing actually a solved problem and we're really just mostly reinventing the wheel and enshittifying perfectly already working systems?

TonyTrapp 2y ago
From my memory, this wildly circulating Windows 3.1 quote is inaccurate. The software they were running was compared to running something like Windows 3.1, but it wasn't actually running on Windows 3.1, as far as I understand. Edit: https://kotaku.com/southwest-airlines-windows-3-1-blue-screen-crowdstrike-1851603013 https://kotaku.com/southwest-airlines-windows-3-1-blue-scree...
Swizec 2y ago
> is computing actually a solved problem and we're really just mostly reinventing the wheel and enshittifying perfectly already working systems? 80% of the work is json bureaucracy The other 80% is adapting to new requirements And if you’re lucky maybe 0.1% of the time you get to build something new. Fear not, a lot of this stuff was perfectly solved with pen and paper long before us computer nerds came to play in the big boy sandbox
deleted 2y ago
[deleted]
nightpool 2y ago
Obviously Southwest is not using Windows 3.1, and you should probably be thinking hard about the trustworthiness of any outlet or article that repeats that claim: https://www.osnews.com/story/140301/no-southwest-airlines-is-not-still-using-windows-3-1/ https://www.osnews.com/story/140301/no-southwest-airlines-is...
m3kw9 2y ago

jujube3 2y ago

Sounds like we saved a lot of tons of CO2.

more_corn 2y ago
Always look on the bright side!
aflag 2y ago
Hard to say, it could actually increased emissions. As when the timings of things don't align correctly it's common to cause an increase of resource usage. Eg. people travelling less optimal route, extra commutes back and to the airport. People having to physically travel to datacentres in order to fix things, just rebooting the machine without need will use more CPU.
- mbreese 2y ago
  Or planes taking less efficient routes or flying at faster speeds to “make up time”.
- deleted 2y ago
  [deleted]
systemtest 2y ago
In my country, companies are required by law to keep track of their CO2 emissions. In the case of CrowdStrike, they would have been able to deduct this event from their emissions for decades.

aftbit 2y ago

One interesting feature of this outage was that "PROD" was generally fine, on account of mostly running on Linux and/or ancient proprietary software, while "CORP" was generally wrecked, on account of mostly running Windows. In other words, the bank systems responsible for moving money mostly worked, while the systems responsible for allowing humans to interact with them (to issue approvals, change configuration, or other ops things) often did not.

brazzy 2y ago
In the original thread there were some reports of people having their Linux systems taken down by Crowdstrike as well. At separate times, of course, and I supposed the greater heterogeneity of Linux distros prevents events of this magnitude. But that would be little consolation when it takes down your systems.
- foobarchu 2y ago
  Those should be considered coincidence until proven otherwise. Crowdstrike is intended to bring down systems when it believes there was an intrusion, after all.
  - brazzy 2y ago
    Not by crashing the kernel...
7thaccount 2y ago
Same thing for a lot of industries actually. PROD runs on Linux and probably has some delay to prevent this. Corp gets hosed.
- LeifCarrotson 2y ago
  Yep, here in manufacturing production/OT PLCs run on Wind River VxWorks from Rockwell, Siemens, and others. The HMI (human-machine interface, basically a touchscreen used to display status and enter setpoints and other data) and SCADA/ERP systems run on Windows. Sometimes, this is an industrial fanless PC with eg. Ignition (Java+Python) software, other times it's a Rockwell Panelview which actually still run Windows CE 6.0. This gets to be a problem when IT wants to get their hooks into OT networks. The PLC is meant to be left alone, and will happily send its Ethernet packet to that servo drive or digital IO card every 10ms for literal decades. There is no reason to update its firmware ever, just don't expose it to the Internet. But corporate wants everything on the Internet. The PLC will reliably run its sequence when you close the contacts on the physical "Cycle Start" pushbutton. But if corporate is down, you can't know what part number you're supposed to make or how many of them, or get a serial number from and report test results to the traceability database.

ks1723 2y ago

I found it quite interesting, that crowdstrike actually exclude a bunch of services explicitly. They also basically say, don’t use, if it needs to be reliable. I don’t know if this is standard for software, but for me this was quite surprising. From crowdstrike terms and services [1]: […] THERE IS NO WARRANTY THAT THE OFFERINGS OR CROWDSTRIKE TOOLS WILL BE ERROR FREE, OR THAT THEY WILL OPERATE WITHOUT INTERRUPTION OR WILL FULFILL ANY OF CUSTOMER’S PARTICULAR PURPOSES OR NEEDS. THE OFFERINGS AND CROWDSTRIKE TOOLS ARE NOT FAULT-TOLERANT AND ARE NOT DESIGNED OR INTENDED FOR USE IN ANY HAZARDOUS ENVIRONMENT REQUIRING FAIL-SAFE PERFORMANCE OR OPERATION. NEITHER THE OFFERINGS NOR CROWDSTRIKE TOOLS ARE FOR USE IN THE OPERATION OF AIRCRAFT NAVIGATION, NUCLEAR FACILITIES, COMMUNICATION SYSTEMS, WEAPONS SYSTEMS, DIRECT OR INDIRECT LIFE-SUPPORT SYSTEMS, AIR TRAFFIC CONTROL, OR ANY APPLICATION OR INSTALLATION WHERE FAILURE COULD RESULT IN DEATH, SEVERE PHYSICAL INJURY, OR PROPERTY DAMAGE. Customer agrees that it is Customer’s responsibility to ensure safe use of an Offering and the CrowdStrike Tools in such applications and installations. CROWDSTRIKE DOES NOT WARRANT ANY THIRD PARTY PRODUCTS OR SERVICES. [1] section 8.6 of https://www.crowdstrike.com/terms-conditions/ https://www.crowdstrike.com/terms-conditions/

objclxt 2y ago
> I don’t know if this is standard for software This is pretty standard. There is almost identical language in the Windows and macOS EULAs, for example.
- ale42 2y ago
  Same for datasheets of most electronic components. The manufacturers don't want the responsibility to avoid possible multi-million lawsuits.
- SoftTalker 2y ago
  So how does it get installed on all the endpoints in 911 dispatch centers?
  - wrs 2y ago
    Because no endpoint protection software exists that doesn’t have the same disclaimer clause. So you install this one and accept the lack of vendor liability. (If such a thing did exist, it would cost a lot more!)

mjevans 2y ago

Outsourcing a core business competency and surely also cutting the contracts to the bone as well to pocket the savings embrittled Delta and I seriously hope the compensation to customers costs more than any savings or profits they made in the interim. It MUST be painful enough that they do not repeat this mistake again. The article quotes https://www.reddit.com/r/delta/comments/1edtfbh/why_did_delta_take_days_to_restore_normal_service/lf9konn/ https://www.reddit.com/r/delta/comments/1edtfbh/why_did_delt... (with improper attribution) topgun966Platinum wrote on Reddit """ These "experts" are completely wrong. The core issue was Delta did NOT have a proper DR plan ready and did NOT have a proper IT business continuity plan ready. UA, AA, and F9 recovered so fast because they had plans on stand-by and engaged them immediately. After the SWA IT problem, UA and AA put in robust DR plans staged everywhere from the server farms, to cloud solutions, to end-user stations at airports. They had plans on how to recover systems. DL outsources a lot of their IT. UA and AA engaged those plans quickly. They did not hold back paying OT for staff. UA and AA have just as much reliance on Windows as Delta. AA was recovered by end of data Friday and resumed normal operations Saturday. UA was about 12 hours behind them having it resolved by Saturday morning resuming normal schedules Saturday afternoon. The ONUS is 100% on DL C+ level in their IT decisions. The problem is that the lower level IT staff is going to get the brunt of the blame and the consequences. """

deleted 2y ago
[deleted]
tiahura 2y ago
That’s why I think the suit against crowdstrike and Ms is mostly a dud. First you have to get around the waiver (much harder for business than a consumer) and then you have to deal with comparative fault - ie delta’s disaster recovery system sucked.

rdtsc 2y ago

From the included link: https://www.techradar.com/pro/security/southwest-airlines-avoided-crowdstrike-microsoft-outage-because-its-still-running-windows-31-fourth-largest-us-airline-remained-free-of-bsod-errors-because-its-os-hasnt-been-updated-in-decades https://www.techradar.com/pro/security/southwest-airlines-av... > To give you an idea of just how outdated this operating system is, Windows 3.1 was originally launched in 1992, and Microsoft ended support for it on December 31, 2001, except for the embedded version, which was officially retired in 2008. I keep hearing the Windows 3.1 story repeated. I mean here it comes from TechRadar and even has the "Pro" in the name, they can't possibly make stuff up, right? But still don't quite believe it. Can anyone working at Southwest confirm that their main scheduling system is running on Windows 3.1?

JumpCrisscross 2y ago
> keep hearing the Windows 3.1 story repeated It’s wrong [1] and serves as a litmus test for whether an outlet independently verifies its claims. (“The systems [Southwest] developed internally, SkySolver and Crew Web Access, look ‘historic like they were designed on Windows 95’.” That got mangled into they run 3.1.) [1] https://www.osnews.com/story/140301/no-southwest-airlines-is-not-still-using-windows-3-1/ https://www.osnews.com/story/140301/no-southwest-airlines-is...
- jjwiseman 2y ago
  Thanks, I updated the post.
  - kragen 2y ago
    i miss the lemonodor blog
- xp84 2y ago
  Wow, that’s even more frustrating considering it’s conflating an unfashionable UI (which I’d argue is a good thing, since all modern UI trends are towards slick, minimalism-worshiping messes which hide everything from users) and old, provably-flawed technological foundations (like a 16-bit system without things like filesystem access control or memory protection). I knew this story was false immediately though because no company ever even in 1993 had production server systems which ran a desktop OS like Win 3.1. It just wasn’t up to the task. They would have used NT if anything.

Zigurd 2y ago

I would like to know if a solid, up to date, well-rehearsed disaster recovery plan saved anyone's butt, or if we're all just raw dogging our machines whether IT is paying for backup and recovery or not?

Zigurd 2y ago
I see just moments after I posted, someone posted this: https://news.ycombinator.com/item?id=41103486 https://news.ycombinator.com/item?id=41103486 So, yeah, lack of DR is why Delta was so screwed.
ta1243 2y ago
Our systems worked fine, we expect things to fail - including software like sentinal one, crowdstrike, etc, and have DR systems which can keep us limping along. We have DR systems which will work should other things happen - say the Thames barrier fails (i.e. no docklands) Unfortunately some of our outsourced suppliers didn't have such attitudes.
paulddraper 2y ago
I've never seen it. Obviously some selection bias there, but I'd love to hear some success stories.
berniedurfee 2y ago
They certainly have DR infrastructure primed and ready to go… with Crowdstrike pre-installed on every DR server.
red-iron-pine 2y ago
Sure has. HSRP and VRRP plus other SD-WAN features definitely made a difference when one of our sites had the fiber pulled by accident. data center tech screwed up bigly and took us plus at least one other of their customers down. definitely saw a blip and stuff had issues for 10 minutes, e.g. pages timed out or had to restart a process, but generally sites failed over and were able to keep limping on while we did triage. got something like a $19 ($21?) service credit and an apology from the data center. our CEO shouted a lot and threatened lawsuits but it never went anywhere. Director of IT Infra quietly thanked all of us for having failover that mostly worked.

xyst 2y ago

> Apparently Southwest Airlines’ ingenious strategy of never upgrading from Windows 3.1 allowed it to remain unscathed. The “ingenious” strategy saved them from a weeks worth of downtime this year. But that same “ingenious” strategy was the primary reason for their meltdown in 2022 [1] https://www.npr.org/2022/12/30/1146377342/5-things-to-know-about-southwests-disastrous-meltdown https://www.npr.org/2022/12/30/1146377342/5-things-to-know-a... [2] https://www.nytimes.com/2022/12/28/travel/southwest-airlines-flight-cancellations.html https://www.nytimes.com/2022/12/28/travel/southwest-airlines...

skrebbel 2y ago
It’s also not true

linotype 2y ago

[flagged]

deleted 2y ago
[deleted]

skrebbel 2y ago

I love that “CrowdStrike” is now a synonym for “global outage”. Not some cute hihi name like “heartbleed”, just the name of the company that did the screwup. Seems fair.

jraph 2y ago
Not sure it's fair, but I am certainly waiting for it to become a verb or a noun. crowdstrike. n. 1. A set of major disruptions caused by an update that was not tested enough, pushed to many devices across the globe. 2. The name of such an update. 3. (by extension) a joke so bad it causes major disruptions. For instance: - Congrats for your crowdstrike! Now my weekend is ruined as I'll be the one who'll be asked to fix this mess. crowdstrike. v. (simple past crowdstruck or crowdstriked¹, past participle crowdstricken, or crowdstruck, or (obsolete, regionalism) crowdstroke²) 1. Action of pushing an update to many devices that causes a global outage or major disruptions in various sectors. For instance: - We've been crowdstruck. Again. crowdstrike. adj. 1. Qualifies an update that, when pushed to many devices across the world, causes major disruptions across the globe. 2. Qualifies such a (set of) event(s). For instance: - We are sorry for the crowdstrike event we caused. We gently remind our kind customers and their end users that per our ToS, we will issue no refund, and that no liability can be held against us. Customers who don't try to contact us in the following month will get a discount for their next contract renewal. You will hear us speak before the Congress, who nicely invited us for some comedy in the hope it will appease you all. Make sure you like the related videos on the various online platforms. We wish you a nice end of the week and nice, relaxing summer holidays. ¹ people have differing but strong opinions on which simple past form is correct, mainly due to regional differences. Some avoid saying crowdstrike and say crowdhit instead. ² some people have tried to push crowdstricken, which first caught on in some areas or particular contexts. The idea that this form likens the qualified subject to the bearer of some sickness has eventually seduced a critical mass of people after some initial push back. Please also see the usage notes for strike for other, rarer, alternative forms [*]. [*] https://en.wiktionary.org/wiki/strike#Usage%20notes https://en.wiktionary.org/wiki/strike#Usage%20notes (Thanks to the contributors in this thread)
- 2y ago

knappe 2y ago

For everyone flabbergasted by Southwest running ~Windows 3.1~ old software, I have bad news about the telecom industry. I worked at Ericsson at an R&D branch and one of the projects in the works was to move one of the main pieces of routing equipment that handled millions of telephony operatorations a day away from an ancient version of Windows. A lot of code lives on much longer than you think. The general attitude we took was that most of the code we were writing would be running for at least 30 years. And that was the attitude at an R&D branch, arguably a side of that industry where we were working on the new tech. Edit: Win 3.1 or something else, the point still stands. There is a lot of old software running out there that will continue to run our core services. Legacy software doesn't just mean v1 versus v2, it can mean v1 versus v41.

llm_nerd 2y ago
>For everyone flabbergasted by Southwest running Windows 3.1 Southwest isn't running Windows 3.1, though. That's some rather lame, but predictable, truth-through-repeated-assertion thing on social media. Not everyone uses CrowdStrike, and in this case SW was the lucky one that didn't.
wrboyce 2y ago
I find this more surprising, even if the Southwest & Win3.1 claims were true, I would expect most Ericsson systems to be Erlang based and thus happily chugging along on a (perhaps ancient) Linux box.
- knappe 2y ago
  I did too. But I knew of only 1 project that was using Erlang. I have always wanted to use it. Instead I saw a lot of MML and (happily) TCL. https://en.m.wikipedia.org/wiki/MML_(programming_language) https://en.m.wikipedia.org/wiki/MML_(programming_language)
ketchupdebugger 2y ago
Good thing a lot of our banking still runs on mainframes, will never be taken out by crowdstrike

deleted 2y ago

[deleted]

otterley 2y ago

It blows my mind how many people actually believed the claim -- clearly in the obvious-joke category -- that SWA is running their mission critical flight systems on Windows 3.1. (Yes, Southwest runs a lot of old tech in their stack, but that claim is patently hyperbolic.) People need to stop believing everything they read on the Internet and have a little bit of skepticism.

deleted 2y ago
[deleted]

beambot 2y ago

Lawsuits inbound. Delta appears to be gearing up for one already: https://finance.yahoo.com/news/delta-air-lines-seek-compensation-211908080.html https://finance.yahoo.com/news/delta-air-lines-seek-compensa...

hypeatei 2y ago
> has hired a law firm and will seek compensation from Microsoft and CrowdStrike Going after Microsoft seems like a misguided move here. What does Microsoft have to do with a third party driver installed by your own IT department?
- travoc 2y ago
  It was too tempting to include damages from accidentally enabling New New Outlook.
- bruce511 2y ago
  I suspect the lawsuit is created by lawyers, not techies. Equally reporting on this whole issue seems to be by journalists, not techies. It's been framed (a lot) as a Windows issue not a Crowd Strike issue. (8.5 million machines were affected, out of 1.4+ billion windows machines [1]) I have one affected customer (10k machines) who assumed I'd suffered like he did, and was surprised when I said we weren't affected. The reporting was consistently that it was a Windows issue, caused by an MS update. Even this article leans into this narrative... "as the New York Times put it, “It is more apt to ask what was not affected.” The answer is Linux, Macs, and phones." Let me add "not to mention 99.4% of windows". So the journalists don't know what happened, or who was affected, and felt "some computers have a problem" was a weak headline. The lawyers get that narrative and run with it. And yes, it's easy to squint and claim the "OS should cope with this", but there's realistically limits on what an OS can do once you install a kernel-level driver on the machine. Should we go after Intel for making the chips? [1] https://www.pcworld.com/article/608447/microsoft-delighted-by-windows-11-usage-but-what-is-it.html https://www.pcworld.com/article/608447/microsoft-delighted-b...
  - 2y ago

azinman2 2y ago

What this also tells me is there are a lot of computers connected to the internet that probably shouldn’t be.

0cf8612b2e1e 2y ago
I suspect it is incredibly challenging to keep a non trivial number of computers available, but somewhat airgapped. Too many ways to unintentionally bridge the networks without extreme diligence. Which is slightly incompatible with something like airlines which employ enormous numbers of people.
- azinman2 2y ago
  These are not small time operations (most of them). They are multi billion dollar companies with complex technical needs. This is doable and the bread and butter of good networking engineers. Having not done this just cost them billions. Now imagine the US at war or another nation state that just wants to cause havoc.
  - unethical_ban 2y ago
    Crowdstrike is a trusted application on every computer in these people's farms. It is not uncommon to have specific rules for these packages to be downloaded directly from the Internet. You're suggesting either that Crowdstrike itself will get used as a vessel for an attack, or that banks and airlines have firewall rules open for enemies.
  - nradov 2y ago
    That's ridiculous. The cost of building air gapped operations systems, and including the resulting loss of productivity and efficiency, would be worse than just accepting an occasional outage. People who lack technical competence and operational experience always tend to over react to software failures. The major airlines may not be "small time operations" but none of them are even in the top 100 US corporations by market cap. They simply don't have the level of IT resources or competence that we see at major tech companies.

nostromo 2y ago

It's insane to me that CrowdStrike's stock is still up 66% year-over-year. With all of the angry customers, lots of incoming lawsuits, and the fact that their "protection" is provably more costly than no protection at all now - I can't imagine why investors aren't dumping it like mad.

parmenidean 2y ago
Good news then! You can short it and make a ton of money if you're confident this share price increase is a mistake.
- concerndc1tizen 2y ago
  [dead]
johndhi 2y ago
My guesses: -no one really cancels their security vendors since security budgets don't shrink -they have a big moat so their customers won't be able to leave them
- kd913 2y ago
  I am confused why they are around to begin with. Companies already trust Microsoft, they buy Windows, Office, Azure. Why would they bother with a 3rd party here when the low effort low risk solution is to pick the tool made by the OS vendor. I.e. windows defender It should be a nobody gets fired for picking IBM situation. How did this random place get so much credibility that people trust them over the manufacturer?
  - htrp 2y ago
    crowdstrike has oracle enterprise sales model. have you ever been to one of their events?
  - kccqzy 2y ago
    Because they provide far more protection than Windows defender. You can write your own custom never-before-seen malware, and CrowdStrike will detect it purely based on behavioral signals. Windows Defender is still largely an antivirus solution.

bandyaboot 2y ago

Anyone know why Minneapolis-St Paul began experiencing cancellations much earlier than other US airports?

jijji 2y ago

basically any airline using linux is not on that list

bruce511 2y ago
It's more accurate to say "any airline not using Crowd Strike is not on that list." Blaming Windows for this outage is like blaming Linux for Apache bugs. The two systems are distinct. It just so happens that Crowd Strike was very successful at selling to large corporates. That includes some airlines. 99.4% of Windows machines were unaffected. Including those of airlines using Windows, but not Crowd Strike.
- jijji 2y ago
  ahhh yes you are correct
misja111 2y ago
Can you name one?
namdnay 2y ago
every airline in the world "uses linux", the core reservation and distribution systems were migrated from TPF to Linux over the past 20 years

bustling-noose 2y ago

> The outage highlighted a different kind of digital divide. On one side, gmail, Facebook, and Twitter kept running, letting us post photos of blue screens located on the other side: the Windows machines responsible for actually doing things in the world like making appointments, opening accounts, and dispatching police. At this point using windows for these tasks seems like using legacy software because training people to use an iPad or a web browser seems too complicated or because no one wants to move their age old systems to a more modern web based system because of costs. Native apps work great, but I think the world is moving to the cloud and that means web based everything should be the norm. Yes AWS AZURE outages can still happen but those can be fixed by spinning up a VM in different clouds. This is also why software jobs aren’t going anywhere thanks for a while. Many systems need to be changed to more modern and robust clouds. It might take decades for this transformation across the globe.

amluto 2y ago
Your “modern and robust cloud” is my “why on Earth doesn’t this thing work offline”. The world is absolutely full of things that have worked for decades to centuries without the Internet, are eventually more or less consistent (remember carbon paper credit card machines?), and did an amazing job of keeping the world running despite, wars, network partitions (the “network” would basically always be partitioned), mistakes, entire branches offline, etc. Sure, a lot of things are easier when centralized, and “the cloud” is incredibly powerful. But it’s not necessarily more robust. Also, depending on any sort of cloud means you’re also depending on the network, and networks are far from infallable. There’s a reason that a lot of stored-value transit systems still track balances on the card and will let people in even if a fare gate cannot connect to a cloud service. And CrowdStrike took out plenty of cloud instances, and recovering them can be worse than recovering physical hardware, as the “robust cloud” has an absolutely terrible ability to do anything outside the happy path of booting an instance normally.
- dailykoder 2y ago
  Okay this sounds all very reasonable, but how do you know when your washing machine is finished, when it's not connected to the cloud and you won't get notified in your app? It sure is not an easy thing and the cloud helps very much here

pimlottc 2y ago

One thing I don't understand from these graphs - why was there a relative uptick in takeoffs starting a short time /before/ the CrowdStrike update was pushed? It's in the overall graph, as well as the graphs for United, American, and especially Delta. I can't think of any reason for this, maybe it's just random noise, or maybe there was something unusual about the previous week at the same time?

mike_hearn 2y ago
It was widely reported to be the busiest travel day for quite a long time, which compounded problems.
account42 2y ago
Yeah, shouldn't have been too hard to add a couple more weeks so you at least get an idea about variance.
jjwiseman 2y ago
One reason I put charts for both absolute numbers of flights and percentage change is to help understand the larger context. Those relative upticks happened just before CrowdStrike hit, so around midnight Eastern time, when traffic was already very low. So a 25% increase might be 12 extra flights taking off in the U.S. in an hour. There's plenty of noise, for sure, and looking at absolute numbers and percent change together can help give you a sense of what was going on. Looking at two days worth of data is probably enough to give you the main themes of the CrowdStrike impact, but not enough to explain every variation.

deleted 2y ago

[deleted]

camillomiller 2y ago

Berlin Brandenburg got hit hard. As a disgruntled BER user, I am NOT surprised they had one of the worse repercussion.

nicbou 2y ago
German IT is often hit hard by such things. Unless of course they're still running on paper. At least they immediately mentioned it on their website, as a banner right at the top. The immigration office's appointment system has been down for over a month, and it took them 3 weeks to just acknowledge it.
- camillomiller 2y ago
  Thanks for your website btw. My partner’s currently renewing her work visa (she’s from Australia) and it’s insane how bad the situation is. To the point where it feels like it should just be illegal for Berlin to operate services like this. Appalling.
ta1243 2y ago
I'm still shocked that Brandenburg is actually open!

roshankhan28 2y ago

i really dont understand, how can my social media have better backup and infrastructure as compared to an OS which is being used by worldwide?

pjc50 2y ago
Windows has always been terrible for reliability. Adding a "security" system which is invasive and always-updated makes the reliability worse.
hiddencost 2y ago
Because one is in a data center that can be controlled, and the other is deployed to user owned hardware that cannot.
Ekaros 2y ago
Because they don't do mass rollouts on the servers. Then again those companies could fail if they had single point of failure with automatic mass deployments... This could happen for anything that supports this type of automatic mass deployment. Just in this case that thing was popular enough and happened on one of the most popular platforms.
goodcanadian 2y ago
Because IT is your social media's business. They know IT inside and out. They understand what can go wrong and how to mitigate it. The business for airlines (for example) is to fly planes. They are pretty damn good at it. IT, however, is just a tool to them that they buy elsewhere. They don't understand it in the same way as social media. They rely on outside contractors to do it right: outside contractors who get the job based on being the cheapest or convincing the buyers their service is "industry best practice."
rr808 2y ago
Meta is one of the most valuable companies in the world with the most resources to buy the best of everything. At 1,280 Billion dollars of market cap it is 30x bigger than American, Delta and United put together. It made $39 Billion last year compared to $7.8 Billion for all US airlines together. Of course it has better systems.

miohtama 2y ago

How to avoid getting rekt > Southwest wasn’t affected because they don’t use CrowdStrike

jandrusk 2y ago

Will be most interesting how this lawsuit by Delta plays out against Microsoft & Crowdstrike: https://www.marketwatch.com/story/delta-hires-law-firm-seeking-damages-from-crowdstrike-microsoft-after-massive-tech-outage-report-a882328a https://www.marketwatch.com/story/delta-hires-law-firm-seeki...

fagrobot 2y ago

[dead]

adrianwest66 2y ago

[dead]