4 ms·

Delta threatened to sue them for their $500M loss. Crowdstrike replied (publicaly) pointing out that their contract limits Crowdstrike's liability to single di

• JumpCrisscross 2y ago
  > they said, "if you sue us, we will dig so deep into your IT practices that it will be more embarrassing for you than us and show that you were at fault" But CrowdStrike said this publicly. If they’d privately relayed it to Delta, it would have been genuine. By performatively relaying it, however, it seems they’re pre-managing optics around the expected suit.
  • gnfargbl 2y ago
    Or attempting to discourage it from becoming a pile-on.
  • magic_man 2y ago
    It doesn't matter it was 100% crowdstrikes fault. Surprised its still worth 60billion dollars.
    • from-nibly 2y ago
      Part of the problem is assuming you can pay a contract to shift your liability completely away.
      • iwontberude 2y ago
        Well if MSFT knew how to write MSAs Crowdstrike would have become property of Microsoft.
      • unyttigfjelltol 2y ago
        Right, the risk structure presumably protects the vendor if just one customer sues, even if the amount of damages claimed is astronomical. Because vendors try to disclaim bet-the-company liability on a single contract.[1] The vendor's game is to make sure the rest of the customer base does not follow this example, because as noted in the linked article while vendors don't accept bet-the-company liability on each contract (or try not to), they do normally have some significant exposure measured in multiples of annual spend. [1] https://www.gs2law.com/blog/current-trends-in-liability-limitations-and-super-caps/ https://www.gs2law.com/blog/current-trends-in-liability-limi...
      • TeMPOraL 2y ago
        The assumption is not only perfectly valid, it's the very reason such contracts are signed in the first place! It's what companies want to buy, and it's what IT security companies exist to sell.
        • from-nibly 2y ago
          Yes, I know that's what everyone wants/thinks, but you actually can't do it. Because at the end of the day, you chose the vendor. So you are still liable for all of it.
    • pknomad 2y ago
      Yes and no. Crowdstrike was the executioner of this epic fail for sure but their archaic infra practices made it even worse. Both Crowdstrike and Microsoft CEOs reached out only to be rebuffed by Delta's own. If I was the CEO - I'd accept any help I can get while you have the benefit of the public opinion. /tin-foil-hat-on Flat out refusal for help makes me think there are other skeletons in the closet that makes Delta look even worse /tin-foil-hat-off
      • JumpCrisscross 2y ago
        > I was the CEO - I'd accept any help I can get while you have the benefit of the public opinion I’d reserve judgement. Delta may have been cautious about giving the arsonists a wider remit.
        • pknomad 2y ago
          Using your analogy - if MS/CS are the arsonists, then Delta are the landlords unsafely storing ammonium nitrate in their own warehouse. Their lack of response to MS/CS isn't coming from a place of reducing potential additional problems but trying to shield their own inadequacies while a potential lawsuit is brewing in the background. https://www.reuters.com/technology/microsoft-blames-delta-its-struggle-recover-global-cyber-outage-2024-08-06/ https://www.reuters.com/technology/microsoft-blames-delta-it...
        • TeMPOraL 2y ago
          In this case, the fire was an accident, and the arsonists happen to be the expert firefighters, and they're very motivated to fix their mistake. They're still the experts in all stuff fire, whereas Delta is not.
        • ImJamal 2y ago
          It doesn't seem like arsonist is the right word. It implies it was intentional, which as far as I can tell there is no proof of. I think the more accurate description would be some firefighters were doing a controlled burn. The burn got out of controlled and then you say that you don't want the firefighters help in put out the fire.
  • cyanydeez 2y ago
    Weirdly, we live in a society
  • wjnc 2y ago
    It’s an argument that hits home at any bigcorp where the execs are entertaining the thought of suing CrowdStrike. Making it public once is a lot more effective than relaying it privately a hundred times. I expect most liability to come from abroad, where parts of the contract might be annulled because not in line with local law. But still I don’t expect it. CrowdStrike delivered the service they promised. The rest is on the customers IT. Hand over the keys and your car may be driven.
    • JumpCrisscross 2y ago
      > It’s an argument that hits home at any bigcorp where the execs are entertaining the thought of suing CrowdStrike Maybe? Discovery is a core element of any lawsuit. It’s also a protected process: you can’t troll through confidential stuff with an intent to make it public to damage the litigant. If anything, I could see Delta pointing to this statement to restrict what CrowdStrike accesses and how [1]. (As well as with the judge when debating what gets redacted or sealed.) [1] https://www.fjc.gov/sites/default/files/2012/ConfidentialDisc.pdf https://www.fjc.gov/sites/default/files/2012/ConfidentialDis...
      • wjnc 2y ago
        Thank you. Nice read. Even given a protective order to keep discovery confidential, the ensuing discussion about the clients lacking IT-policies that exacerbated this crisis is public. Most entertaining would be the discussion where CrowdStrike would argue that based on common IT-risk criteria, you should never hand over the keys to an unaudited party not practicing common IT-risk best practices and (thus) the liability is on the organization. Talk about CrowdStrike managing risks worldwide. They are doing it right now!
  • tedunangst 2y ago
    Seems fair. Delta didn't privately relay their intentions.
• reaperducer 2y ago
  Delta threatened to sue them for their $500M loss. Crowdstrike replied (publicaly) pointing out that their contract limits Crowdstrike's liability to single digit millions. Delta's move seems like an attempt to assuage shareholders and help the C.E.O. save face. Crowdstrike shouldn't be afraid of Delta. Crowdstrike should be afraid of the insurance companies that have to pay out to those businesses that have coverage that includes events like this. Even if the payout to a company is $10,000, a big insurance company may have hundreds or thousands of similar payouts to make. The insurance companies won't just let that go; and they know exactly what to look for, how to find it, and have the people, lawyers, and time to make it happen. Crowdstrike will get its day of reckoning. It won't be today. And it probably won't be public. But the insurance companies will make sure it comes, and it's going to hurt.
  • JumpCrisscross 2y ago
    > the insurance companies will make sure it comes, and it's going to hurt It could be as simple as a reinsurer refusing to renew coverage if a company uses CrowdStrike.
    • vladvasiliu 2y ago
      Which would be funny, since many companies are putting up with Crowdstrike to make insurers happy.
    • dredmorbius 2y ago
      Availability (or not) of insurance coverage is surprisingly effective in enabling or disabling various commercial ventures. The penny dropped for me whilst reading James Burke's Connections on the exceedingly-delayed introduction of the lateen-rigged sail to Europe, largely on the basis that the syndicates which underwrote (and insured) shipping voyages wouldn't provide financing and coverage to ships so rigged. Far more recently we have notions of redlining for both mortgage lending and insurance coverage (title, mortgage, property, casualty) in inner-city housing and retail markets. Co-inventor of packet-based switching writes of his parents' experience with this in Philadelphia: "On the Future Computer Era: Modification of the American Character and the Role of the Engineer, or, A Little Caution in the Haste to Number" (1968) <https://www.rand.org/pubs/papers/P3780.html https://www.rand.org/pubs/papers/P3780.html> (footnote, p. 6). Similarly, government insurance or guarantees (Medicare, SSI, flood insurance, nuclear power plants) has made high-risk prospects possible, or enabled effective services and markets, where laissez-faire approaches would break down. I propose that similar approaches to issues such as privacy violation might be worth investigating. E.g., voiding any insurance policy over damages caused through the harmful use or unintended disclosure of private information. Much of the current surveillance-capitalism sector would instantly become toxic. The principle current barriers to this are that states themselves benefit through such surveillance, and of course the current industry is highly effective at lobbying for its continuance.
• trentnix 2y ago
  They should be providing all that information regularly to auditors anyway. If they don’t have it handy, then their IT leadership should be replaced.
• mc32 2y ago
  That’s odd. One is an internal process which has no obligation to an external party, and the other one who is specifically responsible for being liable for any repercussions due to deviating from their own SDLC process[1]they totally skipped themselves? If I were Delta, I’d get other affected parties and together sue CrowdStrike and get all their dirty laundry out in the open. [1] I haven’t checked but they used to list all their ISO certs, etc. Wonder if those get revoked for such glaring violations…
  • reaperman 2y ago
    Civil suits focus in a large way on determining how much damage is each party’s fault. So Crowdstrike would be saying “Of this $500M in damages, x% was from your own shitty practices not from our mistake”. Thats why it’s all pertinent.
    • otterley 2y ago
      Correct. The legal term is “contributory negligence.”
  • MattGaiser 2y ago
    > One is an internal process which has no obligation to an external party Delta has obligations to their passengers and similarly sidesteps screw ups with similar contractual provisions. How much would Delta owe for not following similar IT practices? Do they now owe customers for their IT failings? Should customers now get to sue Delta for damages related to their poor IT recovery compared to other airlines?
    • mc32 2y ago
      Sure but that’d be something passengers could bring up in a suit against Delta, not someone like CS, who themselves obviously skipped their own internal SDLC and whatever other ISO certs they prominently advertised on their website.
      • dredmorbius 2y ago
• VirusNewbie 2y ago
  They might find out delta does embarrassing things like not testing out of bounds array access or does global deployments without canarying.
• richardw 2y ago
  I’m not sure anything else was material given that the machines were bricked and client roll-out approaches were evaded by Crowdstrike. What client actions would have helped? Surely someone is looking at a class action? People died. The contract can’t make that everyone else’s problem, can it?
  • lupire 2y ago
    If someone's life depends on a networked Windows (or any similar OS) machine you chose to run for that purpose, you are the criminal.
    • 1over137 2y ago
      Indeed. But this is how hospitals run.
  • gizmo686 2y ago
    Sure it can. If every rock climbing company in the country decides that climbing ropes are too expensive and instead decide to by rope from the local hardware store, and that rope has a warning reading "not for use when life or valuable property is at risk", then it is 100% on those climbing companies when people die, because they were using a product in a situation that it was simply not suitable for. The details, of course, depend on the contract and claims that Crowdstrike made. But, in the abstract, you are not responsible for making your product suitable for any use that anyone decides to use it for. If a hospital wants to install software on their life critical infastructure, they are supposed to buy software that is suitable for life critical infastructure.
• mikeocool 2y ago
  It really seems funny that Crowdstrike’s defense is basically “you should have been better prepared for us to knock all of your systems offline.” It’s probably true, but seems like an odd stance to take from a PR perspective or a “selling other clients in the future” perspective.
  • jedberg 2y ago
    In the case of Delta, their outage was much longer than everyone else because they refused help from both Crowdstrike and Microsoft. So their defense is basically "the damages could have been mitigated if you'd listened to us".
    • NavinF 2y ago
      > they refused help from both Crowdstrike and Microsoft Link? Anyway I find it highly amusing that Delta is seeking damages from Microsoft even though Microsoft had nothing to do with it.
      • jedberg 2y ago
        There are many articles about them refusing help, but here is one: https://www.theverge.com/2024/8/6/24214371/microsoft-delta-letter-crowdstrike-response-comments https://www.theverge.com/2024/8/6/24214371/microsoft-delta-l...
      • cratermoon 2y ago
        Delta's position is the Microsoft actively recommended and coordinated with CrowdStrike to the extent that they are co-responsible for outcomes. In a large enterprise like Delta, the vendors do work together in deployment and support. Yes, there's often a great deal of finger-pointing between vendors when something like this happens, but in general vendors so intimately linked have each other on speed-dial. It would not shock me to learn that Delta has email or chat threads involving CrowdStrike, Microsoft, and Delta employees working together during rollouts and upgrades, prior to this event. As far as refusing help, why is that funny? If someone does something stupid and knocks you down, it's perfectly reasonable to distrust the help they offer, especially if that help requires giving them even more trust than what they've already burned.
• icehawk 2y ago
  I'd LOVE to see Crowdstrike do this. The last time I dealt with the specifics of this sort of validation testing for security software was a decade and from what I saw in the RCA Delta can just keep pointing out that whatever they had worked until Crowdstrike failed to understand that the number 20 and the number 21 are not the same: The new IPC Template Type defined 21 input parameter fields, but the integration code that invoked the Content Interpreter with Channel File 291’s Template Instances supplied only 20 input values to match against. This parameter count mismatch evaded multiple layers of build validation and testing, as it was not discovered during the sensor release testing process, the Template Type (using a test Template Instance) stress testing or the first several successful deployments of IPC Template Instances in the field. This combined with the lack of partitioning updates, makes me draw the conclusions they're missing table stakes WRT to validation.
  • mikhael28 2y ago
    Wtf how do you not check for ‘quantity of arguments’ in QA testing?
• otterley 2y ago
  That’s not the way legal process works. CrowdStrike might be permitted to conduct discovery, but that won’t entitle them to share what they might find with the public, embarrassing or otherwise. Business records and other sensitive information relating to parties in civil matters are frequently sealed.