5 ms·

I would like to live in a world where crowdstrike is directly liable for every dollar lost and has to pay for all of it. Only then would these companies start t

• ghostly_s 2y ago
  It's absolutely ludicrous the US airline industry still has no contingency plan for "a bad software update is pushed by a vendor to our Windows systems."
  • SketchySeaBeast 2y ago
    If the computers in a computer reliant massively distributed organization go down, what's the alternative?
    • ThunderSizzle 2y ago
      Computers shouldn't auto update. Updates need to be scheduled and monitored and tested.
      • lazide 2y ago
        Every CTO’s easy answer - cool, we’ll just only do them once every 5 years then. With modern tech stacks, we’re talking hundreds of updates a quarter. If not more.
        • cozzyd 2y ago
          Sounds like they shouldn't be using those tech sticks then.
      • paulmd 2y ago
        That was the defacto state of things up until windows 10 moved to a forced update model. In that world we regularly had massive malware waves and resultant massive botnets constantly disrupting critical infrastructure. We moved away from that for a reason. And the real fix is to make updates not take down the whole system. When was the last time every Linux system in the world was unable to boot all at the same time? Even the crowdstrike issue only took down a very limited subset of installs and the blast radius is far more minimal due to the linux kernel being a lot more sane and cautious.
      • paulddraper 2y ago
        Desktop operating systems, mobile devices, browsers.....there's a lot that doesn't pass your approval.
        • ThunderSizzle 2y ago
          My desktop doesn't auto update. It only updates when I ask it to. My phone does auto update, and nearly every auto update makes it worse and slower. That's been true for nearly every Android I've had.
          • paulddraper 2y ago
            The sheer awesomeness of Android 3 is staggering by now.
    • unstatusthequo 2y ago
      Failover?
      • SketchySeaBeast 2y ago
        Failover to what?
    • amluto 2y ago
      Acknowledge that it’s a distributed system and build accordingly. The distributed systems literature understands failure modes, including Byzantine failure. Old airline systems were built to work with simple protocols and limited communication. A ticket was an independently verifiable object. Reservations were on a printed list. Check-in and gate agents did not need to make a phone call for each passenger. This was expensive, but it worked. One can do the same thing with computers. Build small systems that operate independently, with strictly defined inputs and outputs. Make them un-brickable. Distribute software on read-only media, or outright replace the CPU for each update. You roll back by having the staff unplug the new thing and plug in the old thing. A high-spec CM4 is $90. A small USB stick costs basically nothing. A bespoke CM4-like machine with only ROM could be made in volume for relatively little money. Most of the problem with an approach like this is that the industry doesn’t think this way. Maybe a big buyer should step up, e.g. a major military. Non-CrowdStrike-able tech for, say, the US military seems like it would be extremely valuable.
    • jjav 2y ago
      > If the computers in a computer reliant massively distributed organization go down, what's the alternative? I feel like the answer should be obvious: if you need reliable systems then you don't create a single point of dependency on such fragile systems. The latest blog from Bruce Schneier is a good writeup on this brittleness: https://www.schneier.com/blog/archives/2024/07/the-crowdstrike-outage-and-market-driven-brittleness.html https://www.schneier.com/blog/archives/2024/07/the-crowdstri...
• whoknowsidont 2y ago
  >I would like to live in a world where crowdstrike is directly liable for every dollar lost and has to pay for all of it. That wouldn't make any sense. The blame realistically lies within each company who allowed a critical point of failure just so they get checkbox software and not actually have to expend the effort of making sure the company is able to function with their chosen infrastructure.
  • margalabargala 2y ago
    Exactly. The customer is to blame here, not the seller. Buyer beware. If airlines skimp on maintenance and planes crash, well, all those dead passengers should have picked an airline that takes maintenance more seriously. Next time they'll know better.
    • autoexecbat 2y ago
      It shouldn't be one or the other, but both
    • jjav 2y ago
      > Next time they'll know better. No next time, they're dead, so not a good selection criteria. Agreed that both sides share some liability, but a vendor that says they create security software with root level code injection access must be held to the highest possible standard, and liability. If someone puts up a shack and sells hamburgers and hundreds of people die from the rotten meat, they most certainly are liable. But apparently crowsdstrike can brick computers remotely, causing enormous damage including some deaths, and just say oopsie and walk away without any liability.
  • deleted 2y ago
    [deleted]
  • deleted 2y ago
    [deleted]
• Terretta 2y ago
  > I would like to live in a world where crowdstrike is directly liable for every dollar lost and has to pay for all of it. Or a world where the regulator that required the checkbox — even if the firm can demonstrate a superior way of achieving the same objective — should pay for it.
• leros 2y ago
  Crowdstrike's contracts and terms of service will have clauses about how much liability they have when things go wrong. I have no idea what Crowdstrike's policy is, but pretty often, the liability is limited to the amount of money you've paid them during the outage. I've been involved in procurement at a big corporation and one thing we always modified in contracts was making the vendor 100% liable for any damages caused by their outages, but many vendors wouldn't make that modification.
  • cwilkes 2y ago
    I’ve heard of that for civil engineering firms but the amount of damages is capped at the yearly contract amount. Which in this case is probably a lot less than what these companies are paying in clean up costs.
• paulddraper 2y ago
  That's an excellent way to encourage no one to do anything ever.
  • jjav 2y ago
    We can look outside the software industry to see this is not true. Building bridges is a very overused analogy, but it's still a good one. Structural engineers still design bridges and they get built. What they don't do it throw some drawing together and YOLO it to the builder and let's see what happens, the way software gets built.
    • paulddraper 2y ago
      Absolutely not. If a bridge has maintenance issues causing it to be unusable while it is being fixed, the bridge makers are not on the hook for the total downstream economic loss.
• SkyPuncher 2y ago
  You can. Just be a massive company with a massive contract and demand it as part of your contract negotiations. It’ll probably cost you a pretty penny, but you could make it happen. Realistically, this doesn’t happen because the returns don’t make sense. If vendors are liable for your business, they need to account for your business in their costs. That drives their costs (and prices) up.
  • jjav 2y ago
    > That drives their costs (and prices) up. That is not a bad thing. Today all these shaky companies exist because they can take all the profit and externalize all the downsides, laughing all the way to the bank. If they were made liable, one of two things might happen, both of which are positive outcomes. One, they might upgrade their engineering and quality processes to the point where they can guarantee they won't be the root cause for taking down large parts of the industry. Sure it'll cost more but if they can do it and still make a profit (just less), all is well. Two, maybe it simply can't be profitable to build these root backdoor systems with enough layers of safety, in which case the companies will disappear. This is also good; if the product can't be safe, it should not exist.