12 ms·

I appreciate that we’re finding the humour in this catastrophe but what about the question of liability? I have seen a few stories on HN of the billions lost by

I appreciate that we’re finding the humour in this catastrophe but what about the question of liability? I have seen a few stories on HN of the billions lost by this event but so far not much in the way of lawsuits. What is the situation? Are the licenses so ironclad that customers have no recourse? I could understand this in the case of consumers who might suffer minor inconvenience as their home PC is out of service for a few hours/days but it seems totally unacceptable for industries to accept this level of risk exposure. This is one of the big reasons civil engineering is considered such a serious discipline. If a bridge collapses, there’s not only financial liability but the potential for criminal liability as well. Civil engineering students have it drilled into their heads that if they behave unethically or otherwise take unacceptable risks as an engineer they face jail time for it. Is there any path for software engineers to reach this level of accountability and norms of good practice?

gjsman-1000 2y ago
> Is there any path for software engineers to reach this level of accountability and norms of good practice? Heck, no. Civil engineering doesn’t change. Gravity is a constant. Physics are constants. If Rome wrote an engineering manual, it would still be quite valid today. Imagine if we had standardized software engineering in 2003. Do you think the mandatory training about how to make safe ActiveX controls is going to save you? Do you think the mandatory training on how to safely embed a Java applet will protect your bank? Software is too diverse, too inconsistent, and too rapidly changing to have any chance at standardization. Maybe in several decades when WHATWG hasn’t passed a single new spec into the browser. (Edit: Also, it’s a fool’s errand, as there are literally hundreds of billions of lines of code running in production at this very moment. If you wrote an onerous engineering spec; there would not be enough programmers or lawyers on earth to rewrite and verify it all, even if given decades. This would result in Google, Apple, etc. basically getting grandfathered in while startups get the burden of following the rules - rules that China, India, and other countries happily won’t be enforcing.)
- figassis 2y ago
  Physics will still be the same when your faulty software tells an airplane to dive.
- 2OEH8eoCRo0 2y ago
  I'd imagine we wouldn't have ActiveX controls in the first place.
  - gjsman-1000 2y ago
    Wishful thinking - the IRS is still running on COBOL; our nuclear weapons until a few years ago on Windows 95. The NYC subway still has a lot of OS/2. Standardization does not stop bad engineering. Those who think it does have not witnessed the catastrophe a bad standard can cause. Go download and implement the Microsoft Office OOXML standard - it’s freely available, ISO approved, 6000 pages, and an abomination that not even Google claims to have correctly implemented.
    - 2OEH8eoCRo0 2y ago
      You're making some points for me. You are assuming COBOL, Windows 95, or OS/2 are bad because they're old. Such assumptions are the antithesis of "engineering."
      - gjsman-1000 2y ago
        Old technology isn’t necessarily bad in itself. It’s well documented and understood. Where it’s bad is when the equipment to run that software no longer is manufactured. You can’t get a new computer to run Windows 95. Not even in the military. Your only option is to virtualize, adding a huge possible failure mode that was never considered previously. Where it’s bad is when changes are needed to adapt to modern environments, and nobody’s quite sure about what they are doing anymore. There’s no test suite, never was, the documentation is full of ancient and confusing terminology, mistakes are made. And on and on…
      - jfengel 2y ago
2OEH8eoCRo0 2y ago
> Is there any path for software engineers to reach this level of accountability and norms of good practice? There is no reason that software couldn't be treated with the same care and respect. The only reason we don't is because the industry resists that sort of change. They want to move fast and break things while still calling themselves "engineers." Almost none of this resembles engineering.
- wizzwizz4 2y ago
  > Software engineering, of course, presents itself as another worthy cause, but that is eyewash: if you carefully read its literature and analyse what its devotees actually do, you will discover that software engineering has accepted as its charter "How to program if you cannot.". — Edsger Wybe Dijkstra, 1988. (EWD1036)
  - jfengel 2y ago
    I'm ok with that. I don't want to keep everyone out except just those who happen to have just the right mind set. Programming is about developing software for people, and the more viewpoints are in the room, the better. Some pieces are more important than others. Those are the bits that need to be carefully regulated, as if they were bridges. But not everything we build has lives on the line. If that means we don't get to call ourselves "engineers", I'm good with that. We work with bits, not atoms, and we can develop our own new way of handling that.
    - 2OEH8eoCRo0 2y ago
      I prefer to call it "computer programming." If the title is good enough for Ken Thompson or Don Knuth then it's good enough for me.
    - wizzwizz4 2y ago
      > I don't want to keep everyone out except just those who happen to have just the right mind set. Neither do I. Neither did Dijkstra. EWD1036, “On the cruelty of really teaching computing science”, is about education reform, to enable those who don't "happen to have just the right mind set" to fully participate in actual, effective programming.
belter 2y ago
https://zlk.com/pslra-1/crowdstrike-lawsuit-submission-form?prid=94963&wire=1 https://zlk.com/pslra-1/crowdstrike-lawsuit-submission-form?... https://www.sauderschelkopf.com/wp-content/uploads/2024/07/AS-FILED-Delta-Complaint-8.6.24.pdf https://www.sauderschelkopf.com/wp-content/uploads/2024/07/A...
MattGaiser 2y ago
> Are the licenses so ironclad that customers have no recourse? Even on Hacker News, there was agreement that CrowdStrike screwed up, but then people also blamed IT staff, Microsoft (even after realizing it was a CrowdStrike issue), and the EU/regulators. I imagine responsibility of each entity would need far more clarification than it does now. If you want to define liability, there needs to be a clear line saying who is responsible for what. That doesn’t currently exist in software. There are also considering how people respond to risk. Consider how sesame regulation led to most bread having sesame deliberately put into it. Industry responded by guaranteeing contamination. Crowdstrike and endpoint security firms might respond by saying that only Windows and Mac devices can be secured. Or Microsoft may say that only their solution can provide the requisite security.
miki123211 2y ago
Civil engineering mostly requires you to have a government-verified certificate and to work in the country your infrastructure will be deployed in. Software engineering doesn't, and that makes criminal prosecutions that much harder. There's no path to making it happen. Financial liability for the company in question? Sure, that's probably doable. "Piercing the corporate veil" and punishing the executives who signed off on it? Harder but not impossible. Punishing the engineer who wrote that code, and who lives in a country with no such laws? Won't happen.
- WarOnPrivacy 2y ago
  > Civil engineering mostly requires you to have a government-verified certificate and to work in the country your infrastructure will be deployed in. It's a relatively small (and sharply defined) pool of people who can be called a civil engineer. Are we saying we want to segment software engineering (from coding) - the same way civil engineering is segmented from construction? Otherwise we're talking about placing specialist liability upon a non-specialist group. This seems unethical.
JumpCrisscross 2y ago
> If a bridge collapses, there’s not only financial liability but the potential for criminal liability as well If a bridge collapses people die. To my knowledge, nobody died or was put in mortal peril as a result of the Crowdstrike debacle.
- rightbyte 2y ago
  The deaths if any where probably indirect. E.g. ambulances not turning up in time etc. due to paper and pen fallbacks. With all the hospitals victim of the attack, I would be surprised if the amount of patients that died are zero.
  - JumpCrisscross 2y ago
    > E.g. ambulances not turning up in time etc. due to paper and pen fallbacks Sure. Did this happen? Why were the “emergency management downtime procedures” insufficient [1]? [1] https://www.healthcaredive.com/news/crowdstrike-outage-hits-us-hospitals/721887/ https://www.healthcaredive.com/news/crowdstrike-outage-hits-...
    - tgsovlerkhgsel 2y ago
      If they were equally good as the non-emergency procedures, why wouldn't we use them all the time?
      - JumpCrisscross 2y ago
        > why wouldn't we use them all the time? Because they’re more expensive. They’re all not “equally good,” they’re good enough to keep people alive. (You repurpose resources from elective and billing procedures, et cetera.)
        FireBeyond 2y ago
        This belies a lack of understanding. What resources are you repurposing from elective procedures exactly? Your patient load hasn’t changed, and day surgical instruments and supplies are from the same pool. There’s no “well this pile of equipment is only for elective procedures”. I’m not even sure what “billing procedures you’d repurpose (especially in your context of “keeping people alive”).
LordKeren 2y ago
There is recourse, just not for normal people, as you eluded to. Companies are and will be continuing to sue crowdstrike, and based on the papers that crowdstrike has posted, the impacted companies are extremely likely to be successful. It seems overwhelmingly likely that the companies are going to be able to convince a judge/jury/arbiter that crowdstrike acted grossly negligent and very plainly caused both direct losses and indirect reputational harm to the companies. I’m not sure crowdstrike will even fight it, to be honest. I would assume most of this is going to be settled out of court and we will see crowdstrike crumble in the coming years.
- JumpCrisscross 2y ago
  > not sure crowdstrike will even fight it To my knowledge only Delta is suing and CrowdStrike is kicking and screaming about it [1]. [1] https://www.cnn.com/2024/08/05/business/crowdstrike-fires-back-at-delta/index.html https://www.cnn.com/2024/08/05/business/crowdstrike-fires-ba...
  - LordKeren 2y ago
    It’s a really bad look for crowdstrike to be going down this route. Then again, I don’t think many companies are going to be adopting crowdstrike in the coming years, so I suppose their only option is to defend their stock value at any cost while the company recoils
- dredmorbius 2y ago
  NB: alluded, not eluded. <https://dict.org/bin/Dict?Form=Dict2&Database=gcide&Query=Allude https://dict.org/bin/Dict?Form=Dict2&Database=gcide&Query=Al...> <https://dict.org/bin/Dict?Form=Dict2&Database=gcide&Query=elude https://dict.org/bin/Dict?Form=Dict2&Database=gcide&Query=el...>
  - LordKeren 2y ago
    TIL— thanks! Now it’s time to painfully go through my slack / email history and see how many times I made this mistake :)
jedberg 2y ago
Delta threatened to sue them for their $500M loss. Crowdstrike replied (publicaly) pointing out that their contract limits Crowdstrike's liability to single digit millions. Then then gave them a list of things they would seek in discovery, such as their backup plans, failover plans, testing schedules and results, when their last backup recover exercise was, etc. Basically, they said, "if you sue us, we will dig so deep into your IT practices that it will be more embarrassing for you than us and show that you were at fault".
- JumpCrisscross 2y ago
  > they said, "if you sue us, we will dig so deep into your IT practices that it will be more embarrassing for you than us and show that you were at fault" But CrowdStrike said this publicly. If they’d privately relayed it to Delta, it would have been genuine. By performatively relaying it, however, it seems they’re pre-managing optics around the expected suit.
  - gnfargbl 2y ago
    Or attempting to discourage it from becoming a pile-on.
  - magic_man 2y ago
    It doesn't matter it was 100% crowdstrikes fault. Surprised its still worth 60billion dollars.
    - from-nibly 2y ago
      Part of the problem is assuming you can pay a contract to shift your liability completely away.
      - iwontberude 2y ago
        Well if MSFT knew how to write MSAs Crowdstrike would have become property of Microsoft.
      - 2y ago
minkles 2y ago
Engineering safety culture is built on piles of bodies and suffering unfortunately. I suspect in software the price of failure is mostly low enough that this motivation will never develop.
xyst 2y ago
Probably in a decade or so after the AI crash. I have yet to see anything that comes close to “liability” for the digital realm. US governments and businesses get hacked/infiltrated all the time by foreign adversaries yet we do not declare war. Maybe something happens in the dark or back channels. But we never know.
dylan604 2y ago
A lot of companies have insurance on events causing them to lose sources of income. Whether that's farmers having crop insurance, big box retailers having insurance for catastrophic damage to their big box, I would assume there's something for infrastructure collapse to bring sales to $0 for the duration. Even if everyone that was affected sued ClownStrike for 100% of their losses, it's not like ClownStrike has the revenue to cover those losses. So even if you're a fan of shutting them down, nobody recovers anything close to actual losses. So what would you actually propose? Bug free code is pretty much impossible. Some risk is accepted by the user. Do you seriously think that software should be absolutely 100% bug free before being able to be used? How do you prove that? Of course, the follow up would be how clean is your code that you feel that's even achievable?
- Rinzler89 2y ago
  >Bug free code is pretty much impossible. Some risk is accepted by the user. This wasn't your average SW bug, it was gross negligence on behalf of Crowdstreike, who seems to not have heard of SW testing on actual systems and canary deployment. Big difference. Yeah SW bugs happen all the time but you have to show you took some steps to prevent them, while some dev at Crowdstrike just said "whatever, it works on my machine" and directly pushed to all customer production systems on a Friday. That's the definition of gross negligence that they didn't have any processes in place to prevent something like this. That's like a surgeon not bothering to sterilize his hands and then saying "oh well, hospital infections happen all the time".
  - ncr100 2y ago
    The bug was egregious. Using regexp (edit: in the kernel). (Wtf. It's a bloody language.) And not sanitizing the usage. Then using it differently than testing. And boom. There's people, and there's companies. This company ought to be nuked.
    - cqqxo4zV46cp 2y ago
      Genuinely, what good does that do? It’s all well and good to write dramatic meaningless comments on social networks like Hacker News, but if your desired had actual consequence, can you honestly say that “nuking the company” is a net positive?
ttymck 2y ago
> but so far not much in the way of lawsuits It hasn't been that long? The situation might be that there hasn't been sufficient time to yet gather evidence to commence lawsuits.
cesarb 2y ago
> Is there any path for software engineers to reach this level of accountability and norms of good practice? Yes, time. Civil engineering has thousands of years of history. Software engineering is much newer, the foundations of our craft are still in flux. There have been, at least in my country, legislative proposals for licensure of system analysts, electronic computer programmers, data processing machine operators, and typists(!) since the late 1970s; these laws, if approved, would have set back the progress of software development in my country for several decades (for instance, one proposal would make "manipulation and operation of electronic processing devices or machines, including terminals (digital or visual)" exclusive to those licensed as "data processing machine operator").
- Cpoll 2y ago
  > set back the progress > exclusive to those licensed Sounds to me like it just would've made a lot of money for whatever entities give out the licenses. On the other hand, I've read speculation on here that some countries are short on entrepreneurs entirely due to the difficulty of incorporating a small business, so maybe.
tsujamin 2y ago
I’m interested in what those who suffered outages as a result of crowdstrike told their insurers with respect to “QA’ing production changes” It’d be interesting to see if anyone tries to claim the outage as some sort of insurance event only to lose out because they let Crowdstrike roll updates into a highly regulated environment without testing
crote 2y ago
> Civil engineering students have it drilled into their heads that if they behave unethically or otherwise take unacceptable risks as an engineer they face jail time for it. Is there any path for software engineers to reach this level of accountability and norms of good practice? The problem is that with civil engineering you're designing a physical product. Nothing is ever designed to its absolute limit, and everything is built with a healthy safety margin. You calculate a bridge to carry bumper-to-bumper freight traffic, during a hurricane, when an earthquake hits - and then add 20%. Not entirely sure about whether a beam can handle it? Just size it up! Suddenly it's a lot less critical for your calculations to be exactly accurate - if you're off by 0.5% it just doesn't matter. You made a typo on the design documents? The builder will ask for clarification if you're trying to fit a 150ft beam into a 15.0ft gap. This means a bridge collapse is pretty much guaranteed to be the result of gross negligence. Contrast that to programming. A single "<" instead of "<=" could be the difference between totally fine and billions of dollars of damages. There isn't a single programmer on Earth who could write a 100% bug-free application of nontrivial complexity. Even the seL4 microkernel - whose whole unique selling point is the fact that it has a formal correctness proof - contains bugs! Compilers and proof checkers aren't going to complain if you ask them to do something which is obviously the wrong thing but technically possible. No sane person would accept essentially unlimited liability over even the smallest mistakes. If we want software engineers to have accountability, we first have to find a way to separate innocent run-of-the-mill mistakes from gross negligence - and that's going to be extremely hard to formalize.
- Analemma_ 2y ago
  This entire comment boils down to "we can't be held accountable because it's soooo hard you guys", which isn't even convincing to me as someone in the industry and certainly won't be to someone outside it.
  - heyoni 2y ago
    What a shallow dismissal of a comment that doesn’t even claim that there shouldn’t be accountability.
wannacboatmovie 2y ago
> Is there any path for software engineers to reach this level of accountability and norms Potentially controversial stance here, but most software engineers are not engineers. They study computer science, which doesn't include coursework on engineering ethics among other things. I would say that by design they are less prepared to make ethical decisions and take conservative approaches. Imagine if civil engineers had EULAs for their products. "This bridge has no warranty, implied or otherwise. Cross this bridge AT YOUR OWN RISK. This bridge shall not be used for anything safety critical etc."
lupire 2y ago
How many bridges, would you say, does the average civil engineering firm deliver each year, each on only 1 day notice, in response to a surprise change in requirements due to a newly developed adversarial attack? Crowdstrike does this constantly. You could demand the same level of assurance from software, but in exchange, you don't get to fly, because the capacity won't be there
dakiol 2y ago
> I appreciate that we’re finding the humour in this catastrophe but what about the question of liability? One of the biggest and most used piece of software (the Linux kernel) comes with zero warranties. It can fail, and no one would be liable. Are we fine with that? Is the CS case different because it costs money? From an user perspective we don’t want software failing in the middle of an airplane landing, so whether the software comes from CS or github, it’s of lesser importance.
guax 2y ago
I would find it more useful if liability here we're attributed to the need to purchase such draconian tools. Certifications that require it and C levels who approve it. We would be better by it.
- cqqxo4zV46cp 2y ago
  Oh Christ. Just drop it. A by all accounts legitimate security function of a product targeted at company-owned endpoints. Please don’t devolve this conversation into you being upset about not getting admin rights on your work computer or whatever this is about. Any (esp. larger) org would be criminally negligent to eschew using something like CrowdStrike in order to capitulate to some nerd that thinks that they have ownership over their work equipment.
  - guax 2y ago
    Bruh, chill. I don't give two craps about having admin rights on my work computer. Crowdstrike is bad software and bad way to manage large deployments. They just proved it. I just think that we are also responsible for buying a solution that works that way.
maccard 2y ago
I completely agree. When I've negotiated contracts for my workplace, and we explicitly write in the contract that the vendor is responsible for XYZ, it is my understanding (and confirmed by legal, multiple times) that this means in case of XYZ going wrong, they are liable for up to the amount in the SLA, however that isn't a cap on liability in extenuating circumstances. If this all gets brushed away, it significantly devalues the "well we pay $VENDOR to manage our user data, it's on them if they store it incorrectly" proposition, which would absolutely cause us to renegotiate.
- cqqxo4zV46cp 2y ago
  You aren’t showing us the specific language that you’re referring to, nor do we know what a typical CrowdStrike contract looks like. You could be talking about apples and oranges here. I’ve seen both.
  - maccard 2y ago
    I was pretty sure that someone was going to "ackshually" me here, and here we are. The specific wording doesn't matter. I've negotioated dozens of these contracts and the value add of a vendor managing the data is liability. If they aren't liable for data mis-management, then their managed service is only worth the infra costs + a haircut on top, and we'll renegotiate with this in mind.
wcunning 2y ago
Software (controls) engineers at VW during the emissions scandal went to jail, engineers at GM were held liable for the ignition switch issue (not mostly in software, but still). I expect we'll eventually see some engineers/low level managers thrown under the bus with Boeing. It definitely happens, but not as frequently as it could. That said, I definitely prefer Amazon's response to the AWS East 1 outage back in 2016 -- the engineer wasn't blamed, despite the relatively simple screw up, but the processes/procedures were fixed so that it didn't happen again in the last 8 years. Crowdstrike is a little bit gray on that regard -- people should have known how bad the practice of zero testing on config updates was, but then again, I've seen some investigating saying that the initial description wasn't fully accurate, so I'm waiting for the final community after action report before I really pass judgement.
- deleted 2y ago
  [deleted]
ezoe 2y ago
They totally deserved it. Those who think running third-party closed-source Windows kernel driver(which parse files distributed from Internet in realtime) are good for the security, they must also accept the consequence. I'm sick of these so-called security consultants who always insist check lists like installing proprietary close-source binary blob Linux kernel module to the system consists of otherwise mostly free softwares except for hardware drivers and think they did their job, or executives who pays a lot of money to these idiotic so-called security consultants.
oglop 2y ago
On this website you are asking a population that would be responsible for this, so you will likely only get answers about how hard this is to solve and how it’s not software engineers fault and how we need to understand software engineering is not civil engineering and we need to be careful with this analogy and how it’s not our fault! Don’t blame us when things go wrong, but also, give us all the money when things go right. This is not the place for this question is what I’m saying.
barelysapient 2y ago
Your barber has more licensing requirements than a senior software engineer. Regulations have not caught up with developers (yet).

gjsman-1000 2y ago

> Is there any path for software engineers to reach this level of accountability and norms of good practice? Heck, no. Civil engineering doesn’t change. Gravity is a constant. Physics are constants. If Rome wrote an engineering manual, it would still be quite valid today. Imagine if we had standardized software engineering in 2003. Do you think the mandatory training about how to make safe ActiveX controls is going to save you? Do you think the mandatory training on how to safely embed a Java applet will protect your bank? Software is too diverse, too inconsistent, and too rapidly changing to have any chance at standardization. Maybe in several decades when WHATWG hasn’t passed a single new spec into the browser. (Edit: Also, it’s a fool’s errand, as there are literally hundreds of billions of lines of code running in production at this very moment. If you wrote an onerous engineering spec; there would not be enough programmers or lawyers on earth to rewrite and verify it all, even if given decades. This would result in Google, Apple, etc. basically getting grandfathered in while startups get the burden of following the rules - rules that China, India, and other countries happily won’t be enforcing.)

figassis 2y ago
Physics will still be the same when your faulty software tells an airplane to dive.
2OEH8eoCRo0 2y ago
I'd imagine we wouldn't have ActiveX controls in the first place.
- gjsman-1000 2y ago
  Wishful thinking - the IRS is still running on COBOL; our nuclear weapons until a few years ago on Windows 95. The NYC subway still has a lot of OS/2. Standardization does not stop bad engineering. Those who think it does have not witnessed the catastrophe a bad standard can cause. Go download and implement the Microsoft Office OOXML standard - it’s freely available, ISO approved, 6000 pages, and an abomination that not even Google claims to have correctly implemented.
  - 2OEH8eoCRo0 2y ago
    You're making some points for me. You are assuming COBOL, Windows 95, or OS/2 are bad because they're old. Such assumptions are the antithesis of "engineering."
    - gjsman-1000 2y ago
      Old technology isn’t necessarily bad in itself. It’s well documented and understood. Where it’s bad is when the equipment to run that software no longer is manufactured. You can’t get a new computer to run Windows 95. Not even in the military. Your only option is to virtualize, adding a huge possible failure mode that was never considered previously. Where it’s bad is when changes are needed to adapt to modern environments, and nobody’s quite sure about what they are doing anymore. There’s no test suite, never was, the documentation is full of ancient and confusing terminology, mistakes are made. And on and on…
    - jfengel 2y ago

2OEH8eoCRo0 2y ago

> Is there any path for software engineers to reach this level of accountability and norms of good practice? There is no reason that software couldn't be treated with the same care and respect. The only reason we don't is because the industry resists that sort of change. They want to move fast and break things while still calling themselves "engineers." Almost none of this resembles engineering.

wizzwizz4 2y ago
> Software engineering, of course, presents itself as another worthy cause, but that is eyewash: if you carefully read its literature and analyse what its devotees actually do, you will discover that software engineering has accepted as its charter "How to program if you cannot.". — Edsger Wybe Dijkstra, 1988. (EWD1036)
- jfengel 2y ago
  I'm ok with that. I don't want to keep everyone out except just those who happen to have just the right mind set. Programming is about developing software for people, and the more viewpoints are in the room, the better. Some pieces are more important than others. Those are the bits that need to be carefully regulated, as if they were bridges. But not everything we build has lives on the line. If that means we don't get to call ourselves "engineers", I'm good with that. We work with bits, not atoms, and we can develop our own new way of handling that.
  - 2OEH8eoCRo0 2y ago
    I prefer to call it "computer programming." If the title is good enough for Ken Thompson or Don Knuth then it's good enough for me.
  - wizzwizz4 2y ago
    > I don't want to keep everyone out except just those who happen to have just the right mind set. Neither do I. Neither did Dijkstra. EWD1036, “On the cruelty of really teaching computing science”, is about education reform, to enable those who don't "happen to have just the right mind set" to fully participate in actual, effective programming.

belter 2y ago

https://zlk.com/pslra-1/crowdstrike-lawsuit-submission-form?prid=94963&wire=1 https://zlk.com/pslra-1/crowdstrike-lawsuit-submission-form?... https://www.sauderschelkopf.com/wp-content/uploads/2024/07/AS-FILED-Delta-Complaint-8.6.24.pdf https://www.sauderschelkopf.com/wp-content/uploads/2024/07/A...

MattGaiser 2y ago

> Are the licenses so ironclad that customers have no recourse? Even on Hacker News, there was agreement that CrowdStrike screwed up, but then people also blamed IT staff, Microsoft (even after realizing it was a CrowdStrike issue), and the EU/regulators. I imagine responsibility of each entity would need far more clarification than it does now. If you want to define liability, there needs to be a clear line saying who is responsible for what. That doesn’t currently exist in software. There are also considering how people respond to risk. Consider how sesame regulation led to most bread having sesame deliberately put into it. Industry responded by guaranteeing contamination. Crowdstrike and endpoint security firms might respond by saying that only Windows and Mac devices can be secured. Or Microsoft may say that only their solution can provide the requisite security.

miki123211 2y ago

Civil engineering mostly requires you to have a government-verified certificate and to work in the country your infrastructure will be deployed in. Software engineering doesn't, and that makes criminal prosecutions that much harder. There's no path to making it happen. Financial liability for the company in question? Sure, that's probably doable. "Piercing the corporate veil" and punishing the executives who signed off on it? Harder but not impossible. Punishing the engineer who wrote that code, and who lives in a country with no such laws? Won't happen.

WarOnPrivacy 2y ago
> Civil engineering mostly requires you to have a government-verified certificate and to work in the country your infrastructure will be deployed in. It's a relatively small (and sharply defined) pool of people who can be called a civil engineer. Are we saying we want to segment software engineering (from coding) - the same way civil engineering is segmented from construction? Otherwise we're talking about placing specialist liability upon a non-specialist group. This seems unethical.

JumpCrisscross 2y ago

> If a bridge collapses, there’s not only financial liability but the potential for criminal liability as well If a bridge collapses people die. To my knowledge, nobody died or was put in mortal peril as a result of the Crowdstrike debacle.

rightbyte 2y ago
The deaths if any where probably indirect. E.g. ambulances not turning up in time etc. due to paper and pen fallbacks. With all the hospitals victim of the attack, I would be surprised if the amount of patients that died are zero.
- JumpCrisscross 2y ago
  > E.g. ambulances not turning up in time etc. due to paper and pen fallbacks Sure. Did this happen? Why were the “emergency management downtime procedures” insufficient [1]? [1] https://www.healthcaredive.com/news/crowdstrike-outage-hits-us-hospitals/721887/ https://www.healthcaredive.com/news/crowdstrike-outage-hits-...
  - tgsovlerkhgsel 2y ago
    If they were equally good as the non-emergency procedures, why wouldn't we use them all the time?
    - JumpCrisscross 2y ago
      > why wouldn't we use them all the time? Because they’re more expensive. They’re all not “equally good,” they’re good enough to keep people alive. (You repurpose resources from elective and billing procedures, et cetera.)
      - FireBeyond 2y ago
        This belies a lack of understanding. What resources are you repurposing from elective procedures exactly? Your patient load hasn’t changed, and day surgical instruments and supplies are from the same pool. There’s no “well this pile of equipment is only for elective procedures”. I’m not even sure what “billing procedures you’d repurpose (especially in your context of “keeping people alive”).

LordKeren 2y ago

There is recourse, just not for normal people, as you eluded to. Companies are and will be continuing to sue crowdstrike, and based on the papers that crowdstrike has posted, the impacted companies are extremely likely to be successful. It seems overwhelmingly likely that the companies are going to be able to convince a judge/jury/arbiter that crowdstrike acted grossly negligent and very plainly caused both direct losses and indirect reputational harm to the companies. I’m not sure crowdstrike will even fight it, to be honest. I would assume most of this is going to be settled out of court and we will see crowdstrike crumble in the coming years.

JumpCrisscross 2y ago
> not sure crowdstrike will even fight it To my knowledge only Delta is suing and CrowdStrike is kicking and screaming about it [1]. [1] https://www.cnn.com/2024/08/05/business/crowdstrike-fires-back-at-delta/index.html https://www.cnn.com/2024/08/05/business/crowdstrike-fires-ba...
- LordKeren 2y ago
  It’s a really bad look for crowdstrike to be going down this route. Then again, I don’t think many companies are going to be adopting crowdstrike in the coming years, so I suppose their only option is to defend their stock value at any cost while the company recoils
dredmorbius 2y ago
NB: alluded, not eluded. <https://dict.org/bin/Dict?Form=Dict2&Database=gcide&Query=Allude https://dict.org/bin/Dict?Form=Dict2&Database=gcide&Query=Al...> <https://dict.org/bin/Dict?Form=Dict2&Database=gcide&Query=elude https://dict.org/bin/Dict?Form=Dict2&Database=gcide&Query=el...>
- LordKeren 2y ago
  TIL— thanks! Now it’s time to painfully go through my slack / email history and see how many times I made this mistake :)

jedberg 2y ago

Delta threatened to sue them for their $500M loss. Crowdstrike replied (publicaly) pointing out that their contract limits Crowdstrike's liability to single digit millions. Then then gave them a list of things they would seek in discovery, such as their backup plans, failover plans, testing schedules and results, when their last backup recover exercise was, etc. Basically, they said, "if you sue us, we will dig so deep into your IT practices that it will be more embarrassing for you than us and show that you were at fault".

JumpCrisscross 2y ago
> they said, "if you sue us, we will dig so deep into your IT practices that it will be more embarrassing for you than us and show that you were at fault" But CrowdStrike said this publicly. If they’d privately relayed it to Delta, it would have been genuine. By performatively relaying it, however, it seems they’re pre-managing optics around the expected suit.
- gnfargbl 2y ago
  Or attempting to discourage it from becoming a pile-on.
- magic_man 2y ago
  It doesn't matter it was 100% crowdstrikes fault. Surprised its still worth 60billion dollars.
  - from-nibly 2y ago
    Part of the problem is assuming you can pay a contract to shift your liability completely away.
    - iwontberude 2y ago
      Well if MSFT knew how to write MSAs Crowdstrike would have become property of Microsoft.
    - 2y ago

minkles 2y ago

Engineering safety culture is built on piles of bodies and suffering unfortunately. I suspect in software the price of failure is mostly low enough that this motivation will never develop.

xyst 2y ago

Probably in a decade or so after the AI crash. I have yet to see anything that comes close to “liability” for the digital realm. US governments and businesses get hacked/infiltrated all the time by foreign adversaries yet we do not declare war. Maybe something happens in the dark or back channels. But we never know.

dylan604 2y ago

A lot of companies have insurance on events causing them to lose sources of income. Whether that's farmers having crop insurance, big box retailers having insurance for catastrophic damage to their big box, I would assume there's something for infrastructure collapse to bring sales to $0 for the duration. Even if everyone that was affected sued ClownStrike for 100% of their losses, it's not like ClownStrike has the revenue to cover those losses. So even if you're a fan of shutting them down, nobody recovers anything close to actual losses. So what would you actually propose? Bug free code is pretty much impossible. Some risk is accepted by the user. Do you seriously think that software should be absolutely 100% bug free before being able to be used? How do you prove that? Of course, the follow up would be how clean is your code that you feel that's even achievable?

Rinzler89 2y ago
>Bug free code is pretty much impossible. Some risk is accepted by the user. This wasn't your average SW bug, it was gross negligence on behalf of Crowdstreike, who seems to not have heard of SW testing on actual systems and canary deployment. Big difference. Yeah SW bugs happen all the time but you have to show you took some steps to prevent them, while some dev at Crowdstrike just said "whatever, it works on my machine" and directly pushed to all customer production systems on a Friday. That's the definition of gross negligence that they didn't have any processes in place to prevent something like this. That's like a surgeon not bothering to sterilize his hands and then saying "oh well, hospital infections happen all the time".
- ncr100 2y ago
  The bug was egregious. Using regexp (edit: in the kernel). (Wtf. It's a bloody language.) And not sanitizing the usage. Then using it differently than testing. And boom. There's people, and there's companies. This company ought to be nuked.
  - cqqxo4zV46cp 2y ago
    Genuinely, what good does that do? It’s all well and good to write dramatic meaningless comments on social networks like Hacker News, but if your desired had actual consequence, can you honestly say that “nuking the company” is a net positive?

ttymck 2y ago

> but so far not much in the way of lawsuits It hasn't been that long? The situation might be that there hasn't been sufficient time to yet gather evidence to commence lawsuits.

cesarb 2y ago

> Is there any path for software engineers to reach this level of accountability and norms of good practice? Yes, time. Civil engineering has thousands of years of history. Software engineering is much newer, the foundations of our craft are still in flux. There have been, at least in my country, legislative proposals for licensure of system analysts, electronic computer programmers, data processing machine operators, and typists(!) since the late 1970s; these laws, if approved, would have set back the progress of software development in my country for several decades (for instance, one proposal would make "manipulation and operation of electronic processing devices or machines, including terminals (digital or visual)" exclusive to those licensed as "data processing machine operator").

Cpoll 2y ago
> set back the progress > exclusive to those licensed Sounds to me like it just would've made a lot of money for whatever entities give out the licenses. On the other hand, I've read speculation on here that some countries are short on entrepreneurs entirely due to the difficulty of incorporating a small business, so maybe.

tsujamin 2y ago

I’m interested in what those who suffered outages as a result of crowdstrike told their insurers with respect to “QA’ing production changes” It’d be interesting to see if anyone tries to claim the outage as some sort of insurance event only to lose out because they let Crowdstrike roll updates into a highly regulated environment without testing

crote 2y ago

> Civil engineering students have it drilled into their heads that if they behave unethically or otherwise take unacceptable risks as an engineer they face jail time for it. Is there any path for software engineers to reach this level of accountability and norms of good practice? The problem is that with civil engineering you're designing a physical product. Nothing is ever designed to its absolute limit, and everything is built with a healthy safety margin. You calculate a bridge to carry bumper-to-bumper freight traffic, during a hurricane, when an earthquake hits - and then add 20%. Not entirely sure about whether a beam can handle it? Just size it up! Suddenly it's a lot less critical for your calculations to be exactly accurate - if you're off by 0.5% it just doesn't matter. You made a typo on the design documents? The builder will ask for clarification if you're trying to fit a 150ft beam into a 15.0ft gap. This means a bridge collapse is pretty much guaranteed to be the result of gross negligence. Contrast that to programming. A single "<" instead of "<=" could be the difference between totally fine and billions of dollars of damages. There isn't a single programmer on Earth who could write a 100% bug-free application of nontrivial complexity. Even the seL4 microkernel - whose whole unique selling point is the fact that it has a formal correctness proof - contains bugs! Compilers and proof checkers aren't going to complain if you ask them to do something which is obviously the wrong thing but technically possible. No sane person would accept essentially unlimited liability over even the smallest mistakes. If we want software engineers to have accountability, we first have to find a way to separate innocent run-of-the-mill mistakes from gross negligence - and that's going to be extremely hard to formalize.

Analemma_ 2y ago
This entire comment boils down to "we can't be held accountable because it's soooo hard you guys", which isn't even convincing to me as someone in the industry and certainly won't be to someone outside it.
- heyoni 2y ago
  What a shallow dismissal of a comment that doesn’t even claim that there shouldn’t be accountability.

wannacboatmovie 2y ago

> Is there any path for software engineers to reach this level of accountability and norms Potentially controversial stance here, but most software engineers are not engineers. They study computer science, which doesn't include coursework on engineering ethics among other things. I would say that by design they are less prepared to make ethical decisions and take conservative approaches. Imagine if civil engineers had EULAs for their products. "This bridge has no warranty, implied or otherwise. Cross this bridge AT YOUR OWN RISK. This bridge shall not be used for anything safety critical etc."

lupire 2y ago

How many bridges, would you say, does the average civil engineering firm deliver each year, each on only 1 day notice, in response to a surprise change in requirements due to a newly developed adversarial attack? Crowdstrike does this constantly. You could demand the same level of assurance from software, but in exchange, you don't get to fly, because the capacity won't be there

dakiol 2y ago

> I appreciate that we’re finding the humour in this catastrophe but what about the question of liability? One of the biggest and most used piece of software (the Linux kernel) comes with zero warranties. It can fail, and no one would be liable. Are we fine with that? Is the CS case different because it costs money? From an user perspective we don’t want software failing in the middle of an airplane landing, so whether the software comes from CS or github, it’s of lesser importance.

guax 2y ago

I would find it more useful if liability here we're attributed to the need to purchase such draconian tools. Certifications that require it and C levels who approve it. We would be better by it.

cqqxo4zV46cp 2y ago
Oh Christ. Just drop it. A by all accounts legitimate security function of a product targeted at company-owned endpoints. Please don’t devolve this conversation into you being upset about not getting admin rights on your work computer or whatever this is about. Any (esp. larger) org would be criminally negligent to eschew using something like CrowdStrike in order to capitulate to some nerd that thinks that they have ownership over their work equipment.
- guax 2y ago
  Bruh, chill. I don't give two craps about having admin rights on my work computer. Crowdstrike is bad software and bad way to manage large deployments. They just proved it. I just think that we are also responsible for buying a solution that works that way.

maccard 2y ago

I completely agree. When I've negotiated contracts for my workplace, and we explicitly write in the contract that the vendor is responsible for XYZ, it is my understanding (and confirmed by legal, multiple times) that this means in case of XYZ going wrong, they are liable for up to the amount in the SLA, however that isn't a cap on liability in extenuating circumstances. If this all gets brushed away, it significantly devalues the "well we pay $VENDOR to manage our user data, it's on them if they store it incorrectly" proposition, which would absolutely cause us to renegotiate.

cqqxo4zV46cp 2y ago
You aren’t showing us the specific language that you’re referring to, nor do we know what a typical CrowdStrike contract looks like. You could be talking about apples and oranges here. I’ve seen both.
- maccard 2y ago
  I was pretty sure that someone was going to "ackshually" me here, and here we are. The specific wording doesn't matter. I've negotioated dozens of these contracts and the value add of a vendor managing the data is liability. If they aren't liable for data mis-management, then their managed service is only worth the infra costs + a haircut on top, and we'll renegotiate with this in mind.

wcunning 2y ago

Software (controls) engineers at VW during the emissions scandal went to jail, engineers at GM were held liable for the ignition switch issue (not mostly in software, but still). I expect we'll eventually see some engineers/low level managers thrown under the bus with Boeing. It definitely happens, but not as frequently as it could. That said, I definitely prefer Amazon's response to the AWS East 1 outage back in 2016 -- the engineer wasn't blamed, despite the relatively simple screw up, but the processes/procedures were fixed so that it didn't happen again in the last 8 years. Crowdstrike is a little bit gray on that regard -- people should have known how bad the practice of zero testing on config updates was, but then again, I've seen some investigating saying that the initial description wasn't fully accurate, so I'm waiting for the final community after action report before I really pass judgement.

deleted 2y ago
[deleted]

ezoe 2y ago

They totally deserved it. Those who think running third-party closed-source Windows kernel driver(which parse files distributed from Internet in realtime) are good for the security, they must also accept the consequence. I'm sick of these so-called security consultants who always insist check lists like installing proprietary close-source binary blob Linux kernel module to the system consists of otherwise mostly free softwares except for hardware drivers and think they did their job, or executives who pays a lot of money to these idiotic so-called security consultants.

oglop 2y ago

On this website you are asking a population that would be responsible for this, so you will likely only get answers about how hard this is to solve and how it’s not software engineers fault and how we need to understand software engineering is not civil engineering and we need to be careful with this analogy and how it’s not our fault! Don’t blame us when things go wrong, but also, give us all the money when things go right. This is not the place for this question is what I’m saying.

barelysapient 2y ago

Your barber has more licensing requirements than a senior software engineer. Regulations have not caught up with developers (yet).