4 ms·

IMHO it would send really wrong signals if this doesn't end up with CrowdStrike closing their doors... like if the largest outage in history was caused by you

IMHO it would send really wrong signals if this doesn't end up with CrowdStrike closing their doors... like if the largest outage in history was caused by you due to a config parser failing and it looks as far as I can tell that they didn't follow industry best practices when it comes to config/parsing handling and probably also didn't follow some best practices when it comes to kernel module programming then honestly it would be really strange if you didn't had to declare bankruptcy due to damage payments (which doesn't mean the software is now gone/unmaintained, there are a lot of ways to make sure that doesn't happen, e.g. MS anyway had interest in buying Falcon).

JumpCrisscross 2y ago
> it would send really wrong signals if this doesn't end up with CrowdStrike closing their doors I thought the same until I saw the damage estimates. They’re in the single-digit billions. That’s well below CrowdStrike’s market cap. Unless we’re going hard for retributive justice, liability should be enough.
- saati 2y ago
  That's only a low-ball for fortune 500, it's a small part of the damage they done.
saati 2y ago
They didn't follow testing or deployment best practices either.
- tjpnz 2y ago
  IIRC their QA team was impacted by the most recent round of layoffs. Dumping those responsibilities onto devs isn't a great solution to begin with, and especially not when the product is a complete trash fire.
  - dathinab 2y ago
    yeah you really shouldn't layoff a QA team if you do something like that that's just pure negligence (through yo have to make sure you QA team works properly, i.e. in tandem with your dev not in fight with them, to many QA teams are a mess, but QA is important)
627467 2y ago
I understand that CS doesn't draw much sympathies even before this happened - from myself included - and it is easy to pin point systemic issues to a single failure point and make it liable and financially responsible for all downstream failures. But this only creates excuses for all other responsible players in this systemic issue - or society at large. Just to pick 1 example: I keep reading comments on how profoundly health care providers were affected and that lead to human life losses. I understand that having "tech" involved in health or any sector is important but are we really wanting to build critical services that grind to halt or have huge efficiency impact when a single vendor fails? Are these service providers not responsible for thinking about failure modes?
- HappySweeney 2y ago
  I read that healthcare was mollified by a facility whereby they could choose when and which updates to allow, which CS intentionally bypassed with this latest update. This was during the chaotic day-of though, so possibly false.
  - account42 2y ago
    There can always be lingering bugs that trigger at a certain time on all systems or for certain input data so the only way to have redundancy is to have actually heterogenous systems.
- adrianN 2y ago
  In the end it's a matter of cost. Having two independent supply chains for anything as complex as healthcare infrastructure is essentially unaffordable.
  - account42 2y ago
    > essentially unaffordable [citation needed]
  - 627467 2y ago
    Can't help but think that from this perspective buying a solution from a vendor is less about solving a problem and more about buying a blame-insurance in case something comes crashing down.
jb3689 2y ago
I don't understand why there is so much attention on the deployment and testing side of the coin. Yes, better testing and rollout strategy should have prevented this specific occurrence of a failure. But these strategies aren't bulletproof and things go wrong. You need defense in depth, and some responsibility has to lay on the consumer side for that to happen - particularly for fundamental humane industries like transportation and healthcare. These industries should not be allowed to run any software like this - privileged and without controlled rollouts. I'm all for shaming CrowdStrike's lack of focus on reliability, which they deserve, however there's a bigger issue here of trying to avoid or mitigate risky dependencies in the first place that I hope we also get to explore.
fennecbutt 2y ago
So no responsibility on their clients having a uniform architecture where all key systems use CS software for protection, updates are auto applied and there are no backup systems etc? If aws goes down in a region yeah it sucks but we fail over. If aws goes down ww then it's like well...sometimes that happens. If I've built critical infra like electric utilities, airports etc with swathing vulnerable points then that's the real problem.

JumpCrisscross 2y ago

> it would send really wrong signals if this doesn't end up with CrowdStrike closing their doors I thought the same until I saw the damage estimates. They’re in the single-digit billions. That’s well below CrowdStrike’s market cap. Unless we’re going hard for retributive justice, liability should be enough.

saati 2y ago
That's only a low-ball for fortune 500, it's a small part of the damage they done.

saati 2y ago

They didn't follow testing or deployment best practices either.

tjpnz 2y ago
IIRC their QA team was impacted by the most recent round of layoffs. Dumping those responsibilities onto devs isn't a great solution to begin with, and especially not when the product is a complete trash fire.
- dathinab 2y ago
  yeah you really shouldn't layoff a QA team if you do something like that that's just pure negligence (through yo have to make sure you QA team works properly, i.e. in tandem with your dev not in fight with them, to many QA teams are a mess, but QA is important)

627467 2y ago

I understand that CS doesn't draw much sympathies even before this happened - from myself included - and it is easy to pin point systemic issues to a single failure point and make it liable and financially responsible for all downstream failures. But this only creates excuses for all other responsible players in this systemic issue - or society at large. Just to pick 1 example: I keep reading comments on how profoundly health care providers were affected and that lead to human life losses. I understand that having "tech" involved in health or any sector is important but are we really wanting to build critical services that grind to halt or have huge efficiency impact when a single vendor fails? Are these service providers not responsible for thinking about failure modes?

HappySweeney 2y ago
I read that healthcare was mollified by a facility whereby they could choose when and which updates to allow, which CS intentionally bypassed with this latest update. This was during the chaotic day-of though, so possibly false.
- account42 2y ago
  There can always be lingering bugs that trigger at a certain time on all systems or for certain input data so the only way to have redundancy is to have actually heterogenous systems.
adrianN 2y ago
In the end it's a matter of cost. Having two independent supply chains for anything as complex as healthcare infrastructure is essentially unaffordable.
- account42 2y ago
  > essentially unaffordable [citation needed]
- 627467 2y ago
  Can't help but think that from this perspective buying a solution from a vendor is less about solving a problem and more about buying a blame-insurance in case something comes crashing down.

jb3689 2y ago

I don't understand why there is so much attention on the deployment and testing side of the coin. Yes, better testing and rollout strategy should have prevented this specific occurrence of a failure. But these strategies aren't bulletproof and things go wrong. You need defense in depth, and some responsibility has to lay on the consumer side for that to happen - particularly for fundamental humane industries like transportation and healthcare. These industries should not be allowed to run any software like this - privileged and without controlled rollouts. I'm all for shaming CrowdStrike's lack of focus on reliability, which they deserve, however there's a bigger issue here of trying to avoid or mitigate risky dependencies in the first place that I hope we also get to explore.

fennecbutt 2y ago

So no responsibility on their clients having a uniform architecture where all key systems use CS software for protection, updates are auto applied and there are no backup systems etc? If aws goes down in a region yeah it sucks but we fail over. If aws goes down ww then it's like well...sometimes that happens. If I've built critical infra like electric utilities, airports etc with swathing vulnerable points then that's the real problem.