4 ms·

French here, and working for another french CSP. We lived the OVH incident live and saw the whole aftermath. OVH was held liable because of the data loss, not

French here, and working for another french CSP. We lived the OVH incident live and saw the whole aftermath. OVH was held liable because of the data loss, not for the service interruption. Data loss is something irremediable, permanent, definitive. Some businesses were basically ruined from this incident because they had no more data to operate. To add insult to injury, they sold offsite backups in the datacenter literally meters away. A service interruption, well, shit happens, and this is handled by SLA contracts that both parties agree to. You don't ruin a business (read: close a company) for a few days of outage. I doubt CrowdStrike will be held liable for much; from corporations at least. They cannot repay the damage done, or they close the door. The healthcare sector is another beast, but I think it will come to more regulations for critical entities.

altdataseller 2y ago
If a business closed down because of the OVH incident, how were the damages calculated? 1x annual revenue? Profit? 5x?
dathinab 2y ago
IMHO it would send really wrong signals if this doesn't end up with CrowdStrike closing their doors... like if the largest outage in history was caused by you due to a config parser failing and it looks as far as I can tell that they didn't follow industry best practices when it comes to config/parsing handling and probably also didn't follow some best practices when it comes to kernel module programming then honestly it would be really strange if you didn't had to declare bankruptcy due to damage payments (which doesn't mean the software is now gone/unmaintained, there are a lot of ways to make sure that doesn't happen, e.g. MS anyway had interest in buying Falcon).
- JumpCrisscross 2y ago
  > it would send really wrong signals if this doesn't end up with CrowdStrike closing their doors I thought the same until I saw the damage estimates. They’re in the single-digit billions. That’s well below CrowdStrike’s market cap. Unless we’re going hard for retributive justice, liability should be enough.
  - saati 2y ago
    That's only a low-ball for fortune 500, it's a small part of the damage they done.
- saati 2y ago
  They didn't follow testing or deployment best practices either.
  - tjpnz 2y ago
    IIRC their QA team was impacted by the most recent round of layoffs. Dumping those responsibilities onto devs isn't a great solution to begin with, and especially not when the product is a complete trash fire.
    - dathinab 2y ago
      yeah you really shouldn't layoff a QA team if you do something like that that's just pure negligence (through yo have to make sure you QA team works properly, i.e. in tandem with your dev not in fight with them, to many QA teams are a mess, but QA is important)
- 627467 2y ago
  I understand that CS doesn't draw much sympathies even before this happened - from myself included - and it is easy to pin point systemic issues to a single failure point and make it liable and financially responsible for all downstream failures. But this only creates excuses for all other responsible players in this systemic issue - or society at large. Just to pick 1 example: I keep reading comments on how profoundly health care providers were affected and that lead to human life losses. I understand that having "tech" involved in health or any sector is important but are we really wanting to build critical services that grind to halt or have huge efficiency impact when a single vendor fails? Are these service providers not responsible for thinking about failure modes?
  - HappySweeney 2y ago
    I read that healthcare was mollified by a facility whereby they could choose when and which updates to allow, which CS intentionally bypassed with this latest update. This was during the chaotic day-of though, so possibly false.
    - account42 2y ago
      There can always be lingering bugs that trigger at a certain time on all systems or for certain input data so the only way to have redundancy is to have actually heterogenous systems.
  - adrianN 2y ago
    In the end it's a matter of cost. Having two independent supply chains for anything as complex as healthcare infrastructure is essentially unaffordable.
    - account42 2y ago
      > essentially unaffordable [citation needed]
    - 627467 2y ago
      Can't help but think that from this perspective buying a solution from a vendor is less about solving a problem and more about buying a blame-insurance in case something comes crashing down.
- jb3689 2y ago
  I don't understand why there is so much attention on the deployment and testing side of the coin. Yes, better testing and rollout strategy should have prevented this specific occurrence of a failure. But these strategies aren't bulletproof and things go wrong. You need defense in depth, and some responsibility has to lay on the consumer side for that to happen - particularly for fundamental humane industries like transportation and healthcare. These industries should not be allowed to run any software like this - privileged and without controlled rollouts. I'm all for shaming CrowdStrike's lack of focus on reliability, which they deserve, however there's a bigger issue here of trying to avoid or mitigate risky dependencies in the first place that I hope we also get to explore.
- fennecbutt 2y ago
  So no responsibility on their clients having a uniform architecture where all key systems use CS software for protection, updates are auto applied and there are no backup systems etc? If aws goes down in a region yeah it sucks but we fail over. If aws goes down ww then it's like well...sometimes that happens. If I've built critical infra like electric utilities, airports etc with swathing vulnerable points then that's the real problem.
sam_goody 2y ago
As I understand it, anyone with bitlocker irretrievably lost all their data. Logic dictates that the more critical the data - the more likely it is gone. An informal anecdotal survey says that alot of users use bitlocker - which means a lot of data loss. EDIT: I see that in many cases one can recover Bitlocker encrypted drives. I wonder how much real data loss there is.

altdataseller 2y ago

If a business closed down because of the OVH incident, how were the damages calculated? 1x annual revenue? Profit? 5x?

dathinab 2y ago

IMHO it would send really wrong signals if this doesn't end up with CrowdStrike closing their doors... like if the largest outage in history was caused by you due to a config parser failing and it looks as far as I can tell that they didn't follow industry best practices when it comes to config/parsing handling and probably also didn't follow some best practices when it comes to kernel module programming then honestly it would be really strange if you didn't had to declare bankruptcy due to damage payments (which doesn't mean the software is now gone/unmaintained, there are a lot of ways to make sure that doesn't happen, e.g. MS anyway had interest in buying Falcon).

JumpCrisscross 2y ago
> it would send really wrong signals if this doesn't end up with CrowdStrike closing their doors I thought the same until I saw the damage estimates. They’re in the single-digit billions. That’s well below CrowdStrike’s market cap. Unless we’re going hard for retributive justice, liability should be enough.
- saati 2y ago
  That's only a low-ball for fortune 500, it's a small part of the damage they done.
saati 2y ago
They didn't follow testing or deployment best practices either.
- tjpnz 2y ago
  IIRC their QA team was impacted by the most recent round of layoffs. Dumping those responsibilities onto devs isn't a great solution to begin with, and especially not when the product is a complete trash fire.
  - dathinab 2y ago
    yeah you really shouldn't layoff a QA team if you do something like that that's just pure negligence (through yo have to make sure you QA team works properly, i.e. in tandem with your dev not in fight with them, to many QA teams are a mess, but QA is important)
627467 2y ago
I understand that CS doesn't draw much sympathies even before this happened - from myself included - and it is easy to pin point systemic issues to a single failure point and make it liable and financially responsible for all downstream failures. But this only creates excuses for all other responsible players in this systemic issue - or society at large. Just to pick 1 example: I keep reading comments on how profoundly health care providers were affected and that lead to human life losses. I understand that having "tech" involved in health or any sector is important but are we really wanting to build critical services that grind to halt or have huge efficiency impact when a single vendor fails? Are these service providers not responsible for thinking about failure modes?
- HappySweeney 2y ago
  I read that healthcare was mollified by a facility whereby they could choose when and which updates to allow, which CS intentionally bypassed with this latest update. This was during the chaotic day-of though, so possibly false.
  - account42 2y ago
    There can always be lingering bugs that trigger at a certain time on all systems or for certain input data so the only way to have redundancy is to have actually heterogenous systems.
- adrianN 2y ago
  In the end it's a matter of cost. Having two independent supply chains for anything as complex as healthcare infrastructure is essentially unaffordable.
  - account42 2y ago
    > essentially unaffordable [citation needed]
  - 627467 2y ago
    Can't help but think that from this perspective buying a solution from a vendor is less about solving a problem and more about buying a blame-insurance in case something comes crashing down.
jb3689 2y ago
I don't understand why there is so much attention on the deployment and testing side of the coin. Yes, better testing and rollout strategy should have prevented this specific occurrence of a failure. But these strategies aren't bulletproof and things go wrong. You need defense in depth, and some responsibility has to lay on the consumer side for that to happen - particularly for fundamental humane industries like transportation and healthcare. These industries should not be allowed to run any software like this - privileged and without controlled rollouts. I'm all for shaming CrowdStrike's lack of focus on reliability, which they deserve, however there's a bigger issue here of trying to avoid or mitigate risky dependencies in the first place that I hope we also get to explore.
fennecbutt 2y ago
So no responsibility on their clients having a uniform architecture where all key systems use CS software for protection, updates are auto applied and there are no backup systems etc? If aws goes down in a region yeah it sucks but we fail over. If aws goes down ww then it's like well...sometimes that happens. If I've built critical infra like electric utilities, airports etc with swathing vulnerable points then that's the real problem.

sam_goody 2y ago

As I understand it, anyone with bitlocker irretrievably lost all their data. Logic dictates that the more critical the data - the more likely it is gone. An informal anecdotal survey says that alot of users use bitlocker - which means a lot of data loss. EDIT: I see that in many cases one can recover Bitlocker encrypted drives. I wonder how much real data loss there is.