4 ms·

> mistake Funny way of spelling ‘complete failure in functional testing before going to prod causing billions of dollars worth of damage likely due to criminal

> mistake Funny way of spelling ‘complete failure in functional testing before going to prod causing billions of dollars worth of damage likely due to criminal level negligence’.

tuatoru 2y ago
Billions is probably an underestimate. When you factor on the time spent in meetings discussing endlessly what to do about this in corporations around the world, besides the cancelled flights and banking and hospital and 911 mayhem, a trillion isn't indefensible. Who needs North Korean hackers?
- hsbauauvhabzb 2y ago
  If crowdstrikes internal competency failed to detect a consistent bsod, I doubt they’d notice North Korea dropping malware into their product solarwinds style.
Galanwe 2y ago
I mean, we've seen impacts of these unstaged updates in the deeper layers of banks, airports and hospitals. It's unfair that CrowdStrike gets more than half the blame here. Not to mention, probably half these companies were actually forced to install this. Getting SOC2 is essentially letting a senior-branded KPMG or EY intern tell you how to enforce security policies based on their checklist. There is far more to blame in this incident than CrowdStrike themselves.
- aiisjustanif 2y ago
  Compliance is one thing but a cybersecurity incident response team is very blind without an EDR not to mention you would have less protection from insider threat.
antonvs 2y ago
That's true. But it's also true that companies depending on more or less random third parties to provide them with kernel modules for a monolithic kernel - and then giving them control over auto-updating those kernel modules - is beyond dumb.
HeatrayEnjoyer 2y ago
Undoubtedly killing people too: 911 networks failed. Hospitals were crippled. Short of getting your hands on a nuclear football it's difficult to imagine an act with a higher effort:devastation ratio.
- RCitronsBroker 2y ago
  You’re not wrong about the end result, but the breakdown of systems this complex goes deeper than placing the blame on some CrowdStrike employee. Whoever thought up the great idea to allow auto-update-able kernel modules for something as mission critical as emergency response or healthcare deserves just as much blame. I’ve worked in healthcare for my whole career, this is madness. Not that their process is without flaw, but can we remind ourselves of how stringently we assess medical devices? I cannot imagine it’s controversial to say that emergency response equipment is every bit as critical as a insulin pump. If they fail, someone dies.
  - hsbauauvhabzb 2y ago
    But at the same time, auditing every update to an assurance level beyond ‘it didn’t bsod in test’ is incredibly hard. I don’t disagree with anything you’ve said, but I’d be very interested in solving the problem of actually auditing constant updates from vendors.
    - withinboredom 2y ago
      Even a rudimentary “delay autoupdate by two weeks” would have saved lives here. Let everyone else update first.
    - exe34 2y ago
      staged releases. don't cripple all your systems in one go. hot backups that you only update after the main system isn't dead from an update.
    - com 2y ago
      Automated CI/CD - many of us already do this hundreds of times a day. If you’re an emergency call centre, join a consortium of similar orgs and standardise tech and do it properly. Defer updates. Most things can wait 8-12 hours. Even more can wait 3 weeks (did this for all but security-critical npm package updates in one place). Demand legal changes to ensure fair liability for failure to undertake basic measures by service providers for paid software and services. Demand proper liability for C-suites not ensuring that actual risk management is in place instead of stupid box-ticking. Design better software. Seriously, the kinds of half-baked stuff that costs so much is incredible. It doesn’t take longer, and it doesn’t cost more to do things right, the only change is that management needs to be engaged with outcomes and have skin in the game. Execs should run the risk of going to jail for egregious failures.

> mistake Funny way of spelling ‘complete failure in functional testing before going to prod causing billions of dollars worth of damage likely due to criminal level negligence’.

tuatoru 2y ago

Billions is probably an underestimate. When you factor on the time spent in meetings discussing endlessly what to do about this in corporations around the world, besides the cancelled flights and banking and hospital and 911 mayhem, a trillion isn't indefensible. Who needs North Korean hackers?

hsbauauvhabzb 2y ago
If crowdstrikes internal competency failed to detect a consistent bsod, I doubt they’d notice North Korea dropping malware into their product solarwinds style.

Galanwe 2y ago

I mean, we've seen impacts of these unstaged updates in the deeper layers of banks, airports and hospitals. It's unfair that CrowdStrike gets more than half the blame here. Not to mention, probably half these companies were actually forced to install this. Getting SOC2 is essentially letting a senior-branded KPMG or EY intern tell you how to enforce security policies based on their checklist. There is far more to blame in this incident than CrowdStrike themselves.

aiisjustanif 2y ago
Compliance is one thing but a cybersecurity incident response team is very blind without an EDR not to mention you would have less protection from insider threat.

antonvs 2y ago

That's true. But it's also true that companies depending on more or less random third parties to provide them with kernel modules for a monolithic kernel - and then giving them control over auto-updating those kernel modules - is beyond dumb.

HeatrayEnjoyer 2y ago

Undoubtedly killing people too: 911 networks failed. Hospitals were crippled. Short of getting your hands on a nuclear football it's difficult to imagine an act with a higher effort:devastation ratio.

RCitronsBroker 2y ago
You’re not wrong about the end result, but the breakdown of systems this complex goes deeper than placing the blame on some CrowdStrike employee. Whoever thought up the great idea to allow auto-update-able kernel modules for something as mission critical as emergency response or healthcare deserves just as much blame. I’ve worked in healthcare for my whole career, this is madness. Not that their process is without flaw, but can we remind ourselves of how stringently we assess medical devices? I cannot imagine it’s controversial to say that emergency response equipment is every bit as critical as a insulin pump. If they fail, someone dies.
- hsbauauvhabzb 2y ago
  But at the same time, auditing every update to an assurance level beyond ‘it didn’t bsod in test’ is incredibly hard. I don’t disagree with anything you’ve said, but I’d be very interested in solving the problem of actually auditing constant updates from vendors.
  - withinboredom 2y ago
    Even a rudimentary “delay autoupdate by two weeks” would have saved lives here. Let everyone else update first.
  - exe34 2y ago
    staged releases. don't cripple all your systems in one go. hot backups that you only update after the main system isn't dead from an update.
  - com 2y ago
    Automated CI/CD - many of us already do this hundreds of times a day. If you’re an emergency call centre, join a consortium of similar orgs and standardise tech and do it properly. Defer updates. Most things can wait 8-12 hours. Even more can wait 3 weeks (did this for all but security-critical npm package updates in one place). Demand legal changes to ensure fair liability for failure to undertake basic measures by service providers for paid software and services. Demand proper liability for C-suites not ensuring that actual risk management is in place instead of stupid box-ticking. Design better software. Seriously, the kinds of half-baked stuff that costs so much is incredible. It doesn’t take longer, and it doesn’t cost more to do things right, the only change is that management needs to be engaged with outcomes and have skin in the game. Execs should run the risk of going to jail for egregious failures.