4 ms·

I wrote this in another thread already, but the fuck up was both at crowdstrike (they borked a release) but also and more importantly their customers. Shit happ

I wrote this in another thread already, but the fuck up was both at crowdstrike (they borked a release) but also and more importantly their customers. Shit happens even with the best testing in the world. You do not deploy anything, ever on your entire production fleet at the same time and you do not buy software that does that. It's madness and we're not talking about small companies with tiny IT departments here.

perbu 2y ago
Shit might happen with the best testing, but with decent testing it would not be this serious.
wazzaps 2y ago
Apparently CrowdStrike bypassed clients' staging areas with this update. Source: https://x.com/patrickwardle/status/1814367918425079934 https://x.com/patrickwardle/status/1814367918425079934
owl57 2y ago
> you do not buy software that does that Note how the incident disproportionally affected highly regulated industries, where businesses don't have a choice to screw "best practice".
- TeMPOraL 2y ago
  Only highlighting that "best practice" of cybersecurity is, charitably, total bullshit; less charitably, a racket. This is apparent if you look at the costs to the day-to-day ability of employees to do work, but maybe it'll be more apparent now that people got killed because of it.
  - badgersnake 2y ago
    It’s absolutely a racket.
d1sxeyes 2y ago
That’s a tricky one. CrowdStrike is cybersecurity. Wait until the first customer complains that they were hit by WannaCry v2 because CrowdStrike wanted to wait a few days after they updated a canary fleet. The problem here is that this type of update (a content update) should never be able to cause this however badly it goes. In case the software receives a bad content update, it should fail back to the last known good content update (potentially with a warning fired off to CS, the user, or someone else about the failed update). In principle, updates that could go wrong and cause this kind of issue should absolutely be deployed slowly, but per my understanding, that’s already the practice for non-content updates at CrowdStrike.
- chronid 2y ago
  Windows updates are also cybersecurity, but the customer has (had?) a choice to how to roll those out (with Intune nowadays?). The customer should decide when to update, they own the fleet not the vendor! You do not know if a content update will screw you over and mark all the files of your company as malware. The "It should never happen" situations are the thing you need to prepare for, the reason we talk about security as an onion, the reason we still do staggered production releases with baking times even after tests and QA have passed... "But it's cybersecurity" is not a justification. I know that security departments and IT departments and companies in general love dropping the "responsibility" part on someone else, but in the end of the day the thing getting screwed over is the company fleet. You should retain control and make sure things work properly, the fact those billion dollar revenue companies are unable to do so is a joke. A terrible one, since IT underpins everything nowadays.
  - chrisjj 2y ago
    > The customer should decide when to update, they own the fleet not the vendor! The CS customer has decided to update whenever 24/7 CS says. The alternative is to arrive on Monday morning to an infected fleet.
    - chronid 2y ago
      Sorry, this is untrue. Enterprises have SOCs and oncalls, if there is a high risk they can do at least minimal testing (which would have found this issue as it has a 100% bsod rate) and then fleet rollout. It would have been rolled out by Friday evening in this case without crashing hundred of thousands of servers. The CS customer has decided to offload the responsibility of its fleet to CS. In my opinion that's bullshit and negligence (it doesn't mean I don't understand why they did it), particularly at the scale of some of the customers :)
stef25 2y ago
You'd think that the software would sit in a kind of sandbox so that it couldn't nuke the whole device but only itself. It's crazy that this is possible.
- echoangle 2y ago
  The software basically works as a kernel module as far as I understand, I don’t think there’s a good way to separate that from the OS while still allowing it to have the capabilities it needs to have to surveil all other processes.
  - temac 2y ago
    Something like ebpf.
  - layer8 2y ago
    And even then, you wouldn’t want the system to continue running if the security software crashes. Such a crash might indicate a successful security breach.
KaiserPro 2y ago
> You do not deploy anything, ever on your entire production fleet at the same time and you do not buy software that does that I am sympathetic to that, but its only possible if both policy and staffing allow. for policy, there are lots of places that demand CVEs be patched within x hours depending on severity. A lot of times, that policy comes from the payment integration systems provider/third party. However you are also dependent on programs you install not autoupdating. Now, most have an option to flip that off, but its not always 100% effective.
- chronid 2y ago
  > I am sympathetic to that, but its only possible if both policy and staffing allow. We are not talking about small companies here. We're talking about massive billion revenue enterprises with enormous IT teams and in some cases multiple NOCs and SOCs and probably thousands consultants all around at minimum. I find it hard to be sympathetic to this complete disregard of ownership just to ship responsibility somewhere else (because this is the need at the of the day let's not joke around). I can understand it, sure, and I can believe - to a point - someone did a risk calculation (possibility of crowdstrike upgrade killing all systems vs hack if we don't patch a CVE in <4h), but it's still madness from a reliability standpoint. > for policy, there are lots of places that demand CVEs be patched within x hours depending on severity. I'm pretty sure leadership when they need to choose between production being down for an unspecified amount of time and taking the risk of delaying (of hours in this case) the patching will choose the delay. Partners and payment integration providers can be reasoned with, contracts are not code. A BSOD you cannot talk away. Sure, leadership is also now saying "but we were doing the same thing as everyone else, the consultants told us to and how could have we have known this random software with root on every machine we own could kill us?!" to cover their asses. The problem is solved already, since it impacted everyone, and they're not the ones spending their weekend hammering systems back to life. > However you are also dependent on programs you install not autoupdating. Now, most have an option to flip that off, but its not always 100% effective. You choose what to install on your systems, and you have the option to refuse to engage with companies that don't provide such options. If you don't, you accept the risk.
sateesh 2y ago
Disagree with the part where you put onus on customer. As has been mentioned in other HN thread [1], this update was pushed ignoring whatever the settings customer had configured. The original mistake of the customer, if any, was they didn't read this in fine print of the contract (if this point about updates was explicitly mentioned in the contract). 1. https://news.ycombinator.com/item?id=41003390 https://news.ycombinator.com/item?id=41003390
chrisjj 2y ago
> You do not deploy anything, ever on your entire production fleet at the same time And if an attacker does??

perbu 2y ago

Shit might happen with the best testing, but with decent testing it would not be this serious.

wazzaps 2y ago

Apparently CrowdStrike bypassed clients' staging areas with this update. Source: https://x.com/patrickwardle/status/1814367918425079934 https://x.com/patrickwardle/status/1814367918425079934

owl57 2y ago

> you do not buy software that does that Note how the incident disproportionally affected highly regulated industries, where businesses don't have a choice to screw "best practice".

TeMPOraL 2y ago
Only highlighting that "best practice" of cybersecurity is, charitably, total bullshit; less charitably, a racket. This is apparent if you look at the costs to the day-to-day ability of employees to do work, but maybe it'll be more apparent now that people got killed because of it.
- badgersnake 2y ago
  It’s absolutely a racket.

d1sxeyes 2y ago

That’s a tricky one. CrowdStrike is cybersecurity. Wait until the first customer complains that they were hit by WannaCry v2 because CrowdStrike wanted to wait a few days after they updated a canary fleet. The problem here is that this type of update (a content update) should never be able to cause this however badly it goes. In case the software receives a bad content update, it should fail back to the last known good content update (potentially with a warning fired off to CS, the user, or someone else about the failed update). In principle, updates that could go wrong and cause this kind of issue should absolutely be deployed slowly, but per my understanding, that’s already the practice for non-content updates at CrowdStrike.

chronid 2y ago
Windows updates are also cybersecurity, but the customer has (had?) a choice to how to roll those out (with Intune nowadays?). The customer should decide when to update, they own the fleet not the vendor! You do not know if a content update will screw you over and mark all the files of your company as malware. The "It should never happen" situations are the thing you need to prepare for, the reason we talk about security as an onion, the reason we still do staggered production releases with baking times even after tests and QA have passed... "But it's cybersecurity" is not a justification. I know that security departments and IT departments and companies in general love dropping the "responsibility" part on someone else, but in the end of the day the thing getting screwed over is the company fleet. You should retain control and make sure things work properly, the fact those billion dollar revenue companies are unable to do so is a joke. A terrible one, since IT underpins everything nowadays.
- chrisjj 2y ago
  > The customer should decide when to update, they own the fleet not the vendor! The CS customer has decided to update whenever 24/7 CS says. The alternative is to arrive on Monday morning to an infected fleet.
  - chronid 2y ago
    Sorry, this is untrue. Enterprises have SOCs and oncalls, if there is a high risk they can do at least minimal testing (which would have found this issue as it has a 100% bsod rate) and then fleet rollout. It would have been rolled out by Friday evening in this case without crashing hundred of thousands of servers. The CS customer has decided to offload the responsibility of its fleet to CS. In my opinion that's bullshit and negligence (it doesn't mean I don't understand why they did it), particularly at the scale of some of the customers :)

stef25 2y ago

You'd think that the software would sit in a kind of sandbox so that it couldn't nuke the whole device but only itself. It's crazy that this is possible.

echoangle 2y ago
The software basically works as a kernel module as far as I understand, I don’t think there’s a good way to separate that from the OS while still allowing it to have the capabilities it needs to have to surveil all other processes.
- temac 2y ago
  Something like ebpf.
- layer8 2y ago
  And even then, you wouldn’t want the system to continue running if the security software crashes. Such a crash might indicate a successful security breach.

KaiserPro 2y ago

> You do not deploy anything, ever on your entire production fleet at the same time and you do not buy software that does that I am sympathetic to that, but its only possible if both policy and staffing allow. for policy, there are lots of places that demand CVEs be patched within x hours depending on severity. A lot of times, that policy comes from the payment integration systems provider/third party. However you are also dependent on programs you install not autoupdating. Now, most have an option to flip that off, but its not always 100% effective.

chronid 2y ago
> I am sympathetic to that, but its only possible if both policy and staffing allow. We are not talking about small companies here. We're talking about massive billion revenue enterprises with enormous IT teams and in some cases multiple NOCs and SOCs and probably thousands consultants all around at minimum. I find it hard to be sympathetic to this complete disregard of ownership just to ship responsibility somewhere else (because this is the need at the of the day let's not joke around). I can understand it, sure, and I can believe - to a point - someone did a risk calculation (possibility of crowdstrike upgrade killing all systems vs hack if we don't patch a CVE in <4h), but it's still madness from a reliability standpoint. > for policy, there are lots of places that demand CVEs be patched within x hours depending on severity. I'm pretty sure leadership when they need to choose between production being down for an unspecified amount of time and taking the risk of delaying (of hours in this case) the patching will choose the delay. Partners and payment integration providers can be reasoned with, contracts are not code. A BSOD you cannot talk away. Sure, leadership is also now saying "but we were doing the same thing as everyone else, the consultants told us to and how could have we have known this random software with root on every machine we own could kill us?!" to cover their asses. The problem is solved already, since it impacted everyone, and they're not the ones spending their weekend hammering systems back to life. > However you are also dependent on programs you install not autoupdating. Now, most have an option to flip that off, but its not always 100% effective. You choose what to install on your systems, and you have the option to refuse to engage with companies that don't provide such options. If you don't, you accept the risk.

sateesh 2y ago

Disagree with the part where you put onus on customer. As has been mentioned in other HN thread [1], this update was pushed ignoring whatever the settings customer had configured. The original mistake of the customer, if any, was they didn't read this in fine print of the contract (if this point about updates was explicitly mentioned in the contract). 1. https://news.ycombinator.com/item?id=41003390 https://news.ycombinator.com/item?id=41003390

chrisjj 2y ago

> You do not deploy anything, ever on your entire production fleet at the same time And if an attacker does??