5 ms·

> I find it hard to believe that they didn't test it before deployment. I’m not sure why you find that hard to believe - based on the (admittedly fairly limite

by jmb99 2y ago

> I find it hard to believe that they didn't test it before deployment. I’m not sure why you find that hard to believe - based on the (admittedly fairly limited) evidence we have right now, it’s highly unlikely that this deployment was tested much, if at all. It seems much more likely to me that they were playing fast and loose with definition updates to meet some arbitrary SLAs[1] on zero-day prevention, and it finally caught up with them. Much more likely than somehow every single real-world pc running their software being affected but their test machines somehow all impervious. [1] When my company was considering getting into endpoint security and network anomaly detection, we were required on multiple occasions by multiple potential clients to provide a 4-hour SLA on a wide number of CVE types and severities. That would mean 24/7 on-call security engineers and a sub-4-hour definition creation and deployment. Yes, that 4 hours was for the deployment being available on 100% of the targets. Good luck writing and deploying a high-quality definition for a zero day in 4 hours, let alone running it through a test pipeline, let alone writing new tests to actually cover it. We very quickly noped out of the space, because that was considered “normal” (at least to the potential clients we were discussing). It wouldn’t shock me if CS was working in roughly the same way here.

drooopy 2y ago
This whole f*up was a failure of management and processes at Crowdstrike. "Intern Steve" pushing faulty code to production on a Friday is only a couple of cm of the tip of an enormous iceberg.
- chronid 2y ago
  I wrote this in another thread already, but the fuck up was both at crowdstrike (they borked a release) but also and more importantly their customers. Shit happens even with the best testing in the world. You do not deploy anything, ever on your entire production fleet at the same time and you do not buy software that does that. It's madness and we're not talking about small companies with tiny IT departments here.
  - perbu 2y ago
    Shit might happen with the best testing, but with decent testing it would not be this serious.
  - wazzaps 2y ago
    Apparently CrowdStrike bypassed clients' staging areas with this update. Source: https://x.com/patrickwardle/status/1814367918425079934 https://x.com/patrickwardle/status/1814367918425079934
  - owl57 2y ago
    > you do not buy software that does that Note how the incident disproportionally affected highly regulated industries, where businesses don't have a choice to screw "best practice".
    - TeMPOraL 2y ago
      Only highlighting that "best practice" of cybersecurity is, charitably, total bullshit; less charitably, a racket. This is apparent if you look at the costs to the day-to-day ability of employees to do work, but maybe it'll be more apparent now that people got killed because of it.
      - badgersnake 2y ago
        It’s absolutely a racket.
  - d1sxeyes 2y ago
    That’s a tricky one. CrowdStrike is cybersecurity. Wait until the first customer complains that they were hit by WannaCry v2 because CrowdStrike wanted to wait a few days after they updated a canary fleet. The problem here is that this type of update (a content update) should never be able to cause this however badly it goes. In case the software receives a bad content update, it should fail back to the last known good content update (potentially with a warning fired off to CS, the user, or someone else about the failed update). In principle, updates that could go wrong and cause this kind of issue should absolutely be deployed slowly, but per my understanding, that’s already the practice for non-content updates at CrowdStrike.
- jmb99 2y ago
  Oh absolutely. There’s many levels of failure here. A few that I see as being likely: - Lack of testing of a deployment - Lack of required procedures to validate a deployment - Engineering management prioritizing release pace over stability/testing - Management prioritizing tech debt/pentests/etc far too low - Sales/etc promising fast turnarounds that can’t be feasibly met while following proper standards - Lack of top-down company culture of security and stability first, which should be a must for any security company This outage wasn’t caused only by “the intern pushing release.” It was caused by a poor company culture (read: incorrect direction from the top) resulting in a lack of testing of the program code, lack of testing environment for deployments, lack of formal deployment process, and someone messing up a definition file that was caught by 0 other employees or automated systems.
_moof 2y ago
I can't speak to its veracity but there's a screenshot making its way around in which Crowdstrike discouraged sites from testing due to the urgency of the update.
- AmericanChopper 2y ago
  I don’t work with CS products atm, but my experience with a big CS deployment was exactly like this. They were openly quite hostile to any suggestion of testing their products, we were frequently rebuked for running our prod censors on version n-1. I talked about it a bit in this comment. https://news.ycombinator.com/item?id=%2041002864 https://news.ycombinator.com/item?id=%2041002864 Very much not surprised to see this now.
- jmb99 2y ago
  It’s kind of hard to pitch “zero-day prevention” if you suggest people roll out definitions slowly, over the course of days/weeks. Thus making it a lot harder to charge to the moon for your service. Now, if these sorts of things were battle tested before release, and had a (ideally decade+-long) history of stability with well-documented processes to ensure that stability, you can more easily make the argument that it’s worth it. None of those things are close to true though (and more than likely will never be for any AV/endpoint solution), so it is very hard to justify this sort of configuration.
qaq 2y ago
While true agent should roll back to previous content version if it keeps crashing
- Kwpolska 2y ago
  Detecting system crashes would be hard. You could try logging and comparing timestamps on agent startups and see if the difference is 5 minutes or less. Buggy kernel drivers crash Windows hard and fast.
  - qaq 2y ago
    loading content is pretty specific step so your solution is more or less valid
  - kchr 2y ago
    > Detecting system crashes would be hard. Store something like an `attemptingUpdate` flag before updating, and remove it if the update was successful. Upon system startup, if the flag is present, revert to the previous config and mark the new config bad.

by jmb99 2y ago

drooopy 2y ago

This whole f*up was a failure of management and processes at Crowdstrike. "Intern Steve" pushing faulty code to production on a Friday is only a couple of cm of the tip of an enormous iceberg.

chronid 2y ago
I wrote this in another thread already, but the fuck up was both at crowdstrike (they borked a release) but also and more importantly their customers. Shit happens even with the best testing in the world. You do not deploy anything, ever on your entire production fleet at the same time and you do not buy software that does that. It's madness and we're not talking about small companies with tiny IT departments here.
- perbu 2y ago
  Shit might happen with the best testing, but with decent testing it would not be this serious.
- wazzaps 2y ago
  Apparently CrowdStrike bypassed clients' staging areas with this update. Source: https://x.com/patrickwardle/status/1814367918425079934 https://x.com/patrickwardle/status/1814367918425079934
- owl57 2y ago
  > you do not buy software that does that Note how the incident disproportionally affected highly regulated industries, where businesses don't have a choice to screw "best practice".
  - TeMPOraL 2y ago
    Only highlighting that "best practice" of cybersecurity is, charitably, total bullshit; less charitably, a racket. This is apparent if you look at the costs to the day-to-day ability of employees to do work, but maybe it'll be more apparent now that people got killed because of it.
    - badgersnake 2y ago
      It’s absolutely a racket.
- d1sxeyes 2y ago
  That’s a tricky one. CrowdStrike is cybersecurity. Wait until the first customer complains that they were hit by WannaCry v2 because CrowdStrike wanted to wait a few days after they updated a canary fleet. The problem here is that this type of update (a content update) should never be able to cause this however badly it goes. In case the software receives a bad content update, it should fail back to the last known good content update (potentially with a warning fired off to CS, the user, or someone else about the failed update). In principle, updates that could go wrong and cause this kind of issue should absolutely be deployed slowly, but per my understanding, that’s already the practice for non-content updates at CrowdStrike.
jmb99 2y ago
Oh absolutely. There’s many levels of failure here. A few that I see as being likely: - Lack of testing of a deployment - Lack of required procedures to validate a deployment - Engineering management prioritizing release pace over stability/testing - Management prioritizing tech debt/pentests/etc far too low - Sales/etc promising fast turnarounds that can’t be feasibly met while following proper standards - Lack of top-down company culture of security and stability first, which should be a must for any security company This outage wasn’t caused only by “the intern pushing release.” It was caused by a poor company culture (read: incorrect direction from the top) resulting in a lack of testing of the program code, lack of testing environment for deployments, lack of formal deployment process, and someone messing up a definition file that was caught by 0 other employees or automated systems.

_moof 2y ago

I can't speak to its veracity but there's a screenshot making its way around in which Crowdstrike discouraged sites from testing due to the urgency of the update.

AmericanChopper 2y ago
I don’t work with CS products atm, but my experience with a big CS deployment was exactly like this. They were openly quite hostile to any suggestion of testing their products, we were frequently rebuked for running our prod censors on version n-1. I talked about it a bit in this comment. https://news.ycombinator.com/item?id=%2041002864 https://news.ycombinator.com/item?id=%2041002864 Very much not surprised to see this now.
jmb99 2y ago
It’s kind of hard to pitch “zero-day prevention” if you suggest people roll out definitions slowly, over the course of days/weeks. Thus making it a lot harder to charge to the moon for your service. Now, if these sorts of things were battle tested before release, and had a (ideally decade+-long) history of stability with well-documented processes to ensure that stability, you can more easily make the argument that it’s worth it. None of those things are close to true though (and more than likely will never be for any AV/endpoint solution), so it is very hard to justify this sort of configuration.

qaq 2y ago

While true agent should roll back to previous content version if it keeps crashing

Kwpolska 2y ago
Detecting system crashes would be hard. You could try logging and comparing timestamps on agent startups and see if the difference is 5 minutes or less. Buggy kernel drivers crash Windows hard and fast.
- qaq 2y ago
  loading content is pretty specific step so your solution is more or less valid
- kchr 2y ago
  > Detecting system crashes would be hard. Store something like an `attemptingUpdate` flag before updating, and remove it if the update was successful. Upon system startup, if the flag is present, revert to the previous config and mark the new config bad.