4 ms·

If you consider kernel programming to be inherently unsafe, then you would consider this to be inevitable, meaning it's not really the specific company's fault.

If you consider kernel programming to be inherently unsafe, then you would consider this to be inevitable, meaning it's not really the specific company's fault. They were just the unlucky ones.

efee22 2y ago
Agree, Crowdstrike was an unlucky one, but it is more about the issue in general. If I remember correctly, also others like sysdig user their own kernel modules for collection.
asynchronous 2y ago
I still hold true that testing even improperly would have caught this before it hit worldwide. But I suppose you are right, that doesn’t help the argument being made here.
- ForOldHack 2y ago
  Wasnt that the job of AI/co-pilot/clippy /D.E.P? "Would you like me to try and execute a random blank file?" And of course QA. I was unaffected, but was fielding calls from customers. My update Tuesday is the week after, so in-between MS and my updates, I am very suspicious of everything. I was also unaffected by 22H2, and spent time fielding calls.
lordnacho 2y ago
They could have helped their luck by doing some of the common sense things suggested in the article. For instance, why not find a subset of your customers that are low risk, push it out to them, and see what happens? Or perhaps have your own fleet of example installations to run things on first. None of which depends on any specific technology.
- hello_moto 2y ago
  "find a subset of low risk customers" and use them as test subject? Repeat that a few times to understand the repercussions. If I were the customers and I found out that I was used as test subject, how would I feel?
  - whynotminot 2y ago
    Canary deployments are already an industry accepted practice and it’s shocking Crowdstrike apparently doesn’t do them.
    - hello_moto 2y ago
      Which industry? Cybersecurity or Cloud software?
      - whynotminot 2y ago
        Any industry that wants to reliably deliver software that doesn’t brick systems at scale? I’m confused by your question. Are you telling me the cybersecurity scene is special and shouldn’t follow best practices for software deployment?
        hello_moto 2y ago
        Canary deployment for subset of Salesforce customers won't see much of revolt from customers compare to AV definition rollout (not software, but AV definition) in Cybersecurity where gaps between 0day and rollout means you're exposed. If customers found out that some are getting roll out faster than the others, essentially splitting the group into 2, there will be a need for customer opt-in/opt-out. If everyone is opting-out because of Friday, your Canary deployment becomes meaningless. Any proof that other Cybersecurity vendors do Canary deployment for their AV definition? :) PS: not to say that the company should test more internally...
        whynotminot 2y ago
        Canary deployment doesn’t necessarily mean massive gaps between deployment waves. You can fast-follow. Sure, there may be scenarios with especially severe vulnerabilities where time is of the essence. I’m out of the loop if this crowdstrike update was such a scenario where best practices for software deployment were worth bypassing. If this is just how they roll with regular definition updates, then their deployment practices are garbage and this kind of large scale disaster was inevitable.
- gtsop 2y ago
  Why even do that? We have virtualization, they could emulate real clients and networks of clients. This particular bug would have been prevented for sure
  - lordnacho 2y ago
    Yeah I thought maybe the VM thing might not catch the bug for some reason, but it seems like the natural thing to do. Spin up VM, see if there's a crash. I heard the technical reason had something to do with a file being full of nulls, but that sort of thing you should catch. Honestly, the most generous excuse I can think of is that CS were informed of some sort of vulnerability that would have profound consequences immediately, and that necessitated a YOLO push. But even that doesn't seem too likely.
brendangregg 2y ago
Right, and we wanted to talk about all security solutions and not make this about one company. We also wanted to avoid shaming since they have been seriously working on eBPF adoption, so in that regard they are at the forefront of doing the right thing.
deleted 2y ago
[deleted]

If you consider kernel programming to be inherently unsafe, then you would consider this to be inevitable, meaning it's not really the specific company's fault. They were just the unlucky ones.

efee22 2y ago

Agree, Crowdstrike was an unlucky one, but it is more about the issue in general. If I remember correctly, also others like sysdig user their own kernel modules for collection.

asynchronous 2y ago

I still hold true that testing even improperly would have caught this before it hit worldwide. But I suppose you are right, that doesn’t help the argument being made here.

ForOldHack 2y ago
Wasnt that the job of AI/co-pilot/clippy /D.E.P? "Would you like me to try and execute a random blank file?" And of course QA. I was unaffected, but was fielding calls from customers. My update Tuesday is the week after, so in-between MS and my updates, I am very suspicious of everything. I was also unaffected by 22H2, and spent time fielding calls.

lordnacho 2y ago

They could have helped their luck by doing some of the common sense things suggested in the article. For instance, why not find a subset of your customers that are low risk, push it out to them, and see what happens? Or perhaps have your own fleet of example installations to run things on first. None of which depends on any specific technology.

hello_moto 2y ago
"find a subset of low risk customers" and use them as test subject? Repeat that a few times to understand the repercussions. If I were the customers and I found out that I was used as test subject, how would I feel?
- whynotminot 2y ago
  Canary deployments are already an industry accepted practice and it’s shocking Crowdstrike apparently doesn’t do them.
  - hello_moto 2y ago
    Which industry? Cybersecurity or Cloud software?
    - whynotminot 2y ago
      Any industry that wants to reliably deliver software that doesn’t brick systems at scale? I’m confused by your question. Are you telling me the cybersecurity scene is special and shouldn’t follow best practices for software deployment?
      - hello_moto 2y ago
        Canary deployment for subset of Salesforce customers won't see much of revolt from customers compare to AV definition rollout (not software, but AV definition) in Cybersecurity where gaps between 0day and rollout means you're exposed. If customers found out that some are getting roll out faster than the others, essentially splitting the group into 2, there will be a need for customer opt-in/opt-out. If everyone is opting-out because of Friday, your Canary deployment becomes meaningless. Any proof that other Cybersecurity vendors do Canary deployment for their AV definition? :) PS: not to say that the company should test more internally...
        whynotminot 2y ago
        Canary deployment doesn’t necessarily mean massive gaps between deployment waves. You can fast-follow. Sure, there may be scenarios with especially severe vulnerabilities where time is of the essence. I’m out of the loop if this crowdstrike update was such a scenario where best practices for software deployment were worth bypassing. If this is just how they roll with regular definition updates, then their deployment practices are garbage and this kind of large scale disaster was inevitable.
gtsop 2y ago
Why even do that? We have virtualization, they could emulate real clients and networks of clients. This particular bug would have been prevented for sure
- lordnacho 2y ago
  Yeah I thought maybe the VM thing might not catch the bug for some reason, but it seems like the natural thing to do. Spin up VM, see if there's a crash. I heard the technical reason had something to do with a file being full of nulls, but that sort of thing you should catch. Honestly, the most generous excuse I can think of is that CS were informed of some sort of vulnerability that would have profound consequences immediately, and that necessitated a YOLO push. But even that doesn't seem too likely.

brendangregg 2y ago

Right, and we wanted to talk about all security solutions and not make this about one company. We also wanted to avoid shaming since they have been seriously working on eBPF adoption, so in that regard they are at the forefront of doing the right thing.

deleted 2y ago

[deleted]