5 ms·

This "channel file" is equivalent to an AV signature file. Crowdstrike is the company, the product here is "Falcon" which does behavioral monitoring of processe

This "channel file" is equivalent to an AV signature file. Crowdstrike is the company, the product here is "Falcon" which does behavioral monitoring of processes both on the device and using logs collected from the device in the cloud. I can see your perspective, but you should consider this: They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in the last 15 years of their operation until this one outage. You can't take days testing gradual roll outs for this type of content, because that's how long customers are left unprotected by that content. Although the root cause is on the channel files, I feel like the driver that processes them should have been able to handle the "logic bug" in question so we'll find out more over time I guess. For example, with windows defender which runs on virtually all windows systems, the signature updates on billions of devices are pushed immediately (with exception to enterprise systems, but even then there is usually not much testing on signature files themselves, if at all). As far as the devops process Crowdstrike uses to test the channel files, I think it's best to leave commentary on that to actual insiders but these updates happen several times a day sometimes and get pushed to every Crowdstrike customer.

zug_zug 2y ago
>> You can't take days testing gradual roll outs for this type of content, because that's how long customers are left unprotected by that content. If you can't take days to do it then do a gradual rollout in hours. It's not a high bar.
- notepad0x90 2y ago
  they reverted it after about one hour. but sure, they didn't need to target all customers all at once, that's a good point.
lotr5 2y ago
they are dumb enough to process their "channel files" in kernel, this should be only done in usermode
- vladvasiliu 2y ago
  While I can understand both arguments for and against a gradual rollout, this is the main issue: why do these things need to be processed in kernel? And if there’s a good reason to do it, why isn’t there some kind of circuit breaker?
  - lotr5 2y ago
    probably they didn't find solution where they fully trust information coming from usermode process
  - notepad0x90 2y ago
    because the thing that uses them is in kernel mode, and the sensor needs to be performant. at some point, the content must be consumed by the kernel mode sensor. user mode edr's exist but bypassing them is trivial, intercepting syscalls rootkit style and monitoring kernel+usermode memory is the best and most performant way to monitor the whole system.
    - calf 2y ago
      Apple documentation argues the opposite: "Developers can use frameworks such as DriverKit and NetworkExtension to write USB and human interface drivers, endpoint security tools (like data loss prevention or other endpoint agents), and VPN and network tools, all without needing to write kexts. Third-party security agents should be used only if they take advantage of these APIs or have a robust road map to transition to them and away from kernel extensions." Specifically the 2nd sentence above says security software should use the APIs, not Apple's kernel extensions.
      - notepad0x90 2y ago
        well, this is windows not macos. I don't know what you can do with driverkit for example. maybe microsoft should learn from apple?
        2y ago
joaomacp 2y ago
> They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in the last 15 years of their operation I certainly don't want to know (through disaster news) about the construction company that built the bridge I drive through everyday, not for another 15 years, not ever! This kind of software simply should not fail, with such a massive install base on so many sensitive industries. We're better than that, the software industry is starting to mature and there are simple and widely-known procedures that could have been used to prevent it. I have no idea how CrowdStrike stock has only dropped 10% to the values of 2 months ago. Actually, if the financial troubles you get into are only these, take back what I said, software should be failing a lot (why spend money on robustness when you don't lose money on bugs?)
- notepad0x90 2y ago
  working in software, you should know how insanely complex software is, even google, amazon, microsoft, cloudflare and such have outages. mistakes happen because humans are involved. it is the nature and risk of depending complex systems. bridges by comparison are not that complicated. I actually expected their stock to drop a lot more than this, but goes to show you how valuable they are. investors know that any dip is only temporary because no one is getting rid of crowdstrike. Think of the security landscape as early 90's new york city at night and crowdstrike as the big bulky guy with lots of guns who protects you for a fee, if he makes a mistakes and hurts you, you will be mad but in the end your need for protection does not suddenly go away and it was a one time mistake.
  - TheOtherHobbes 2y ago
    In which case "Are you awake and sane?" would be a sensible reality check before heading out. You're trying to hand-wave away the inexcusable. The outage is a symptom. The problem is the lack of even the most basic testing. Clearly these files are sent out without even a minimal sanity check. That is a problem, and it's not something that can be hand-waved away.
    - 2y ago
tail_exchange 2y ago
Another commenter said that this change was a malformed configuration that crashed the application. If this is the case, you wouldn't need days to see this problem manifest, but only a few minutes. If they had rolled it out to 1% of their customers and waited for a couple hours before releasing it everywhere, they probably would have caught it.
- dalyons 2y ago
  A couple of hours is a long time in the world of automated attacks
  - TheOtherHobbes 2y ago
    Not as long as the weeks it's going to take to undo this.
  - mynameisvlad 2y ago
    A couple of hours is absolutely nothing compared to the massive worldwide effort that many people have to put in to fix the problem of a company’s shitty product and release practices. This is inexcusable, point blank. “A couple of hours is a long time” is not a valid excuse when the alternative, as clearly evidenced, is millions of computers and critical systems simultaneously failing hard. This might have been different if it was a small subset of computers, but this clearly could have been caught in minutes with any sort of sensible testing or canary rollout practices.
    - notepad0x90 2y ago
      I'm guessing they didn't expect content updates to cause such an impact, they've been doing this for 15 years, it is that uncommon. a couple of hours in their world is a long time because their concern is protecting customers as soon as possible. I'm sure they'll do all kinds of tests going forward and be transparent about it. Keep in mind how easy it is for you or I to come to conclusions without understanding or knowing the context they operate in, maybe it will be more clear soon enough.
  - 2y ago
Marsymars 2y ago
> They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in the last 15 years of their operation until this one outage. They certainly run their software on those many customers' systems, but but based on my experience with them, "protect" isn't a descriptor I'm willing to grant them. We don't have the counter-factual where Crowdstrike doesn't exist, but I'm not convinced that they've been a net economic or security benefit to the world over the span of their existence.
- notepad0x90 2y ago
  Yes, we do have a counter factual, they catch actual APT's they investigated the DNC hack in the 2016 elections and stopped many more attacks. You are utterly clueless in this area to make a comment like that honestly, I don't mean that as an insult but you are talking about a world they don't exist in as if every company has them. Most of their customers get them after getting pwned and learning their lesson the hard way. And availability isn't the only security property their customers desire, keeping information out of threat actors' hands and preventing them from tampering things is also desirable. I really hope you understand that in your hypothetical world without crowdstrike, threat actors still exist.
  - Marsymars 2y ago
    > Most of their customers get them after getting pwned and learning their lesson the hard way. Sure, that applies to my company, but the counter-factual isn't "nothing is done and we keep getting pwned", the counter-factual is that instead of the resources spent on crowdstrike and their various problems (which have been regular since we adopted them, the recent mess was just the biggest), those resources are spent on improving security infrastructure without crowdstrike.

zug_zug 2y ago

>> You can't take days testing gradual roll outs for this type of content, because that's how long customers are left unprotected by that content. If you can't take days to do it then do a gradual rollout in hours. It's not a high bar.

notepad0x90 2y ago
they reverted it after about one hour. but sure, they didn't need to target all customers all at once, that's a good point.

lotr5 2y ago

they are dumb enough to process their "channel files" in kernel, this should be only done in usermode

vladvasiliu 2y ago
While I can understand both arguments for and against a gradual rollout, this is the main issue: why do these things need to be processed in kernel? And if there’s a good reason to do it, why isn’t there some kind of circuit breaker?
- lotr5 2y ago
  probably they didn't find solution where they fully trust information coming from usermode process
- notepad0x90 2y ago
  because the thing that uses them is in kernel mode, and the sensor needs to be performant. at some point, the content must be consumed by the kernel mode sensor. user mode edr's exist but bypassing them is trivial, intercepting syscalls rootkit style and monitoring kernel+usermode memory is the best and most performant way to monitor the whole system.
  - calf 2y ago
    Apple documentation argues the opposite: "Developers can use frameworks such as DriverKit and NetworkExtension to write USB and human interface drivers, endpoint security tools (like data loss prevention or other endpoint agents), and VPN and network tools, all without needing to write kexts. Third-party security agents should be used only if they take advantage of these APIs or have a robust road map to transition to them and away from kernel extensions." Specifically the 2nd sentence above says security software should use the APIs, not Apple's kernel extensions.
    - notepad0x90 2y ago
      well, this is windows not macos. I don't know what you can do with driverkit for example. maybe microsoft should learn from apple?
      - 2y ago

joaomacp 2y ago

> They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in the last 15 years of their operation I certainly don't want to know (through disaster news) about the construction company that built the bridge I drive through everyday, not for another 15 years, not ever! This kind of software simply should not fail, with such a massive install base on so many sensitive industries. We're better than that, the software industry is starting to mature and there are simple and widely-known procedures that could have been used to prevent it. I have no idea how CrowdStrike stock has only dropped 10% to the values of 2 months ago. Actually, if the financial troubles you get into are only these, take back what I said, software should be failing a lot (why spend money on robustness when you don't lose money on bugs?)

notepad0x90 2y ago
working in software, you should know how insanely complex software is, even google, amazon, microsoft, cloudflare and such have outages. mistakes happen because humans are involved. it is the nature and risk of depending complex systems. bridges by comparison are not that complicated. I actually expected their stock to drop a lot more than this, but goes to show you how valuable they are. investors know that any dip is only temporary because no one is getting rid of crowdstrike. Think of the security landscape as early 90's new york city at night and crowdstrike as the big bulky guy with lots of guns who protects you for a fee, if he makes a mistakes and hurts you, you will be mad but in the end your need for protection does not suddenly go away and it was a one time mistake.
- TheOtherHobbes 2y ago
  In which case "Are you awake and sane?" would be a sensible reality check before heading out. You're trying to hand-wave away the inexcusable. The outage is a symptom. The problem is the lack of even the most basic testing. Clearly these files are sent out without even a minimal sanity check. That is a problem, and it's not something that can be hand-waved away.
  - 2y ago

tail_exchange 2y ago

Another commenter said that this change was a malformed configuration that crashed the application. If this is the case, you wouldn't need days to see this problem manifest, but only a few minutes. If they had rolled it out to 1% of their customers and waited for a couple hours before releasing it everywhere, they probably would have caught it.

dalyons 2y ago
A couple of hours is a long time in the world of automated attacks
- TheOtherHobbes 2y ago
  Not as long as the weeks it's going to take to undo this.
- mynameisvlad 2y ago
  A couple of hours is absolutely nothing compared to the massive worldwide effort that many people have to put in to fix the problem of a company’s shitty product and release practices. This is inexcusable, point blank. “A couple of hours is a long time” is not a valid excuse when the alternative, as clearly evidenced, is millions of computers and critical systems simultaneously failing hard. This might have been different if it was a small subset of computers, but this clearly could have been caught in minutes with any sort of sensible testing or canary rollout practices.
  - notepad0x90 2y ago
    I'm guessing they didn't expect content updates to cause such an impact, they've been doing this for 15 years, it is that uncommon. a couple of hours in their world is a long time because their concern is protecting customers as soon as possible. I'm sure they'll do all kinds of tests going forward and be transparent about it. Keep in mind how easy it is for you or I to come to conclusions without understanding or knowing the context they operate in, maybe it will be more clear soon enough.
- 2y ago

Marsymars 2y ago

> They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in the last 15 years of their operation until this one outage. They certainly run their software on those many customers' systems, but but based on my experience with them, "protect" isn't a descriptor I'm willing to grant them. We don't have the counter-factual where Crowdstrike doesn't exist, but I'm not convinced that they've been a net economic or security benefit to the world over the span of their existence.

notepad0x90 2y ago
Yes, we do have a counter factual, they catch actual APT's they investigated the DNC hack in the 2016 elections and stopped many more attacks. You are utterly clueless in this area to make a comment like that honestly, I don't mean that as an insult but you are talking about a world they don't exist in as if every company has them. Most of their customers get them after getting pwned and learning their lesson the hard way. And availability isn't the only security property their customers desire, keeping information out of threat actors' hands and preventing them from tampering things is also desirable. I really hope you understand that in your hypothetical world without crowdstrike, threat actors still exist.
- Marsymars 2y ago
  > Most of their customers get them after getting pwned and learning their lesson the hard way. Sure, that applies to my company, but the counter-factual isn't "nothing is done and we keep getting pwned", the counter-factual is that instead of the resources spent on crowdstrike and their various problems (which have been regular since we adopted them, the recent mess was just the biggest), those resources are spent on improving security infrastructure without crowdstrike.