12 ms·

Throwaway account... CrowdStrike in this context is a NT kernel loadable module (a .sys file) which does syscall level interception and logs then to a separate

Throwaway account... CrowdStrike in this context is a NT kernel loadable module (a .sys file) which does syscall level interception and logs then to a separate process on the machine. It can also STOP syscalls from working if they are trying to connect out to other nodes and accessing files they shouldn't be (using some drunk ass heuristics). What happened here was they pushed a new kernel driver out to every client without authorization to fix an issue with slowness and latency that was in the previous Falcon sensor product. They have a staging system which is supposed to give clients control over this but they pissed over everyone's staging and rules and just pushed this to production. This has taken us out and we have 30 people currently doing recovery and DR. Most of our nodes are boot looping with blue screens which in the cloud is not something you can just hit F8 and remove the driver. We have to literally take each node down, attach the disk to a working node, delete the .sys file and bring it up. Either that or bring up a new node entirely from a snapshot. This is fine but EC2 is rammed with people doing this now so it's taking forever. Storage latency is through the roof. I fought for months to keep this shit out of production because of this reason. I am now busy but vindicated. Edit: to all the people moaning about windows, we've had no problems with Windows. This is not a windows issue. This is a third party security vendor shitting in the kernel.

jakebleiberg 2y ago
Hello: I'm a reporter with Bloomberg News covering cybersecurity. I'm trying to learn more about this Crowdstrike update potentially bypassing staging rules and would love to hear about your experience. Would you be open to a coversation? I'm reachable by email at jbleiberg2@bloomberg.net or on Signal at JakeBleiberg.24. Here's my Bloomberg author page: https://www.bloomberg.com/authors/AWuCZUVX-Pc/jake-bleiberg https://www.bloomberg.com/authors/AWuCZUVX-Pc/jake-bleiberg. Thank you. Jake
tamimio 2y ago
> give clients control over this but they pissed over everyone's staging and rules and just pushed this to production. Interesting.. > We have to literally take each node down, attach the disk to a working node.. Probably the easiest solution for you is to go back in time to a previous scheduled snapshot, if you have that setup already.
- teeheelol 2y ago
  That would make sense but it appears everyone is doing EBS snapshots in our regions like mad so they aren't restoring. Spoke to our AWS account manager (we are a big big big org) and they have contention issues everywhere. I really want our cages, C7000's and VMware back at this point.
  - bflesch 2y ago
    Beautiful
  - CoastalCoder 2y ago
    > Spoke to our AWS account manager (we are a big big big org) Is this how you got the inside scoop on the rollout fiasco?
  - khrystoph 2y ago
    Netflix big? Bigger or Smaller? I'm betting I have a good idea of one of the possible orgs you work for, since I used to work specifically with the largest 100 customers during my ~3yr stint in premium support
    - teeheelol 2y ago
      Smaller. No one has heard of us :)
    - OhMeadhbh 2y ago
      Netflix isn't really that big. two organizations ago our reverse proxy used 40k cores. netflixes is less than 5k. of course, that could just mean our nginx extensions are 8 times crappier than netflix.
amluto 2y ago
I did approximately this recently, but on a Linux machine on GCP. It sucked far worse than it should have: apparently GCP cannot reliably “stop” a VM in a timely manner. And you can’t detach a boot disk from a VM that isn’t “stopped”, nor can you multi-attach it, nor can you (AFAICT) convince a VM to boot off an alternate disk. I used to have this crazy idea that fancy cloud vendors had competent management tools. Like maybe I could issue an API call to boot an existing instance from an alternate disk or HTTPS netboot URL. Or to insta-stop a VM and get block-level access to its disk via API, even if I had to pay for the instance while doing this. And I’m not sure that it’s possible to do this sort of recovery at all without blowing away local SSD. There’s a “preview” feature for this on GCP, which seems to be barely supported, and I bet it adds massive latency to the process. Throwing away one’s local SSD on every single machine in a deployment sounds like a great way to cause potentially catastrophic resource usage when everything starts back up. Hmm, I wonder if you’re even guaranteed to be able to get your instance back after stopping it. WTF. Why can’t I have any means to access the boot disk of an instance, in a timely manner? Or any better means to recover an instance? Is AWS any better?
- teeheelol 2y ago
  AWS is not any better really on this. In fact 2 years ago (to the day!) we had a complete AZ outage in our local AWS region. This resulted in their control plane going nuts and being unable to shut down or start new instances. Then capacity problems.
  - khrystoph 2y ago
    That's happened several times, actually. That's probably just the latest one. The really fun one was when S3 went down in 2017 in Virginia. Caused global outages of multiple services because most services were housed out of Virginia and when EC2 and other services went offline due to dependency on S3, everything cascade failed across multiple regions (in terms of start/stop/delete...ie. api actions. Stuff that was running was, for the most part, still working in some places). ...I remember that day pretty well. It was a busy day.
- khrystoph 2y ago
HelloNurse 2y ago
Before reaching the "pushed out to every client without authorization" stage, a kernel driver/module should have been tested. Tested by Microsoft, not by "a third party security vendor shitting in the kernel" that some criminally negligent manager decided to trust.
- teeheelol 2y ago
  Yeah we have a staging and test process where we run their updated Falcon sensor releasees. They shit all over our controls and went to production. This says we don't control it and should not trust it. It is being removed.
  - JonChesterfield 2y ago
    > It is being removed. Congratulations on actually fixing the root cause, as opposed to hand wringing and hoping they don't break you again. I'm expecting "oh noes, better keep it on anyway to be safe" to be the popular choice.
    - khrystoph 2y ago
      yeah, I agree. I think most places will at least keep it until the existing contract comes time for renegotiation and most will probably keep using cs. It's far easier for IT departments to just keep using it than it is to switch and managers will complain about "the cost of migrating" and "the time to evaluate and test a new solution" or "other products don't have feature X that we need" (even when they don't need that feature, but THINK they do).
- Shorn 2y ago
  > Tested by Microsoft MS don't have testers any more. Where do you think CS learned their radically effective test-in-prod approach?
  - teeheelol 2y ago
    I think they learned it from Freedesktop developers.
spectrumero 2y ago
I always said if you want to create real chaos, don't write malware. Get on the inside of a security product like this, and push out a bad update, and you can take most of the world down.
- valval 2y ago
  So… Write malware?
  - Namari 2y ago
    *Malicious code on a legit software
Reason077 2y ago
> ”This is not a windows issue. This is a third party security vendor shitting in the kernel.“ Sure, but Windows shares some portion of the blame for allowing third-party security vendors to “shit in the kernel”. Compare to macOS which has banned third-party kernel extensions on Apple Silicon. Things that once ran as kernel extensions, including CrowdStrike, now run in userspace as “system extensions”.
- corny_penguin88 2y ago
  [dead]
- deepsummer 2y ago
  ...but still, if the user space process is broken, MacOS will fail as well. Maybe it's a bit easier to recover, but any broken process with non-trivial privileges can interrupt the whole system.
  - Reason077 2y ago
    It's certainly not supposed to work like that. In the kernel, a crash brings down the entire system by design. But in userspace, failed services can be restarted and continued without affecting other services. If a failure in a userspace service can crash the entire system, that's a bug.
    - deepsummer 2y ago
      It's kind of inevitable that a security system can crash the system. It just needs to claim than one essential binary is infected with malware, and the system won't run.
- phendrenad2 2y ago
  Yes... in the same sense that if a user bricks their own system by deleting system32 then Windows shares some small sliver of the blame. In other words, not much.
  - 2y ago
numbsafari 2y ago
> This is not a windows issue. Honest question, I've seen comments in these various threads about people having similar issues (from a few months/weeks back) with kernel extension based deployments of CrowdStrike on Debian/Ubuntu systems. I haven't seen anything similar regarding Mac OS, which no longer allows kernel extensions. Is Mac OS not impacted by these kinds of issues with CrowdStrike's product, or have we just not heard about it due to the small scale? Personally, it's a shared responsibility issue. MS should build a product that is "open to extension but closed for modification". > they pissed over everyone's staging and rules and just pushed this to production. I am guessing that act alone is going to create a massive liability for CrowdStrike over this issue. You've made other comments that your organization is actively removing CrowdStrike. I'm curious how this plays out. Did CrowdStrike just SolarWind themselves? Will we see their CISO/CTO/CEO do time? This is just the first part of this saga.
- teeheelol 2y ago
  The issue is where it is integrated. You could arguably implement CrowdStrike in BPF on Linux. On NT they literally hook NT syscalls in the kernel from a driver they inject into kernel space which is much bad juju. As for macOS, you have no access to the kernel. There is no shared responsibility. CrowdStrike pushed a broken driver out, then triggered the breakage, overriding customer requirement and configuration for staging. It is a faulty product with no viable security controls or testing.
  - meeper22 2y ago
    Yep, it's extremely lame that CS has been pushing the "Windows" narrative to frame it as a Windows issue in the press, so everyone will just default blame Microsoft (which everyone knows) and not Crowdstrike (which only IT/cybersec people are familiar with). And then you get midwits who blame Microsoft for allowing kernel access in the first place. Yes Apple deprecated kexts on macOS; that's a hell of a lot easier to do when you control the entire hardware ecosystem. Go ahead and switch to Apple then. If you want to build your own machines or pick your hardware vendor, guess what, people are going to need to write drivers, and they are probably going to want kernel mode, and the endpoint security people like CrowdStrike will want to get in there too because the threat is there. There's no way for Microsoft or Linux for that matter to turn on a dime and deny kernel access to all the thousands upon thousands of drivers and system software running on billions of machines in billions of potential configurations. That requires completely reworking the system architecture.
Twirrim 2y ago
> Most of our nodes are boot looping with blue screens which in the cloud is not something you can just hit F8 and remove the driver. It took a bit to figure out with some customers, but we provide optional VNC access to instances at OCI, and with VNC the trick seems to be to hit esc and then F8, at the right stage in the boot process. Timing seems to be the devil in the details there, though. Getting that timing right is frustrating. People seem to be developing a knack for it though.
sphar1970 2y ago
But it is Windows because the kernel should be able to roll back a bad update, there should NEVER be BSODs.
- teeheelol 2y ago
  Windows does do that. Crowdstrike sticks it back in at the UEFI level by the looks, because you know, "security".
- OhMeadhbh 2y ago
  pish! this isn't VM/SP! commodity OSes and hardware took over because customers didn't want to pay firms to staff people who grokked risk management. linux supplanted mature OSes because some dork implied even security bugs were shallow with all those billions of eyes. It's a weird world when MSFT does a security stand down in 2003 and in 2008 starts widening security holes because the new "secure" OS they wrote was a no-go for third parties who didn't want to pay $100 to hire someone who knew how to rub two primes together. I miss my AS/400. This might be a decent place to recount the experience I had when interviewing for office security architect in 2003. my background is mainframe VM system design and large system risk management modeling which I had been doing since the late 80s at IBM, DEC, then Digital Switch and Bell Canada. My resume was pretty decent at the time. I don't like Python and tell VP/Eng's they have a problem when they can't identify benefits from JIRA/SCRUM, so I don't get a lot of job offers these days. Just a crusty greybeard bitching... But anyway... so I'm up in Redmond and I have a decent couple of interviews with people and then the 3rd most senior dev in all of MSFT comes in and asks "how's your QA skills?" and I start to answer about how QA and Safety/Security/Risk Management are different things. QA is about insuring the code does what it's supposed to, software security, et al is about making sure the code doesn't do what it's not supposed to and the philosophic sticky wicket you enter when trying to prove a negative (worth a google deep dive if you're unfamiliar.) Dude cuts me off and says "meh. security is stupid. in a month, Bill will end this stupid security stand down and we'll get back to writing code and I need to put you somewhere and I figured QA is the right place." When I hear that MSFT has systems that expose inadequate risk management abstractions, I think of the culture that promoted that guy to his senior position... I'm sure he was a capable engineer, but the culture in Redmond discounts the business benefits of risk management (to the point they outsource critical system infrastructure to third parties) because senior engineers don't want to be bothered to learn new tricks. Culture eats strategy for breakfast, and MSFT has been fed on a cultural diet of junk food for almost half a century. At least from the perspective of doing business in the modern world.
crustycoder 2y ago
Although it's a .sys file, it's not a device driver. "Although Channel Files end with the SYS extension, they are not kernel drivers." https://www.crowdstrike.com/blog/technical-details-on-todays-outage/ https://www.crowdstrike.com/blog/technical-details-on-todays...
- teeheelol 2y ago
  Yeah it's a way of delivering a payload to the driver, which promptly crashed. Which is horrible!
  - crustycoder 2y ago
    Horrible for sure, not least because hackers now know that the channel file parser is fragile and perhaps exploitable. I haven't seen any significant discussion about follow-on attacks, it's all been about rolling back the config file rather than addressing the root cause, which is the shonky device driver.
    - OhMeadhbh 2y ago
      I suspect the wiley hackors have known how fragile that code is for years.
willmadden 2y ago
It's a shitty C++ hack job within CloudStrike with a null pointer. Because the software has root access, Windows shuts it down as a security precaution. A simple unit test would have caught this, or any number of tools that look for null pointers in C++, not even full QA. It's unbelievable incompetence.

jakebleiberg 2y ago

Hello: I'm a reporter with Bloomberg News covering cybersecurity. I'm trying to learn more about this Crowdstrike update potentially bypassing staging rules and would love to hear about your experience. Would you be open to a coversation? I'm reachable by email at jbleiberg2@bloomberg.net or on Signal at JakeBleiberg.24. Here's my Bloomberg author page: https://www.bloomberg.com/authors/AWuCZUVX-Pc/jake-bleiberg https://www.bloomberg.com/authors/AWuCZUVX-Pc/jake-bleiberg. Thank you. Jake

tamimio 2y ago

> give clients control over this but they pissed over everyone's staging and rules and just pushed this to production. Interesting.. > We have to literally take each node down, attach the disk to a working node.. Probably the easiest solution for you is to go back in time to a previous scheduled snapshot, if you have that setup already.

teeheelol 2y ago
That would make sense but it appears everyone is doing EBS snapshots in our regions like mad so they aren't restoring. Spoke to our AWS account manager (we are a big big big org) and they have contention issues everywhere. I really want our cages, C7000's and VMware back at this point.
- bflesch 2y ago
  Beautiful
- CoastalCoder 2y ago
  > Spoke to our AWS account manager (we are a big big big org) Is this how you got the inside scoop on the rollout fiasco?
- khrystoph 2y ago
  Netflix big? Bigger or Smaller? I'm betting I have a good idea of one of the possible orgs you work for, since I used to work specifically with the largest 100 customers during my ~3yr stint in premium support
  - teeheelol 2y ago
    Smaller. No one has heard of us :)
  - OhMeadhbh 2y ago
    Netflix isn't really that big. two organizations ago our reverse proxy used 40k cores. netflixes is less than 5k. of course, that could just mean our nginx extensions are 8 times crappier than netflix.

amluto 2y ago

I did approximately this recently, but on a Linux machine on GCP. It sucked far worse than it should have: apparently GCP cannot reliably “stop” a VM in a timely manner. And you can’t detach a boot disk from a VM that isn’t “stopped”, nor can you multi-attach it, nor can you (AFAICT) convince a VM to boot off an alternate disk. I used to have this crazy idea that fancy cloud vendors had competent management tools. Like maybe I could issue an API call to boot an existing instance from an alternate disk or HTTPS netboot URL. Or to insta-stop a VM and get block-level access to its disk via API, even if I had to pay for the instance while doing this. And I’m not sure that it’s possible to do this sort of recovery at all without blowing away local SSD. There’s a “preview” feature for this on GCP, which seems to be barely supported, and I bet it adds massive latency to the process. Throwing away one’s local SSD on every single machine in a deployment sounds like a great way to cause potentially catastrophic resource usage when everything starts back up. Hmm, I wonder if you’re even guaranteed to be able to get your instance back after stopping it. WTF. Why can’t I have any means to access the boot disk of an instance, in a timely manner? Or any better means to recover an instance? Is AWS any better?

teeheelol 2y ago
AWS is not any better really on this. In fact 2 years ago (to the day!) we had a complete AZ outage in our local AWS region. This resulted in their control plane going nuts and being unable to shut down or start new instances. Then capacity problems.
- khrystoph 2y ago
  That's happened several times, actually. That's probably just the latest one. The really fun one was when S3 went down in 2017 in Virginia. Caused global outages of multiple services because most services were housed out of Virginia and when EC2 and other services went offline due to dependency on S3, everything cascade failed across multiple regions (in terms of start/stop/delete...ie. api actions. Stuff that was running was, for the most part, still working in some places). ...I remember that day pretty well. It was a busy day.
khrystoph 2y ago

HelloNurse 2y ago

Before reaching the "pushed out to every client without authorization" stage, a kernel driver/module should have been tested. Tested by Microsoft, not by "a third party security vendor shitting in the kernel" that some criminally negligent manager decided to trust.

teeheelol 2y ago
Yeah we have a staging and test process where we run their updated Falcon sensor releasees. They shit all over our controls and went to production. This says we don't control it and should not trust it. It is being removed.
- JonChesterfield 2y ago
  > It is being removed. Congratulations on actually fixing the root cause, as opposed to hand wringing and hoping they don't break you again. I'm expecting "oh noes, better keep it on anyway to be safe" to be the popular choice.
  - khrystoph 2y ago
    yeah, I agree. I think most places will at least keep it until the existing contract comes time for renegotiation and most will probably keep using cs. It's far easier for IT departments to just keep using it than it is to switch and managers will complain about "the cost of migrating" and "the time to evaluate and test a new solution" or "other products don't have feature X that we need" (even when they don't need that feature, but THINK they do).
Shorn 2y ago
> Tested by Microsoft MS don't have testers any more. Where do you think CS learned their radically effective test-in-prod approach?
- teeheelol 2y ago
  I think they learned it from Freedesktop developers.

spectrumero 2y ago

I always said if you want to create real chaos, don't write malware. Get on the inside of a security product like this, and push out a bad update, and you can take most of the world down.

valval 2y ago
So… Write malware?
- Namari 2y ago
  *Malicious code on a legit software

Reason077 2y ago

> ”This is not a windows issue. This is a third party security vendor shitting in the kernel.“ Sure, but Windows shares some portion of the blame for allowing third-party security vendors to “shit in the kernel”. Compare to macOS which has banned third-party kernel extensions on Apple Silicon. Things that once ran as kernel extensions, including CrowdStrike, now run in userspace as “system extensions”.

corny_penguin88 2y ago
[dead]
deepsummer 2y ago
...but still, if the user space process is broken, MacOS will fail as well. Maybe it's a bit easier to recover, but any broken process with non-trivial privileges can interrupt the whole system.
- Reason077 2y ago
  It's certainly not supposed to work like that. In the kernel, a crash brings down the entire system by design. But in userspace, failed services can be restarted and continued without affecting other services. If a failure in a userspace service can crash the entire system, that's a bug.
  - deepsummer 2y ago
    It's kind of inevitable that a security system can crash the system. It just needs to claim than one essential binary is infected with malware, and the system won't run.
phendrenad2 2y ago
Yes... in the same sense that if a user bricks their own system by deleting system32 then Windows shares some small sliver of the blame. In other words, not much.
- 2y ago

numbsafari 2y ago

> This is not a windows issue. Honest question, I've seen comments in these various threads about people having similar issues (from a few months/weeks back) with kernel extension based deployments of CrowdStrike on Debian/Ubuntu systems. I haven't seen anything similar regarding Mac OS, which no longer allows kernel extensions. Is Mac OS not impacted by these kinds of issues with CrowdStrike's product, or have we just not heard about it due to the small scale? Personally, it's a shared responsibility issue. MS should build a product that is "open to extension but closed for modification". > they pissed over everyone's staging and rules and just pushed this to production. I am guessing that act alone is going to create a massive liability for CrowdStrike over this issue. You've made other comments that your organization is actively removing CrowdStrike. I'm curious how this plays out. Did CrowdStrike just SolarWind themselves? Will we see their CISO/CTO/CEO do time? This is just the first part of this saga.

teeheelol 2y ago
The issue is where it is integrated. You could arguably implement CrowdStrike in BPF on Linux. On NT they literally hook NT syscalls in the kernel from a driver they inject into kernel space which is much bad juju. As for macOS, you have no access to the kernel. There is no shared responsibility. CrowdStrike pushed a broken driver out, then triggered the breakage, overriding customer requirement and configuration for staging. It is a faulty product with no viable security controls or testing.
- meeper22 2y ago
  Yep, it's extremely lame that CS has been pushing the "Windows" narrative to frame it as a Windows issue in the press, so everyone will just default blame Microsoft (which everyone knows) and not Crowdstrike (which only IT/cybersec people are familiar with). And then you get midwits who blame Microsoft for allowing kernel access in the first place. Yes Apple deprecated kexts on macOS; that's a hell of a lot easier to do when you control the entire hardware ecosystem. Go ahead and switch to Apple then. If you want to build your own machines or pick your hardware vendor, guess what, people are going to need to write drivers, and they are probably going to want kernel mode, and the endpoint security people like CrowdStrike will want to get in there too because the threat is there. There's no way for Microsoft or Linux for that matter to turn on a dime and deny kernel access to all the thousands upon thousands of drivers and system software running on billions of machines in billions of potential configurations. That requires completely reworking the system architecture.

Twirrim 2y ago

> Most of our nodes are boot looping with blue screens which in the cloud is not something you can just hit F8 and remove the driver. It took a bit to figure out with some customers, but we provide optional VNC access to instances at OCI, and with VNC the trick seems to be to hit esc and then F8, at the right stage in the boot process. Timing seems to be the devil in the details there, though. Getting that timing right is frustrating. People seem to be developing a knack for it though.

sphar1970 2y ago

But it is Windows because the kernel should be able to roll back a bad update, there should NEVER be BSODs.

teeheelol 2y ago
Windows does do that. Crowdstrike sticks it back in at the UEFI level by the looks, because you know, "security".
OhMeadhbh 2y ago
pish! this isn't VM/SP! commodity OSes and hardware took over because customers didn't want to pay firms to staff people who grokked risk management. linux supplanted mature OSes because some dork implied even security bugs were shallow with all those billions of eyes. It's a weird world when MSFT does a security stand down in 2003 and in 2008 starts widening security holes because the new "secure" OS they wrote was a no-go for third parties who didn't want to pay $100 to hire someone who knew how to rub two primes together. I miss my AS/400. This might be a decent place to recount the experience I had when interviewing for office security architect in 2003. my background is mainframe VM system design and large system risk management modeling which I had been doing since the late 80s at IBM, DEC, then Digital Switch and Bell Canada. My resume was pretty decent at the time. I don't like Python and tell VP/Eng's they have a problem when they can't identify benefits from JIRA/SCRUM, so I don't get a lot of job offers these days. Just a crusty greybeard bitching... But anyway... so I'm up in Redmond and I have a decent couple of interviews with people and then the 3rd most senior dev in all of MSFT comes in and asks "how's your QA skills?" and I start to answer about how QA and Safety/Security/Risk Management are different things. QA is about insuring the code does what it's supposed to, software security, et al is about making sure the code doesn't do what it's not supposed to and the philosophic sticky wicket you enter when trying to prove a negative (worth a google deep dive if you're unfamiliar.) Dude cuts me off and says "meh. security is stupid. in a month, Bill will end this stupid security stand down and we'll get back to writing code and I need to put you somewhere and I figured QA is the right place." When I hear that MSFT has systems that expose inadequate risk management abstractions, I think of the culture that promoted that guy to his senior position... I'm sure he was a capable engineer, but the culture in Redmond discounts the business benefits of risk management (to the point they outsource critical system infrastructure to third parties) because senior engineers don't want to be bothered to learn new tricks. Culture eats strategy for breakfast, and MSFT has been fed on a cultural diet of junk food for almost half a century. At least from the perspective of doing business in the modern world.

crustycoder 2y ago

Although it's a .sys file, it's not a device driver. "Although Channel Files end with the SYS extension, they are not kernel drivers." https://www.crowdstrike.com/blog/technical-details-on-todays-outage/ https://www.crowdstrike.com/blog/technical-details-on-todays...

teeheelol 2y ago
Yeah it's a way of delivering a payload to the driver, which promptly crashed. Which is horrible!
- crustycoder 2y ago
  Horrible for sure, not least because hackers now know that the channel file parser is fragile and perhaps exploitable. I haven't seen any significant discussion about follow-on attacks, it's all been about rolling back the config file rather than addressing the root cause, which is the shonky device driver.
  - OhMeadhbh 2y ago
    I suspect the wiley hackors have known how fragile that code is for years.

willmadden 2y ago

It's a shitty C++ hack job within CloudStrike with a null pointer. Because the software has root access, Windows shuts it down as a security precaution. A simple unit test would have caught this, or any number of tools that look for null pointers in C++, not even full QA. It's unbelievable incompetence.