7 ms·

This is a good writeup, but to be fair it's just not a matter of banking regulations. Basically all big companies are under similar obligations regarding endpo

This is a good writeup, but to be fair it's just not a matter of banking regulations. Basically all big companies are under similar obligations regarding endpoint protection.

candiddevmike 2y ago
Should endpoint protection require kernel level access? At what point does it stop becoming protection and start becoming a liability? Obligatory who watches/protects the watchmen/protector...
- greenn 2y ago
  With the current model kernel level access is required. Real security products have to be able to operate above userland. Ideally in the future there can be a layer in between userland and kernel for this sort of thing. Maybe we use some of those extra protection rings?
  - ghusto 2y ago
    Couldn't you just ask some OS APIs provided by something in kernelspace for what you need? In fact, isn't this how macOS does things?
    - mywittyname 2y ago
      Microsoft was on their way to doing this, but was shot down by EU regulators because the APIs weren't available to all third-party vendors.
      - armada651 2y ago
        I think it's kind of ridiculous to then blame the regulators for the fact that Microsoft decided not to go ahead with a more competitor-friendly design. The fact that Microsoft abandoned it as soon as a regulator pointed out how anti-competitive the design of the API was makes you wonder what Microsoft's true intention was. To me that implies the anti-competitive design was its main feature and to Microsoft it would've been pointless to continue without it.
        mywittyname 2y ago
        Maybe. Not working at MS I can't say what their reasons were. But another way of looking at this would be that perhaps they wanted to be the beta testers of the API themselves because opening it up would have been a maintenance liability for the company. Microsoft tends to be pretty good about backwards compatibility in ways that Apple is not. We also don't know that these APIs were cancelled, they may make it into future versions of windows.
    - btilly 2y ago
      You could, and in fact this is what Microsoft wanted to do. The EU said that they couldn't. And the reason why not is simple. Anything that Microsoft thinks is a good thing to add to the API, they'll add for themselves. When the new API is released, their software is released with it. This gives them a competitive advantage over competitors who have to wait for Microsoft to have the idea that they want, and then scramble to implement it after Microsoft does. The EU is suspicious of this for the simple reason that Microsoft has a several decade history of doing exactly that. Repeatedly. My favorite example being the release of Windows 95 with Microsoft Word available at the same time, and with WordPerfect unable to run. By the time WordPerfect had figured out how to port their software to Windows 95, they were no longer the market leader.
      - ghusto 2y ago
        Interesting! I guess there's no way to fix this with further regulation either, since it would be some work to prove MS had access to the API contracts before they released them. The ultimate lesson then is to stop using MS stuff.
      - lucianbr 2y ago
        The way I see it, Microsoft sells some antivirus software, and also gets to decide who is allowed or not to compete with their antivirus software, by providing or denying access to the API. Obviously unfair.
        bluGill 2y ago
        I think anti-virus should be part of the core os. This does kill all third party vendors - good riddance to most of them, sorry if there is one that isn't evil (I'm not aware of it)
        lucianbr 2y ago
        Once the AV vendors exist, killing them, especially by Microsoft, is clearly anticompetitive. If you could prevail on a government to decide that, maybe it could work. One thing I see, is that AV has a component of maintaining a DB of signatures of bad things. This does not seem at all the job of the core os. Would the Debian team maintain such a DB?
  - jen20 2y ago
    > With the current model kernel level access is required. On Windows.
    - c0balt 2y ago
      Note: At least on Linux the main alternatives for this, either eBPF (e.g., pulsar or falcon) or a kernel module, both require this too.
      - kelnos 2y ago
        eBPF is at least somewhat sandboxed, no? So it doesn't quite have the access required to accidentally stomp on any portion of kernel memory it wants?
        c0balt 2y ago
        Indeed it's executed via a Jit on something like a VM. However it can still, make your system quite disfunctional if, e.g., all filesystem or network calls are blocked.
        vel0city 2y ago
        The version of the CrowdStrike sensor that caused kernel panics on RHEL/Rocky was using eBPF. It living in eBPF doesn't mean it can't cause system instability. And as mentioned elsewhere, an eBPF module behaving badly but in valid ways can still make your system pretty unusable.
      - jen20 2y ago
        macOS does not require this however.
  - tmm 2y ago
    > Maybe we use some of those extra protection rings? Maybe not. Intel is considering removing rings 1 and 2 for a future 64-bit only x86 architecture, because they "are unused by modern software". https://www.intel.com/content/www/us/en/developer/articles/technical/envisioning-future-simplified-architecture.html https://www.intel.com/content/www/us/en/developer/articles/t...
    - bluGill 2y ago
      I don't think those extra rings would be useful for what is needed anyway.
- jabroni_salad 2y ago
  If you don't do it, someone else will. Unless the OS is locked down to the point that even its owner cannot do that. Actually, this is something I like about Operational Technology, you run into a lot of doodads where the elevation process requires turning a physical key, and the device's main functionality is disabled while it is in service mode. Ofc the doodad has to be engineered to operate reliably, perpetually, for years, and you cant really expect that from a desktop computer.
  - bluGill 2y ago
    I have said for 20 years now that Microsoft Word should have a check on startup, if the current user is administrator it should put up a message that administrators are not allowed to use a Word Process, login as someone else. This one change would solve a lot of problems. Even on home machines where no user has a password, having to do something special to get into administrator mode will stop several attacks just because people will slow down and ask.
    - codewench 2y ago
      That's pretty much what Microsoft tried with the UAC prompts, and that was fairly universally disliked. Not that I disagree with you, running as admin by default is a terrible practice, but it's a tough sell to the general public
    - Dalewyn 2y ago
      Administrators can and should be able to do anything and everything, that is literally an administrator's job description. Also, if you want to stop everyone from using administrator accounts, the simplest way is to not have the Windows installer/OOBE setup make an administrator account first. Windows has a built-in Administrator account already not unlike Root in Linux, there is no reason (other than tradition and absolute convenience) the Windows installer/OOBE setup needs to make an administrator account for the user installing/setting up.
    - Vogtinator 2y ago
      Would that actually have a positive effect? Running malicious software in the only user's context can already cause maximum damage: https://xkcd.com/1200/ https://xkcd.com/1200/ This would just result in more UAC prompts and thus annoyed users who get taught to click on "Allow" whenever a dialog pops up.
- adrr 2y ago
  How else would you monitor a windows box? EU won't allow Microsoft to lock down their kernel and provide MacOS type solution with APIs for trust publishers.
  - imtringued 2y ago
    From what I have heard, Microsoft is allowed to do that, they merely aren't allowed to be a competitor to the software that uses the API.
- supriyo-biswas 2y ago
  > At what point does it stop becoming protection and start becoming a liability? If such outages were more frequent, then it could definitely become a liability. But such risks have to be balanced against the risk of being compromised and leaking customer data and other confidential trade secrets, and the risk posed by the latter one is far higher, not to say it's also more common.
- SkyPuncher 2y ago
  Yes. Absolutely yes. It's the only way to detect certain types of advanced threats.
- notepad0x90 2y ago
  Yes, it needs kernel access given the userspace api's available in windows. Period. not a single person who knows how the tool works and the threats it protects against has said other wise. userpace can't disable or tamper kernel space but an admin/root process in userspace can.
  - candiddevmike 2y ago
    FWIW I asked if it should require access, not what the current status quo is/what limitations exist within the OS.
    - notepad0x90 2y ago
      It isn't a status quo, it is the design of the windows operating system, as well as Linux. Macos does it's own thing but it is somewhat effective because you need to go into recovery before you can disable sysexts as root. Imagine needing to go to windows recovery environment to disable drivers, that won't fly. Apple can do that because they control the hardware and software, you rarely need to mess with sysexts as part of troubleshooting as a result. Unlike normal software development, anti-malware software has to be resilient against all kinds of tampering. The price for having an os that isn't heavily locked down and tamper resistant due to hardware enabled checks is having to rely on kernel mode code to enforce tamper resistance. Evasion is another issue, you can already hook api calls from user space (some EDRs do this) but evading it as a privileged user is trivial. It boils down to how on x86/x64 the cpu enforces 3 major privilege rings, by design things that are integrated with the OS that require OS level privileges and system wide access must run in the same ring as the OS (ring0/kernel mode). There are many ways to tackle this but I haven't heard of any (even from Microsoft's blogs/proposals after the incident) that won't reduce the capabilities and tamper/evasion resiliency of these security softwares. if x64 had a "secure world" concept like ARM for example, that would be different but it doesn't.
SoftTalker 2y ago
If not regulations, then demands by insurers for cyberattack insurance coverage.
SkyPuncher 2y ago
Basically all B2B companies are under some sort of obligation to have endpoint protection. All of these requirements essentially become transitive across a company's entire supply chain. * Big bank needs to comply with X, so do all of their vendors. * Vendor wants to sell to big bank, so they comply with X. They also need all of their vendors to comply with X. * So on and so on. ---- Ultimately, there are a lot more options than CrowdStrike, but this is a case of "Nobody gets fired for buying IBM". Even if CrowdStrike isn't the "best", it's good enough. Because it's use is sooo widespread, an issue with it often affects dozens and dozens of other companies when you're affected. One of the great things about this effect is everyone "goes down at the same time", so people don't tend to point fingers at you. In fact, they might not have any clue you're down because some other, more critical system is down internally and preventing them from accessing you. I remember a similiar situation happening a few years back. A big outage hit large parts of the internet. A pretty major part of our app got taken offline with this outage. This was a known risk and something that we accepted. We expected some backlash and inquires if this situation should ever happen. It was a calculated risk to dedicate more effort towards building customer-facing value. I think we got one inquiry. It was basically just an FYI. This person had so many things broken on their end that "one more thing" being broken was just a drop in the bucket.
- pantulis 2y ago
  Yes, this is a good summary of the situation. As a matter of fact, I guess there were quite a lot of systems and services that went down even though they were not using Crowdstrike themselves, but some part of their cloud supply chain was. I see Salesforce and Adobe were impacted in some way, probably due to the collateral Azure disruption. On the other hand, count me surprised at the sales prowess of Crowdstrike, I did not know how big they were.

This is a good writeup, but to be fair it's just not a matter of banking regulations. Basically all big companies are under similar obligations regarding endpoint protection.

candiddevmike 2y ago

Should endpoint protection require kernel level access? At what point does it stop becoming protection and start becoming a liability? Obligatory who watches/protects the watchmen/protector...

greenn 2y ago
With the current model kernel level access is required. Real security products have to be able to operate above userland. Ideally in the future there can be a layer in between userland and kernel for this sort of thing. Maybe we use some of those extra protection rings?
- ghusto 2y ago
  Couldn't you just ask some OS APIs provided by something in kernelspace for what you need? In fact, isn't this how macOS does things?
  - mywittyname 2y ago
    Microsoft was on their way to doing this, but was shot down by EU regulators because the APIs weren't available to all third-party vendors.
    - armada651 2y ago
      I think it's kind of ridiculous to then blame the regulators for the fact that Microsoft decided not to go ahead with a more competitor-friendly design. The fact that Microsoft abandoned it as soon as a regulator pointed out how anti-competitive the design of the API was makes you wonder what Microsoft's true intention was. To me that implies the anti-competitive design was its main feature and to Microsoft it would've been pointless to continue without it.
      - mywittyname 2y ago
        Maybe. Not working at MS I can't say what their reasons were. But another way of looking at this would be that perhaps they wanted to be the beta testers of the API themselves because opening it up would have been a maintenance liability for the company. Microsoft tends to be pretty good about backwards compatibility in ways that Apple is not. We also don't know that these APIs were cancelled, they may make it into future versions of windows.
  - btilly 2y ago
    You could, and in fact this is what Microsoft wanted to do. The EU said that they couldn't. And the reason why not is simple. Anything that Microsoft thinks is a good thing to add to the API, they'll add for themselves. When the new API is released, their software is released with it. This gives them a competitive advantage over competitors who have to wait for Microsoft to have the idea that they want, and then scramble to implement it after Microsoft does. The EU is suspicious of this for the simple reason that Microsoft has a several decade history of doing exactly that. Repeatedly. My favorite example being the release of Windows 95 with Microsoft Word available at the same time, and with WordPerfect unable to run. By the time WordPerfect had figured out how to port their software to Windows 95, they were no longer the market leader.
    - ghusto 2y ago
      Interesting! I guess there's no way to fix this with further regulation either, since it would be some work to prove MS had access to the API contracts before they released them. The ultimate lesson then is to stop using MS stuff.
    - lucianbr 2y ago
      The way I see it, Microsoft sells some antivirus software, and also gets to decide who is allowed or not to compete with their antivirus software, by providing or denying access to the API. Obviously unfair.
      - bluGill 2y ago
        I think anti-virus should be part of the core os. This does kill all third party vendors - good riddance to most of them, sorry if there is one that isn't evil (I'm not aware of it)
        lucianbr 2y ago
        Once the AV vendors exist, killing them, especially by Microsoft, is clearly anticompetitive. If you could prevail on a government to decide that, maybe it could work. One thing I see, is that AV has a component of maintaining a DB of signatures of bad things. This does not seem at all the job of the core os. Would the Debian team maintain such a DB?
- jen20 2y ago
  > With the current model kernel level access is required. On Windows.
  - c0balt 2y ago
    Note: At least on Linux the main alternatives for this, either eBPF (e.g., pulsar or falcon) or a kernel module, both require this too.
    - kelnos 2y ago
      eBPF is at least somewhat sandboxed, no? So it doesn't quite have the access required to accidentally stomp on any portion of kernel memory it wants?
      - c0balt 2y ago
        Indeed it's executed via a Jit on something like a VM. However it can still, make your system quite disfunctional if, e.g., all filesystem or network calls are blocked.
      - vel0city 2y ago
        The version of the CrowdStrike sensor that caused kernel panics on RHEL/Rocky was using eBPF. It living in eBPF doesn't mean it can't cause system instability. And as mentioned elsewhere, an eBPF module behaving badly but in valid ways can still make your system pretty unusable.
    - jen20 2y ago
      macOS does not require this however.
- tmm 2y ago
  > Maybe we use some of those extra protection rings? Maybe not. Intel is considering removing rings 1 and 2 for a future 64-bit only x86 architecture, because they "are unused by modern software". https://www.intel.com/content/www/us/en/developer/articles/technical/envisioning-future-simplified-architecture.html https://www.intel.com/content/www/us/en/developer/articles/t...
  - bluGill 2y ago
    I don't think those extra rings would be useful for what is needed anyway.
jabroni_salad 2y ago
If you don't do it, someone else will. Unless the OS is locked down to the point that even its owner cannot do that. Actually, this is something I like about Operational Technology, you run into a lot of doodads where the elevation process requires turning a physical key, and the device's main functionality is disabled while it is in service mode. Ofc the doodad has to be engineered to operate reliably, perpetually, for years, and you cant really expect that from a desktop computer.
- bluGill 2y ago
  I have said for 20 years now that Microsoft Word should have a check on startup, if the current user is administrator it should put up a message that administrators are not allowed to use a Word Process, login as someone else. This one change would solve a lot of problems. Even on home machines where no user has a password, having to do something special to get into administrator mode will stop several attacks just because people will slow down and ask.
  - codewench 2y ago
    That's pretty much what Microsoft tried with the UAC prompts, and that was fairly universally disliked. Not that I disagree with you, running as admin by default is a terrible practice, but it's a tough sell to the general public
  - Dalewyn 2y ago
    Administrators can and should be able to do anything and everything, that is literally an administrator's job description. Also, if you want to stop everyone from using administrator accounts, the simplest way is to not have the Windows installer/OOBE setup make an administrator account first. Windows has a built-in Administrator account already not unlike Root in Linux, there is no reason (other than tradition and absolute convenience) the Windows installer/OOBE setup needs to make an administrator account for the user installing/setting up.
  - Vogtinator 2y ago
    Would that actually have a positive effect? Running malicious software in the only user's context can already cause maximum damage: https://xkcd.com/1200/ https://xkcd.com/1200/ This would just result in more UAC prompts and thus annoyed users who get taught to click on "Allow" whenever a dialog pops up.
adrr 2y ago
How else would you monitor a windows box? EU won't allow Microsoft to lock down their kernel and provide MacOS type solution with APIs for trust publishers.
- imtringued 2y ago
  From what I have heard, Microsoft is allowed to do that, they merely aren't allowed to be a competitor to the software that uses the API.
supriyo-biswas 2y ago
> At what point does it stop becoming protection and start becoming a liability? If such outages were more frequent, then it could definitely become a liability. But such risks have to be balanced against the risk of being compromised and leaking customer data and other confidential trade secrets, and the risk posed by the latter one is far higher, not to say it's also more common.
SkyPuncher 2y ago
Yes. Absolutely yes. It's the only way to detect certain types of advanced threats.
notepad0x90 2y ago
Yes, it needs kernel access given the userspace api's available in windows. Period. not a single person who knows how the tool works and the threats it protects against has said other wise. userpace can't disable or tamper kernel space but an admin/root process in userspace can.
- candiddevmike 2y ago
  FWIW I asked if it should require access, not what the current status quo is/what limitations exist within the OS.
  - notepad0x90 2y ago
    It isn't a status quo, it is the design of the windows operating system, as well as Linux. Macos does it's own thing but it is somewhat effective because you need to go into recovery before you can disable sysexts as root. Imagine needing to go to windows recovery environment to disable drivers, that won't fly. Apple can do that because they control the hardware and software, you rarely need to mess with sysexts as part of troubleshooting as a result. Unlike normal software development, anti-malware software has to be resilient against all kinds of tampering. The price for having an os that isn't heavily locked down and tamper resistant due to hardware enabled checks is having to rely on kernel mode code to enforce tamper resistance. Evasion is another issue, you can already hook api calls from user space (some EDRs do this) but evading it as a privileged user is trivial. It boils down to how on x86/x64 the cpu enforces 3 major privilege rings, by design things that are integrated with the OS that require OS level privileges and system wide access must run in the same ring as the OS (ring0/kernel mode). There are many ways to tackle this but I haven't heard of any (even from Microsoft's blogs/proposals after the incident) that won't reduce the capabilities and tamper/evasion resiliency of these security softwares. if x64 had a "secure world" concept like ARM for example, that would be different but it doesn't.

SoftTalker 2y ago

If not regulations, then demands by insurers for cyberattack insurance coverage.

SkyPuncher 2y ago

Basically all B2B companies are under some sort of obligation to have endpoint protection. All of these requirements essentially become transitive across a company's entire supply chain. * Big bank needs to comply with X, so do all of their vendors. * Vendor wants to sell to big bank, so they comply with X. They also need all of their vendors to comply with X. * So on and so on. ---- Ultimately, there are a lot more options than CrowdStrike, but this is a case of "Nobody gets fired for buying IBM". Even if CrowdStrike isn't the "best", it's good enough. Because it's use is sooo widespread, an issue with it often affects dozens and dozens of other companies when you're affected. One of the great things about this effect is everyone "goes down at the same time", so people don't tend to point fingers at you. In fact, they might not have any clue you're down because some other, more critical system is down internally and preventing them from accessing you. I remember a similiar situation happening a few years back. A big outage hit large parts of the internet. A pretty major part of our app got taken offline with this outage. This was a known risk and something that we accepted. We expected some backlash and inquires if this situation should ever happen. It was a calculated risk to dedicate more effort towards building customer-facing value. I think we got one inquiry. It was basically just an FYI. This person had so many things broken on their end that "one more thing" being broken was just a drop in the bucket.

pantulis 2y ago
Yes, this is a good summary of the situation. As a matter of fact, I guess there were quite a lot of systems and services that went down even though they were not using Crowdstrike themselves, but some part of their cloud supply chain was. I see Salesforce and Adobe were impacted in some way, probably due to the collateral Azure disruption. On the other hand, count me surprised at the sales prowess of Crowdstrike, I did not know how big they were.