4 ms·

How do you actively detect a malware agent running in user space using stealth or a kernel. Authors of such are fully aware of Linux hardening like SELinux / Ap

How do you actively detect a malware agent running in user space using stealth or a kernel. Authors of such are fully aware of Linux hardening like SELinux / AppArmor and work around it.

amluto 2y ago
> How do you actively detect a malware agent running in user space using stealth or a kernel. You start with correct design. The system has a root of trust (ideally you skip the insane level of complexity that is Secure Boot + TPM and use something simple, testable, and verifiable — this isn’t actually that hard). Only authorized images will boot, and, more importantly, nothing else on the network trusts the machine until it proves it’s running the right image. Then you make the image immutable. Want to edit a system file? You can’t. Maybe in developer mode you can edit an overlay. All configuration is stored in a designated place, and that configuration is minimized. A stock image from the distro vendor has zero configuration, so there is no incomprehensible soup in /etc to audit. Configuration is also attested. Persistent data is separate from configuration. All persistent data is considered suspect. Any bug that allows malicious persistent data to compromise anything is a blocker, including corrupt filesystem metadata. A root-of-trust attestation has limited lifetime. The system forcibly re-verifies periodically. This either means rebooting or doing a runtime “dynamic root of trust” attenuation. The latter is complex. Complicated messes like kernel “lockdown” and the stock Secure Boot signatures have no place. Usermode root and the kernel are approximately equally trusted. SELinux is barely necessary, if at all, unless the actual user code wants it to control access to persistent data. But there are simpler, better schemes that are easier to reason about. Sadly the industry doesn’t think this way. I’m regularly surprised that Apple hasn’t gone in this direction more aggressively than they are with their MacOS products.
- worthless-trash 2y ago
  What you've answered is a great (if not the best) way to defend against attackers, but not what was asked,They asked how to detect. I'll strongly disagree on selinux, I have seen it work in practice to defeat attackers many times, that provide features that seccomp and cgroups etc do not.
  - amluto 2y ago
    re: SELinux, I think it depends on your use case. If your system is a flight information display, then you may well have two userspace processes that do anything of significance: the display manager and the actual app. There is no persistent state. At this point, SELinux is purely overhead and extra attack surface — what would it even protect. If your payload is a container (database server, microservice, whatever), and you’re doing some form of best-practice volume management, then only the database’s own data is mounted for it. SELinux is a real PITA to get working in a context like this, and it’s not really clear what it would add. (Okay, maybe you get fancy and use it to restrict what can talk to the microservice. Or maybe you use network namespaces.) If you’re running a desktop or a more conventional server setup, then, sure, MAC policy has its place.
    - ungamedplayer 2y ago
      Absolutely depends on the use case. I'm attempting to talk in the generic case. If you limit policy to the minimum attack surface from outside the process including permissions and capabilities which are significantly more fine grained in selinux compared to normal Unix permissions, you reduce the the capability of the attacker once they gain access to the system. Imagine if they got access to local code execution... Binding to sctp protocol would instantiate the whole protocol in kernel. Effectively opening up whole new attack vectors. I can't see any other techniques (other than selinux like AC) that enables this kind of attack space reduction as easily. I am aware that you can blacklist modules,etc but this is just one of many examples.
      - ongy 2y ago
        You can use SEccomp for some of it as well. But for SEccomp something in the hierarchy needs to do this actively While SELinux can be set up somewhat orthogonal to the running system. OTOH systemd should make it easy to confirm every service process
      - amluto 2y ago
        For this sort of kernel attack surface reduction, I would use a combination of seccomp and runtime module loading restriction. In the specific example of sctp, one can turn off loading of modules at runtime entirely.
- simiones 2y ago
  You haven't answered anything interesting. Any software system that anyone cares about operates on state - user documents, a database, other bespoke systems etc. If the operator of that system accidentally deploys malware to it, how to you ensure that this malware doesn't destroy, replace, or exfiltrate this state the the system normally operates on? Malicious code doesn't need to run as root in order to completely destroy a business. Not to mention, all of the things you describe are very nice if the kernel is perfectly secure. But it's not, so it's always possible and even likely that compromising any user on the system is equivalent to compromising root. And if you compromise one system, you can then exploit bugs in other systems' kernels that might allow RCE through well-crafted packets or other exploits that gain access without running through any user-space code that might validate those attestations. And finally, when a vulnerability is found allowing such exploits, you now need to update all of these readonly systems - and this happens at least once a month. Do you go with a USB stick to each of 10k systems on five continents to update them? This kind of smug "I know better than the rest of the industry, security is easy if you do things my way" is rarely productive or applicable.
- ongy 2y ago
  > Only authorized images will boo How do you do this on modern commodity hardware without secure boot? Or do you assume something in the category of embedded systems that allow to blow some efuses to get similar trusted boot?
  - amluto 2y ago
    > How do you do this on modern commodity hardware without secure boot? It’s not necessarily easy without Secure Boot, sadly. The actual straightforward solution is boot ROM. It would be nifty if someone made SD cards, eMMC devices and such meant for this use case for independent use. Most Android vendors manage to use boot ROM.
worthless-trash 2y ago
> How do you actively detect a malware agent running in user space using stealth Depending how advanced the attacker is, check the executing binary maps back to the actual expected name and location on disk. Make sure the executable and libraries used at runtime are the correct ones matching hashes of known good qualities. Ensure the process tree structure has an expected structure, ie "bash" isnt starting a process called apache. Make sure the selinux policy is correct for the process that is running. (I have no idea about apparmor) Check to see if its linking to the expected binaries, that its not using 'hidden' files (starting with a dot or directory with a dot), or deleted files. Confirm that the process is opening sockets and files that. you expect it to (ie, apache shouldn't open files that are outside its configuration directive). The process should not be making outgoing socket connections unless it is a client. It should not be running with capabilities(7) that it does not require. It should not be executing from a setuid binary. Check the process name, quite often attackers rename the running executable, so you'll see /proc/pid/cmdline renamed with a bunch of null bytes at the end. Some malware has 'anti debugging' tactics, ie, they have traced themselves to prevent you tracing them, you can find this as one of the lines in /proc/pid/status iirc. There are more, but thats the few off the top of my head. > or a kernel. This is a MUCH harder problem, because attackers can always disable any security mechanism assuming they kernel code execution. However, assuming they are not too focused.. If the system is booting in secureboot mode, it should be enabled, and no extra / unused / out of date kernel modules loaded. I know that code injection at the memory level means that attackers can inject unsigned code, so in this case you would want to periodically sample the code and ensure that execution context would only have the processers EIP in known areas where the kernel would map executable code. You could do an additional check to see if the areas are mapped by userspace processes (it might be too late) so you can find offending attackers. If the host is virtualized, this becomes easier to do and mapping and comparing memory from the guest kernel for the executable code sections means that its harder for an attacker to work around by being able to disable a mechanism. Usually attacker kernel exploits do not persist long temr in kernel space, (they abuse kernel space to allow for userspace privilege escalation ie make a binary setuid or modify permissions on a /dev/) because the longer they are there the more likely they are to panic the system. Some of the more advanced attacks I have seen are from people uploading system kernel panic images, where I have a 'snapshot' of the running system and can work around attackers mitigation techniques.
worthless-trash 2y ago
If you have good examples, I'd love to see it, A writeup even more so on the techniques they used. My findings so far in the wild (and on my honeypot) is really amateur level garbage. I spent a weekend and abused a c&c infrastructure server to fix the clients and remove the flaw and malware. I see very little sophistication there.
deleted 2y ago
[deleted]
fulafel 2y ago
This is a much harder problem than prevention, which is what the OS should be doing.

How do you actively detect a malware agent running in user space using stealth or a kernel. Authors of such are fully aware of Linux hardening like SELinux / AppArmor and work around it.

amluto 2y ago

> How do you actively detect a malware agent running in user space using stealth or a kernel. You start with correct design. The system has a root of trust (ideally you skip the insane level of complexity that is Secure Boot + TPM and use something simple, testable, and verifiable — this isn’t actually that hard). Only authorized images will boot, and, more importantly, nothing else on the network trusts the machine until it proves it’s running the right image. Then you make the image immutable. Want to edit a system file? You can’t. Maybe in developer mode you can edit an overlay. All configuration is stored in a designated place, and that configuration is minimized. A stock image from the distro vendor has zero configuration, so there is no incomprehensible soup in /etc to audit. Configuration is also attested. Persistent data is separate from configuration. All persistent data is considered suspect. Any bug that allows malicious persistent data to compromise anything is a blocker, including corrupt filesystem metadata. A root-of-trust attestation has limited lifetime. The system forcibly re-verifies periodically. This either means rebooting or doing a runtime “dynamic root of trust” attenuation. The latter is complex. Complicated messes like kernel “lockdown” and the stock Secure Boot signatures have no place. Usermode root and the kernel are approximately equally trusted. SELinux is barely necessary, if at all, unless the actual user code wants it to control access to persistent data. But there are simpler, better schemes that are easier to reason about. Sadly the industry doesn’t think this way. I’m regularly surprised that Apple hasn’t gone in this direction more aggressively than they are with their MacOS products.

worthless-trash 2y ago
What you've answered is a great (if not the best) way to defend against attackers, but not what was asked,They asked how to detect. I'll strongly disagree on selinux, I have seen it work in practice to defeat attackers many times, that provide features that seccomp and cgroups etc do not.
- amluto 2y ago
  re: SELinux, I think it depends on your use case. If your system is a flight information display, then you may well have two userspace processes that do anything of significance: the display manager and the actual app. There is no persistent state. At this point, SELinux is purely overhead and extra attack surface — what would it even protect. If your payload is a container (database server, microservice, whatever), and you’re doing some form of best-practice volume management, then only the database’s own data is mounted for it. SELinux is a real PITA to get working in a context like this, and it’s not really clear what it would add. (Okay, maybe you get fancy and use it to restrict what can talk to the microservice. Or maybe you use network namespaces.) If you’re running a desktop or a more conventional server setup, then, sure, MAC policy has its place.
  - ungamedplayer 2y ago
    Absolutely depends on the use case. I'm attempting to talk in the generic case. If you limit policy to the minimum attack surface from outside the process including permissions and capabilities which are significantly more fine grained in selinux compared to normal Unix permissions, you reduce the the capability of the attacker once they gain access to the system. Imagine if they got access to local code execution... Binding to sctp protocol would instantiate the whole protocol in kernel. Effectively opening up whole new attack vectors. I can't see any other techniques (other than selinux like AC) that enables this kind of attack space reduction as easily. I am aware that you can blacklist modules,etc but this is just one of many examples.
    - ongy 2y ago
      You can use SEccomp for some of it as well. But for SEccomp something in the hierarchy needs to do this actively While SELinux can be set up somewhat orthogonal to the running system. OTOH systemd should make it easy to confirm every service process
    - amluto 2y ago
      For this sort of kernel attack surface reduction, I would use a combination of seccomp and runtime module loading restriction. In the specific example of sctp, one can turn off loading of modules at runtime entirely.
simiones 2y ago
You haven't answered anything interesting. Any software system that anyone cares about operates on state - user documents, a database, other bespoke systems etc. If the operator of that system accidentally deploys malware to it, how to you ensure that this malware doesn't destroy, replace, or exfiltrate this state the the system normally operates on? Malicious code doesn't need to run as root in order to completely destroy a business. Not to mention, all of the things you describe are very nice if the kernel is perfectly secure. But it's not, so it's always possible and even likely that compromising any user on the system is equivalent to compromising root. And if you compromise one system, you can then exploit bugs in other systems' kernels that might allow RCE through well-crafted packets or other exploits that gain access without running through any user-space code that might validate those attestations. And finally, when a vulnerability is found allowing such exploits, you now need to update all of these readonly systems - and this happens at least once a month. Do you go with a USB stick to each of 10k systems on five continents to update them? This kind of smug "I know better than the rest of the industry, security is easy if you do things my way" is rarely productive or applicable.
ongy 2y ago
> Only authorized images will boo How do you do this on modern commodity hardware without secure boot? Or do you assume something in the category of embedded systems that allow to blow some efuses to get similar trusted boot?
- amluto 2y ago
  > How do you do this on modern commodity hardware without secure boot? It’s not necessarily easy without Secure Boot, sadly. The actual straightforward solution is boot ROM. It would be nifty if someone made SD cards, eMMC devices and such meant for this use case for independent use. Most Android vendors manage to use boot ROM.

worthless-trash 2y ago

> How do you actively detect a malware agent running in user space using stealth Depending how advanced the attacker is, check the executing binary maps back to the actual expected name and location on disk. Make sure the executable and libraries used at runtime are the correct ones matching hashes of known good qualities. Ensure the process tree structure has an expected structure, ie "bash" isnt starting a process called apache. Make sure the selinux policy is correct for the process that is running. (I have no idea about apparmor) Check to see if its linking to the expected binaries, that its not using 'hidden' files (starting with a dot or directory with a dot), or deleted files. Confirm that the process is opening sockets and files that. you expect it to (ie, apache shouldn't open files that are outside its configuration directive). The process should not be making outgoing socket connections unless it is a client. It should not be running with capabilities(7) that it does not require. It should not be executing from a setuid binary. Check the process name, quite often attackers rename the running executable, so you'll see /proc/pid/cmdline renamed with a bunch of null bytes at the end. Some malware has 'anti debugging' tactics, ie, they have traced themselves to prevent you tracing them, you can find this as one of the lines in /proc/pid/status iirc. There are more, but thats the few off the top of my head. > or a kernel. This is a MUCH harder problem, because attackers can always disable any security mechanism assuming they kernel code execution. However, assuming they are not too focused.. If the system is booting in secureboot mode, it should be enabled, and no extra / unused / out of date kernel modules loaded. I know that code injection at the memory level means that attackers can inject unsigned code, so in this case you would want to periodically sample the code and ensure that execution context would only have the processers EIP in known areas where the kernel would map executable code. You could do an additional check to see if the areas are mapped by userspace processes (it might be too late) so you can find offending attackers. If the host is virtualized, this becomes easier to do and mapping and comparing memory from the guest kernel for the executable code sections means that its harder for an attacker to work around by being able to disable a mechanism. Usually attacker kernel exploits do not persist long temr in kernel space, (they abuse kernel space to allow for userspace privilege escalation ie make a binary setuid or modify permissions on a /dev/) because the longer they are there the more likely they are to panic the system. Some of the more advanced attacks I have seen are from people uploading system kernel panic images, where I have a 'snapshot' of the running system and can work around attackers mitigation techniques.

worthless-trash 2y ago

If you have good examples, I'd love to see it, A writeup even more so on the techniques they used. My findings so far in the wild (and on my honeypot) is really amateur level garbage. I spent a weekend and abused a c&c infrastructure server to fix the clients and remove the flaw and malware. I see very little sophistication there.

deleted 2y ago

[deleted]

fulafel 2y ago

This is a much harder problem than prevention, which is what the OS should be doing.