7 ms·

Linux Threat Hunting: ‘Syslogk’ a kernel rootkit found in the wild

140 points by rmdoss 4y ago

28304283409234 4y ago
Seems to only relate to RHEL 6, or derivatives of, such as CentOS 6. Yes: 6. Which is as EOL as enterprise software gets: https://access.redhat.com/support/policy/updates/errata#Life_Cycle_Dates https://access.redhat.com/support/policy/updates/errata#Life...
- Xylakant 4y ago
  RHEL 6 is in Extended Lifecycle Support until June 2024 (that is: customers with suitable subscriptions still get critical patches). It’s a zombie, but it’s not quite dead yet. I’d bet that there are still enough (+) people out there running it. (+) or rather too many.
  - scns 4y ago
    There are paying customers. It might not be shiny/fun, but there is a reason Red Hat became the first one-billion dollar open-source company in 2012.
    - Jedd 4y ago
      Wouldn't IBM be the title holders there?
      - deleted 4y ago
        [deleted]
      - capableweb 4y ago
        "open-source company" is kind of an subjective term at this point I guess, so hard to say. But what can be said, is that even though IBM contributes a lot to open source, not many would claim IBM is a "open-source company" I think, at least when compared to Red Hat.
        Jedd 4y ago
        Yeah, I get that distinction, and the vagueness of the phrase 'open source' doesn't help with this kind of definition. If IBM put US$1B into 'Linux' 22 years ago - but this was a small part of their operating budget at the time - do we look at absolute or comparative value? If IBM buys, three years ago, RedHat for US$34B, does that mean IBM is now the biggest 'open source' company? (If so, the next question is obvious.)
  - quags 4y ago
    For centos6 there is cloudlinux also providing extended release support (paid) for those who don't have a RHEL subscription until 2024.
- sheepdestroyer 4y ago
  Comparatively, RHEL 6 is still kind of fine, at least it is still officially supported as virtualized OS in oVirt... We run a lot of CentOS 5 virtual machines (and some physical ones! ; and some RHEL4! , and a few Fedora core 8 and 4 !!!), with no end in sight... :( It is a huge concern for the Infra team, a source of many headaches, and we need to go through oops to keep them running, but: - Clients don't want to move from OLDPRODUCT that requires extremely old php. - Dev team is not interested in migrating OLDPRODUCT to a modern platform, or even try to put it in a container. Their eyes are turned to the shiny NEWPRODUCT that is seemingly never fully coming to production (only one client has signed for it). - New clients are still regularly signed on OLDPRODUCT. - No one in the org wants to pay for for a migration anyway. - Since some clients have complained about poor apparent security, what was visible was just hidden behind newer haproxy.
  - semi-extrinsic 4y ago
    OT, but: why don't you just compile up an old php version from source on a new OS? It's a bit of a hassle the first time you do it, sure, but less than the hassle of running multiple legacy OS?
    - bombcar 4y ago
      The legacy OS is even less hassle, because it never gets any updates and just sits there. If you compile old software on a new OS, every single update to that OS has a chance to blow up your compiled old version, so it takes more hand-holding. Can be worth it at times, but other times it's just easier to firewall and hope for the best.
  - carlmr 4y ago
    >NEWPRODUCT that is seemingly never fully coming to production >- New clients are still regularly signed on OLDPRODUCT. I mean what's the WHY behind that? Why don't even new customers sign on to the new product? Why is the new product not in production? Is that the same reason?
    - bombcar 4y ago
- kibwen 4y ago
  Despite that, RHEL 6 appears to be entrenched. Rust has been proposing to update its baseline Linux and glibc versions to circa-2012 vintage, which would exclude RHEL 6, and has been receiving pushback from people whose customers still use RHEL 6.
wazari972 4y ago
> To load the rootkit into kernel space, it is necessary to approximately match the kernel version used for compiling; it does not have to be strictly the same. >> vermagic=2.6.32-696.23.1.el6.x86_64 SMP mod_unload modversions do you know why they say "approximately match"? I thought it had to match exactly so that the kernel accepts to load the module
- chr15p 4y ago
  A kernel module doesn't have to match the kernel version, it has to be able to resolve all the symbols (function calls, variables etc) it uses into valid symbols supplied by the kernel you are loading on. The greater the difference between the kernel version you compiled for, and the kernel version you are trying to load it on, the greater the chance something you are relying on changed and the module loader cant resolve all the symbols and so it fails. So saying a kmod has to match the kernel version is good practice but the reality is not quite as strict. Red Hat has a list of "white listed" symbols that they try to maintain across a major version of RHEL so if your kmod only relies on them and nothing else then it should load on any kernel version within that release. But that's a Red Hat thing, not a Linux kernel thing.
  - xyzzy123 4y ago
    Perhaps also worth noting that rootkits don't have to follow the usual rules; you don't have to rely on the kernel linker if you don't want to. (Tradeoff of runtime DIY symbol resolution / code grovelling being it's more work, and more likely to be crashy). As a rootkit author you have considerably more flexibility than most module authors who are constrained by "sanity", maintainability, accepted practice and licensing terms.
- yjftsjthsd-h 4y ago
  I don't know the exact rules, but note that this is targeting RHEL6 and Red Hat makes a deliberate effort to preserve kernel ABI compatibility so it is probably a lot easier than on most Linux distributions.
rollcat 4y ago
OpenBSD has removed loadable kernel modules back in 2014; macOS is aggressively moving in the same direction. Meanwhile - is running a Linux system without module support even viable these days? $ du -sh /lib/modules/$(uname -r) 294M /lib/modules/5.10.0-15-amd64
- lapsis_beeftech 4y ago
  Certainly viable but your non-modular system might not support all the features you want. The Linux system I am using to post this comment is running without CONFIG_MODULES and have been so for years, but I am not using ZFS nor anything from NVIDIA.
- shaded-enmity 4y ago
  How is the vector of persistence of any significance here? At some point the attackers got root access on your system, game over.
  - cb321 4y ago
    I believe the concern is that the attackers gain root access on system A but hide their presence/activity - even in the presence of logs to remote, more trusted server B. https://github.com/c-blake/kslog https://github.com/c-blake/kslog has maybe a little more color on this topic, though I'm sure there are whole volumes written about it elsewhere. :) EDIT: But maybe your "game over" point is just that it is kind of a pipe dream to hope to block all concealment tactics? That may be fair, but I think a lot of security folks cling to that dream. :)
    - shaded-enmity 4y ago
      > I believe the concern is that the attackers gain root access on system A but hide their presence/activity - even in the presence of logs to remote, more trusted server B. That's generally called pivoting and has nothing to do with method of persistence of the malicious code. OP makes a point that certain systems move or have moved away from giving root user the ability to extend/modify kernel code at runtime via kernel modules, my argument is that none of that matters since root user can still extend/modify kernel code at runtime via binary patching.

140 points by rmdoss 4y ago

Seems to only relate to RHEL 6, or derivatives of, such as CentOS 6. Yes: 6. Which is as EOL as enterprise software gets: https://access.redhat.com/support/policy/updates/errata#Life_Cycle_Dates https://access.redhat.com/support/policy/updates/errata#Life...

Xylakant 4y ago
RHEL 6 is in Extended Lifecycle Support until June 2024 (that is: customers with suitable subscriptions still get critical patches). It’s a zombie, but it’s not quite dead yet. I’d bet that there are still enough (+) people out there running it. (+) or rather too many.
- scns 4y ago
  There are paying customers. It might not be shiny/fun, but there is a reason Red Hat became the first one-billion dollar open-source company in 2012.
  - Jedd 4y ago
    Wouldn't IBM be the title holders there?
    - deleted 4y ago
      [deleted]
    - capableweb 4y ago
      "open-source company" is kind of an subjective term at this point I guess, so hard to say. But what can be said, is that even though IBM contributes a lot to open source, not many would claim IBM is a "open-source company" I think, at least when compared to Red Hat.
      - Jedd 4y ago
        Yeah, I get that distinction, and the vagueness of the phrase 'open source' doesn't help with this kind of definition. If IBM put US$1B into 'Linux' 22 years ago - but this was a small part of their operating budget at the time - do we look at absolute or comparative value? If IBM buys, three years ago, RedHat for US$34B, does that mean IBM is now the biggest 'open source' company? (If so, the next question is obvious.)
- quags 4y ago
  For centos6 there is cloudlinux also providing extended release support (paid) for those who don't have a RHEL subscription until 2024.
sheepdestroyer 4y ago
Comparatively, RHEL 6 is still kind of fine, at least it is still officially supported as virtualized OS in oVirt... We run a lot of CentOS 5 virtual machines (and some physical ones! ; and some RHEL4! , and a few Fedora core 8 and 4 !!!), with no end in sight... :( It is a huge concern for the Infra team, a source of many headaches, and we need to go through oops to keep them running, but: - Clients don't want to move from OLDPRODUCT that requires extremely old php. - Dev team is not interested in migrating OLDPRODUCT to a modern platform, or even try to put it in a container. Their eyes are turned to the shiny NEWPRODUCT that is seemingly never fully coming to production (only one client has signed for it). - New clients are still regularly signed on OLDPRODUCT. - No one in the org wants to pay for for a migration anyway. - Since some clients have complained about poor apparent security, what was visible was just hidden behind newer haproxy.
- semi-extrinsic 4y ago
  OT, but: why don't you just compile up an old php version from source on a new OS? It's a bit of a hassle the first time you do it, sure, but less than the hassle of running multiple legacy OS?
  - bombcar 4y ago
    The legacy OS is even less hassle, because it never gets any updates and just sits there. If you compile old software on a new OS, every single update to that OS has a chance to blow up your compiled old version, so it takes more hand-holding. Can be worth it at times, but other times it's just easier to firewall and hope for the best.
- carlmr 4y ago
  >NEWPRODUCT that is seemingly never fully coming to production >- New clients are still regularly signed on OLDPRODUCT. I mean what's the WHY behind that? Why don't even new customers sign on to the new product? Why is the new product not in production? Is that the same reason?
  - bombcar 4y ago
kibwen 4y ago
Despite that, RHEL 6 appears to be entrenched. Rust has been proposing to update its baseline Linux and glibc versions to circa-2012 vintage, which would exclude RHEL 6, and has been receiving pushback from people whose customers still use RHEL 6.

wazari972 4y ago

> To load the rootkit into kernel space, it is necessary to approximately match the kernel version used for compiling; it does not have to be strictly the same. >> vermagic=2.6.32-696.23.1.el6.x86_64 SMP mod_unload modversions do you know why they say "approximately match"? I thought it had to match exactly so that the kernel accepts to load the module

chr15p 4y ago
A kernel module doesn't have to match the kernel version, it has to be able to resolve all the symbols (function calls, variables etc) it uses into valid symbols supplied by the kernel you are loading on. The greater the difference between the kernel version you compiled for, and the kernel version you are trying to load it on, the greater the chance something you are relying on changed and the module loader cant resolve all the symbols and so it fails. So saying a kmod has to match the kernel version is good practice but the reality is not quite as strict. Red Hat has a list of "white listed" symbols that they try to maintain across a major version of RHEL so if your kmod only relies on them and nothing else then it should load on any kernel version within that release. But that's a Red Hat thing, not a Linux kernel thing.
- xyzzy123 4y ago
  Perhaps also worth noting that rootkits don't have to follow the usual rules; you don't have to rely on the kernel linker if you don't want to. (Tradeoff of runtime DIY symbol resolution / code grovelling being it's more work, and more likely to be crashy). As a rootkit author you have considerably more flexibility than most module authors who are constrained by "sanity", maintainability, accepted practice and licensing terms.
yjftsjthsd-h 4y ago
I don't know the exact rules, but note that this is targeting RHEL6 and Red Hat makes a deliberate effort to preserve kernel ABI compatibility so it is probably a lot easier than on most Linux distributions.

rollcat 4y ago

OpenBSD has removed loadable kernel modules back in 2014; macOS is aggressively moving in the same direction. Meanwhile - is running a Linux system without module support even viable these days? $ du -sh /lib/modules/$(uname -r) 294M /lib/modules/5.10.0-15-amd64

lapsis_beeftech 4y ago
Certainly viable but your non-modular system might not support all the features you want. The Linux system I am using to post this comment is running without CONFIG_MODULES and have been so for years, but I am not using ZFS nor anything from NVIDIA.
shaded-enmity 4y ago
How is the vector of persistence of any significance here? At some point the attackers got root access on your system, game over.
- cb321 4y ago
  I believe the concern is that the attackers gain root access on system A but hide their presence/activity - even in the presence of logs to remote, more trusted server B. https://github.com/c-blake/kslog https://github.com/c-blake/kslog has maybe a little more color on this topic, though I'm sure there are whole volumes written about it elsewhere. :) EDIT: But maybe your "game over" point is just that it is kind of a pipe dream to hope to block all concealment tactics? That may be fair, but I think a lot of security folks cling to that dream. :)
  - shaded-enmity 4y ago
    > I believe the concern is that the attackers gain root access on system A but hide their presence/activity - even in the presence of logs to remote, more trusted server B. That's generally called pivoting and has nothing to do with method of persistence of the malicious code. OP makes a point that certain systems move or have moved away from giving root user the ability to extend/modify kernel code at runtime via kernel modules, my argument is that none of that matters since root user can still extend/modify kernel code at runtime via binary patching.