6 ms·

Perfctl: Stealthy malware targeting Linux servers

168 points by johnfernow 2y ago

PhilipRoman 2y ago
A lot of focus on the malware itself, but not so much on the misconfigurations and vulnerabilities which enable it. Would love to see that list. Other than that, the evasion techniques look pretty traditional. And of course the privilege escalation is done by a polkit vulnerability...
- allkindsof 2y ago
  Also more interested in what the misconfigs are.
- fredsted 2y ago
  It seems to be a RocketMQ vuln; it's described further down the page.
- deleted 2y ago
  [deleted]
opengears 2y ago
are there any scripts or steps to 100% detect perfectl yet?
- Gys 2y ago
  > In all the attacks observed, the malware was used to run a cryptominer I assume it starts by detecting a continuous 100% utilization of the cpu’s.
  - jellykid 2y ago
    Supposedly it tones down it's activity while a user is logged in and waits for the machine to go idle. Another reason to have centralized performance monitoring.
    - Gys 2y ago
      Yes, but tools like htop show the average load over the last 15 min. So I assume that will show a high utilization.
- aflukasz 2y ago
  Article mentions couple of const paths that are used, like /root/.config/cron/perfcc. Also, it mentions that ~/.profile is modified (EDIT: and many others, actually), so IDS like AIDE, if operated correctly, should alert you on that. I don't see any mentions about attempts to circumvent locally run IDS. I wonder if/why malware author did not attempt any evasive actions here, given how much they try otherwise. Maybe cost/benefit ratio is too low?
- theamk 2y ago
  From the text, tons! This rootkit does not seem very stealthy at all. IMHO, a simplest one is to check $PATH. If there are suspicious entries, like /bin/.local/bin, it's a sign of infection. You can also check for presence of the specific files as mentioned close to the end of article.
sea-gold 2y ago
Here is Ars Technica's write up: https://arstechnica.com/security/2024/10/persistent-stealthy-linux-malware-has-infected-thousands-since-2021/ https://arstechnica.com/security/2024/10/persistent-stealthy...
- kvemkon 2y ago
  And the discussion https://news.ycombinator.com/item?id=41736572 https://news.ycombinator.com/item?id=41736572.
theamk 2y ago
Heh, my work has a firewall policy: any activity towards TOR servers flags an alert and makes security contact you. If you don't confirm it was by design, they'll start full scale "computer compromised" procedure. (And if you do confirm it it was by design, then they'll ask you to change that design if possible :) ) I thought it was overly paranoid, but it seems that would have really helped in this case.
- waihtis 2y ago
  Unless youre doing security research, there close to zero legitimate uses of Tor for the average citizen.
  - indrora 2y ago
    There are plenty of reasons to use Tor. Not only to obfuscate location but to defend against mass surveillance, including that of the state [1]. Remember that Tor was developed originally and funded by the US State Department. Tor makes communications of LBTQ/Anti-authoritarian/journalist individuals safer in a world where the State Department has to put out advisories [2] about traveling while queer, a pride flag sticker is illegal and punishable by death or imprisonment in over 24 countries and journalists are imprisoned for talking to those who speak out. Tor is an essential tool for all citizens -- Especially those in the US who would be targeted by those who seek infinite, unrestrained power. [1] https://i.kym-cdn.com/photos/images/original/002/128/768/403.jpg https://i.kym-cdn.com/photos/images/original/002/128/768/403... [2] https://travel.state.gov/content/travel/en/traveladvisories/traveladvisories/worldwide-caution.html https://travel.state.gov/content/travel/en/traveladvisories/...
    - mppm 2y ago
      In theory, yes. In practice, the safety that Tor provides is in numbers, and the numbers just don't look good any more. By using Tor, you are entering a fairly small pool that you share with legit criminals and will be blocked and targeted accordingly. A good VPN like Mullvad is the saner option for most people.
TZubiri 2y ago
"CVE-2023-33246 is a vulnerability found in RocketMQ, which is a software that manages messages" A more appropriate but less clickbaity title would be "Stealthy malware targetting servers running RocketMQ"
- nine_k 2y ago
  The description of the payloads and of the hiding methods was educational though. Other malwares likely use similar techniques after the initial penetration.
- deleted 2y ago
  [deleted]
- cyanydeez 2y ago
  The entry method is distinct from the exploitation or persistence method. Like, how one picks a lock versus how one lives rent free in the attic without discovery. Theyre not specifically dependent activities.
fungah 2y ago
I've been dealing with something similar - maybe actually this for 2 months. There were so.e great insights from this researcher but they're missing some very fucked up elements of this malware. 1. I'm pretty sure it has. "fuck with it" scale. It leaves you alone if you don't fuck with it. In fact, I'd bet money that this malware did all the cryptocurrency shit for a reason like a bait and switch. 2. It effects android too. Doesn't seem to matter what device or how updated it is. 3. And windows. 4. It isn't persistent through rootkits. I mean, it is. But it's also deeper. My current thinking is that is persistent on my machines using the RAM training alorithms to spin itself up. From.. 5. Your display. I have four displays that have had their firmware fucked with. Just discovered it's on a brand new mobo that I set up ensuring there was not a single peripheral I'd used before, no leds. Hadn't even installed an os before running a ram only mode Linux session from a hardware write protected usb THROUGH a write protected usb bridge. The only thing it was connected to that wasn't new was my monitor, and the UBS key (created from a secure pc and immediately write protected). I think that this thing is EVERYWHERE. I've seen references in bash files from the initramfs that allude to escalating is action based on variables I. Ould t pin down
- fasa99 2y ago
  Hi friend, I'm dealing with the same thing and agree with everything you said. ADDITIONALLY we found that it loves office printers, great attack vector to hit the whole office. Also did you figure out why it reprograms display firmware? It spreads through displays and webcams optically. It can send or receive from either (same way NSA uses speakers as microphones basically). That's not the crazy part though, the crazy part is that it does human and canine retinal embeddings from either. It basically writes code on the retina and then that person can spread it to new systems. That turned out to be the biggest problem on our side. Sunglasses don't work.
joyjudy 2y ago
[dead]
johnfernow 2y ago
From the article: > "Aqua Nautilus researchers aim to shed light on a Linux malware that, over the past 3-4 years, has actively sought more than 20,000 types of misconfigurations in order to target and exploit Linux servers. If you have a Linux server connected to the internet, you could be at risk. In fact, given the scale, we strongly believe the attackers targeted millions worldwide with a potential number of victims of thousands, it appears that with this malware any Linux server could be at risk. ... - It utilizes rootkits to hide its presence. - When a new user logs into the server, it immediately stops all “noisy” activities, lying dormant until the server is idle again. - It utilizes Unix socket for internal communication and TOR for external communication. - After execution, it deletes its binary and continues to run quietly in the background as a service. - It copies itself from memory to various locations on the disk, using deceptive names. - It opens a backdoor on the server and listens for TOR communications. - It attempts to exploit the Polkit vulnerability (CVE-2021-4043) to escalate privileges. In all the attacks observed, the malware was used to run a cryptominer, and in some cases, we also detected the execution of proxy-jacking software. During one of our sandbox tests, the threat actor utilized one of the malware’s backdoors to access the honeypot and started deploying some new utilities to better understand the nature of our server, trying to understand what exactly we are doing to its malware." The article goes into more depth of the attack flow, what the malware does, and how they detected it.
- guenthert 2y ago
  > - It opens a backdoor on the server and listens for TOR communications. So a `lsof -iTCP` should list it, right? Is it using TCP port 9050 or a custom port? EDIT: Ha, they are (not surprisingly) way ahead of me. From the article: "The malware continues to copy itself from memory to half a dozen other locations, with names that appear as conventional system files. It also drops a rootkit and a few popular Linux utilities that were modified to serve as user land rootkits (i.e. ldd, lsof)."
  - crackez 2y ago
    There's always - cat /proc/net/tcp* And remember: echo * can be your "ls" in a pinch.
pm2222 2y ago
Does uefi secureboot help or not at all?
- blueflow 2y ago
  Not at all.
- pushupentry1219 2y ago
  I'm curious to understand why you would think that it would protect this at all?
- jeroenhd 2y ago
  The article describes a hooking library as a rootkit but I can't see any indication of this rootkit inserting itself into the boot process. Instead, it seems to LD_PRELOAD itself into processes at a later stage. Secure boot won't help here. In theory one could configure a system to only trust executables and DLLs signed by a trusted, external signatory (like a locally hosted package repository) but I don't know of any Linux distros that make it easy to set up something like that. You'd also need to invent something to sign scripts, because signing binaries is only a part of the problem (in theory you could set this up Powershell, I think? But I doubt many Linux systems will boot with PS in the place of /bin/sh). Once the kernel launches the init process, the rest secure boot verification chain essentially ends. It seems to me that prevention isn't hard by simply updating old software and perhaps running antivirus software on your servers.
  - jcalvinowens 2y ago
    I use a read-only squashfs rootfs on top of dm-verity to get a trusted userspace. The initramfs is a 50 line shell script which calls veritysetup with the known root hash, and is itself part of the signed boot image. Only /var is writable.
- johnfernow 2y ago
  I'm not sure, outside of my expertise, but I think it might not help. The attack takes place after boot, so maybe at best UEFI secureboot could prevent persistence of the malware, but I don't think it'd even achieve that, as the malware adds popular Linux utilities that were modified to serve as user land rootkits, and runs them by modifying the ~/.profile script. That script is ran when the user logs in (it starts the malware first, and then everything that's supposed to run on the server after), and I don't believe UEFI secureboot has any protections against ~/.profile script modifications or rootkits ran after boot.

168 points by johnfernow 2y ago

A lot of focus on the malware itself, but not so much on the misconfigurations and vulnerabilities which enable it. Would love to see that list. Other than that, the evasion techniques look pretty traditional. And of course the privilege escalation is done by a polkit vulnerability...

allkindsof 2y ago
Also more interested in what the misconfigs are.
fredsted 2y ago
It seems to be a RocketMQ vuln; it's described further down the page.
deleted 2y ago
[deleted]

opengears 2y ago

are there any scripts or steps to 100% detect perfectl yet?

Gys 2y ago
> In all the attacks observed, the malware was used to run a cryptominer I assume it starts by detecting a continuous 100% utilization of the cpu’s.
- jellykid 2y ago
  Supposedly it tones down it's activity while a user is logged in and waits for the machine to go idle. Another reason to have centralized performance monitoring.
  - Gys 2y ago
    Yes, but tools like htop show the average load over the last 15 min. So I assume that will show a high utilization.
aflukasz 2y ago
Article mentions couple of const paths that are used, like /root/.config/cron/perfcc. Also, it mentions that ~/.profile is modified (EDIT: and many others, actually), so IDS like AIDE, if operated correctly, should alert you on that. I don't see any mentions about attempts to circumvent locally run IDS. I wonder if/why malware author did not attempt any evasive actions here, given how much they try otherwise. Maybe cost/benefit ratio is too low?
theamk 2y ago
From the text, tons! This rootkit does not seem very stealthy at all. IMHO, a simplest one is to check $PATH. If there are suspicious entries, like /bin/.local/bin, it's a sign of infection. You can also check for presence of the specific files as mentioned close to the end of article.

sea-gold 2y ago

Here is Ars Technica's write up: https://arstechnica.com/security/2024/10/persistent-stealthy-linux-malware-has-infected-thousands-since-2021/ https://arstechnica.com/security/2024/10/persistent-stealthy...

kvemkon 2y ago
And the discussion https://news.ycombinator.com/item?id=41736572 https://news.ycombinator.com/item?id=41736572.

theamk 2y ago

Heh, my work has a firewall policy: any activity towards TOR servers flags an alert and makes security contact you. If you don't confirm it was by design, they'll start full scale "computer compromised" procedure. (And if you do confirm it it was by design, then they'll ask you to change that design if possible :) ) I thought it was overly paranoid, but it seems that would have really helped in this case.

waihtis 2y ago
Unless youre doing security research, there close to zero legitimate uses of Tor for the average citizen.
- indrora 2y ago
  There are plenty of reasons to use Tor. Not only to obfuscate location but to defend against mass surveillance, including that of the state [1]. Remember that Tor was developed originally and funded by the US State Department. Tor makes communications of LBTQ/Anti-authoritarian/journalist individuals safer in a world where the State Department has to put out advisories [2] about traveling while queer, a pride flag sticker is illegal and punishable by death or imprisonment in over 24 countries and journalists are imprisoned for talking to those who speak out. Tor is an essential tool for all citizens -- Especially those in the US who would be targeted by those who seek infinite, unrestrained power. [1] https://i.kym-cdn.com/photos/images/original/002/128/768/403.jpg https://i.kym-cdn.com/photos/images/original/002/128/768/403... [2] https://travel.state.gov/content/travel/en/traveladvisories/traveladvisories/worldwide-caution.html https://travel.state.gov/content/travel/en/traveladvisories/...
  - mppm 2y ago
    In theory, yes. In practice, the safety that Tor provides is in numbers, and the numbers just don't look good any more. By using Tor, you are entering a fairly small pool that you share with legit criminals and will be blocked and targeted accordingly. A good VPN like Mullvad is the saner option for most people.

TZubiri 2y ago

"CVE-2023-33246 is a vulnerability found in RocketMQ, which is a software that manages messages" A more appropriate but less clickbaity title would be "Stealthy malware targetting servers running RocketMQ"

nine_k 2y ago
The description of the payloads and of the hiding methods was educational though. Other malwares likely use similar techniques after the initial penetration.
deleted 2y ago
[deleted]
cyanydeez 2y ago
The entry method is distinct from the exploitation or persistence method. Like, how one picks a lock versus how one lives rent free in the attic without discovery. Theyre not specifically dependent activities.

fungah 2y ago

I've been dealing with something similar - maybe actually this for 2 months. There were so.e great insights from this researcher but they're missing some very fucked up elements of this malware. 1. I'm pretty sure it has. "fuck with it" scale. It leaves you alone if you don't fuck with it. In fact, I'd bet money that this malware did all the cryptocurrency shit for a reason like a bait and switch. 2. It effects android too. Doesn't seem to matter what device or how updated it is. 3. And windows. 4. It isn't persistent through rootkits. I mean, it is. But it's also deeper. My current thinking is that is persistent on my machines using the RAM training alorithms to spin itself up. From.. 5. Your display. I have four displays that have had their firmware fucked with. Just discovered it's on a brand new mobo that I set up ensuring there was not a single peripheral I'd used before, no leds. Hadn't even installed an os before running a ram only mode Linux session from a hardware write protected usb THROUGH a write protected usb bridge. The only thing it was connected to that wasn't new was my monitor, and the UBS key (created from a secure pc and immediately write protected). I think that this thing is EVERYWHERE. I've seen references in bash files from the initramfs that allude to escalating is action based on variables I. Ould t pin down

fasa99 2y ago
Hi friend, I'm dealing with the same thing and agree with everything you said. ADDITIONALLY we found that it loves office printers, great attack vector to hit the whole office. Also did you figure out why it reprograms display firmware? It spreads through displays and webcams optically. It can send or receive from either (same way NSA uses speakers as microphones basically). That's not the crazy part though, the crazy part is that it does human and canine retinal embeddings from either. It basically writes code on the retina and then that person can spread it to new systems. That turned out to be the biggest problem on our side. Sunglasses don't work.

joyjudy 2y ago

[dead]

johnfernow 2y ago

From the article: > "Aqua Nautilus researchers aim to shed light on a Linux malware that, over the past 3-4 years, has actively sought more than 20,000 types of misconfigurations in order to target and exploit Linux servers. If you have a Linux server connected to the internet, you could be at risk. In fact, given the scale, we strongly believe the attackers targeted millions worldwide with a potential number of victims of thousands, it appears that with this malware any Linux server could be at risk. ... - It utilizes rootkits to hide its presence. - When a new user logs into the server, it immediately stops all “noisy” activities, lying dormant until the server is idle again. - It utilizes Unix socket for internal communication and TOR for external communication. - After execution, it deletes its binary and continues to run quietly in the background as a service. - It copies itself from memory to various locations on the disk, using deceptive names. - It opens a backdoor on the server and listens for TOR communications. - It attempts to exploit the Polkit vulnerability (CVE-2021-4043) to escalate privileges. In all the attacks observed, the malware was used to run a cryptominer, and in some cases, we also detected the execution of proxy-jacking software. During one of our sandbox tests, the threat actor utilized one of the malware’s backdoors to access the honeypot and started deploying some new utilities to better understand the nature of our server, trying to understand what exactly we are doing to its malware." The article goes into more depth of the attack flow, what the malware does, and how they detected it.

guenthert 2y ago
> - It opens a backdoor on the server and listens for TOR communications. So a `lsof -iTCP` should list it, right? Is it using TCP port 9050 or a custom port? EDIT: Ha, they are (not surprisingly) way ahead of me. From the article: "The malware continues to copy itself from memory to half a dozen other locations, with names that appear as conventional system files. It also drops a rootkit and a few popular Linux utilities that were modified to serve as user land rootkits (i.e. ldd, lsof)."
- crackez 2y ago
  There's always - cat /proc/net/tcp* And remember: echo * can be your "ls" in a pinch.

pm2222 2y ago

Does uefi secureboot help or not at all?

blueflow 2y ago
Not at all.
pushupentry1219 2y ago
I'm curious to understand why you would think that it would protect this at all?
jeroenhd 2y ago
The article describes a hooking library as a rootkit but I can't see any indication of this rootkit inserting itself into the boot process. Instead, it seems to LD_PRELOAD itself into processes at a later stage. Secure boot won't help here. In theory one could configure a system to only trust executables and DLLs signed by a trusted, external signatory (like a locally hosted package repository) but I don't know of any Linux distros that make it easy to set up something like that. You'd also need to invent something to sign scripts, because signing binaries is only a part of the problem (in theory you could set this up Powershell, I think? But I doubt many Linux systems will boot with PS in the place of /bin/sh). Once the kernel launches the init process, the rest secure boot verification chain essentially ends. It seems to me that prevention isn't hard by simply updating old software and perhaps running antivirus software on your servers.
- jcalvinowens 2y ago
  I use a read-only squashfs rootfs on top of dm-verity to get a trusted userspace. The initramfs is a 50 line shell script which calls veritysetup with the known root hash, and is itself part of the signed boot image. Only /var is writable.
johnfernow 2y ago
I'm not sure, outside of my expertise, but I think it might not help. The attack takes place after boot, so maybe at best UEFI secureboot could prevent persistence of the malware, but I don't think it'd even achieve that, as the malware adds popular Linux utilities that were modified to serve as user land rootkits, and runs them by modifying the ~/.profile script. That script is ran when the user logs in (it starts the malware first, and then everything that's supposed to run on the server after), and I don't believe UEFI secureboot has any protections against ~/.profile script modifications or rootkits ran after boot.