6 ms·

I expect a lot of people will be doing a whole lot of thinking along these lines over the next months. Code review? Some kind of behavioral analysis? IMO the

by wepple 3y ago

I expect a lot of people will be doing a whole lot of thinking along these lines over the next months. Code review? Some kind of behavioral analysis? IMO the call to system() was kind of sloppy, and a binary capabilities scanner could have potentially identified a path to that.

tux3 3y ago
I think behavioral analysis could be promising. There's a lot of weird stuff this code does on startup that any reasonable Debian package on the average install should not be doing in a million years. Games and proprietary software will sometimes ship with DRM protection layers that do insane things in the name of obfuscation, making it hard to distinguish from malware. But (with only a couple exceptions) there's no reason for a binary or library in a Debian package to ever try to write the PLT outside of the normal mechanism, to try to overwrite symbols in other modules, to add LD audit hooks on startup, to try to resolve things manually by walking ELF structures, to do anti-debug tricks, or just to have any kind of obfuscation or packing that free software packaged for a distro is not supposed to have. Some of these may be (much) more difficult to detect than others, some might not be realistic. But there are several plausible different ways a scanner could have detected something weird going on in memory during ssh startup. No one wants a Linux antivirus. But I think everyone would benefit from throwing all the behavioral analysis we can come up with at new Debian package uploads. We're very lucky someone noticed this one, we may not have the same luck next time.
- raggi 3y ago
  Except had we been doing that they would have put guards in place to detect it - as they already had guards to avoid the code path when a debugger is attached, to avoid building the payload in when it's not one of the target systems, and so on. Their evasion was fairly extensive, so we'd need many novel dynamic systems to stand a chance, and we'd have to guard those systems extremely tightly - the author got patches into oss-fuzz as well to "squash false positives". All in all, adding more arms to the arms race does raise the bar, but the bar they surpassed already demonstrated tenacity, long term thinking, and significant defense and detection evasion efforts.
  - tux3 3y ago
    I broadly agree, but I think we can draw a parallel with the arms race of new exploit techniques versus exploit protection. People still manage to write exploits today, but now you must find an ASLR leak, you must chain enough primitives to work around multiple layers of protection, it's generally a huge pain to write exploits compared to the 90s. Today the dynamic detection that we have for Linux packages seems thin to non-existent, like the arms race has not even started yet. I think there is a bit of low-hanging fruit to make attacker lives harder (and some much higher-hanging fruit that would be a real headache). Luckily there is an asymmetry in favor of the defenders (for once). If we create a scanner, we do not _have_ to publish every type of scan it knows how to do. Much like companies fighting spammers and fraud don't detail exactly how they catch bad actors. (Or, for another example, I know the Tor project has a similar asymmetry to detect bad relays. They collaborate on their relay scanner internally, but no one externally knows all the details.)
    - raggi 3y ago
      Yeah, perhaps something akin to an OSS variant of virustotal's multi-vendor analysis. I'm still not sure it would catch this, but as you say, raising the bar isn't something we tend to regret.
    - saagarjha 3y ago
      This is an arms race that is largely won by attackers, actually. Sophisticated attacks are caught by them sometimes but usually the author has far more knowledge or cleverer tricks than the person implementing the checks, who is limited by their imagination of what they think an attacker might do.
      - wepple 3y ago
        I’m not sure that attackers “win” this arms race; the cost has gone through the roof and their chance of detection constantly increases. Indeed, defenders are often outgunned, but at a minimum we’ve pushed the level of effort way into nation-state territory, which is a good start.
- snnn 3y ago
  > to try to overwrite symbols in other modules, to add LD audit hooks on startup, to try to resolve things manually by walking ELF structures I want to name one thing: when Windows failed to load a DLL because a dependency was missing, it doesn't tell you what was missed. To get the information, you have to interact with the DLL loader with low level Windows APIs. In some circumstances Linux apps may also have the need. Like for printing a user friendly error message or recovery from a non-fatal error. For example, the patchelf tool that is used for building portable python packages. > No one wants a Linux antivirus It is not true. Actually these software are very popular in enterprise settings.
- ffsm8 3y ago
  > No one wants a Linux antivirus ClamAV has been around for a very long time at this point. It's just not installed on servers, usually
  - kemotep 3y ago
    To say nothing of enterprise EDR/XDR solutions that have linux versions. These things aren’t bulletproof but can be 1 layer in your multilayer security posture.
  - EasyMark 3y ago
    don't most people who use that just use it for scanning incoming email attachments usually?
    - snnn 3y ago
      ClamAV also has a lot of findings when scanning some open source project's source code. For example, LLVM project's test data. Because some of the test data are meant to check if a known security bug is fixed, from a antivirus software perspective these data files can be seen as exploits. ClamAV is commonly used. Or, I would suggest adding it to every CI build pipeline. Most time it wouldn't have any finding, but it is better than nothing. I would like to offer free help if an open source project has the need to harden their build pipelines and their release process.
      - everybackdoor 3y ago
        [dead]
  - snnn 3y ago
    Does not have to be installed. See this: https://learn.microsoft.com/en-us/azure/defender-for-cloud/concept-agentless-data-collection#how-agentless-scanning-works https://learn.microsoft.com/en-us/azure/defender-for-cloud/c... A cloud provider can take snapshots of running VMs then run antivirus scan offline to minimize the impact to the customers. Similarly, many applications are containerized and the containers are stateless, we can scan the docker images instead. This approach has been quite mature.
- robocat 3y ago
  > we may not have the same luck next time If the prior is 1 was out there (this one), the chances that there is 1+ still undetected seems fairly high to me. To behaviourally detect this requires many independent actors to be looking in independent ways(e.g. security researchers, internal teams). Edit: I mean with private code & tests (not open source, nor purchasable antivirus). It's not easy to donate to Google Zero. Some of the best funded and most skilled teams seem to be antivirus vendors (and high value person protection). I hate the antivirus industry yet I've been helped by it (the anti-tragedy of the commons). Commonly public detection code (e.g. open source) is likely to be defeated by attackers with a lot of resources. Hard to protect ourselves against countries where the individuals are safe from prosecution. Even nefarious means like assasination likely only work against individuals and not teams.
  - bandrami 3y ago
    > If the prior is 1 That would surprise me greatly
    - lazyasciiart 3y ago
      I think you’re saying “I would be surprised if there is only 1 exploit like this that already exists” which is what the previous comment was also saying. “If the prior is one” is often used to mean “we know for sure that there is one”.
- cryptonector 3y ago
  > But I think everyone would benefit from throwing all the behavioral analysis we can come up with at new Debian package uploads. Why "new uploads" and not also "all existing"?
- cipherzero 3y ago
  Sorry, I'm unfamiliar with PLT what does stand for?
  - intelVISA 3y ago
    procedure linkage table
x0x0 3y ago
[flagged]
- raggi 3y ago
  Think whatever you shall about systemd of course, but please stop with the blind belief mud slinging: - systemd didn't create the patch to include libsystemd, distros did - current systemd versions already remove liblzma from their dependencies, the affected distros are behind on systemd updates though - you can implement notify in standalone code in about the same effort as it takes to use the dependency, there wasn't really a good reason for distro's to be adding this dependency to such a critical binary. systemd documents the protocol independently to make this easy. distros having sketchy patches to sshd has a long history, remember the debian weak key fiasco?
  - Denvercoder9 3y ago
    > - current systemd versions already remove liblzma from their dependencies, the affected distros are behind on systemd updates though The affected distros aren't behind on systemd updates, the change to systemd you describe has been merged but not yet released.
    - raggi 3y ago
      Ah, thank you for the correction!
      - x0x0 3y ago
        Maybe you should reconsider your "blind belief mud slinging" eh? Be better.
  - everybackdoor 3y ago
    [flagged]
    - rcxdude 3y ago
jnwatson 3y ago
The real problem was doing expensive math for every connection. If it had relied on a cookie or some simpler-to-compute pre-filter, no one would have been the wiser.
- anarazel 3y ago
  The slowdown is actually in the startup of the backdoor, not when it's actually performing authentication. Note how in the original report even sshd -h (called in the right environment to circumvent countermeasures) is slow.
  - klabb3 3y ago
    Wow. Given the otherwise extreme sophistication this is such a blunder. I imagine the adversary is tearing their hair out over this. 2-3 years of full time infiltration work down the drain, for probably more than a single person. As for the rest of us, we got lucky. In fact, it’s quite hilarious that some grump who’s thanklessly perf testing other people’s code is like “no like, exploit makes my system slower”.
    - mitjam 3y ago
      If it was not fulltime work I wonder what else they have been working on with different accounts.
    - mslot 3y ago
      You're responding to said grump ;) Andres is one of the most prolific PostgreSQL committers and his depth of understanding of systems performance is second to none. I wouldn't have guessed he would one day save the world with it, but there you go.
raggi 3y ago
the call to system is obfuscated, static analysis wouldn't see it
londons_explore 3y ago
I'm really surprised they did a call to system() rather than just implement a tiny bytecode interpreter. A bytecode interpreter that can call syscalls can be just a few hundred bytes of code, and means you can avoid calling system() (whose calls might be logged), and avoid calling mprotect to make code executable (also something likely to raise security red flags). The only downside of a bytecode interpreter is the whole of the rest of your malware needs to be compiled to your custom bytecode to get the benefits, and you will take a pretty big performance hit. Unless you're streaming the users webcam, that probably isn't an issue tho.
- brokenmachine 3y ago
  XZ backdoor v2.0 is sure to have that now.
XorNot 3y ago
If you think about it this is a data-providence problem though. The exploit was hidden in "test" code which gets included in release code by compiler flags. Now, if there was a proper chain of accountability for data, then this wouldn't have been possible to hide the way it is - any amount of pre-processing resulting in the release tarball including derived products of "test" files would be suspicious. The problem is we don't actually track data providence like this - no build system does. The most we do is <git hash in> -> <some deterministic bits out>. But we don't include the human readable data which explains how that transform happens at enough levels.
- paulmd 3y ago
  You don’t need to go to that extent even - simply properly segregating test resources from dist resources would have prevented this, and that’s something Java has been doing for 20 years. It’s not sufficient against a determined attacker, but it does demonstrate just how unserious the C world is about their build engineering. I literally can’t think of a single time in 15 years of work that I’ve ever seen a reason for a dist build to need test resources. That’s at best a bug - if it’s a dist resource it goes in the dist resources, not test. And if the tooling doesn’t do a good job of making that mistake difficult… it’s bad tooling.
someguydave 3y ago
Dynamic linking was a mistake and should be eliminated
- intelVISA 3y ago
  yep, but how to reverse such a blunder?
  - someguydave 3y ago
    Well step one is to just stop doing it? I guess someone will need to start publishing a static-only linux distro
    - LtWorf 3y ago
      I think you don't fully understand how the attack works. Had sshd been written in go or rust, the attack would have still been possible.
      - someguydave 3y ago
        of course, I never wrote otherwise?
- kbenson 3y ago
  That this was dynamically linked is the least interesting thing about it IMO. It was a long term I filtration where they got legitimate commit access to a well used library. If xz was statically linked in some way, or just used as an executa Le to compress something (like the kernel), the same problems exist and no dynamic linking would need to be involved.
  - Cloudef 3y ago
    Not true, it would be much harder to hook into openssl functions if the final executable was static [1], the only way is that if the openssl function this attack targeted, actually called a function from libxz. [1] https://sourceware.org/glibc/wiki/GNU_IFUNC https://sourceware.org/glibc/wiki/GNU_IFUNC Dynamic loading is relic of the past and cause of many headaches in linux ecosystem, in this case it also just obfuscates the execution path of the code more so you can't really rely on the code you are reading. Unfortunately I don't think it's possible to completely get rid of dynamic loading as some components such as GPU drivers require it, but it should be reduced to minimum.
ashishbijlani 3y ago
I’ve been building Packj [1] to detect malicious PyPI/NPM/Ruby/PHP/etc. dependencies using behavioral analysis. It uses static+dynamic code analysis to scan for indicators of compromise (e.g., spawning of shell, use of SSH keys, network communication, use of decode+eval, etc). It also checks for several metadata attributes to detect bad actors (e.g., typo squatting). 1. https://github.com/ossillate-inc/packj https://github.com/ossillate-inc/packj

by wepple 3y ago

tux3 3y ago

I think behavioral analysis could be promising. There's a lot of weird stuff this code does on startup that any reasonable Debian package on the average install should not be doing in a million years. Games and proprietary software will sometimes ship with DRM protection layers that do insane things in the name of obfuscation, making it hard to distinguish from malware. But (with only a couple exceptions) there's no reason for a binary or library in a Debian package to ever try to write the PLT outside of the normal mechanism, to try to overwrite symbols in other modules, to add LD audit hooks on startup, to try to resolve things manually by walking ELF structures, to do anti-debug tricks, or just to have any kind of obfuscation or packing that free software packaged for a distro is not supposed to have. Some of these may be (much) more difficult to detect than others, some might not be realistic. But there are several plausible different ways a scanner could have detected something weird going on in memory during ssh startup. No one wants a Linux antivirus. But I think everyone would benefit from throwing all the behavioral analysis we can come up with at new Debian package uploads. We're very lucky someone noticed this one, we may not have the same luck next time.

raggi 3y ago
Except had we been doing that they would have put guards in place to detect it - as they already had guards to avoid the code path when a debugger is attached, to avoid building the payload in when it's not one of the target systems, and so on. Their evasion was fairly extensive, so we'd need many novel dynamic systems to stand a chance, and we'd have to guard those systems extremely tightly - the author got patches into oss-fuzz as well to "squash false positives". All in all, adding more arms to the arms race does raise the bar, but the bar they surpassed already demonstrated tenacity, long term thinking, and significant defense and detection evasion efforts.
- tux3 3y ago
  I broadly agree, but I think we can draw a parallel with the arms race of new exploit techniques versus exploit protection. People still manage to write exploits today, but now you must find an ASLR leak, you must chain enough primitives to work around multiple layers of protection, it's generally a huge pain to write exploits compared to the 90s. Today the dynamic detection that we have for Linux packages seems thin to non-existent, like the arms race has not even started yet. I think there is a bit of low-hanging fruit to make attacker lives harder (and some much higher-hanging fruit that would be a real headache). Luckily there is an asymmetry in favor of the defenders (for once). If we create a scanner, we do not _have_ to publish every type of scan it knows how to do. Much like companies fighting spammers and fraud don't detail exactly how they catch bad actors. (Or, for another example, I know the Tor project has a similar asymmetry to detect bad relays. They collaborate on their relay scanner internally, but no one externally knows all the details.)
  - raggi 3y ago
    Yeah, perhaps something akin to an OSS variant of virustotal's multi-vendor analysis. I'm still not sure it would catch this, but as you say, raising the bar isn't something we tend to regret.
  - saagarjha 3y ago
    This is an arms race that is largely won by attackers, actually. Sophisticated attacks are caught by them sometimes but usually the author has far more knowledge or cleverer tricks than the person implementing the checks, who is limited by their imagination of what they think an attacker might do.
    - wepple 3y ago
      I’m not sure that attackers “win” this arms race; the cost has gone through the roof and their chance of detection constantly increases. Indeed, defenders are often outgunned, but at a minimum we’ve pushed the level of effort way into nation-state territory, which is a good start.
snnn 3y ago
> to try to overwrite symbols in other modules, to add LD audit hooks on startup, to try to resolve things manually by walking ELF structures I want to name one thing: when Windows failed to load a DLL because a dependency was missing, it doesn't tell you what was missed. To get the information, you have to interact with the DLL loader with low level Windows APIs. In some circumstances Linux apps may also have the need. Like for printing a user friendly error message or recovery from a non-fatal error. For example, the patchelf tool that is used for building portable python packages. > No one wants a Linux antivirus It is not true. Actually these software are very popular in enterprise settings.
ffsm8 3y ago
> No one wants a Linux antivirus ClamAV has been around for a very long time at this point. It's just not installed on servers, usually
- kemotep 3y ago
  To say nothing of enterprise EDR/XDR solutions that have linux versions. These things aren’t bulletproof but can be 1 layer in your multilayer security posture.
- EasyMark 3y ago
  don't most people who use that just use it for scanning incoming email attachments usually?
  - snnn 3y ago
    ClamAV also has a lot of findings when scanning some open source project's source code. For example, LLVM project's test data. Because some of the test data are meant to check if a known security bug is fixed, from a antivirus software perspective these data files can be seen as exploits. ClamAV is commonly used. Or, I would suggest adding it to every CI build pipeline. Most time it wouldn't have any finding, but it is better than nothing. I would like to offer free help if an open source project has the need to harden their build pipelines and their release process.
    - everybackdoor 3y ago
      [dead]
- snnn 3y ago
  Does not have to be installed. See this: https://learn.microsoft.com/en-us/azure/defender-for-cloud/concept-agentless-data-collection#how-agentless-scanning-works https://learn.microsoft.com/en-us/azure/defender-for-cloud/c... A cloud provider can take snapshots of running VMs then run antivirus scan offline to minimize the impact to the customers. Similarly, many applications are containerized and the containers are stateless, we can scan the docker images instead. This approach has been quite mature.
robocat 3y ago
> we may not have the same luck next time If the prior is 1 was out there (this one), the chances that there is 1+ still undetected seems fairly high to me. To behaviourally detect this requires many independent actors to be looking in independent ways(e.g. security researchers, internal teams). Edit: I mean with private code & tests (not open source, nor purchasable antivirus). It's not easy to donate to Google Zero. Some of the best funded and most skilled teams seem to be antivirus vendors (and high value person protection). I hate the antivirus industry yet I've been helped by it (the anti-tragedy of the commons). Commonly public detection code (e.g. open source) is likely to be defeated by attackers with a lot of resources. Hard to protect ourselves against countries where the individuals are safe from prosecution. Even nefarious means like assasination likely only work against individuals and not teams.
- bandrami 3y ago
  > If the prior is 1 That would surprise me greatly
  - lazyasciiart 3y ago
    I think you’re saying “I would be surprised if there is only 1 exploit like this that already exists” which is what the previous comment was also saying. “If the prior is one” is often used to mean “we know for sure that there is one”.
cryptonector 3y ago
> But I think everyone would benefit from throwing all the behavioral analysis we can come up with at new Debian package uploads. Why "new uploads" and not also "all existing"?
cipherzero 3y ago
Sorry, I'm unfamiliar with PLT what does stand for?
- intelVISA 3y ago
  procedure linkage table

x0x0 3y ago

[flagged]

raggi 3y ago
Think whatever you shall about systemd of course, but please stop with the blind belief mud slinging: - systemd didn't create the patch to include libsystemd, distros did - current systemd versions already remove liblzma from their dependencies, the affected distros are behind on systemd updates though - you can implement notify in standalone code in about the same effort as it takes to use the dependency, there wasn't really a good reason for distro's to be adding this dependency to such a critical binary. systemd documents the protocol independently to make this easy. distros having sketchy patches to sshd has a long history, remember the debian weak key fiasco?
- Denvercoder9 3y ago
  > - current systemd versions already remove liblzma from their dependencies, the affected distros are behind on systemd updates though The affected distros aren't behind on systemd updates, the change to systemd you describe has been merged but not yet released.
  - raggi 3y ago
    Ah, thank you for the correction!
    - x0x0 3y ago
      Maybe you should reconsider your "blind belief mud slinging" eh? Be better.
- everybackdoor 3y ago
  [flagged]
  - rcxdude 3y ago

jnwatson 3y ago

The real problem was doing expensive math for every connection. If it had relied on a cookie or some simpler-to-compute pre-filter, no one would have been the wiser.

anarazel 3y ago
The slowdown is actually in the startup of the backdoor, not when it's actually performing authentication. Note how in the original report even sshd -h (called in the right environment to circumvent countermeasures) is slow.
- klabb3 3y ago
  Wow. Given the otherwise extreme sophistication this is such a blunder. I imagine the adversary is tearing their hair out over this. 2-3 years of full time infiltration work down the drain, for probably more than a single person. As for the rest of us, we got lucky. In fact, it’s quite hilarious that some grump who’s thanklessly perf testing other people’s code is like “no like, exploit makes my system slower”.
  - mitjam 3y ago
    If it was not fulltime work I wonder what else they have been working on with different accounts.
  - mslot 3y ago
    You're responding to said grump ;) Andres is one of the most prolific PostgreSQL committers and his depth of understanding of systems performance is second to none. I wouldn't have guessed he would one day save the world with it, but there you go.

raggi 3y ago

the call to system is obfuscated, static analysis wouldn't see it

londons_explore 3y ago

I'm really surprised they did a call to system() rather than just implement a tiny bytecode interpreter. A bytecode interpreter that can call syscalls can be just a few hundred bytes of code, and means you can avoid calling system() (whose calls might be logged), and avoid calling mprotect to make code executable (also something likely to raise security red flags). The only downside of a bytecode interpreter is the whole of the rest of your malware needs to be compiled to your custom bytecode to get the benefits, and you will take a pretty big performance hit. Unless you're streaming the users webcam, that probably isn't an issue tho.

brokenmachine 3y ago
XZ backdoor v2.0 is sure to have that now.

XorNot 3y ago

If you think about it this is a data-providence problem though. The exploit was hidden in "test" code which gets included in release code by compiler flags. Now, if there was a proper chain of accountability for data, then this wouldn't have been possible to hide the way it is - any amount of pre-processing resulting in the release tarball including derived products of "test" files would be suspicious. The problem is we don't actually track data providence like this - no build system does. The most we do is <git hash in> -> <some deterministic bits out>. But we don't include the human readable data which explains how that transform happens at enough levels.

paulmd 3y ago
You don’t need to go to that extent even - simply properly segregating test resources from dist resources would have prevented this, and that’s something Java has been doing for 20 years. It’s not sufficient against a determined attacker, but it does demonstrate just how unserious the C world is about their build engineering. I literally can’t think of a single time in 15 years of work that I’ve ever seen a reason for a dist build to need test resources. That’s at best a bug - if it’s a dist resource it goes in the dist resources, not test. And if the tooling doesn’t do a good job of making that mistake difficult… it’s bad tooling.

someguydave 3y ago

Dynamic linking was a mistake and should be eliminated

intelVISA 3y ago
yep, but how to reverse such a blunder?
- someguydave 3y ago
  Well step one is to just stop doing it? I guess someone will need to start publishing a static-only linux distro
  - LtWorf 3y ago
    I think you don't fully understand how the attack works. Had sshd been written in go or rust, the attack would have still been possible.
    - someguydave 3y ago
      of course, I never wrote otherwise?
kbenson 3y ago
That this was dynamically linked is the least interesting thing about it IMO. It was a long term I filtration where they got legitimate commit access to a well used library. If xz was statically linked in some way, or just used as an executa Le to compress something (like the kernel), the same problems exist and no dynamic linking would need to be involved.
- Cloudef 3y ago
  Not true, it would be much harder to hook into openssl functions if the final executable was static [1], the only way is that if the openssl function this attack targeted, actually called a function from libxz. [1] https://sourceware.org/glibc/wiki/GNU_IFUNC https://sourceware.org/glibc/wiki/GNU_IFUNC Dynamic loading is relic of the past and cause of many headaches in linux ecosystem, in this case it also just obfuscates the execution path of the code more so you can't really rely on the code you are reading. Unfortunately I don't think it's possible to completely get rid of dynamic loading as some components such as GPU drivers require it, but it should be reduced to minimum.

ashishbijlani 3y ago

I’ve been building Packj [1] to detect malicious PyPI/NPM/Ruby/PHP/etc. dependencies using behavioral analysis. It uses static+dynamic code analysis to scan for indicators of compromise (e.g., spawning of shell, use of SSH keys, network communication, use of decode+eval, etc). It also checks for several metadata attributes to detect bad actors (e.g., typo squatting). 1. https://github.com/ossillate-inc/packj https://github.com/ossillate-inc/packj