10 ms·

One have question on this is, if the backdoor would not been discovered due to performance issue (which was as I understood it purely an oversight/fixable defic

One have question on this is, if the backdoor would not been discovered due to performance issue (which was as I understood it purely an oversight/fixable deficiency in the code), what are the chances of discovering this backdoor later, or are there tools that would have picked it up? Those questions are IMO relevant to understand if this kind of backdoor is the first one of the kind, or the first one that was uncovered.

wepple 3y ago
I expect a lot of people will be doing a whole lot of thinking along these lines over the next months. Code review? Some kind of behavioral analysis? IMO the call to system() was kind of sloppy, and a binary capabilities scanner could have potentially identified a path to that.
- tux3 3y ago
  I think behavioral analysis could be promising. There's a lot of weird stuff this code does on startup that any reasonable Debian package on the average install should not be doing in a million years. Games and proprietary software will sometimes ship with DRM protection layers that do insane things in the name of obfuscation, making it hard to distinguish from malware. But (with only a couple exceptions) there's no reason for a binary or library in a Debian package to ever try to write the PLT outside of the normal mechanism, to try to overwrite symbols in other modules, to add LD audit hooks on startup, to try to resolve things manually by walking ELF structures, to do anti-debug tricks, or just to have any kind of obfuscation or packing that free software packaged for a distro is not supposed to have. Some of these may be (much) more difficult to detect than others, some might not be realistic. But there are several plausible different ways a scanner could have detected something weird going on in memory during ssh startup. No one wants a Linux antivirus. But I think everyone would benefit from throwing all the behavioral analysis we can come up with at new Debian package uploads. We're very lucky someone noticed this one, we may not have the same luck next time.
  - raggi 3y ago
    Except had we been doing that they would have put guards in place to detect it - as they already had guards to avoid the code path when a debugger is attached, to avoid building the payload in when it's not one of the target systems, and so on. Their evasion was fairly extensive, so we'd need many novel dynamic systems to stand a chance, and we'd have to guard those systems extremely tightly - the author got patches into oss-fuzz as well to "squash false positives". All in all, adding more arms to the arms race does raise the bar, but the bar they surpassed already demonstrated tenacity, long term thinking, and significant defense and detection evasion efforts.
    - tux3 3y ago
      I broadly agree, but I think we can draw a parallel with the arms race of new exploit techniques versus exploit protection. People still manage to write exploits today, but now you must find an ASLR leak, you must chain enough primitives to work around multiple layers of protection, it's generally a huge pain to write exploits compared to the 90s. Today the dynamic detection that we have for Linux packages seems thin to non-existent, like the arms race has not even started yet. I think there is a bit of low-hanging fruit to make attacker lives harder (and some much higher-hanging fruit that would be a real headache). Luckily there is an asymmetry in favor of the defenders (for once). If we create a scanner, we do not _have_ to publish every type of scan it knows how to do. Much like companies fighting spammers and fraud don't detail exactly how they catch bad actors. (Or, for another example, I know the Tor project has a similar asymmetry to detect bad relays. They collaborate on their relay scanner internally, but no one externally knows all the details.)
      - raggi 3y ago
        Yeah, perhaps something akin to an OSS variant of virustotal's multi-vendor analysis. I'm still not sure it would catch this, but as you say, raising the bar isn't something we tend to regret.
      - saagarjha 3y ago
        This is an arms race that is largely won by attackers, actually. Sophisticated attacks are caught by them sometimes but usually the author has far more knowledge or cleverer tricks than the person implementing the checks, who is limited by their imagination of what they think an attacker might do.
- x0x0 3y ago
  [flagged]
  - raggi 3y ago
    Think whatever you shall about systemd of course, but please stop with the blind belief mud slinging: - systemd didn't create the patch to include libsystemd, distros did - current systemd versions already remove liblzma from their dependencies, the affected distros are behind on systemd updates though - you can implement notify in standalone code in about the same effort as it takes to use the dependency, there wasn't really a good reason for distro's to be adding this dependency to such a critical binary. systemd documents the protocol independently to make this easy. distros having sketchy patches to sshd has a long history, remember the debian weak key fiasco?
    - Denvercoder9 3y ago
      > - current systemd versions already remove liblzma from their dependencies, the affected distros are behind on systemd updates though The affected distros aren't behind on systemd updates, the change to systemd you describe has been merged but not yet released.
      - raggi 3y ago
        Ah, thank you for the correction!
        x0x0 3y ago
        Maybe you should reconsider your "blind belief mud slinging" eh? Be better.
    - everybackdoor 3y ago
      [flagged]
      - rcxdude 3y ago
- jnwatson 3y ago
  The real problem was doing expensive math for every connection. If it had relied on a cookie or some simpler-to-compute pre-filter, no one would have been the wiser.
  - anarazel 3y ago
    The slowdown is actually in the startup of the backdoor, not when it's actually performing authentication. Note how in the original report even sshd -h (called in the right environment to circumvent countermeasures) is slow.
    - klabb3 3y ago
      Wow. Given the otherwise extreme sophistication this is such a blunder. I imagine the adversary is tearing their hair out over this. 2-3 years of full time infiltration work down the drain, for probably more than a single person. As for the rest of us, we got lucky. In fact, it’s quite hilarious that some grump who’s thanklessly perf testing other people’s code is like “no like, exploit makes my system slower”.
      - mitjam 3y ago
        If it was not fulltime work I wonder what else they have been working on with different accounts.
      - mslot 3y ago
        You're responding to said grump ;) Andres is one of the most prolific PostgreSQL committers and his depth of understanding of systems performance is second to none. I wouldn't have guessed he would one day save the world with it, but there you go.
- raggi 3y ago
  the call to system is obfuscated, static analysis wouldn't see it
- londons_explore 3y ago
  I'm really surprised they did a call to system() rather than just implement a tiny bytecode interpreter. A bytecode interpreter that can call syscalls can be just a few hundred bytes of code, and means you can avoid calling system() (whose calls might be logged), and avoid calling mprotect to make code executable (also something likely to raise security red flags). The only downside of a bytecode interpreter is the whole of the rest of your malware needs to be compiled to your custom bytecode to get the benefits, and you will take a pretty big performance hit. Unless you're streaming the users webcam, that probably isn't an issue tho.
  - brokenmachine 3y ago
    XZ backdoor v2.0 is sure to have that now.
- XorNot 3y ago
  If you think about it this is a data-providence problem though. The exploit was hidden in "test" code which gets included in release code by compiler flags. Now, if there was a proper chain of accountability for data, then this wouldn't have been possible to hide the way it is - any amount of pre-processing resulting in the release tarball including derived products of "test" files would be suspicious. The problem is we don't actually track data providence like this - no build system does. The most we do is <git hash in> -> <some deterministic bits out>. But we don't include the human readable data which explains how that transform happens at enough levels.
  - paulmd 3y ago
    You don’t need to go to that extent even - simply properly segregating test resources from dist resources would have prevented this, and that’s something Java has been doing for 20 years. It’s not sufficient against a determined attacker, but it does demonstrate just how unserious the C world is about their build engineering. I literally can’t think of a single time in 15 years of work that I’ve ever seen a reason for a dist build to need test resources. That’s at best a bug - if it’s a dist resource it goes in the dist resources, not test. And if the tooling doesn’t do a good job of making that mistake difficult… it’s bad tooling.
- someguydave 3y ago
  Dynamic linking was a mistake and should be eliminated
  - intelVISA 3y ago
    yep, but how to reverse such a blunder?
    - someguydave 3y ago
      Well step one is to just stop doing it? I guess someone will need to start publishing a static-only linux distro
      - LtWorf 3y ago
        I think you don't fully understand how the attack works. Had sshd been written in go or rust, the attack would have still been possible.
        someguydave 3y ago
        of course, I never wrote otherwise?
  - kbenson 3y ago
    That this was dynamically linked is the least interesting thing about it IMO. It was a long term I filtration where they got legitimate commit access to a well used library. If xz was statically linked in some way, or just used as an executa Le to compress something (like the kernel), the same problems exist and no dynamic linking would need to be involved.
    - Cloudef 3y ago
      Not true, it would be much harder to hook into openssl functions if the final executable was static [1], the only way is that if the openssl function this attack targeted, actually called a function from libxz. [1] https://sourceware.org/glibc/wiki/GNU_IFUNC https://sourceware.org/glibc/wiki/GNU_IFUNC Dynamic loading is relic of the past and cause of many headaches in linux ecosystem, in this case it also just obfuscates the execution path of the code more so you can't really rely on the code you are reading. Unfortunately I don't think it's possible to completely get rid of dynamic loading as some components such as GPU drivers require it, but it should be reduced to minimum.
- ashishbijlani 3y ago
  I’ve been building Packj [1] to detect malicious PyPI/NPM/Ruby/PHP/etc. dependencies using behavioral analysis. It uses static+dynamic code analysis to scan for indicators of compromise (e.g., spawning of shell, use of SSH keys, network communication, use of decode+eval, etc). It also checks for several metadata attributes to detect bad actors (e.g., typo squatting). 1. https://github.com/ossillate-inc/packj https://github.com/ossillate-inc/packj
joeyh 3y ago
Since a liblzma backdoor could be used to modify compiler packages that are installed on some distributions, it gets right back to a trusting trust attack. Although initial detection via eg strace would be possible, if the backdoor was later removed or went quiescentit would be full trusting trust territory.
- ghostpepper 3y ago
  How would this be possible? This backdoor works because lzma is loaded into sshd (by a roundabout method involving systemd). I don't think gcc or clang links lzma.
  - slim 3y ago
    servers hosting gcc binaries are accessed using ssh
  - joeyh 3y ago
    dpkg-deb is linked with liblzma
  - dist-epoch 3y ago
    When the backdoor is loaded by sshd it could modify the gcc/clang install, or some system header file.
  - xorcist 3y ago
    To be fair neither does sshd. But I'm sure someone somewhere has a good reason for gcc to write status via journald or something like that? There's however no reason to limit yourself to gcc for a supply chain attack like this. In any non trivial build system, there's going to be lots of third party things involved. Especially when you include tests in the build. Is Python invoked somewhere along the build chain? That's like a dozen libraries loaded already. Nothing is gained from protecting against an exact replica of this attack, but from this family of attacks.
  - Bulat_Ziganshin 3y ago
    the installation process itself executes xz scripts which can make any (?) modifications to the system
rdtsc 3y ago
At least for some comic relief I'd like to imagine Jia's boss slapping him and saying something like "you idiot, we worked on this for so many years and you couldn't have checked for any perf issues?" But seriously, we could have found ourselves with this in all stable repos: RHEL, Debian, Ubuntu, IoT devices 5 years from now and it would have been a much larger shit show.
- chris_wot 3y ago
  Surely this is something the FBI should be involved with? Or some authority?
  - colinsane 3y ago
    sure. what makes you think they aren't?
    - sitkack 3y ago
      Why would the FBI investigate the NSA? We have zero idea who the actors involved are.
      - colinsane 3y ago
        my comment allows for the NSA to be involved in this and for the FBI to not be investigating them.
      - pxc 3y ago
        It's not actually unusual for three-letter US agencies to be at odds with one another. But one possible reason is if the FBI is convinced that something the NSA is doing is illegal. They may not always be inclined to tolerate that.
        sitkack 3y ago
        You would have to agree that it was possible for the government to break the law. And what the repercussions are when that happens.
quatrefoil 3y ago
If the exploit wasn't baing used, the odds would would be pretty low. They picked the right place to bury it (i.e., effectively outside the codebase, where no auditor ever looks). That said, if you're not using it, it defeats the purpose. And the more you're using it, the higher the likelihood you will be detected down the line. Compare to Solarwinds.
- londons_explore 3y ago
  I suspect I could have used this exact attack against 10,000 random SSH servers spread all over the world, and not be detected. Most people don't log TCP connections, and those that do don't go through their logs looking for odd certificates in ssh connections. And no common logging at the ssh/pam level would have picked this up. Your only chance is some sysadmin who has put 'tripwires' on certain syscalls like system(), fork() or mmap() looking for anything unusual. Even then, they might detect the attack, yet have no chance at actually finding how the malicious code loaded itself.
  - matrix_overload 3y ago
    With sufficient data points, you can do A/B and see that all affected systems run a specific version of Linux distro, and eventually track it down to a particular package.
    - bastawhiz 3y ago
      Unless you're the bad actor, you have no way to trigger the exploit, so you can't really do an a/b test. You can only confirm which versions of which distros are vulnerable. And that assumes you have sufficient instrumentation in place to know the exploit has been triggered. Even then, who actually has a massive fleet of publicly exposed servers all running a mix of distros/versions? You might run a small handful of distros, but I suspect anyone running a fleet large enough to actually collect a substantial amount of data probably also has tools to upgrade the whole fleet (or at least large swaths) in one go. Certainly there are companies where updates are the wild west, but the odds that they're all accessible to and controllable by a single motivated individual who can detect the exploit is essentially zero.
formerly_proven 3y ago
I think this would’ve been difficult to catch because the patching of sshd happens during linking, when it’s permissible, and if this is correct then it’s not a master key backdoor, so there is no regular login audit trail. And sshd would of course be allowed to start other processes. A very tight SELinux policy could catch sshd executing something that ain’t a shell but hardening to that degree would be extremely rare I assume. As for being discovered outside the target, well we tried that exercise already, didn’t we? A bunch of people stared at the payload with valgrind et al and didn’t see it. It’s also fairly well protected from being discovered in debugging environments, because the overt infrastructure underlying the payload is incompatible with ASan and friends. And even if it is linked in, the code runs long before main(), so even if you were prodding around near or in liblzma with a debugger you wouldn’t normally observe it execute. e: sibling suggests strace, yes you can see all syscalls after the process is spawned and you can watch the linker work. But from what I’ve gathered the payload isn’t making any syscalls at that stage to determine whether to activate, it’s just looking at argv and environ etc.
- tux3 3y ago
  One idea may be to create a patched version of ld-linux itself with added sanity checks while the process loads. For something much more heavy-handed, force the pages in sensitive sections to fault, either in the kernel or in a hypervisor. Then look at where the access is coming from in the page fault handler. I don't think you can reliably differentiate a backdoor executing a command, and a legitimate user logged in with ssh running a command once the backdoor is already installed. But the way backdoors install themselves is where they really break the rules.
- Deathcrow 3y ago
  >A very tight SELinux policy could catch sshd executing something that ain’t a shell but hardening to that degree would be extremely rare I assume. Huh, ssh executes things that aren't shells all the time during normal operation. No? i.e. 'ssh myserver.lan cat /etc/fstab'
  - sega_sai 3y ago
deleted 3y ago
[deleted]
nurple 3y ago
I think one interesting corollary here is how the Ken Thompson attack was discovered at PWB[0] because it had a memory performance bug[1]. [0] https://en.wikipedia.org/wiki/PWB/UNIX [1] https://news.ycombinator.com/item?id=38020792
xlii 3y ago
Working for about a year in an environment that was exposed to high volume of malevolent IT actors (and some pretty scary ones) I’d say: discovery chances very always pretty high. Keeping veil of secrecy requires unimaginable amount of energy. Same goes with truth consistency. One little slip and everything goes to nothing. Sometimes single sentence can start a chain of reaction and uncover meticulous crafted plan. That’s how crime if fought every day. Whereas police work has limited resources, software is analyzed daily by hobbyists as a hobby, professionals who still do it for a hobby, and professionals for professional reasons. Discovery was bound to happen eventually. XZ attack was very well executed. It’s a master piece. I wouldn’t be surprised if some state agencies would be involved. But it also was incredibly lucky. I know for sure for myself, but also many of my colleagues would go into long journey if found any of issues that are flagged right now. One takeaway is that chance of finding such issue would be impossible if xz/liblzma wouldn’t be open source (and yes I am also aware it enabled it in the first place) but imagine this existing in Windows or MacOS.
- rigid 3y ago
  > it enabled it in the first place it took roughly two years including social engineering. I'd say the same approach is much easier in a big software company.
  - lazyasciiart 3y ago
    How do you mean?
    - rigid 3y ago
      I bet in the majority of cases, there's no need to pressure for merging. In a big company it's much easier to slip it in. Code seemingly less relevant for security is often not reviewed by a lot of people. Also, often people don't really care and just sign it off without a closer look. And when it's merged, no one will ever look at it again, other than with FOSS.
      - 3y ago
geggo98 3y ago
Using a jump host could help, only allowing port forwarding. Ideally it would be heavily monitored and create a new instance for every connection (e.g., inside a container). The attacker would then be stuck inside the jump host and would have to probe where to connect next. This hopefully would then trigger an alert, causing some suspicion. A shared instance would allow the attacker to just wait for another connection and then follow its traces, without risking triggering an alert by probing. The ideal jump host would allow to freeze the running ssh process on an alert, either with a snapshot (VM based) or checkpointing (container based), so it can be analyzed later.
amelius 3y ago
Backdoors can be placed in any type of software. For example, a GIMP plugin could connect to your display and read keystrokes, harvest passwords, etcetera. Utilities run by the superuser are of course even more potentially dangerous. Supply-chain attacks like these are just bound to happen. Perhaps not as often in SSH which is heavily scrutinized, but the consequences can be serious nevertheless.
x-yl 3y ago
Can I ask for why it wouldn't have been discovered if the obvious delay wasn't present? Wouldn't anyone profiling a running sshd (which I have to imagine someone out there is doing) see it spending all its crypto time in liblzma?
- xorcist 3y ago
  The situation certainly wouldn't be helped by the fact that this exploit targeted the systemd integration used by Debian and Red Hat. OpenSSH developers aren't likely to run that since they already rejected that patch for the increased attack surface. Hard to argue against, in retrospect. The attack also avoids activation under those conditions a profiler or debugger would run under.

wepple 3y ago

I expect a lot of people will be doing a whole lot of thinking along these lines over the next months. Code review? Some kind of behavioral analysis? IMO the call to system() was kind of sloppy, and a binary capabilities scanner could have potentially identified a path to that.

tux3 3y ago
I think behavioral analysis could be promising. There's a lot of weird stuff this code does on startup that any reasonable Debian package on the average install should not be doing in a million years. Games and proprietary software will sometimes ship with DRM protection layers that do insane things in the name of obfuscation, making it hard to distinguish from malware. But (with only a couple exceptions) there's no reason for a binary or library in a Debian package to ever try to write the PLT outside of the normal mechanism, to try to overwrite symbols in other modules, to add LD audit hooks on startup, to try to resolve things manually by walking ELF structures, to do anti-debug tricks, or just to have any kind of obfuscation or packing that free software packaged for a distro is not supposed to have. Some of these may be (much) more difficult to detect than others, some might not be realistic. But there are several plausible different ways a scanner could have detected something weird going on in memory during ssh startup. No one wants a Linux antivirus. But I think everyone would benefit from throwing all the behavioral analysis we can come up with at new Debian package uploads. We're very lucky someone noticed this one, we may not have the same luck next time.
- raggi 3y ago
  Except had we been doing that they would have put guards in place to detect it - as they already had guards to avoid the code path when a debugger is attached, to avoid building the payload in when it's not one of the target systems, and so on. Their evasion was fairly extensive, so we'd need many novel dynamic systems to stand a chance, and we'd have to guard those systems extremely tightly - the author got patches into oss-fuzz as well to "squash false positives". All in all, adding more arms to the arms race does raise the bar, but the bar they surpassed already demonstrated tenacity, long term thinking, and significant defense and detection evasion efforts.
  - tux3 3y ago
    I broadly agree, but I think we can draw a parallel with the arms race of new exploit techniques versus exploit protection. People still manage to write exploits today, but now you must find an ASLR leak, you must chain enough primitives to work around multiple layers of protection, it's generally a huge pain to write exploits compared to the 90s. Today the dynamic detection that we have for Linux packages seems thin to non-existent, like the arms race has not even started yet. I think there is a bit of low-hanging fruit to make attacker lives harder (and some much higher-hanging fruit that would be a real headache). Luckily there is an asymmetry in favor of the defenders (for once). If we create a scanner, we do not _have_ to publish every type of scan it knows how to do. Much like companies fighting spammers and fraud don't detail exactly how they catch bad actors. (Or, for another example, I know the Tor project has a similar asymmetry to detect bad relays. They collaborate on their relay scanner internally, but no one externally knows all the details.)
    - raggi 3y ago
      Yeah, perhaps something akin to an OSS variant of virustotal's multi-vendor analysis. I'm still not sure it would catch this, but as you say, raising the bar isn't something we tend to regret.
    - saagarjha 3y ago
      This is an arms race that is largely won by attackers, actually. Sophisticated attacks are caught by them sometimes but usually the author has far more knowledge or cleverer tricks than the person implementing the checks, who is limited by their imagination of what they think an attacker might do.
x0x0 3y ago
[flagged]
- raggi 3y ago
  Think whatever you shall about systemd of course, but please stop with the blind belief mud slinging: - systemd didn't create the patch to include libsystemd, distros did - current systemd versions already remove liblzma from their dependencies, the affected distros are behind on systemd updates though - you can implement notify in standalone code in about the same effort as it takes to use the dependency, there wasn't really a good reason for distro's to be adding this dependency to such a critical binary. systemd documents the protocol independently to make this easy. distros having sketchy patches to sshd has a long history, remember the debian weak key fiasco?
  - Denvercoder9 3y ago
    > - current systemd versions already remove liblzma from their dependencies, the affected distros are behind on systemd updates though The affected distros aren't behind on systemd updates, the change to systemd you describe has been merged but not yet released.
    - raggi 3y ago
      Ah, thank you for the correction!
      - x0x0 3y ago
        Maybe you should reconsider your "blind belief mud slinging" eh? Be better.
  - everybackdoor 3y ago
    [flagged]
    - rcxdude 3y ago
jnwatson 3y ago
The real problem was doing expensive math for every connection. If it had relied on a cookie or some simpler-to-compute pre-filter, no one would have been the wiser.
- anarazel 3y ago
  The slowdown is actually in the startup of the backdoor, not when it's actually performing authentication. Note how in the original report even sshd -h (called in the right environment to circumvent countermeasures) is slow.
  - klabb3 3y ago
    Wow. Given the otherwise extreme sophistication this is such a blunder. I imagine the adversary is tearing their hair out over this. 2-3 years of full time infiltration work down the drain, for probably more than a single person. As for the rest of us, we got lucky. In fact, it’s quite hilarious that some grump who’s thanklessly perf testing other people’s code is like “no like, exploit makes my system slower”.
    - mitjam 3y ago
      If it was not fulltime work I wonder what else they have been working on with different accounts.
    - mslot 3y ago
      You're responding to said grump ;) Andres is one of the most prolific PostgreSQL committers and his depth of understanding of systems performance is second to none. I wouldn't have guessed he would one day save the world with it, but there you go.
raggi 3y ago
the call to system is obfuscated, static analysis wouldn't see it
londons_explore 3y ago
I'm really surprised they did a call to system() rather than just implement a tiny bytecode interpreter. A bytecode interpreter that can call syscalls can be just a few hundred bytes of code, and means you can avoid calling system() (whose calls might be logged), and avoid calling mprotect to make code executable (also something likely to raise security red flags). The only downside of a bytecode interpreter is the whole of the rest of your malware needs to be compiled to your custom bytecode to get the benefits, and you will take a pretty big performance hit. Unless you're streaming the users webcam, that probably isn't an issue tho.
- brokenmachine 3y ago
  XZ backdoor v2.0 is sure to have that now.
XorNot 3y ago
If you think about it this is a data-providence problem though. The exploit was hidden in "test" code which gets included in release code by compiler flags. Now, if there was a proper chain of accountability for data, then this wouldn't have been possible to hide the way it is - any amount of pre-processing resulting in the release tarball including derived products of "test" files would be suspicious. The problem is we don't actually track data providence like this - no build system does. The most we do is <git hash in> -> <some deterministic bits out>. But we don't include the human readable data which explains how that transform happens at enough levels.
- paulmd 3y ago
  You don’t need to go to that extent even - simply properly segregating test resources from dist resources would have prevented this, and that’s something Java has been doing for 20 years. It’s not sufficient against a determined attacker, but it does demonstrate just how unserious the C world is about their build engineering. I literally can’t think of a single time in 15 years of work that I’ve ever seen a reason for a dist build to need test resources. That’s at best a bug - if it’s a dist resource it goes in the dist resources, not test. And if the tooling doesn’t do a good job of making that mistake difficult… it’s bad tooling.
someguydave 3y ago
Dynamic linking was a mistake and should be eliminated
- intelVISA 3y ago
  yep, but how to reverse such a blunder?
  - someguydave 3y ago
    Well step one is to just stop doing it? I guess someone will need to start publishing a static-only linux distro
    - LtWorf 3y ago
      I think you don't fully understand how the attack works. Had sshd been written in go or rust, the attack would have still been possible.
      - someguydave 3y ago
        of course, I never wrote otherwise?
- kbenson 3y ago
  That this was dynamically linked is the least interesting thing about it IMO. It was a long term I filtration where they got legitimate commit access to a well used library. If xz was statically linked in some way, or just used as an executa Le to compress something (like the kernel), the same problems exist and no dynamic linking would need to be involved.
  - Cloudef 3y ago
    Not true, it would be much harder to hook into openssl functions if the final executable was static [1], the only way is that if the openssl function this attack targeted, actually called a function from libxz. [1] https://sourceware.org/glibc/wiki/GNU_IFUNC https://sourceware.org/glibc/wiki/GNU_IFUNC Dynamic loading is relic of the past and cause of many headaches in linux ecosystem, in this case it also just obfuscates the execution path of the code more so you can't really rely on the code you are reading. Unfortunately I don't think it's possible to completely get rid of dynamic loading as some components such as GPU drivers require it, but it should be reduced to minimum.
ashishbijlani 3y ago
I’ve been building Packj [1] to detect malicious PyPI/NPM/Ruby/PHP/etc. dependencies using behavioral analysis. It uses static+dynamic code analysis to scan for indicators of compromise (e.g., spawning of shell, use of SSH keys, network communication, use of decode+eval, etc). It also checks for several metadata attributes to detect bad actors (e.g., typo squatting). 1. https://github.com/ossillate-inc/packj https://github.com/ossillate-inc/packj

joeyh 3y ago

Since a liblzma backdoor could be used to modify compiler packages that are installed on some distributions, it gets right back to a trusting trust attack. Although initial detection via eg strace would be possible, if the backdoor was later removed or went quiescentit would be full trusting trust territory.

ghostpepper 3y ago
How would this be possible? This backdoor works because lzma is loaded into sshd (by a roundabout method involving systemd). I don't think gcc or clang links lzma.
- slim 3y ago
  servers hosting gcc binaries are accessed using ssh
- joeyh 3y ago
  dpkg-deb is linked with liblzma
- dist-epoch 3y ago
  When the backdoor is loaded by sshd it could modify the gcc/clang install, or some system header file.
- xorcist 3y ago
  To be fair neither does sshd. But I'm sure someone somewhere has a good reason for gcc to write status via journald or something like that? There's however no reason to limit yourself to gcc for a supply chain attack like this. In any non trivial build system, there's going to be lots of third party things involved. Especially when you include tests in the build. Is Python invoked somewhere along the build chain? That's like a dozen libraries loaded already. Nothing is gained from protecting against an exact replica of this attack, but from this family of attacks.
- Bulat_Ziganshin 3y ago
  the installation process itself executes xz scripts which can make any (?) modifications to the system

rdtsc 3y ago

At least for some comic relief I'd like to imagine Jia's boss slapping him and saying something like "you idiot, we worked on this for so many years and you couldn't have checked for any perf issues?" But seriously, we could have found ourselves with this in all stable repos: RHEL, Debian, Ubuntu, IoT devices 5 years from now and it would have been a much larger shit show.

chris_wot 3y ago
Surely this is something the FBI should be involved with? Or some authority?
- colinsane 3y ago
  sure. what makes you think they aren't?
  - sitkack 3y ago
    Why would the FBI investigate the NSA? We have zero idea who the actors involved are.
    - colinsane 3y ago
      my comment allows for the NSA to be involved in this and for the FBI to not be investigating them.
    - pxc 3y ago
      It's not actually unusual for three-letter US agencies to be at odds with one another. But one possible reason is if the FBI is convinced that something the NSA is doing is illegal. They may not always be inclined to tolerate that.
      - sitkack 3y ago
        You would have to agree that it was possible for the government to break the law. And what the repercussions are when that happens.

quatrefoil 3y ago

If the exploit wasn't baing used, the odds would would be pretty low. They picked the right place to bury it (i.e., effectively outside the codebase, where no auditor ever looks). That said, if you're not using it, it defeats the purpose. And the more you're using it, the higher the likelihood you will be detected down the line. Compare to Solarwinds.

londons_explore 3y ago
I suspect I could have used this exact attack against 10,000 random SSH servers spread all over the world, and not be detected. Most people don't log TCP connections, and those that do don't go through their logs looking for odd certificates in ssh connections. And no common logging at the ssh/pam level would have picked this up. Your only chance is some sysadmin who has put 'tripwires' on certain syscalls like system(), fork() or mmap() looking for anything unusual. Even then, they might detect the attack, yet have no chance at actually finding how the malicious code loaded itself.
- matrix_overload 3y ago
  With sufficient data points, you can do A/B and see that all affected systems run a specific version of Linux distro, and eventually track it down to a particular package.
  - bastawhiz 3y ago
    Unless you're the bad actor, you have no way to trigger the exploit, so you can't really do an a/b test. You can only confirm which versions of which distros are vulnerable. And that assumes you have sufficient instrumentation in place to know the exploit has been triggered. Even then, who actually has a massive fleet of publicly exposed servers all running a mix of distros/versions? You might run a small handful of distros, but I suspect anyone running a fleet large enough to actually collect a substantial amount of data probably also has tools to upgrade the whole fleet (or at least large swaths) in one go. Certainly there are companies where updates are the wild west, but the odds that they're all accessible to and controllable by a single motivated individual who can detect the exploit is essentially zero.

formerly_proven 3y ago

I think this would’ve been difficult to catch because the patching of sshd happens during linking, when it’s permissible, and if this is correct then it’s not a master key backdoor, so there is no regular login audit trail. And sshd would of course be allowed to start other processes. A very tight SELinux policy could catch sshd executing something that ain’t a shell but hardening to that degree would be extremely rare I assume. As for being discovered outside the target, well we tried that exercise already, didn’t we? A bunch of people stared at the payload with valgrind et al and didn’t see it. It’s also fairly well protected from being discovered in debugging environments, because the overt infrastructure underlying the payload is incompatible with ASan and friends. And even if it is linked in, the code runs long before main(), so even if you were prodding around near or in liblzma with a debugger you wouldn’t normally observe it execute. e: sibling suggests strace, yes you can see all syscalls after the process is spawned and you can watch the linker work. But from what I’ve gathered the payload isn’t making any syscalls at that stage to determine whether to activate, it’s just looking at argv and environ etc.

tux3 3y ago
One idea may be to create a patched version of ld-linux itself with added sanity checks while the process loads. For something much more heavy-handed, force the pages in sensitive sections to fault, either in the kernel or in a hypervisor. Then look at where the access is coming from in the page fault handler. I don't think you can reliably differentiate a backdoor executing a command, and a legitimate user logged in with ssh running a command once the backdoor is already installed. But the way backdoors install themselves is where they really break the rules.
Deathcrow 3y ago
>A very tight SELinux policy could catch sshd executing something that ain’t a shell but hardening to that degree would be extremely rare I assume. Huh, ssh executes things that aren't shells all the time during normal operation. No? i.e. 'ssh myserver.lan cat /etc/fstab'
- sega_sai 3y ago

deleted 3y ago

[deleted]

nurple 3y ago

I think one interesting corollary here is how the Ken Thompson attack was discovered at PWB[0] because it had a memory performance bug[1]. [0] https://en.wikipedia.org/wiki/PWB/UNIX [1] https://news.ycombinator.com/item?id=38020792

xlii 3y ago

Working for about a year in an environment that was exposed to high volume of malevolent IT actors (and some pretty scary ones) I’d say: discovery chances very always pretty high. Keeping veil of secrecy requires unimaginable amount of energy. Same goes with truth consistency. One little slip and everything goes to nothing. Sometimes single sentence can start a chain of reaction and uncover meticulous crafted plan. That’s how crime if fought every day. Whereas police work has limited resources, software is analyzed daily by hobbyists as a hobby, professionals who still do it for a hobby, and professionals for professional reasons. Discovery was bound to happen eventually. XZ attack was very well executed. It’s a master piece. I wouldn’t be surprised if some state agencies would be involved. But it also was incredibly lucky. I know for sure for myself, but also many of my colleagues would go into long journey if found any of issues that are flagged right now. One takeaway is that chance of finding such issue would be impossible if xz/liblzma wouldn’t be open source (and yes I am also aware it enabled it in the first place) but imagine this existing in Windows or MacOS.

rigid 3y ago
> it enabled it in the first place it took roughly two years including social engineering. I'd say the same approach is much easier in a big software company.
- lazyasciiart 3y ago
  How do you mean?
  - rigid 3y ago
    I bet in the majority of cases, there's no need to pressure for merging. In a big company it's much easier to slip it in. Code seemingly less relevant for security is often not reviewed by a lot of people. Also, often people don't really care and just sign it off without a closer look. And when it's merged, no one will ever look at it again, other than with FOSS.
    - 3y ago

geggo98 3y ago

Using a jump host could help, only allowing port forwarding. Ideally it would be heavily monitored and create a new instance for every connection (e.g., inside a container). The attacker would then be stuck inside the jump host and would have to probe where to connect next. This hopefully would then trigger an alert, causing some suspicion. A shared instance would allow the attacker to just wait for another connection and then follow its traces, without risking triggering an alert by probing. The ideal jump host would allow to freeze the running ssh process on an alert, either with a snapshot (VM based) or checkpointing (container based), so it can be analyzed later.

amelius 3y ago

Backdoors can be placed in any type of software. For example, a GIMP plugin could connect to your display and read keystrokes, harvest passwords, etcetera. Utilities run by the superuser are of course even more potentially dangerous. Supply-chain attacks like these are just bound to happen. Perhaps not as often in SSH which is heavily scrutinized, but the consequences can be serious nevertheless.

x-yl 3y ago

Can I ask for why it wouldn't have been discovered if the obvious delay wasn't present? Wouldn't anyone profiling a running sshd (which I have to imagine someone out there is doing) see it spending all its crypto time in liblzma?

xorcist 3y ago
The situation certainly wouldn't be helped by the fact that this exploit targeted the systemd integration used by Debian and Red Hat. OpenSSH developers aren't likely to run that since they already rejected that patch for the increased attack surface. Hard to argue against, in retrospect. The attack also avoids activation under those conditions a profiler or debugger would run under.