27 ms·

XZ backdoor: "It's RCE, not auth bypass, and gated/unreplayable."

• SubiculumCode 3y ago
  So is this backdoor active in Ubuntu distributions?
  • dboreham 3y ago
    It appears not.
• deleted 3y ago
  [deleted]
• junon 3y ago
  EDIT: Here's some more RE work on the matter. Has some symbol remapping information that was extracted from the prefix trie the backdoor used to hide strings. Looks like it tried to hide itself even from RE/analysis, too. https://gist.github.com/smx-smx/a6112d54777845d389bd7126d6e9f504 https://gist.github.com/smx-smx/a6112d54777845d389bd7126d6e9... Full list of decoded strings here: https://gist.github.com/q3k/af3d93b6a1f399de28fe194add452d01 https://gist.github.com/q3k/af3d93b6a1f399de28fe194add452d01 -- For someone unfamiliar with openssl's internals (like me): The N value, I presume, is pulled from the `n` field of `rsa_st`: https://github.com/openssl/openssl/blob/56e63f570bd5a479439bc6f6a2499f6b86ded341/crypto/rsa/rsa_local.h#L62C5-L62C12 https://github.com/openssl/openssl/blob/56e63f570bd5a479439b... Which is a `BIGNUM`: https://github.com/openssl/openssl/blob/56e63f570bd5a479439bc6f6a2499f6b86ded341/crypto/bn/bn_local.h#L245 https://github.com/openssl/openssl/blob/56e63f570bd5a479439b... Which appears to be a variable length type. The back door pulls this from the certificate received from a remote attacker, attempts to decrypt it with ChaCha20, and if it decrypts successfully, passed to `system()`, which is essentially a simple wrapper that executes a line of shellscript under whichever user the process is currently executing. If I'm understanding things correctly, this is worse than a public key bypass (which myself and I think a number of others presumed it might be) - a public key bypass would, in theory, only allow you access as the user you're logging in with. Assumedly, hardened SSH configurations would disallow root access. However, since this is an RCE in the context of e.g. an sshd process itself, this means that sshd running as root would allow the payload to itself run as root. Wild. This is about as bad as a widespread RCE can realistically get.
  • jeroenhd 3y ago
    > However, since this is an RCE in the context of e.g. an sshd process itself, this means that sshd running as root would allow the payload to itself run as root. With the right sandboxing techniques, SELinux and mitigations could prevent the attacker from doing anything with root permissions. However, applying a sandbox to an SSH daemon effectively is very difficult.
    • junon 3y ago
      Right, though if I'm understanding correctly, this is targeting openssl, not just sshd. So there's a larger set of circumstances where this could have been exploited. I'm not sure if it's yet been confirmed that this is confined only to sshd.
      • jeroenhd 3y ago
        The exploit, as currently found, seems to target OpenSSH specifically. It's possible that everything involving xz has been compromised, but I haven't read any reports that there is a path to malware execution outside of OpenSSH. A quote from the first analysis that I know of (https://www.openwall.com/lists/oss-security/2024/03/29/4 https://www.openwall.com/lists/oss-security/2024/03/29/4): > Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked. This appears to be part of some countermeasures to make analysis harder. > a) TERM environment variable is not set > b) argv[0] needs to be /usr/sbin/sshd > c) LD_DEBUG, LD_PROFILE are not set > d) LANG needs to be set > e) Some debugging environments, like rr, appear to be detected. Plain gdb appears to be detected in some situations, but not others
    • onedognight 3y ago
      > With the right sandboxing techniques, SELinux and mitigations could prevent the attacker from doing anything with root permissions. Please review this commit[0] where the sandbox detection was “improved”. [0] https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2bbb81307644efdb58db2c422d9ba7 https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2...
• deleted 3y ago
  [deleted]
• Animats 3y ago
  Is there a git diff which shows this going in?
  • tempodox 3y ago
    Apparently it’s not in the original repo, but in a build script in a distribution tar.
    • joshcryer 3y ago
      They also used social engineering to disable fuzzing which would have caught the discrepancy: https://github.com/google/oss-fuzz/pull/10667 https://github.com/google/oss-fuzz/pull/10667
      • codetrotter 3y ago
        It’s pretty funny how a bunch of people come piling reaction emojis onto the comments in the PR, after it has all become publicly known. I’m like.. bro, adding reaction emojis after the fact as if that makes any sort of difference to anything.
        • happosai 3y ago
          Honestly, it's harrasment at this point.
        • qudat 3y ago
          Feels almost like tampering with evidence at a crime scene
          • lazide 3y ago
            It’s just adding your initials on the tunnel someone famous just died in.
          • cqqxo4zV46cp 3y ago
            That’s absurd. Elaborate.
• jtchang 3y ago
  So this basically means to scan for this exploit remotely we'd need the private key of the attacker which we don't have. Only other option is to run detection scripts locally. Yikes.
  • tialaramex 3y ago
    One completely awful thing some scanners might choose to do is if you're offering RSA auth (which most SSH servers are and indeed the SecSH RFC says this is Mandatory To Implement) then you're "potentially vulnerable" which would encourage people to do password auth instead. Unless we find that this problem has somehow infested a lot of real world systems that seems to me even worse than the time similar "experts" decided that it was best to demand people rotate their passwords every year or so thereby ensuring the real security is reduced while on paper you claim you improved it.
    • danparsonson 3y ago
      Have to admit I've never understood why password auth is considered so much worse than using a cert - surely a decent password (long, random, etc) is for all practical purposes unguessable, and so you're either using a private RSA key that no-one can guess, or a password that no-one can guess, and then what's the difference? With the added inconvenience of having to pass around a certificate if you want to login to the same account from from multiple sources.
      • denysvitali 3y ago
        The difference, IMHO, is that it's easier to pick an easy guessable password than it is to create an insecure RSA key (on purpose).
      • dboreham 3y ago
        It depends what happens to the password. Typically it's sent as a bearer credential. But there are auth schemes (not widely used these days) where the password isn't sent over the wire.
• deleted 3y ago
  [deleted]
• gavinhoward 3y ago
  If that's true, then I am 100% certain that this backdoor is from a nation state-level actor.
  • junon 3y ago
    That's what this feels like. That, or someone who wanted to sell this to one. Can't imagine the ABC's are sleeping on this one at this point.
  • jeroenhd 3y ago
    I already felt like this was way too sophisticated for a random cybercriminal. It's not like making up fake internet identities is very difficult, but someone has pretended to be a good-faith contributor for ages, in a surprisingly long-term operation. You need some funding and a good reason to pull off something like that. This could also be a ransomware group hoping to break into huge numbers of servers, though. Ransomware groups have been getting more sophisticated and they will already infiltrate their targets for months at a time (to make sure all old backups are useless when they strike), so I wouldn't put it past them to infiltrate the server authentication mechanism directly.
    • TheBlight 3y ago
      Whoever this was is going after a government or a crypto exchange. Don't think anything else merits this effort.
      • delfinom 3y ago
        Otoh, maybe they just wanted to create a cryptomining farm. Lol. Don't underestimate the drive some people have to make a buck.
        • 1oooqooq 3y ago
          people downvoting you already forgot the cia fundraiser in the 80s...
      • 3y ago
• chadsix 3y ago
  [dead]
• emeraldd 3y ago
  If I'm reading this right, would there be any persistent evidence of the executed payload? I can't think of a reason anything would have to go to disk in this situation, so a compromise could easily look like an Auth failure in the logs .... maybe a difference in timings .. but that's about it ...
  • junon 3y ago
    Unless the payload did something that produced evidence, or if the program using openssl that was affected was, for some reason, having all of its actions logged, then no, probably not.
• noncoml 3y ago
  Unpopular opinion, but I cannot but admire the whole operation. Condemn it of course, but still admire it. It was a piece of art! From conception to execution, masterful! We got extremely lucky that it was caught so early.
  • dc-programmer 3y ago
    I agree, but the social engineering parts do feel particularly cruel
    • nocoiner 3y ago
      I felt really bad for the original maintainer getting dog-piled by people who berated him for not doing his (unpaid) job and basically just bring shame and discredit to himself and the community. Definitely cruel. Though… do we know that the maintainer at that point was the same individual as the one who started the project? Goes deep, man.
      • StefanBatory 3y ago
        Even if it's not his fault the maintainer at this point won't be trusted at all. I feel for him, I think even finding a job at this moment for him would be impossible. Why would you hire someone that could be suspected for that?
        • polio 3y ago
          This could've happened to anybody, frankly. The attacker was advanced and persistent. I cannot help but feel sympathetic for the original maintainer here.
          • consumer451 3y ago
            From TFA's profile: https://bsky.app/profile/filippo.abyssdomain.expert/post/3kovq64a2g32g https://bsky.app/profile/filippo.abyssdomain.expert/post/3ko... This is a profound realization, isn't it? How much more paranoid should/will maintainers be going forward?
• wiktor-k 3y ago
  > OpenSSH certs are weird in that they include the signer's public key. OpenSSH signatures in general contain signer's public key, which I personally think it's not weird but rather cool since it allows verifying the signature without out of the band key delivery (like in OpenPGP). The authentication of the public key is a separate subject but at least some basic checks can be done with an OpenSSH signature only.
  • 1oooqooq 3y ago
    > cool since it allows verifying the signature without out of the band key delivery hope you do key selection sanitization instead of the default (nobody does). otherwise you're accepting random keys you have laying around (like github) when logging to secret.example.com
    • slooonz 3y ago
      What do you mean ?
    • dolmen 3y ago
      Using an SSH key used with GitHub for other purposes than GitHub is not a good practice (even if it's common). https://github.com/dolmen/github-keygen https://github.com/dolmen/github-keygen
      • dmonitor 3y ago
        I’m confused. I make a unique private key for each machine I use. How is using that machine-specific key on multiple hosts insecure?
        • dolmen 3y ago
          Your SSH public keys used on GitHub are very publicly exposed. This information could be used by SSH servers you are connecting to. You might think you are connecting anonymously, while in fact your SSH client is sending your public key which could then be resolved to your GitHub account.
• asveikau 3y ago
  I have found it irritating how in the community, in recent years, it's popular to say that if a project doesn't have recent commits or releases that something is seriously wrong. This is a toxic attitude. There was nothing wrong with "unmaintained" lzma two years ago. The math of the lzma algorithm doesn't change. The library was "done" and that's ok. The whiny mailing list post from the sock puppet, complaining about the lack of speedy releases, which was little more than ad hominem attacks on the part time maintainer, is all too typical and we shouldn't assume those people are "right" or have any validity to their opinion.
  • snnn 3y ago
    I mostly agree with you, but I think your argument is wrong. Last month I found a tiny bug in Unix's fgrep program(the bug has no risk). The program implements Aho Corasick algorithm, which hasn't changed much over decades. However, at least when the code was released to 4.4BSD, the bug still existed. It is not much a concern as nowadays most fgrep progroms are just an alias of grep. They do not use the old Unix code anymore. The old Unix code, and much part of FreeBSD, really couldn't meet today's security standard.For example, many text processing programs are vulnerable to DoS attacks when processing well-crafted input strings. I agree with you that in many cases we really don't need to touch the old code. However, it is not just because the algorithm didn't change.
  • deleted 3y ago
    [deleted]
  • cesarb 3y ago
    > The math of the lzma algorithm doesn't change. The library was "done" and that's ok. Playing devil's advocate: the math doesn't change, but the environment around it does. Just off the top of my head, we have: the 32-bit to 64-bit transition, the removal of pre-C89 support (https://fedoraproject.org/wiki/Changes/PortingToModernC https://fedoraproject.org/wiki/Changes/PortingToModernC) which requires an autotools update, the periodic tightening of undefined behaviors, new architectures like RISC-V, the increasing amount of cores and a slowdown in the increase of per-core speed, the periodic release of new and exciting vector instructions, and exotic security features like CHERI which require more care with things like pointer provenance.
• bawolff 3y ago
  The headline seems like a distinction without a difference. Bypassing ssh auth means getting a root shell. There is no significant difference between that and running system(). At most maybe system() has less logging.
  • armitron 3y ago
    > Bypassing ssh auth means getting a root shell Only if you're allowed to login as root, which is definitely not the case everywhere.
    • colinsane 3y ago
      not only that, but logins show up in logs.
    • juliusdavies 3y ago
      My sense was this backdoor gets to execute whatever it wants using whatever "user" sshd is running as. So even if root logins are disabled, this backdoor doesn't care.
    • wepple 3y ago
      Plus, detection is likely to be be very different.
  • deleted 3y ago
    [deleted]
• jmward01 3y ago
  Where is the law enforcement angle on this? This individual/organization needs to be on the top of every country's most wanted lists.
  • ganeshkrishnan 3y ago
    [flagged]
    • stephc_int13 3y ago
      How do you know that?
      • deleted 3y ago
        [deleted]
      • delfinom 3y ago
        [flagged]
        • deleted 3y ago
          [deleted]
  • greyface- 3y ago
    I understand the impulse to seek justice, but what crime have they committed? It's illegal to gain unauthorized access, but not to write vulnerable code. Is there evidence that this is being exploited in the wild?
    • jmward01 3y ago
      I am definitely not a lawyer so I have no claim to knowing what is or is not a crime. However, if backdooring SSH on a potentially wide scale doesn't trip afoul of laws then we need to seriously have a discussion about the modern world. I'd argue that investigating this as a crime is likely in the best interest of public safety and even (I hesitate to say this) national security considering the potential scale of this. Finally, I would say there is a distinction between writing vulnerable code and creating a backdoor with malicious intent. It appears (from the articles I have been reading so far) that this was malicious, not an accident or lack of skill. We will see over the next few days though as more experts get eyes on this.
• paolomainardi 3y ago
  I am wondering if reinstalling the entire Archlinux installation would be a wise choice.
  • sebiw 3y ago
    Arch Linux uses a native/unpatched version of OpenSSH without dependency on libsystemd and thus without dependency on xz-utils, resulting in no exploitable code path. This means that at least the currently talked about vulnerability/exploit via SSH did presumably not work on Arch. Disclaimer: This is my understanding of the currently circulating facts. Additional fallout might be possible, as the reverse engineering of the backdoor is ongoing.
    • pas 3y ago
      there are other ways for liblzma to get into ssh (via PAM and libselinux)
    • Deathcrow 3y ago
      This is only correct if the sshd backdoor is the only malicious code introduced into the library.
  • Phelinofist 3y ago
    Just to extend the sibling comment with an excerpt of the Arch announce mail regarding the backdoor: >From the upstream report [1]: > openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma. Arch does not directly link openssh to liblzma, and thus this attack vector is not possible. You can confirm this by issuing the following command: ``` ldd "$(command -v sshd)" ``` However, out of an abundance of caution, we advise users to remove the malicious code from their system by upgrading either way. This is because other yet-to-be discovered methods to exploit the backdoor could exist.
• bagels 3y ago
  Why does xz need new features at this point?
• sneak 3y ago
  The questions this backdoor raises: - what other ones exist by this same team or similar teams? - how many such teams are operating? - how many such dependencies are vulnerable to such infiltration attacks? what is our industry’s attack surface for such covert operations? I think making a graph of all major network services (apache httpd, postgres, mysql, nginx, openssh, dropbear ssh, haproxy, varnish, caddy, squid, postfix, etc) and all of their dependencies and all of the committers to all of those dependencies might be the first step in seeing which parts are the most high value and have attracted the least scrutiny. This can’t be the first time someone attempted this - this is just the first unsuccessful time. (Yes, I know about the attempted/discovered backdoor in the linux kernel - this is remote and is a horse of a different color).
  • almostnormal 3y ago
    Why did they decide to create a backdoor, instead of using a zeroday like everyone else? Why did they implement a fully-featured backdoor and attempted to hide the way it is deployed, instead of deploying something innocent-looking that might as well be a bug if detected? These must have been conscious decisions. The reasons might provide a hint what the goals might have been.
    • formerly_proven 3y ago
      Ed448 is orders of magnitude better NOBUS than hoping that nobody else stumbles over the zero-day you found.
    • Ekaros 3y ago
      If they seemingly almost succeeded how many others have already done similar backdoor? Or was this actually just poking on things seeing if it was possible to inject this sort of behaviour?
    • bdd8f1df777b 3y ago
      Presumably because other people can also utilize the “bug” they create intentionally but looking inadvertently. This backdoor however is activated by the private key only the attacker has so it’s airtight.
• sega_sai 3y ago
  One have question on this is, if the backdoor would not been discovered due to performance issue (which was as I understood it purely an oversight/fixable deficiency in the code), what are the chances of discovering this backdoor later, or are there tools that would have picked it up? Those questions are IMO relevant to understand if this kind of backdoor is the first one of the kind, or the first one that was uncovered.
  • wepple 3y ago
    I expect a lot of people will be doing a whole lot of thinking along these lines over the next months. Code review? Some kind of behavioral analysis? IMO the call to system() was kind of sloppy, and a binary capabilities scanner could have potentially identified a path to that.
    • tux3 3y ago
      I think behavioral analysis could be promising. There's a lot of weird stuff this code does on startup that any reasonable Debian package on the average install should not be doing in a million years. Games and proprietary software will sometimes ship with DRM protection layers that do insane things in the name of obfuscation, making it hard to distinguish from malware. But (with only a couple exceptions) there's no reason for a binary or library in a Debian package to ever try to write the PLT outside of the normal mechanism, to try to overwrite symbols in other modules, to add LD audit hooks on startup, to try to resolve things manually by walking ELF structures, to do anti-debug tricks, or just to have any kind of obfuscation or packing that free software packaged for a distro is not supposed to have. Some of these may be (much) more difficult to detect than others, some might not be realistic. But there are several plausible different ways a scanner could have detected something weird going on in memory during ssh startup. No one wants a Linux antivirus. But I think everyone would benefit from throwing all the behavioral analysis we can come up with at new Debian package uploads. We're very lucky someone noticed this one, we may not have the same luck next time.
      • raggi 3y ago
        Except had we been doing that they would have put guards in place to detect it - as they already had guards to avoid the code path when a debugger is attached, to avoid building the payload in when it's not one of the target systems, and so on. Their evasion was fairly extensive, so we'd need many novel dynamic systems to stand a chance, and we'd have to guard those systems extremely tightly - the author got patches into oss-fuzz as well to "squash false positives". All in all, adding more arms to the arms race does raise the bar, but the bar they surpassed already demonstrated tenacity, long term thinking, and significant defense and detection evasion efforts.
• KennyBlanken 3y ago
  [flagged]
  • skybrian 3y ago
    Does the Go SDK even have any third-party dependencies? How could something like the XZ backdoor happen? Also, given that Go binaries are statically linked, if you can't rebuild from source, you're screwed. You need to get good at building from source.
  • fbdab103 3y ago
    I do not know why I read Golang issue threads. I always get angry at the we-know-better attitude. With just a dash of, "Well, that's not how Google does it. Why don't you just do better internally?"
• chatmasta 3y ago
  Can someone explain succinctly what the backdoor does? Do we even know yet? The backdoor itself is not a payload, right? Does it need a malicious archive to exploit it? Or does it hook into the sshd process to listen for malicious packets from a remote attacker? The OP makes it sound like an attacker can send a malicious payload in the pre-auth phase of an SSH session - but why does he say that an exploit might never be available? Surely if we can reverse the code we can write a PoC? Basically, how does an attacker control a machine with this backdoor on it?
  • snnn 3y ago
    That's the most interesting part. No, we don't know it yet. The backdoor is so sophisticated that none of us can fully understand it. It is not a “usual” security bug.
    • heresWaldo 3y ago
      Yeah these types of security issues will be used by politicians to force hardware makers to lockdown hardware, embed software in chips. The go fast startups habit of “import the world to make my company products” is a huge security issue IT workers ignore. The only solution politics and big tech will chase is obsolete said job market by pulling more of the stack into locked down hardware, with updates only allowed to come from the gadget vendor.
      • WesolyKubeczek 3y ago
        Which will make updates either expensive or impossible. You will be able to write books about exploitable bugs in the hardware, and those books will easily survive several editions.
      • avidiax 3y ago
        The NSA demands that Intel and AMD provide backdoor ways to turn off the IME/PSP, which are basically a small OS running in a small processor inside your processor. So the precedent is that the government wants less embedded software in their hardware, at least for themselves. If we relied on gadget vendors to maintain such software, I think we can just look at any IoT or router manufacturer to get an idea of just how often and for how long they will update the software. So that idea will probably backfire spectacularly if implemented.
• dolmen 3y ago
  git.tukaani.org runs sshd. If that sshd was upgraded with the xz backdoor, we cannot exclude that the host was compromised as it could be have been a obvious target for the backdoor author.
  • bostik 3y ago
    Rather unlikely. The bad actor never had access to git.tukaani.org, and the sshd version running on that host is: SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u3 That is, a stable Debian release. Definitely not one with liblzma5:5.6.x
• justinsaccount 3y ago
  > Apparently the backdoor reverts back to regular operation if the payload is malformed or the signature from the attacker's key doesn't verify. Does this mean it's possible to send every ssh server on the internet a malformed payload to get it to disable the backdoor if it was vulnerable?
  • denysvitali 3y ago
    It just reverts it for the specific connection - most likely to not raise suspicions on the fact that SSH doesn't accept RSA keys anymore
• pcthrowaway 3y ago
  Has anyone proposed a name for this exploit yet?
  • kstrauser 3y ago
    Dragon Gate. There’s my contribution.
  • pixelbeat__ 3y ago
    rxzec
  • aaronmdjones 3y ago
    CVE-2024-3094
    • chr-s 3y ago
      "Backdoor in upstream xz/liblzma leading to ssh server compromise"[0] isn't bad either. [0]https://www.openwall.com/lists/oss-security/2024/03/29/4 https://www.openwall.com/lists/oss-security/2024/03/29/4
  • denysvitali 3y ago
    Pick one: xzdoor, backdoorssh
  • TheSmoke 3y ago
    doordassh :)
    • pcthrowaway 3y ago
      ooh I like this one. I'd propose: backdoordassh But I wonder if they'd run into trademark issues
  • LeoPanthera 3y ago
    LZMAO
    • 3y ago
• martinohansen 3y ago
  Imagine a future where state actors have hundreds of AI agents fixing bugs, gaining reputation while they slowly introduce backdoors. I really hope open source models succeed.
  • pas 3y ago
    ... if we want security it needs trust anyway. it doesn't matter if it's amazing Code GPT or Chad NSA, the PR needs to be reviewed by someone we trust. it's the trust that's the problem. web of trust purists were right just ahead of the time.
    • pvg 3y ago
      Chad NSA It's called the ANS is Chad.
    • cqqxo4zV46cp 3y ago
      That just…doesn’t make any sense. Everyone starts from zero and works their way up.
    • bee_rider 3y ago
      It would actually be sort of interesting if multiple adversarial intelligence agencies could review and sign commits. We might not trust any particular intelligence agency, but I bet the NSA and China would both be interested in not letting much through, if they knew the other guy was looking.
      • ecshafer 3y ago
        That is an interesting solution. If China, US, Russia, EU, etc all sign off and say "yep this is secure" we should trust it. Since if they think they found an exploit, they might assume the other people found an exploit. This is a little bit like the idea of a fair cut for a cake. If you have two people that want the last slice of cake, you have one cut and the other choose the first slice, since the chooser will choose the biggest slice, so the slicer knowing they will get the smaller will make it as equal as possible. In this case the NSA makes the cut (the code), and Russia / China chooses if its allowed in.
• deleted 3y ago
  [deleted]
• transpute 3y ago
  SE Linux comment by @poettering, https://news.ycombinator.com/item?id=39867126 https://news.ycombinator.com/item?id=39867126 > Libselinux pulls in liblzma too and gets linked into tons more programs than libsystemd. And will end up in sshd too (at the very least via libpam/pam_selinux). And most of the really big distros tend do support selinux at least to some level. Hence systemd or not, sshd remains vulnerable by this specific attack. Devuan comment by @capitainenemo, https://news.ycombinator.com/item?id=39866190 https://news.ycombinator.com/item?id=39866190 > The sshd in Devuan does link to a libsystemd stub - this is to cut down on their maintenance of upstream packages. However that stub does not link to lzma. Future proofing, https://gynvael.coldwind.pl/?lang=en&id=782 https://gynvael.coldwind.pl/?lang=en&id=782 Stage 2 "extension" mechanism This whole thing basically looks like an "extension/patching" system that would allow adding future scripts to be run in the context of Stage 2, without having to modify the original payload-carrying test files. Which makes sense, as modyfing a "bad" and "good" test files over and over again is pretty suspicious. So the plan seemed to be to just add new test files instead, which would have been picked up, deciphered, and executed.
  • deleted 3y ago
    [deleted]
• thegrim000 3y ago
  I've seen a lot of discussion on the topic but have yet to see someone just specify which versions of xz are likely affected so that I can verify whether I'm running them or not ..
  • aftbit 3y ago
    Ask your distro. On Arch, it's 5.6.0-1 and 5.6.1-1. Most of Debian (outside sid/unstable) is unaffected, as allegedly is Red Hat. https://archlinux.org/news/the-xz-package-has-been-backdoored/ https://archlinux.org/news/the-xz-package-has-been-backdoore...
• dboreham 3y ago
  So this is called the house of cards backdoor now?
• rany_ 3y ago
  If my server doesn't have any RSA public keys in its authorized_keys, only ed25519 keys; does this backdoor just not work?
  • keiba 3y ago
    This backdoor does not care about any of the authorisation configuration set by the user. It is executed before that step. So just make sure you are not affected.
    • rany_ 3y ago
      It was just that it hooks to `RSA_public_decrypt` which threw me off, I didn't really understand this backdoor much. I only have one Debian sid machine which was vulnerable and accessible via a public IPv4 ssh, I'm not sure if I should just wipe it.
  • adtac 3y ago
    It will still work if the connecting client offers a RSA key. The only real way to be sure it's not on your system is if your liblzma version is strictly less than 5.6.0 (first infected version): ls -al $(ldd $(which sshd) | grep lzma | awk '{ print $3 }')
    • rany_ 3y ago
      Thanks for the reply, I was just curious because `RSA_public_decrypt` threw me off.
      • adtac 3y ago
        FWIW RSA_public_decrypt is an 90s way of saying RSA_signature_validate
• deleted 3y ago
  [deleted]
• creato 3y ago
  One of the takeaways from this to me is that there is way too much sketchy bullshit happening in critical system software. Prebuilt binary blobs [1]? Rewriting calls to SIMD enhanced versions at runtime [2]? Disabling sanitizers [3]? Incomprehensible build scripts [4]? All of this was either at least strongly frowned upon, if not outright unacceptable, on every project I've ever worked on, either professionally or for fun. And the stakes were far lower for those projects than critical linux system software. 1. https://lwn.net/Articles/967442/ https://lwn.net/Articles/967442/ 2, 3: https://github.com/google/oss-fuzz/pull/10667 https://github.com/google/oss-fuzz/pull/10667 4. https://news.ycombinator.com/item?id=39866161 https://news.ycombinator.com/item?id=39866161
  • cryptonector 3y ago
    > Rewriting calls to SIMD enhanced versions at runtime? That's something that's been done for decades. It's pretty normal. What's not normal is for that to get re-done after startup. That is, one library should not be able to get that resolution process to be re-done after it's been done once. Malicious code that knows the run-time linker-loader's data structures could still re-resolve things anyways, which means that even removing this feature altogether from the run-time linker-loader wouldn't prevent this particular aspect of this attack. I.e., you're barking up the wrong tree with (2).
  • nwallin 3y ago
    > Prebuilt binary blobs [1]? My understanding is that the binary blobs were test data. Find a bug that happens on certain input. Craft a payload that both triggers the bug and does the malicious thing you want to do. Add the binary blob to /tests/files/. Then write a legitimate test to ensure that the bug goes away. Then do some build script bullshit to somehow get that binary into the build.
  • ParetoOptimal 3y ago
    We should probably look into other fundamental dependencies like xz as well to see if similar practices are happening.
• senoralligator 3y ago
  [dead]
• cryptonector 3y ago
  I'm surprised the attackers used Ed448 instead of Ed25519.
  • supposemaybe 3y ago
    Maybe they know something we don’t about Ed25519? [Queue X-Files theme tune]
  • t0mas88 3y ago
    Maybe their organisation has a policy that requires stronger encryption? Possibly because that organisation is also in the business of cracking such encryption...
    • hosteur 3y ago
      Is Ed448 stronger than Ed25519?
      • d-z-m 3y ago
        Quantitatively, yes(2^224 security target vs 2^128 security target respectively wrt. discrete log calculation). Qualitatively, 2^128 is already computationally infeasible(barring some advance in quantum computing), so the meaningful difference in security is debatable, assuming no weaknesses in the underlying curve.
• Edidiongben9 3y ago
  [dead]
• snnn 3y ago
  Maybe we should consider moving more and more system process to webassembly. wasmtime has a nice sandbox. Surely it will decrease the performance, but performance is not always that important. For example, on my dev machine even if SSHD or apache's performance dropped 3x because of that, I wouldn't mind. If I really care, spend more money to get a more powerful CPU.
• tevon 3y ago
  Currently if you visit the xz repository it is disabled for violating github's TOS. While it should clearly be disabled, I feel like github should leave the code and history up, while displaying a banner (and disabled any features that could be exploited), so that researchers and others can learn about the exploit. In more minor situations when a library is hosting malicious code, if I found the repo to be down I might not think anything of it.
  • jhfdbkofdchk 3y ago
    I imagine they don’t want automation downloading it.
  • Randalthorro 3y ago
    You can find GitHub events from the repo as a csv here https://github.com/emirkmo/xz-backdoor-github https://github.com/emirkmo/xz-backdoor-github If you are interested in the source code that is easy to find. This code and git repo are linked all over the world, in many git repos, and the source is bundled many times in releases as well.
  • shp0ngle 3y ago
    xz has its own git mirror where you can see all the commits
    • junon 3y ago
      Notably only writable by Lasse who I personally believe is a Good Actor here.
• xuhu 3y ago
  It just seems implausible that the malicious x86 code would not have shown up in strace, perf record, or some backtrace. Once this ended up in all the major distros, some syscall or glibc call would have eventually looked like a red flag to someone before long.
• pmcao 3y ago
  [dead]
• probably_satan 3y ago
  [dead]
• lyu07282 3y ago
  God the amount of damage this would've caused, nightmarish, we are so unbelievably lucky. In a few months it would've been in every deb&rpm distribution. Thank God we found it early!
  • supposemaybe 3y ago
    Found it early?.. I found the backdoor on five of my Vultr servers as well as my MacBook Pro this evening. I certainly didn’t catch it early. So if that’s the state of it, it could very well be too late for many many companies. Not to mention folks who rely on TOR for their safety - there could be entire chains of backdoored entry, middle and exit nodes exposing vast numbers of TOR users over the past month or so (spies included!).
    • echoangle 3y ago
      Which Distro did you use on the affected devices?
    • Klonoar 3y ago
      ...maybe list the distros or macOS point releases/paths that you found it on macOS. ;P
      • fredoralive 3y ago
        Homebrew had updated to the backdoored version, so although it doesn’t appear to trigger on Mac OS, you should update things to ‘upgrade’ From 5.6.1 to 5.4.6.
    • lyu07282 3y ago
      It was only in rolling release/testing/unstable distributions, a pretty small subset of systems in the grand scheme of things, is why I said that. It was introduced in February 23 release of xz. This could've been years until discovered. Never use unstable/testing on real servers, that's a bad idea for entirely different reasons.
    • broadsidepicnic 3y ago
• tigerlily 3y ago
  As a de facto maintainer of an obscure open source game, I see devs come and go. I just merge all the worthwhile contributions. Some collaborators go pretty deep with their features, with a variety of coding styles, in a mishmash of C and C++. I'm not always across the implementation details, but in the back of my mind I'm thinking, man, anyone could just code up some real nasty backdoor and the project would be screwed. Lucky the game is so obscure and the attack surface minuscule, but it did stop me from any temptation to sign Windows binaries out of any sense of munificence. This xz backdoor is just the most massive nightmare, and I really feel for the og devs, and anyone who got sucked in by this.
  • p-e-w 3y ago
    > but in the back of my mind I'm thinking, man, anyone could just code up some real nasty backdoor and the project would be screwed That's true of course, but it's not a problem specific to software. In fact, I'm not even sure it's a "problem" in a meaningful sense at all. When you're taking a walk on a forest road, any car that comes your way could just run you over. Chances are the driver would never get caught. There is nothing you can do to protect yourself against it. Police aren't around to help you. This horror scenario, much worse than a software backdoor, is actually the minimum viable danger that you need to accept in order to be able to do anything at all. And yes, sometimes it does really happen. But at the end of the day, the vast majority of people just don't seek to actively harm others. Everything humans do relies on that assumption, and always has. The fantasy that if code review was just a little tighter, if more linters, CI mechanisms, and pattern matching were employed, if code signing was more widespread, if we verified people's identities etc., if all these things were implemented, then such scenarios could be prevented, that fantasy is the real problem. It's symptomatic of the insane Silicon Valley vision that the world can and should be managed and controlled at every level of detail. Which is a "cure" that would be much worse than any disease it could possibly prevent.
    • consumer451 3y ago
      > But at the end of the day, the vast majority of people just don't seek to actively harm others. Everything humans do relies on that assumption, and always has. https://en.wikipedia.org/wiki/Normalcy_bias https://en.wikipedia.org/wiki/Normalcy_bias ? > It's symptomatic of the insane Silicon Valley vision that the world can and should be managed and controlled at every level of detail. Which is a "cure" that would be much worse than any disease it could possibly prevent. What "cure" would you recommend?
• DrTung 3y ago
  If the RCE is Russian, it could be used a communication kill-switch on the morning of an attack outside Ukraine, similar to the Viasat hack https://en.m.wikipedia.org/wiki/Viasat_hack https://en.m.wikipedia.org/wiki/Viasat_hack
  • KnightHawk3 3y ago
    Why does it have to be russian? Could be any country, even USA, China. I don't see why this sort of attack is limited to Russia.
    • spxneo 3y ago
      Exactly I don't understand the obsession with Russia Literally just a country with people like you and me. "Jia Tan" does not sound Russian https://boehs.org/node/everything-i-know-about-the-xz-backdoor https://boehs.org/node/everything-i-know-about-the-xz-backdo...
      • jsploit 3y ago
        > "Jia Tan" does not sound Russian It's not a real name.
        • trallnag 3y ago
          What about X Æ A-12?
    • timeon 3y ago
      > Why does it have to be russian? Could be any country, even USA, China. Are you sure you are replying to right comment? Because they did not implied it was not other country or actor. It even directly starts with 'IF'.
      • KnightHawk3 3y ago
        doesnt track if russian -> could be ... why would being russian make it this? it could be if it was made in any country. they did this attack once, okay? but its not like other countries dont pay attention.
• circusfly 3y ago
  Could the backdoor have targeted Wireguard instead of ssh?
  • supposemaybe 3y ago
    Ssh is shell, whereas wireguard is a vpn. You will still be vulnerable as you can connect to an ssh server through your wireguard tunnel.
    • circusfly 3y ago
      Well, yes, I can, but unlike ssh which is open to the world, my VPN is only open to me and the family. It seems like that greatly reduces the potential attack surface.
    • d-z-m 3y ago
      I don't think that's what OP is asking. I think OP is asking if wireguard functions could be hooked in the same way as sshd functions are in this exploit.
  • circusfly 3y ago
    Down voted for asking a valid question? Or is every reader of every HN post expected to be an in depth expert in every article posted every minute of every day of the year? What kind of asshole who has earned the right to down vote comments on HN would down vote a legitimate question?
  • d-z-m 3y ago
    My first instinct would be no, as wireguard runs in kernel space(if you're using kernel wireguard, not wireguard-go/some other userspace implementation),and couldn't link in liblzma, a userspace component.
• lettergram 3y ago
  What I’d like to understand is it’s proven intentional? My understanding is it was a few added characters in a header file. I can’t tell you the number of times I was tired and clicked an extra key before committing, or my cat walked across the keyboard while I was out of the room.
  • userbinator 3y ago
    That level of sophistication is certainly intentional.
    • lettergram 3y ago
      That’s not an explanation about exactly how intention was derived. I suppose I’m asking for the chain of events that led to the conclusion. I see lots of technical hot takes for how something could work, with no validation it does, nor intent behind it. I’d like to understand what steps we know were taken and how that presents itself.
      • willbush 3y ago
        I think you're probably missing a lot context about the situation. Here's some useful links https://x0f.org/@FreePietje/112187047353892463 https://x0f.org/@FreePietje/112187047353892463 also https://gynvael.coldwind.pl/?lang=en&id=782 https://gynvael.coldwind.pl/?lang=en&id=782
        • lettergram 3y ago
          Appreciate, that’s the context I was looking for
  • adtac 3y ago
    It was a few added characters in a header file to make it possible to deliver the actual payload: 80+ kilobytes of machine code. There's no way to actually tell, but I'd estimate the malware source code to be O(10000) lines in C. It's actually pretty sophisticated. You don't accidentally write a in-memory ELF program header parser.
• brcmthrowaway 3y ago
  So this is why Apple hates open source. Random anons committing to your project.
  • Bnichs 3y ago
    I think their problem with open source is more that they can't have complete control and make every user's decision for them, security is just a nice tag along to that.
• tambourine_man 3y ago
  I’d like to know what we mortals – those that run Ubuntu LTS on VMs, for instance — need to do, if anything.
  • Valord 3y ago
    Same!
    • somethingsaid 3y ago
      Right now, nothing. The issue didn’t reach mainstream builds except nightly Red Hat and Fedora 41. The xz version affected has already been pulled and won’t be included in any future software versions.
  • pilgrim0 3y ago
    I would, by routine, advise that publicly available boxes are configured to accept connections only from whitelisted sources, doing that at the lowest possible level on the stack. That’s usually how secure environments such as those used in PCI compliant topologies are specified.
  • geggo98 3y ago
    My suggestion: Put your SSH behind WireGuard and/or behind a jump host (with only port forwarding allowed, no shell). If you don’t have a separate host, use a Docker container. If you use a jump host, consider a different OS (e.g., BSD vs Linux). Remember this analogy with slices of Swiss cheese used during the pandemics? If one slice has a hole, the next slice hopefully won’t have a hole on the same position. The more slices you have, the better for you. Although for remote management, you don’t want to have too many “slices” you have to manage and that can fail.
• willtemperley 3y ago
  Another good reason to use a firewall with an IP address allowlist for SSH.
  • adtac 3y ago
    And now for my next trick: smuggling a backdoor into iptables.
    • willtemperley 3y ago
      Ha yes. I rely on cloud-provided security groups for that. So I blindly have to believe my provider is immune to supply chain attacks.
• ngcc_hk 3y ago
  Whilst we are in 3bp mode, what happen if a state actor want to harm open source … not to destroy it but to pollute it … as said you can check and test all. Legal … they do not care. I wonder. … We are all good vs even Buddha has evil nature mental model.
• sgammon 3y ago
  There appears to be a string encoded in the binary payload: https://gist.github.com/q3k/af3d93b6a1f399de28fe194add452d01#file-hashes-txt-L115 https://gist.github.com/q3k/af3d93b6a1f399de28fe194add452d01... Which functions as a killswitch: https://piaille.fr/@zeno/112185928685603910 https://piaille.fr/@zeno/112185928685603910 If that is indeed the case, one mitigation might be ``` echo "yolAbejyiejuvnup=Evjtgvsh5okmkAvj" | sudo tee -a /etc/environment ```
  • sgammon 3y ago
    Make absolutely sure to include `-a` so it doesn't nuke your env file, and generally speaking, one should upgrade to a version without the malicious code and restart, of course.
    • chii 3y ago
      > without the malicious code and restart i wonder if the malicious code would've installed a more permanent backdoor elsewhere that would remain after a restart. I recall things like on windows where malware would replace your keyboard drivers or mouse drivers with their own ones that had the malware/virus, so that even if the original malware is removed, the system is never safe again. You'd have to wipe. And this is not even counting any firmware that might've been dropped.
  • dannyw 3y ago
    That's so strange. This reeks of nation state actors, wanting ways to protect their own systems.
    • ddalex 3y ago
      any competent malware dev would have a panic switch...
    • chii 3y ago
      if you need to test your own malware that you're developing, do you really want to just run it and disrupt your own system? It's not uncommon to put in a check that allows the malware to run but be a noop.
• ivlad 3y ago
  Doesn’t certificate verification occur in PrevSep process? If so, this would only grant RCE with nobody’s privilege.
• wtznc 3y ago
  [flagged]
  • deleted 3y ago
    [deleted]
• bandrami 3y ago
  This was so bizarre I had to check twice that it isn't actually April 1 (I'm in Asia so I get a lot of off-by-one-day stories from the US)
• nprateem 3y ago
  We basically need to analyse dependencies used in critical code paths (e.g. network attached services like sshd) and then start a process to add more rigorous controls around them. Some kind of enhanced scrutiny, certification and governance instead of relying on repos with individual maintainers and no meaningful code reviews, branch protection, etc. And by we I mean at international/governmental level. Free software needs to stop being $free. Society needs to start paying for the critical foundations upon which we all stand.
• juliusdavies 3y ago
  Lucky the XZ license switched from "Public Domain" to 0BSD in February (just before these 5.6.0 and 5.6.1 releases)! 0BSD has no clauses, but it does have this: > IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
  • perbu 3y ago
    Gross negligence and willfull misconduct cannot be waved away like this, in certain legislations, at least.
    • juliusdavies 3y ago
      Yes, but hopefully it does protect Lasse Collin.
      • broadsidepicnic 3y ago
        From what?
        • juliusdavies 3y ago
          Litigation including criminal prosecution
          • rwmj 3y ago
            He had nothing to do with this, no one is going to prosecute him.
          • broadsidepicnic 3y ago
            Uh what would be his charge? I cannot fathom how and based on what he would be charged. Maybe this is an american thing, but he's not an US citizen to start with
• kzrdude 3y ago
  It's weird now that "it's right there in the open", Listed among the most recent commits there is "update two test files" https://git.tukaani.org/?p=xz.git;a=commitdiff;h=6e636819e8f070330d835fce46289a3ff72a7b89 https://git.tukaani.org/?p=xz.git;a=commitdiff;h=6e636819e8f... And it's kind of smart to attack a compression library - you have plausible deniability for these opaque binary blobs - they are supposedly test cases, but in reality encode parts of the backdoor.
• rubymamis 3y ago
  It baffles me how such an important package that so many Linux servers use every day is unmaintained by the original author due to insufficient funds. Something gotta change in OSS. I think one solution could be in licenses that force companies/business of certain sizes to pay maintenance fees. One idea from the top of my head.
  • nicce 3y ago
    > I think one solution could be in licenses that force companies/business of certain sizes to pay maintenance fees. One idea from the top of my head. Yet people have huge opposition for those licenses. The big scream of "not free anymore" starts and the entity gets cancelled.
    • rubymamis 3y ago
      I think focusing on big organizations (e.g. above certain revenue/profit should help). There must be some sweet spot, after all, the organizations that rely on it should want it to be maintained as well.
      • chii 3y ago
        tragedy of the commons. If someone else can pay to maintain it, but you get the benefits, then it's the obvious strategy to use. And also, there's zero evidence that proprietary software won't have these backdoors. In fact, you can't even check them for it!
        • rubymamis 3y ago
          That's why I think such OSS packages should use licenses that force large companies to pay (moderate) fees for maintenance. I assume such sums of money won't even tickle them. Imagine 10 large companies, each pay $1000 a month for critical packages they use. For each developer, that's $10,000 they can either use to quit their current job or hire another person to share the burden. We need to normalize this.
• grepLeigh 3y ago
  I'm assuming nation states and similar actors monitor mailing lists for phrases like "I'm feeling burnt out" or "not enough bandwidth, can you open a PR?" According to the timeline here, trust was established in "only" a few years. https://boehs.org/node/everything-i-know-about-the-xz-backdoor https://boehs.org/node/everything-i-know-about-the-xz-backdo... So I imagine major actors already have other assets in at-risk open source projects, either for the source code or distro patch/packaging level. Is that too tinfoil hat? I only know enough about secops to be dangerous to myself and everyone around me.
  • junon 3y ago
    > Is that too tinfoil hat? Not at all, no. This is probably happening way more than just this instance, unfortunately.
• alsadi 3y ago
  Untill the entire scope of this is identified I suggest activating the kill switch https://bcksp.blogspot.com/2024/03/how-to-disable-xz-backdoor-or-alike-in.html?m=1 https://bcksp.blogspot.com/2024/03/how-to-disable-xz-backdoo...
• tonymet 3y ago
  Who was the author ?
  • tonymet 3y ago
    A subsequent investigation found that the backdoor was a culmination of approximately 3 years of effort by a user going by the name Jia Tan and the nickname JiaT75, who appears to have made a concentrated effort to gain access to a position of trust within the xz project, by putting pressure on the head maintainer to step down and hand over the control of the project.[3]
• christo 3y ago
  Does anyone know if a honeypot has been set up for this? In the event that exploit code has already been deployed, while these exploit attempts should now thankfully be futile, is there any valuable information that can be gained about the network sources of these exploit attempts? We might assume that since this attack was foiled, exploit attempts won't happen, but if this is an automated botnet project, there may already be other operational elements in the wild that are knocking? It's possible that there is an infection detection component to the project which is already measuring baseline accessibility, possibly using this and several alternate vectors. After all, effort was made to evade exploit attempt detection which would enable valuable active monitoring since it shouldn't trigger suspicions.
• twisteddiana 3y ago
  Would be awesome if this was an April fool's day joke!
• geenat 3y ago
  Similar exploit vector in Swoole and exactly why it was forked for Open Swoole: https://old.reddit.com/r/PHP/comments/1b9acx3/article_intro_to_swoole_and_async_php/ktxfboi/ https://old.reddit.com/r/PHP/comments/1b9acx3/article_intro_... Binary blobs under the guise of "testing"- exploit targeting a dependent library. Thankfully this only effects Swoole users, not all of PHP.