8 ms·

It's pretty interesting that they didn't just introduce an RCE that anyone can exploit, it requires the attacker's private key. It's ironically a very security

It's pretty interesting that they didn't just introduce an RCE that anyone can exploit, it requires the attacker's private key. It's ironically a very security conscious vulnerability.

haswell 3y ago
I suspect the original rationale is about preserving the longevity of the backdoor. If you blow a hole wide open that anyone can enter, it’s going to be found and shut down quickly. If this hadn’t had the performance impact that brought it quickly to the surface, it’s possible that this would have lived quietly for a long time exactly because it’s not widely exploitable.
- declan_roberts 3y ago
  I agree that this is probably about persistence. Initially I thought the developer was playing the long-con to dump some crypto exchange and make off with literally a billion dollars or more. But if that was the case they wouldn't bother with the key. It'd be a one-and-done situation. It would be a stop-the-world event. Now it looks more like nation-state spycraft.
  - meowface 3y ago
    >But if that was the case they wouldn't bother with the key. It'd be a one-and-done situation. It would be a stop-the-world event. Why not? It's possible someone else could've discovered the exploit before the big attack but decided not to disclose it. Or that they could've disclosed it and caused a lot of damage the attacker didn't necessarily want. And they easily could've been planning both a long-term way to access a huge swath of machines and also biding their time for a huge heist. They have no reason to not restrict the backdoor to their personal use. And it probably is spycraft of some sort, and I think more likely than not it's a nation-state, but not necessarily. I could see a talented individual or group wanting to pull this off.
    - varenc 3y ago
      I think we need to consider the context. The attacker ultimately only had control over the lzma library. I'm skeptical that there's an innocent looking way that lzma could have in the open introduced an "accidental" RCE vuln that'd affect sshd. Of course I agree that they also wanted an explicit stealth backdoor for all the other reasons, but I don't think a plausibly deniable RCE or authentication bypass vuln would have even been possible.
  - btown 3y ago
    It's worth also noting that the spycraft involved a coordinated harassment campaign of the original maintainer, with multiple writing styles, to accelerate a transition of maintainership to the attacker: https://www.mail-archive.com/xz-devel@tukaani.org/msg00566.html https://www.mail-archive.com/xz-devel@tukaani.org/msg00566.h... https://www.mail-archive.com/xz-devel@tukaani.org/msg00568.html https://www.mail-archive.com/xz-devel@tukaani.org/msg00568.h... https://www.mail-archive.com/xz-devel@tukaani.org/msg00569.html https://www.mail-archive.com/xz-devel@tukaani.org/msg00569.h... While this doesn't prove nation-state involvement, it certainly borrows from a wide-ranging playbook of techniques.
    - allanbreyes 3y ago
      Ugh, that this psyops sockpuppetry may have started or contributed to the maintainer's mental health issues seems like the most depressing part of all this. Maintaining OSS is hard enough.
      - __MatrixMan__ 3y ago
        No good deed goes unpunished, what a shame.
      - btown 3y ago
        I hope that one takeaway from this entire situation is that if you're a maintainer and your users are pushing you outside your levels of comfort, that's a reflection on them, not on you - and that it could be reflective of something far, far worse than just their impatience. If you, as a maintainer, value stability of not only your software but also your own mental health, it is entirely something you can be proud of to resist calls for new features, scope increases, and rapid team additions.
      - Delk 3y ago
        Probably didn't start them, considering that he already mentioned (long-term) mental health issues in the mailing list discussion in which the (likely) sock puppets started making demands. But it's hard to see the whole thing helping, and it is some combination of depressing and infuriating. I hope he's doing ok.
  - juitpykyk 3y ago
    You're talking as if securing a backdoor with public cryptography is some unimaginable feat of technology. It's literally a couple hours work.
    - kbenson 3y ago
      I don't think they were using complexity as the reason for that assumption, but instead goals. Adding security doesn't require a nation state's level of resources, but it is a more attractive feature for a nation state that wants to preserve it over time and prevent adversaries from making use of it.
      - neodymiumphish 3y ago
        And on the contrary, creating a vulnerability that’s not identifiable to a limited attack group provides for a bit more deniability and anonymity. It’s hard to say which is more favorable by a nation-state actor.
        usrusr 3y ago
        An interesting angle: if this was somehow observed in use instead of getting discovered without observing use, there would be a glaring bright cui bono associated with what it was being used for. Theoretically, I could even imagine that public key security motivated by some "mostly benign" actor who fell in love with the question "could I pull this off?" dreading the scenario of their future hacker superpower falling in the wrong hands. It's not entirely unthinkable that an alternative timeline where this was never discovered would only ever see this superbackdoor getting used for harmless pranks, winning bets and the like. In this unlikely scenario, the discovery, through the state actor and crime syndicate activities undoubtedly inspired by the discovery, would paradoxically lead to a less safe computing world than the alternative timeline where the person with the key had free RCE almost everywhere.
      - heavyset_go 3y ago
        This makes sense in closed source products where you'll never get to audit the source for such exploits, but little sense in open source projects where anyone can audit it. That's to say an enterprise router or switch would likely have secured exploits put there by corporate and national security agencies, whereas open source exploits would benefit from the probable deniability.
  - heavyset_go 3y ago
    > Now it looks more like nation-state spycraft. I disagree, plenty of exploits implemented WireGuard-like crypto key access to them.
- eli 3y ago
  More to the point it prevents your enemies from using the exploit against friendly targets. The tradeoff is that, once you find it, it's very clearly a backdoor. No way you can pretend this was an innocent bug.
- chpatrick 3y ago
  Also people can come and immediately undo whatever you did if it's not authenticated.
- Beijinger 3y ago
  This is just a safety measure that it does not blow up in your own face (country).
  - CommitSyn 3y ago
    Not only did it mandate authentication, but there was also a Killswitch, and there's no other reason for that than because you have reasons to suspect someone may have access to this sensitive world-stopper exploit and use it against you. That leaves simply a (reasonably, but still) paranoid person, and people worried about that+consequences are unlikely to have the mental well-being to pull this off, or, a group of people.
- deleted 3y ago
  [deleted]
- deleted 3y ago
  [deleted]
Alifatisk 3y ago
For real, it's almost like a state-sponsored exploit. It's crafted and executed incredibly well, the performance issue feels like pure luck it got found.
- jdewerd 3y ago
  Yeah, we should probably expect that there are roughly 1/p(found) more of these lurking out there. Not a pleasant thought.
- andersa 3y ago
  It'd make total sense if it was. This way you get to have the backdoor without your enemies being able to use it against your own companies.
- hnthrowaway0328 3y ago
  Do we have a detailed technical analysis of the code? I read a few analysis but they all seem preliminary. It is very useful to learn from the code.
  - coldpie 3y ago
    There's a few links down at the bottom of the OP to quite detailed analysis. From there you could join a Discord where discussion is ongoing.
    - hnthrowaway0328 3y ago
      Thanks coldpie.
- stingraycharles 3y ago
  I like the theory that actually, it wasn’t luck but was picked up on by detection tools of a large entity (Google / Microsoft / NSA / whatever), and they’re just presenting the story like this to keep their detection methods a secret. It’s what I would do.
linsomniac 3y ago
Am I reading it correctly that the payload signature includes the target SSH host key? So you can't just spray it around to servers, it's fairly computationally expensive to send it to a host.
- miduil 3y ago
  *host key fingerprint, but I assume what you've meant. It's practically a good backdoor then, crypto graphically protected and safe against "re-play" attacks.
  - amluto 3y ago
    Not quite. It still looks vulnerable: an attacker A without the private key impersonates a victim server V and reports their host key. A careless attacker B with the key tries to attack A, but an ends up recovering a valid payload targeting V.
    - Denvercoder9 3y ago
      I'm not too familar with the SSH protocol, but is it possible to impersonate a victim server V without having the private key to their host key?
      - pxx 3y ago
        This stuff is pre-auth. You can just treat the entire thing as opaque and proxy everything to the host you're trying to compromise; as soon as you have an exploit string for a given host you can just replay it.
        robryk 3y ago
        Ssh does client authentication after handshake. The server is required to sign the handshake result with its private key, so you won't get past handshake if you are a server that claims to have a public key that you don't know the private key for. E: see RFC 4253, sections 8 and 10, and RFC 4252 for corroboration
nialv7 3y ago
how are you going to sell it if anyone can get in?
- ryanmerket 3y ago
  this is probably the right answer. hacking group getting a 0day to sell to nation states on a per-use agreement.
cesarb 3y ago
This is called NOBUS: https://en.wikipedia.org/wiki/NOBUS https://en.wikipedia.org/wiki/NOBUS
- halJordan 3y ago
  This is not that concept. That concept is no one but us can technically complete the exploit. Technical feasibility in that you need a supercomputer to do it, not protecting a backdoor with the normal cia triad
  - justinclift 3y ago
    That doesn't seem correct: If they determine the vulnerability is only exploitable by the NSA for reasons such as computational resources, budget, or skill set, they label it as NOBUS and will not move to patch it, but rather leave it open to exploit against current or future targets. If (!) the NSA regards ssh keys as secure, then from that article it sounds like the NOBUS thing would fit.
    - CommitSyn 3y ago
      That would only fit "If (!) the NSA regards ssh keys as secure >for everyone but them<
      - justinclift 3y ago
        Not sure why? My thinking in this hypothetical scenario is that the NSA would have the private key, which is why it would be a NOBUS thing. If they didn't have the key though, then yeah it doesn't fit. Unless they can walk through ssh key security anyway. :)
    - bed99 3y ago
      Is it legal and possible for the NSA to socially attack someone with mental health issues this way? I'd like to rule-out some state actors by the mere heartbreaking act of socially engineering and targeting a person in pain. Not even considering the original maintainer may be a US citizen (idk).
takeda 3y ago
If you think of it as a state sponsored attack it makes a lot of sense to have a "secure" vulnerability in system that your own citizens might use. It looks like the whole contribution to xz was an effort to just inject that backdoor. For example the author created the whole test framework where he could hide the malicious payload. Before he started work on xz, he made contribution to libarchive in BSD which created a vulnerability.
- pxx 3y ago
  The libarchive diff didn't create any vulnerability. The fprintf calls were consistent with others in the same repository.
  - takeda 3y ago
    They still preferred to revert it as it looks very suspicious. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1068047 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1068047 https://github.com/libarchive/libarchive/pull/2101 https://github.com/libarchive/libarchive/pull/2101
    - asveikau 3y ago
      If I read that correctly the problem is that it prints a filename that might include terminal control sequences that come from an attacker-controlled file name. Comment in your second link: https://github.com/libarchive/libarchive/pull/1609#issuecomment-2028125584 https://github.com/libarchive/libarchive/pull/1609#issuecomm...
  - jhugo 3y ago
    It did, actually: the filename can contain terminal control characters, which thanks to the change from safe_fprintf to fprintf, were printed without escaping, which allows the creator of the archive being extracted to control the terminal of the user extracting the archive.
    - pxx 3y ago
aaron695 3y ago
[dead]
userbinator 3y ago
IMHO it's not that surprising; asymmetric crypto has been common in ransomware for a long time, and of course ransomware in general is based on securing data from its owner. "It's not only the good guys who have guns."
Taniwha 3y ago
I wonder if anyone has packet logs from the past few weeks that show attempts at sshd - might be some incriminating IP addresses to start hunting with
whirlwin 3y ago
Imagine how much the private key is worth on the black market
bayindirh 3y ago
It's a (failed) case study in "what if we backdoor it in a way only good guys can use but bad guys can't?" Computers don't know who's / what's good or bad. They're deterministic machines which responds to commands. I don't know whether there'll be any clues about who did this, but this will be the poster child of "you can't have backdoors and security in a single system" argument (which I strongly support).

It's pretty interesting that they didn't just introduce an RCE that anyone can exploit, it requires the attacker's private key. It's ironically a very security conscious vulnerability.

haswell 3y ago

I suspect the original rationale is about preserving the longevity of the backdoor. If you blow a hole wide open that anyone can enter, it’s going to be found and shut down quickly. If this hadn’t had the performance impact that brought it quickly to the surface, it’s possible that this would have lived quietly for a long time exactly because it’s not widely exploitable.

declan_roberts 3y ago
I agree that this is probably about persistence. Initially I thought the developer was playing the long-con to dump some crypto exchange and make off with literally a billion dollars or more. But if that was the case they wouldn't bother with the key. It'd be a one-and-done situation. It would be a stop-the-world event. Now it looks more like nation-state spycraft.
- meowface 3y ago
  >But if that was the case they wouldn't bother with the key. It'd be a one-and-done situation. It would be a stop-the-world event. Why not? It's possible someone else could've discovered the exploit before the big attack but decided not to disclose it. Or that they could've disclosed it and caused a lot of damage the attacker didn't necessarily want. And they easily could've been planning both a long-term way to access a huge swath of machines and also biding their time for a huge heist. They have no reason to not restrict the backdoor to their personal use. And it probably is spycraft of some sort, and I think more likely than not it's a nation-state, but not necessarily. I could see a talented individual or group wanting to pull this off.
  - varenc 3y ago
    I think we need to consider the context. The attacker ultimately only had control over the lzma library. I'm skeptical that there's an innocent looking way that lzma could have in the open introduced an "accidental" RCE vuln that'd affect sshd. Of course I agree that they also wanted an explicit stealth backdoor for all the other reasons, but I don't think a plausibly deniable RCE or authentication bypass vuln would have even been possible.
- btown 3y ago
  It's worth also noting that the spycraft involved a coordinated harassment campaign of the original maintainer, with multiple writing styles, to accelerate a transition of maintainership to the attacker: https://www.mail-archive.com/xz-devel@tukaani.org/msg00566.html https://www.mail-archive.com/xz-devel@tukaani.org/msg00566.h... https://www.mail-archive.com/xz-devel@tukaani.org/msg00568.html https://www.mail-archive.com/xz-devel@tukaani.org/msg00568.h... https://www.mail-archive.com/xz-devel@tukaani.org/msg00569.html https://www.mail-archive.com/xz-devel@tukaani.org/msg00569.h... While this doesn't prove nation-state involvement, it certainly borrows from a wide-ranging playbook of techniques.
  - allanbreyes 3y ago
    Ugh, that this psyops sockpuppetry may have started or contributed to the maintainer's mental health issues seems like the most depressing part of all this. Maintaining OSS is hard enough.
    - __MatrixMan__ 3y ago
      No good deed goes unpunished, what a shame.
    - btown 3y ago
      I hope that one takeaway from this entire situation is that if you're a maintainer and your users are pushing you outside your levels of comfort, that's a reflection on them, not on you - and that it could be reflective of something far, far worse than just their impatience. If you, as a maintainer, value stability of not only your software but also your own mental health, it is entirely something you can be proud of to resist calls for new features, scope increases, and rapid team additions.
    - Delk 3y ago
      Probably didn't start them, considering that he already mentioned (long-term) mental health issues in the mailing list discussion in which the (likely) sock puppets started making demands. But it's hard to see the whole thing helping, and it is some combination of depressing and infuriating. I hope he's doing ok.
- juitpykyk 3y ago
  You're talking as if securing a backdoor with public cryptography is some unimaginable feat of technology. It's literally a couple hours work.
  - kbenson 3y ago
    I don't think they were using complexity as the reason for that assumption, but instead goals. Adding security doesn't require a nation state's level of resources, but it is a more attractive feature for a nation state that wants to preserve it over time and prevent adversaries from making use of it.
    - neodymiumphish 3y ago
      And on the contrary, creating a vulnerability that’s not identifiable to a limited attack group provides for a bit more deniability and anonymity. It’s hard to say which is more favorable by a nation-state actor.
      - usrusr 3y ago
        An interesting angle: if this was somehow observed in use instead of getting discovered without observing use, there would be a glaring bright cui bono associated with what it was being used for. Theoretically, I could even imagine that public key security motivated by some "mostly benign" actor who fell in love with the question "could I pull this off?" dreading the scenario of their future hacker superpower falling in the wrong hands. It's not entirely unthinkable that an alternative timeline where this was never discovered would only ever see this superbackdoor getting used for harmless pranks, winning bets and the like. In this unlikely scenario, the discovery, through the state actor and crime syndicate activities undoubtedly inspired by the discovery, would paradoxically lead to a less safe computing world than the alternative timeline where the person with the key had free RCE almost everywhere.
    - heavyset_go 3y ago
      This makes sense in closed source products where you'll never get to audit the source for such exploits, but little sense in open source projects where anyone can audit it. That's to say an enterprise router or switch would likely have secured exploits put there by corporate and national security agencies, whereas open source exploits would benefit from the probable deniability.
- heavyset_go 3y ago
  > Now it looks more like nation-state spycraft. I disagree, plenty of exploits implemented WireGuard-like crypto key access to them.
eli 3y ago
More to the point it prevents your enemies from using the exploit against friendly targets. The tradeoff is that, once you find it, it's very clearly a backdoor. No way you can pretend this was an innocent bug.
chpatrick 3y ago
Also people can come and immediately undo whatever you did if it's not authenticated.
Beijinger 3y ago
This is just a safety measure that it does not blow up in your own face (country).
- CommitSyn 3y ago
  Not only did it mandate authentication, but there was also a Killswitch, and there's no other reason for that than because you have reasons to suspect someone may have access to this sensitive world-stopper exploit and use it against you. That leaves simply a (reasonably, but still) paranoid person, and people worried about that+consequences are unlikely to have the mental well-being to pull this off, or, a group of people.
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]

Alifatisk 3y ago

For real, it's almost like a state-sponsored exploit. It's crafted and executed incredibly well, the performance issue feels like pure luck it got found.

jdewerd 3y ago
Yeah, we should probably expect that there are roughly 1/p(found) more of these lurking out there. Not a pleasant thought.
andersa 3y ago
It'd make total sense if it was. This way you get to have the backdoor without your enemies being able to use it against your own companies.
hnthrowaway0328 3y ago
Do we have a detailed technical analysis of the code? I read a few analysis but they all seem preliminary. It is very useful to learn from the code.
- coldpie 3y ago
  There's a few links down at the bottom of the OP to quite detailed analysis. From there you could join a Discord where discussion is ongoing.
  - hnthrowaway0328 3y ago
    Thanks coldpie.
stingraycharles 3y ago
I like the theory that actually, it wasn’t luck but was picked up on by detection tools of a large entity (Google / Microsoft / NSA / whatever), and they’re just presenting the story like this to keep their detection methods a secret. It’s what I would do.

linsomniac 3y ago

Am I reading it correctly that the payload signature includes the target SSH host key? So you can't just spray it around to servers, it's fairly computationally expensive to send it to a host.

miduil 3y ago
*host key fingerprint, but I assume what you've meant. It's practically a good backdoor then, crypto graphically protected and safe against "re-play" attacks.
- amluto 3y ago
  Not quite. It still looks vulnerable: an attacker A without the private key impersonates a victim server V and reports their host key. A careless attacker B with the key tries to attack A, but an ends up recovering a valid payload targeting V.
  - Denvercoder9 3y ago
    I'm not too familar with the SSH protocol, but is it possible to impersonate a victim server V without having the private key to their host key?
    - pxx 3y ago
      This stuff is pre-auth. You can just treat the entire thing as opaque and proxy everything to the host you're trying to compromise; as soon as you have an exploit string for a given host you can just replay it.
      - robryk 3y ago
        Ssh does client authentication after handshake. The server is required to sign the handshake result with its private key, so you won't get past handshake if you are a server that claims to have a public key that you don't know the private key for. E: see RFC 4253, sections 8 and 10, and RFC 4252 for corroboration

nialv7 3y ago

how are you going to sell it if anyone can get in?

ryanmerket 3y ago
this is probably the right answer. hacking group getting a 0day to sell to nation states on a per-use agreement.

cesarb 3y ago

This is called NOBUS: https://en.wikipedia.org/wiki/NOBUS https://en.wikipedia.org/wiki/NOBUS

halJordan 3y ago
This is not that concept. That concept is no one but us can technically complete the exploit. Technical feasibility in that you need a supercomputer to do it, not protecting a backdoor with the normal cia triad
- justinclift 3y ago
  That doesn't seem correct: If they determine the vulnerability is only exploitable by the NSA for reasons such as computational resources, budget, or skill set, they label it as NOBUS and will not move to patch it, but rather leave it open to exploit against current or future targets. If (!) the NSA regards ssh keys as secure, then from that article it sounds like the NOBUS thing would fit.
  - CommitSyn 3y ago
    That would only fit "If (!) the NSA regards ssh keys as secure >for everyone but them<
    - justinclift 3y ago
      Not sure why? My thinking in this hypothetical scenario is that the NSA would have the private key, which is why it would be a NOBUS thing. If they didn't have the key though, then yeah it doesn't fit. Unless they can walk through ssh key security anyway. :)
  - bed99 3y ago
    Is it legal and possible for the NSA to socially attack someone with mental health issues this way? I'd like to rule-out some state actors by the mere heartbreaking act of socially engineering and targeting a person in pain. Not even considering the original maintainer may be a US citizen (idk).

takeda 3y ago

If you think of it as a state sponsored attack it makes a lot of sense to have a "secure" vulnerability in system that your own citizens might use. It looks like the whole contribution to xz was an effort to just inject that backdoor. For example the author created the whole test framework where he could hide the malicious payload. Before he started work on xz, he made contribution to libarchive in BSD which created a vulnerability.

pxx 3y ago
The libarchive diff didn't create any vulnerability. The fprintf calls were consistent with others in the same repository.
- takeda 3y ago
  They still preferred to revert it as it looks very suspicious. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1068047 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1068047 https://github.com/libarchive/libarchive/pull/2101 https://github.com/libarchive/libarchive/pull/2101
  - asveikau 3y ago
    If I read that correctly the problem is that it prints a filename that might include terminal control sequences that come from an attacker-controlled file name. Comment in your second link: https://github.com/libarchive/libarchive/pull/1609#issuecomment-2028125584 https://github.com/libarchive/libarchive/pull/1609#issuecomm...
- jhugo 3y ago
  It did, actually: the filename can contain terminal control characters, which thanks to the change from safe_fprintf to fprintf, were printed without escaping, which allows the creator of the archive being extracted to control the terminal of the user extracting the archive.
  - pxx 3y ago

aaron695 3y ago

[dead]

userbinator 3y ago

IMHO it's not that surprising; asymmetric crypto has been common in ransomware for a long time, and of course ransomware in general is based on securing data from its owner. "It's not only the good guys who have guns."

Taniwha 3y ago

I wonder if anyone has packet logs from the past few weeks that show attempts at sshd - might be some incriminating IP addresses to start hunting with

whirlwin 3y ago

Imagine how much the private key is worth on the black market

bayindirh 3y ago

It's a (failed) case study in "what if we backdoor it in a way only good guys can use but bad guys can't?" Computers don't know who's / what's good or bad. They're deterministic machines which responds to commands. I don't know whether there'll be any clues about who did this, but this will be the poster child of "you can't have backdoors and security in a single system" argument (which I strongly support).