8 ms·

Everything I Know About the XZ Backdoor

195 points by internetter 3y ago

tomrod 3y ago
Wow. Timeline-wise this appears to be a clearly coordinated attempt, unless I'm missing something.
- goalieca 3y ago
  The timeline suggests this is a patient hacker. The most patient of which are nation-states because they're not worried about cashflow and think in terms of years.
  - imiric 3y ago
    Makes one wonder how many instances of this goes unnoticed. Maybe this attacker got sloppy, but how many of them don't? We were lucky that one person noticed suspicious behavior and had the technical skills and patience to investigate, but how many of us don't?
    - fmajid 3y ago
      Many: https://www.theregister.com/2022/02/23/chinese_nsa_linux/ https://www.theregister.com/2022/02/23/chinese_nsa_linux/
    - goalieca 3y ago
      Undoubtedly they do. This is an example of why defence in depth is important. Things fail all the time for intentional and non-intentional reasons. A single point of failure can’t be allowed.
    - Kluggy 3y ago
      It wasn't sloppy. It was just luck that someone noticed a half a second extra latency on the second connection of a newly run sshd process and went down the rabbit hole. Had they just shrugged and moved onto more "important" tasks/deliverables, it would most likely have landed in production across the world. I'm a tad reminded of https://xkcd.com/705/ https://xkcd.com/705/ We got so lucky here. We won't get lucky every time. We will have a massive breach one of these days.
      - 31337Logic 3y ago
        Kinda like Clifford Stoll!
      - sunk1st 3y ago
        I don’t think it was luck. I think some people are so in tune with their systems that investigating an anomaly like this is a frequent occurrence. This particular anomaly just happened to have an explosive ending.
        jeltz 3y ago
        Yes, I have met Andres in real life and I can totally believe that he is that in tune with his system. He wrote that he found this while benchmark PostgreSQL and saw weird load from ssh. He does a lot of benchmarking of PostgreSQL patches. But I would say it was also luck. If Andres hadn't been benchmarking on Debian Testing (or whatever system he found this on) this might have taken longer time to discover.
      - account42 3y ago
        It may not sound sloppy if you are used to todays apps and websites but half a second is an eternity in CPU time. Half a second is also very much a significant amount of time compared to normal ssh connection times with low network latency - if not Freund then someone else would have noticed, complained and this would have eventually been investigated. The only luck part here is it taking less than two months for this to happen but the attacker could have prevented this avenue for detection entirely by optimizing the exploit not to slow down the ssh proces.
    - bananapub 3y ago
      > Maybe this attacker got sloppy this attacker did not get sloppy, they got control of a critical piece of free software, and got a still-not-fully-understood piece of malware in to, past whatever peer review we like to imagine we have, then got it uploaded into at least two critical linux distributions, past whatever peer review we like to imagine we have, and was only found, two years in to the operation, by pure luck and a very dedicated engineer.
      - asveikau 3y ago
        It was actually found less than two months after it was introduced in 5.6.0. Now, the attacker maintained the project for 2 years before that, and we don't know what else they inserted, but the known vulnerability was only in a release since February.
      - account42 3y ago
        They did get sloppy though - the injected code slows down sshd startup enough that someone noticed and investigated. This could have been avoided.
adeon 3y ago
This mailing thread is quoted in the 2022 part on the page on the xz maintainer being pressured to add new maintainers for xz: https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.html https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.h... Seems like the usual story of some lone maintainer maintaining a popular project and losing their time and energy. The post doesn't spell it out but I wonder if they implied there's a suspicion that the person doing the pressuring there is also another sock puppet of JiaT75 as some scheme of getting access to xz. That would seem particularly cruel; take advantage of a tired maintainer with mental health issues to use their project to smuggle security exploits to the world. Regardless, be nice to people who are doing "unpaid hobby projects" your work depends on. Reading the thread made me sad.
- Hakkin 3y ago
  > The post doesn't spell it out but I wonder if they implied there's a suspicion that the person doing the pressuring there is also another sock puppet of JiaT75 as some scheme of getting access to xz You can get the user's e-mail by clicking the "Reply via e-mail to" on the page. It matches the <firstname><lastname><number>@protonmail.com of the other sockpuppets, and the PGP key for the account was made 1 day prior to their first communication on that mailing list.
  - radicaldreamer 3y ago
    This is an intelligence agency trying to add backdoors to archiving software... likely targeting other utilities as well with same MO
    - Nextgrid 3y ago
      Doubt the target is archiving software itself - presumably the reason these libraries got picked is because they already have high penetration across many layers of the stack which would ensure the backdoor has wide coverage.
    - Tuna-Fish 3y ago
      The backdoor targets OpenSSH. The reason it's added to xz is that because of a complex dependency chain, it ends up being compiled to build OpenSSH. As far as I can tell, the payload doesn't get deployed into anything else.
deleted 3y ago
[deleted]
LegionMammal978 3y ago
Regarding the libarchive change: > This commit does not do what it says it does. All it does, in fact, is replace safe_fprint with an unsafe variant, potentially introducing another vulnerability. The code was merged without any discussion, and lives on to this day. libarchive should also be considered compromised until proven otherwise. This is very far from the truth, from what I can tell. If you look at the PR, it does, in fact, add strerror(errno) to the end of the error line, as it claims in its description. And the switch from safe_fprintf() to regular fprintf() for the archive_error_string matches the rest of the codebase, which is scattered with calls to lafe_errc() and lafe_warnc() with archive_error_string (forwarding to vfprintf()). If the contents of that string were considered sensitive to print out, then it would hardly be a new vulnerability. Especially compare the code in tar/write.c, which calls fprintf(stderr) to output the archive_error_string in precisely the same way as the PR in question does. Note that safe_fprintf() has nothing at all to do with memory safety: its only purpose is to escape unprintable characters before printing them. Indeed, a stack overflow bug within the safe_fprintf() implementation was the subject of a CVE in 2022 [0]. [0] https://github.com/advisories/GHSA-c42q-jv3x-rq38 https://github.com/advisories/GHSA-c42q-jv3x-rq38
- internetter 3y ago
  Thanks, in my haste I misread. I removed the part claiming it didn't do what it says, though I maintain the changes made simultaneously is very suspect as there is no apparent motivation
  - Bulat_Ziganshin 3y ago
    the motivation is probably "get foot in the door". the attacker also made a few documentation-only PRs in various repos, but having code PR will make him more creditable, and also would help to add more backdoors to libarchive in the future and librachive is now part of Windows 10, being used as a universal archive decompression library in the Explorer
dang 3y ago
Main ongoing thread: Backdoor in upstream xz/liblzma leading to SSH server compromise - https://news.ycombinator.com/item?id=39865810 https://news.ycombinator.com/item?id=39865810 - March 2024 (744 comments)
hnthrowaway0328 3y ago
This might be the tip of the iceberg, especially if this is part of the work of a nation-state agency. Very persistent and clever though.
hnthrowaway0328 3y ago
Some other thoughts: how many popular open source projects only have 1-2 maintainers? I guess there is a way to do a search. I assume 5+ is safer, if they are of different humans.
cyberturtle2 3y ago
[dead]

195 points by internetter 3y ago

tomrod 3y ago

Wow. Timeline-wise this appears to be a clearly coordinated attempt, unless I'm missing something.

goalieca 3y ago
The timeline suggests this is a patient hacker. The most patient of which are nation-states because they're not worried about cashflow and think in terms of years.
- imiric 3y ago
  Makes one wonder how many instances of this goes unnoticed. Maybe this attacker got sloppy, but how many of them don't? We were lucky that one person noticed suspicious behavior and had the technical skills and patience to investigate, but how many of us don't?
  - fmajid 3y ago
    Many: https://www.theregister.com/2022/02/23/chinese_nsa_linux/ https://www.theregister.com/2022/02/23/chinese_nsa_linux/
  - goalieca 3y ago
    Undoubtedly they do. This is an example of why defence in depth is important. Things fail all the time for intentional and non-intentional reasons. A single point of failure can’t be allowed.
  - Kluggy 3y ago
    It wasn't sloppy. It was just luck that someone noticed a half a second extra latency on the second connection of a newly run sshd process and went down the rabbit hole. Had they just shrugged and moved onto more "important" tasks/deliverables, it would most likely have landed in production across the world. I'm a tad reminded of https://xkcd.com/705/ https://xkcd.com/705/ We got so lucky here. We won't get lucky every time. We will have a massive breach one of these days.
    - 31337Logic 3y ago
      Kinda like Clifford Stoll!
    - sunk1st 3y ago
      I don’t think it was luck. I think some people are so in tune with their systems that investigating an anomaly like this is a frequent occurrence. This particular anomaly just happened to have an explosive ending.
      - jeltz 3y ago
        Yes, I have met Andres in real life and I can totally believe that he is that in tune with his system. He wrote that he found this while benchmark PostgreSQL and saw weird load from ssh. He does a lot of benchmarking of PostgreSQL patches. But I would say it was also luck. If Andres hadn't been benchmarking on Debian Testing (or whatever system he found this on) this might have taken longer time to discover.
    - account42 3y ago
      It may not sound sloppy if you are used to todays apps and websites but half a second is an eternity in CPU time. Half a second is also very much a significant amount of time compared to normal ssh connection times with low network latency - if not Freund then someone else would have noticed, complained and this would have eventually been investigated. The only luck part here is it taking less than two months for this to happen but the attacker could have prevented this avenue for detection entirely by optimizing the exploit not to slow down the ssh proces.
  - bananapub 3y ago
    > Maybe this attacker got sloppy this attacker did not get sloppy, they got control of a critical piece of free software, and got a still-not-fully-understood piece of malware in to, past whatever peer review we like to imagine we have, then got it uploaded into at least two critical linux distributions, past whatever peer review we like to imagine we have, and was only found, two years in to the operation, by pure luck and a very dedicated engineer.
    - asveikau 3y ago
      It was actually found less than two months after it was introduced in 5.6.0. Now, the attacker maintained the project for 2 years before that, and we don't know what else they inserted, but the known vulnerability was only in a release since February.
    - account42 3y ago
      They did get sloppy though - the injected code slows down sshd startup enough that someone noticed and investigated. This could have been avoided.

adeon 3y ago

This mailing thread is quoted in the 2022 part on the page on the xz maintainer being pressured to add new maintainers for xz: https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.html https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.h... Seems like the usual story of some lone maintainer maintaining a popular project and losing their time and energy. The post doesn't spell it out but I wonder if they implied there's a suspicion that the person doing the pressuring there is also another sock puppet of JiaT75 as some scheme of getting access to xz. That would seem particularly cruel; take advantage of a tired maintainer with mental health issues to use their project to smuggle security exploits to the world. Regardless, be nice to people who are doing "unpaid hobby projects" your work depends on. Reading the thread made me sad.

Hakkin 3y ago
> The post doesn't spell it out but I wonder if they implied there's a suspicion that the person doing the pressuring there is also another sock puppet of JiaT75 as some scheme of getting access to xz You can get the user's e-mail by clicking the "Reply via e-mail to" on the page. It matches the <firstname><lastname><number>@protonmail.com of the other sockpuppets, and the PGP key for the account was made 1 day prior to their first communication on that mailing list.
- radicaldreamer 3y ago
  This is an intelligence agency trying to add backdoors to archiving software... likely targeting other utilities as well with same MO
  - Nextgrid 3y ago
    Doubt the target is archiving software itself - presumably the reason these libraries got picked is because they already have high penetration across many layers of the stack which would ensure the backdoor has wide coverage.
  - Tuna-Fish 3y ago
    The backdoor targets OpenSSH. The reason it's added to xz is that because of a complex dependency chain, it ends up being compiled to build OpenSSH. As far as I can tell, the payload doesn't get deployed into anything else.

deleted 3y ago

[deleted]

LegionMammal978 3y ago

Regarding the libarchive change: > This commit does not do what it says it does. All it does, in fact, is replace safe_fprint with an unsafe variant, potentially introducing another vulnerability. The code was merged without any discussion, and lives on to this day. libarchive should also be considered compromised until proven otherwise. This is very far from the truth, from what I can tell. If you look at the PR, it does, in fact, add strerror(errno) to the end of the error line, as it claims in its description. And the switch from safe_fprintf() to regular fprintf() for the archive_error_string matches the rest of the codebase, which is scattered with calls to lafe_errc() and lafe_warnc() with archive_error_string (forwarding to vfprintf()). If the contents of that string were considered sensitive to print out, then it would hardly be a new vulnerability. Especially compare the code in tar/write.c, which calls fprintf(stderr) to output the archive_error_string in precisely the same way as the PR in question does. Note that safe_fprintf() has nothing at all to do with memory safety: its only purpose is to escape unprintable characters before printing them. Indeed, a stack overflow bug within the safe_fprintf() implementation was the subject of a CVE in 2022 [0]. [0] https://github.com/advisories/GHSA-c42q-jv3x-rq38 https://github.com/advisories/GHSA-c42q-jv3x-rq38

internetter 3y ago
Thanks, in my haste I misread. I removed the part claiming it didn't do what it says, though I maintain the changes made simultaneously is very suspect as there is no apparent motivation
- Bulat_Ziganshin 3y ago
  the motivation is probably "get foot in the door". the attacker also made a few documentation-only PRs in various repos, but having code PR will make him more creditable, and also would help to add more backdoors to libarchive in the future and librachive is now part of Windows 10, being used as a universal archive decompression library in the Explorer

dang 3y ago

Main ongoing thread: Backdoor in upstream xz/liblzma leading to SSH server compromise - https://news.ycombinator.com/item?id=39865810 https://news.ycombinator.com/item?id=39865810 - March 2024 (744 comments)

hnthrowaway0328 3y ago

This might be the tip of the iceberg, especially if this is part of the work of a nation-state agency. Very persistent and clever though.

hnthrowaway0328 3y ago

Some other thoughts: how many popular open source projects only have 1-2 maintainers? I guess there is a way to do a search. I assume 5+ is safer, if they are of different humans.

cyberturtle2 3y ago

[dead]