7 ms·

For those panicking, here are some key things to look for, based on the writeup: - A very recent version of liblzma5 - 5.6.0 or 5.6.1. This was added in the l

by bonyt 3y ago

For those panicking, here are some key things to look for, based on the writeup: - A very recent version of liblzma5 - 5.6.0 or 5.6.1. This was added in the last month or so. If you're not on a rolling release distro, your version is probably older. - A debian or RPM based distro of Linux on x86_64. In an apparent attempt to make reverse engineering harder, it does not seem to apply when built outside of deb or rpm packaging. It is also specific to Linux. - Running OpenSSH sshd from systemd. OpenSSH as patched by some distros only pulls in libsystemd for logging functionality, which pulls in the compromised liblzma5. Debian testing already has a version called '5.6.1+really5.4.5-1' that is really an older version 5.4, repackaged with a newer version to convince apt that it is in fact an upgrade. It is possible there are other flaws or backdoors in liblzma5, though.

treffer 3y ago
Ubuntu still ships 5.4.5 on 24.03 (atm). I did a quick diff of the source (.orig file from packages.ubuntu.com) and the content mostly matched the 5.4.5 github tag except for Changelog and some translation files. It does match the tarball content, though. So for 5.4.5 the tagged release and download on github differ. It does change format strings, e.g. +#: src/xz/args.c:735 +#, fuzzy +#| msgid "%s: With --format=raw, --suffix=.SUF is required unless writing to stdout" +msgid "With --format=raw, --suffix=.SUF is required unless writing to stdout" +msgstr "%s: amb --format=raw, --suffix=.SUF és necessari si no s'escriu a la sortida estàndard" There is no second argument to that printf for example. I think there is at least a format string injection in the older tarballs. [Edit] formatting
- fransje26 3y ago
  Thanks for the heads up.
- chasil 3y ago
  RHEL9 is shipping 5.2.5; RHEL8 is on 5.2.4.
- mort96 3y ago
  FYI, your formatting is broken. Hacker News doesn't support backtick code blocks, you have to indent code. Anyway, so... the xz project has been compromised for a long time, at least since 5.4.5. I see that this JiaT75 guy has been the primary guy in charge of at least the GitHub releases for years. Should we view all releases after he got involved as probably compromised?
  - treffer 3y ago
    Thank you, formatting fixed. My TLDR is that I would regard all commits by JiaT75 as potentially compromised. Given the ability to manipulate gitnhistory I am not sure if a simple time based revert is enough. It would be great to compare old copies of the repo with the current state. There is no guarantee that the history wasn't tampered with. Overall the only safe action would IMHO to establish a new upstream from an assumed good state, then fully audit it. At that point we should probably just abandon it and use zstd instead.
    - tomrod 3y ago
      Not just Jia. There are some other accounts of concern with associated activity or short term/bot-is names.
      - jdright 3y ago
        yes, like this one: https://github.com/facebook/folly/pull/2153 https://github.com/facebook/folly/pull/2153
        account42 3y ago
        Interesting, this would suggest exploits other than the known sshd one.
    - ogurechny 3y ago
      Zstd belongs to the class of speed-optimized compressors providing “tolerable” compression ratios. Their intended use case is wrapping some easily compressible data with negligible (in the grand scale) performance impact. So when you have a server which sends gigabits of text per second, or caches gigabytes of text, or processes a queue with millions of text protocol messages, you can add compression on one side and decompression on the other to shrink them without worrying too much about CPU usage. Xz is an implant of 7zip's LZMA(2) compression into traditional Unix archiver skeleton. It trades long compression times and giant dictionaries (that need lots of memory) for better (“much-better-than-deflate”) compression ratios. Therefore, zstd, no matter how fashionable that name might be in some circles, is not a replacement for xz. It should also be noted that those LZMA-based archive formats might not be considered state-of-the-art today. If you worry about data density, there are options for both faster compression at the same size, and better compression in the same amount of time (provided that data is generally compressible). 7zip and xz are widespread and well tested, though, and allow decompression to be fast, which might be important in some cases. Alternatives often decompress much slowly. This is also a trade-off between total time spent on X nodes compressing data, and Y nodes decompressing data. When X is 1, and Y is in the millions (say, software distribution), you can spend A LOT of time compressing even for relatively minuscule gains without affecting the scales. It should also be noted that many (or most) decoders of top compressing archivers are implemented as virtual machines executing chains of transform and unpack operations defined in archive file over pieces of data also saved there. Or, looking from a different angle, complex state machines initializing their state using complex data in the archive. Compressor tries to find most suitable combination of basic steps based on input data, and stores the result in the archive. (This is logically completed in neural network compression tools which learn what to do with data from data itself.) As some people may know, implementing all that byte juggling safely and effectively is a herculean task, and compression tools had exploits in the past because of that. Switching to a better solution might introduce a lot more potentially exploited bugs.
- jwilk 3y ago
  "#, fuzzy" means the translation is out-of-date and it will be discarded at compile time.
  - treffer 3y ago
    I tried to get the translation to trigger by switching to french and it does not show. You are right. So it's just odd that the tags and release tarballs diverge.
hostyle 3y ago
$ dpkg-query -W liblzma5 liblzma5:amd64 5.4.1-0.2
fransje26 3y ago
I did notice that my debian-based system got noticeably slower and unresponsive at times the last two weeks, without obvious reasons. Could it be related? I read through the report, but what wasn't directly clear to me was: what does the exploit actually do? My normal internet connection has such an appalling upload that I don't think anything relevant could be uploaded. But I will change my ssh keys asap.
- anarazel 3y ago
  > I did notice that my debian-based system got noticeably slower and unresponsive at times the last two weeks, without obvious reasons. Could it be related? Possible but unlikely. > I read through the report, but what wasn't directly clear to me was: what does the exploit actually do? It injects code that runs early during sshd connection establishment. Likely allowing remote code execution if you know the right magic to send to the server.
  - fransje26 3y ago
    Thank you for the explanation.
- cpach 3y ago
  Are you on stable/testing/unstable? With our current knowledge, stable shouldn’t be affected by this.
  - fransje26 3y ago
    Stable, luckily. Thank you for the information.
idoubtit 3y ago
The article gives a link to a simple shell script that detects the signature of the compromised function. > Running OpenSSH sshd from systemd I think this is irrelevant. From the article: "Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked." If I understand correctly the whole section, the behavior of OpenSSH may have differed when launched from systemd, but the backdoor was there in both cases. Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available.
- bonyt 3y ago
  I think the distributions that do use systemd are the ones that add the libsystemd code, which in turn brings in the liblzma5 code. So, it may not be entirely relevant how it is run, but it needs to be a version of OpenSSH patched.
- anarazel 3y ago
  > From the article: "Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked." If I understand correctly the whole section, the behavior of OpenSSH may have differed when launched from systemd, but the backdoor was there in both cases. It looks like the backdoor "deactivates" itself when it detects being started interactively, as a security researcher might. I was eventually able to circumvent that, but unless you do so, it'll not be active when started interactively. However, the backdoor would also be active if you started it with an shell script (as the traditional sys-v rc scripts did) outside the context of an interactive shell, as TERM wouldn't be set either in that context. > Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available. There's no xz code in openssh.
- nwallin 3y ago
  > Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available. OpenSSH is developed by the OpenBSD project, and systemd is not compatible with OpenBSD. The upstream project has no systemd or liblzma code to strip. If your sshd binary links to liblzma, it's because the package maintainers for your distro have gone out of their way to add systemd's patch to your sshd binary. > From the article: "Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked." If I understand correctly the whole section, the behavior of OpenSSH may have differed when launched from systemd, but the backdoor was there in both cases. From what I understand, the backdoor detects if it's in any of a handful of different debug environments. If it's in a debug environment or not launched by systemd, it won't hook itself up. ("nothing to see here folks...") But if sshd isn't linked to liblzma to begin with, none of the backdoor's code even exists in the processes' page maps. I'm still downgrading to an unaffected version, of course, but it's nice to know I was never vulnerable just by typing 'ldd `which sshd`' and not seeing liblzma.so.
stbenjam 3y ago
> Debian testing already has a version called '5.6.1+really5.4.5-1' that is really an older version 5.4, repackaged with a newer version to convince apt that it is in fact an upgrade. I'm surprised .deb doesn't have a better approach. RPM has epoch for this purpose http://novosial.org/rpm/epoch/index.html http://novosial.org/rpm/epoch/index.html
- pja 3y ago
  Debian packages can have epochs too. I’m not sure why the maintainers haven’t just bumped the epoch here. Maybe they’re expecting a 5.6.x release shortly that fixes all these issues & don’t want to add an epoch for a very short term packaging issue?
- nicolas_17 3y ago
  .deb has epochs too, but I think Debian developers avoid it where possible because 1:5.4.5 is interpreted as newer than anything without a colon, so it would break eg. packages that depend on liblzma >= 5.0, < 6. There may be more common cases that aren't coming to mind now.
  - account42 3y ago
    Seems like debian is mixing too many things into the package version - version used for deciding on upgrades and abi version for dependencies should be decoupled like it is in modern RPM distros.
    - nicolas_17 3y ago
      If a binary library ABI is backwards-incompatible, they change the package name. I was just guessing at the reason epoch is avoided and that <6 is probably an awful example. So now I actually bothered to look it up, and it turns out the actual reason is that the epoch changes what version is considered "greater", but it's not part of the .deb filename, so you still can't reuse version numbers used in the past. If you release 5.0, then 5.1, then you want to rollback and release 1:5.0, it's going to break things in the Debian archives. https://www.debian.org/doc/debian-policy/ch-binary.html#uniqueness-of-version-numbers https://www.debian.org/doc/debian-policy/ch-binary.html#uniq... Additionally, once you add an epoch you're stuck with it forever, while if you use 5.1+really5.0, you can get rid of the kludge when 5.2 is out. https://www.debian.org/doc/debian-policy/ch-controlfields.html#epochs-should-be-used-sparingly https://www.debian.org/doc/debian-policy/ch-controlfields.ht...
deleted 3y ago
[deleted]
NotPractical 3y ago
> If you're not on a rolling release distro, your version is probably older. Ironic considering security is often advertised as a feature of rolling release distros. I suppose in most instances it does provide better security, but there are some advantages to Debian's approach (stable Debian, that is).
- javajosh 3y ago
  >Ironic considering security is often advertised as a feature of rolling release distros. Security is a feature of rolling release. But supply-chain attacks like this are the exception to the rule.
  - yreg 3y ago
    Isn't that what security-updates-only is for? This particular backdoor is not shipped inside of a security patch, right?
- leeoniya 3y ago
  i mean, rolling implies rolling 0-days, too.
pdw 3y ago
Focusing on sshd is the wrong approach. The backdoor was in liblzma5. It was discovered to attack sshd, but it very likely had other targets as well. The payload hasn't been analyzed yet, but _almost everything_ links to libzma5. Firefox and Chromium do. Keepassxc does. And it might have made arbitrary changes to your system, so installing the security update might not remove the backdoor.
- junon 3y ago
  From what I'm understanding it's trying to patch itself into the symbol resolution step of ld.so specifically for libcrypto under systemd on x86_64. Am I misreading the report? That's a strong indication it's targeting sshd specifically.
  - pdw 3y ago
    Lots of software links both liblzma and libcrypto. As I read Andres Freund's report, there is still a lot of uncertainty: "There's lots of stuff I have not analyzed and most of what I observed is purely from observation rather than exhaustively analyzing the backdoor code." "There are other checks I have not fully traced."
  - lostmsu 3y ago
    As mentioned many times in other places now, this account had control over xz code for 2 years. The discovered CVE might be just a tip of an iceberg.
- saagarjha 3y ago
  It checks for argv[0] == "sshd"
rathermucker 3y ago
Tumbleweed has a package: liblzma5-5.6.1.revertto5.4-3.2.x86_64 FYI
- account42 3y ago
  revertto probably just means "revert to" but it does sound quite italian lol.
deleted 3y ago
[deleted]

by bonyt 3y ago

treffer 3y ago

Ubuntu still ships 5.4.5 on 24.03 (atm). I did a quick diff of the source (.orig file from packages.ubuntu.com) and the content mostly matched the 5.4.5 github tag except for Changelog and some translation files. It does match the tarball content, though. So for 5.4.5 the tagged release and download on github differ. It does change format strings, e.g. +#: src/xz/args.c:735 +#, fuzzy +#| msgid "%s: With --format=raw, --suffix=.SUF is required unless writing to stdout" +msgid "With --format=raw, --suffix=.SUF is required unless writing to stdout" +msgstr "%s: amb --format=raw, --suffix=.SUF és necessari si no s'escriu a la sortida estàndard" There is no second argument to that printf for example. I think there is at least a format string injection in the older tarballs. [Edit] formatting

fransje26 3y ago
Thanks for the heads up.
chasil 3y ago
RHEL9 is shipping 5.2.5; RHEL8 is on 5.2.4.
mort96 3y ago
FYI, your formatting is broken. Hacker News doesn't support backtick code blocks, you have to indent code. Anyway, so... the xz project has been compromised for a long time, at least since 5.4.5. I see that this JiaT75 guy has been the primary guy in charge of at least the GitHub releases for years. Should we view all releases after he got involved as probably compromised?
- treffer 3y ago
  Thank you, formatting fixed. My TLDR is that I would regard all commits by JiaT75 as potentially compromised. Given the ability to manipulate gitnhistory I am not sure if a simple time based revert is enough. It would be great to compare old copies of the repo with the current state. There is no guarantee that the history wasn't tampered with. Overall the only safe action would IMHO to establish a new upstream from an assumed good state, then fully audit it. At that point we should probably just abandon it and use zstd instead.
  - tomrod 3y ago
    Not just Jia. There are some other accounts of concern with associated activity or short term/bot-is names.
    - jdright 3y ago
      yes, like this one: https://github.com/facebook/folly/pull/2153 https://github.com/facebook/folly/pull/2153
      - account42 3y ago
        Interesting, this would suggest exploits other than the known sshd one.
  - ogurechny 3y ago
    Zstd belongs to the class of speed-optimized compressors providing “tolerable” compression ratios. Their intended use case is wrapping some easily compressible data with negligible (in the grand scale) performance impact. So when you have a server which sends gigabits of text per second, or caches gigabytes of text, or processes a queue with millions of text protocol messages, you can add compression on one side and decompression on the other to shrink them without worrying too much about CPU usage. Xz is an implant of 7zip's LZMA(2) compression into traditional Unix archiver skeleton. It trades long compression times and giant dictionaries (that need lots of memory) for better (“much-better-than-deflate”) compression ratios. Therefore, zstd, no matter how fashionable that name might be in some circles, is not a replacement for xz. It should also be noted that those LZMA-based archive formats might not be considered state-of-the-art today. If you worry about data density, there are options for both faster compression at the same size, and better compression in the same amount of time (provided that data is generally compressible). 7zip and xz are widespread and well tested, though, and allow decompression to be fast, which might be important in some cases. Alternatives often decompress much slowly. This is also a trade-off between total time spent on X nodes compressing data, and Y nodes decompressing data. When X is 1, and Y is in the millions (say, software distribution), you can spend A LOT of time compressing even for relatively minuscule gains without affecting the scales. It should also be noted that many (or most) decoders of top compressing archivers are implemented as virtual machines executing chains of transform and unpack operations defined in archive file over pieces of data also saved there. Or, looking from a different angle, complex state machines initializing their state using complex data in the archive. Compressor tries to find most suitable combination of basic steps based on input data, and stores the result in the archive. (This is logically completed in neural network compression tools which learn what to do with data from data itself.) As some people may know, implementing all that byte juggling safely and effectively is a herculean task, and compression tools had exploits in the past because of that. Switching to a better solution might introduce a lot more potentially exploited bugs.
jwilk 3y ago
"#, fuzzy" means the translation is out-of-date and it will be discarded at compile time.
- treffer 3y ago
  I tried to get the translation to trigger by switching to french and it does not show. You are right. So it's just odd that the tags and release tarballs diverge.

hostyle 3y ago

$ dpkg-query -W liblzma5 liblzma5:amd64 5.4.1-0.2

fransje26 3y ago

I did notice that my debian-based system got noticeably slower and unresponsive at times the last two weeks, without obvious reasons. Could it be related? I read through the report, but what wasn't directly clear to me was: what does the exploit actually do? My normal internet connection has such an appalling upload that I don't think anything relevant could be uploaded. But I will change my ssh keys asap.

anarazel 3y ago
> I did notice that my debian-based system got noticeably slower and unresponsive at times the last two weeks, without obvious reasons. Could it be related? Possible but unlikely. > I read through the report, but what wasn't directly clear to me was: what does the exploit actually do? It injects code that runs early during sshd connection establishment. Likely allowing remote code execution if you know the right magic to send to the server.
- fransje26 3y ago
  Thank you for the explanation.
cpach 3y ago
Are you on stable/testing/unstable? With our current knowledge, stable shouldn’t be affected by this.
- fransje26 3y ago
  Stable, luckily. Thank you for the information.

idoubtit 3y ago

The article gives a link to a simple shell script that detects the signature of the compromised function. > Running OpenSSH sshd from systemd I think this is irrelevant. From the article: "Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked." If I understand correctly the whole section, the behavior of OpenSSH may have differed when launched from systemd, but the backdoor was there in both cases. Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available.

bonyt 3y ago
I think the distributions that do use systemd are the ones that add the libsystemd code, which in turn brings in the liblzma5 code. So, it may not be entirely relevant how it is run, but it needs to be a version of OpenSSH patched.
anarazel 3y ago
> From the article: "Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked." If I understand correctly the whole section, the behavior of OpenSSH may have differed when launched from systemd, but the backdoor was there in both cases. It looks like the backdoor "deactivates" itself when it detects being started interactively, as a security researcher might. I was eventually able to circumvent that, but unless you do so, it'll not be active when started interactively. However, the backdoor would also be active if you started it with an shell script (as the traditional sys-v rc scripts did) outside the context of an interactive shell, as TERM wouldn't be set either in that context. > Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available. There's no xz code in openssh.
nwallin 3y ago
> Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available. OpenSSH is developed by the OpenBSD project, and systemd is not compatible with OpenBSD. The upstream project has no systemd or liblzma code to strip. If your sshd binary links to liblzma, it's because the package maintainers for your distro have gone out of their way to add systemd's patch to your sshd binary. > From the article: "Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked." If I understand correctly the whole section, the behavior of OpenSSH may have differed when launched from systemd, but the backdoor was there in both cases. From what I understand, the backdoor detects if it's in any of a handful of different debug environments. If it's in a debug environment or not launched by systemd, it won't hook itself up. ("nothing to see here folks...") But if sshd isn't linked to liblzma to begin with, none of the backdoor's code even exists in the processes' page maps. I'm still downgrading to an unaffected version, of course, but it's nice to know I was never vulnerable just by typing 'ldd `which sshd`' and not seeing liblzma.so.

stbenjam 3y ago

> Debian testing already has a version called '5.6.1+really5.4.5-1' that is really an older version 5.4, repackaged with a newer version to convince apt that it is in fact an upgrade. I'm surprised .deb doesn't have a better approach. RPM has epoch for this purpose http://novosial.org/rpm/epoch/index.html http://novosial.org/rpm/epoch/index.html

pja 3y ago
Debian packages can have epochs too. I’m not sure why the maintainers haven’t just bumped the epoch here. Maybe they’re expecting a 5.6.x release shortly that fixes all these issues & don’t want to add an epoch for a very short term packaging issue?
nicolas_17 3y ago
.deb has epochs too, but I think Debian developers avoid it where possible because 1:5.4.5 is interpreted as newer than anything without a colon, so it would break eg. packages that depend on liblzma >= 5.0, < 6. There may be more common cases that aren't coming to mind now.
- account42 3y ago
  Seems like debian is mixing too many things into the package version - version used for deciding on upgrades and abi version for dependencies should be decoupled like it is in modern RPM distros.
  - nicolas_17 3y ago
    If a binary library ABI is backwards-incompatible, they change the package name. I was just guessing at the reason epoch is avoided and that <6 is probably an awful example. So now I actually bothered to look it up, and it turns out the actual reason is that the epoch changes what version is considered "greater", but it's not part of the .deb filename, so you still can't reuse version numbers used in the past. If you release 5.0, then 5.1, then you want to rollback and release 1:5.0, it's going to break things in the Debian archives. https://www.debian.org/doc/debian-policy/ch-binary.html#uniqueness-of-version-numbers https://www.debian.org/doc/debian-policy/ch-binary.html#uniq... Additionally, once you add an epoch you're stuck with it forever, while if you use 5.1+really5.0, you can get rid of the kludge when 5.2 is out. https://www.debian.org/doc/debian-policy/ch-controlfields.html#epochs-should-be-used-sparingly https://www.debian.org/doc/debian-policy/ch-controlfields.ht...

deleted 3y ago

[deleted]

NotPractical 3y ago

> If you're not on a rolling release distro, your version is probably older. Ironic considering security is often advertised as a feature of rolling release distros. I suppose in most instances it does provide better security, but there are some advantages to Debian's approach (stable Debian, that is).

javajosh 3y ago
>Ironic considering security is often advertised as a feature of rolling release distros. Security is a feature of rolling release. But supply-chain attacks like this are the exception to the rule.
- yreg 3y ago
  Isn't that what security-updates-only is for? This particular backdoor is not shipped inside of a security patch, right?
leeoniya 3y ago
i mean, rolling implies rolling 0-days, too.

pdw 3y ago

Focusing on sshd is the wrong approach. The backdoor was in liblzma5. It was discovered to attack sshd, but it very likely had other targets as well. The payload hasn't been analyzed yet, but _almost everything_ links to libzma5. Firefox and Chromium do. Keepassxc does. And it might have made arbitrary changes to your system, so installing the security update might not remove the backdoor.

junon 3y ago
From what I'm understanding it's trying to patch itself into the symbol resolution step of ld.so specifically for libcrypto under systemd on x86_64. Am I misreading the report? That's a strong indication it's targeting sshd specifically.
- pdw 3y ago
  Lots of software links both liblzma and libcrypto. As I read Andres Freund's report, there is still a lot of uncertainty: "There's lots of stuff I have not analyzed and most of what I observed is purely from observation rather than exhaustively analyzing the backdoor code." "There are other checks I have not fully traced."
- lostmsu 3y ago
  As mentioned many times in other places now, this account had control over xz code for 2 years. The discovered CVE might be just a tip of an iceberg.
saagarjha 3y ago
It checks for argv[0] == "sshd"

rathermucker 3y ago

Tumbleweed has a package: liblzma5-5.6.1.revertto5.4-3.2.x86_64 FYI

account42 3y ago
revertto probably just means "revert to" but it does sound quite italian lol.

deleted 3y ago

[deleted]