5 ms·

Excellent succinct breakdown of the xz mess, from an OpenBSD developer

107 points by signa11 3y ago

speps 3y ago
This one was much clearer to me: https://research.swtch.com/xz-script https://research.swtch.com/xz-script Discussion: https://news.ycombinator.com/item?id=39903685 https://news.ycombinator.com/item?id=39903685
PreInternet01 3y ago
> much of the emerging narrative about this backdoor that you can read all over the net is based on idle speculation ...which sums up the situation quite nicely. This was a pretty bad attack on a certain ecosystem. The ecosystem will recover, or not, regardless of your feelings. Unless you're in a position to truly make a difference, just sit back and enjoy the ride...
- fmajid 3y ago
  There is some protection to be had in using non-mainstream platforms (in my case, Alpine Linux, OpenBSD and Illumos, though I do have some Ubuntu or Pop-OS! laptops).
deleted 3y ago
[deleted]
waynesonfire 3y ago
> Liblzma ends up dynamically linked to sshd because of a systemd-related extension added by many Linux packagers that pulls in liblzma as an unrelated dependency kinda wish this was unpacked a bit more, why exactly is a service executable dynamically linking to a library without using any of its symbols or functions, because of systemd. and a follow up if some openbsd folks can comment. over the years i've read about various unique security capabilities in openbsd, it seems natural to ask, what kernel or OS capabilities does openbsd provide to thwart the stage 2 efforts for this class of injection techniques?
- MrStonedOne 3y ago
  [dead]
- zokier 3y ago
  Libsystemd is a grabbag utility library, and completely optional: > The libsystemd library provides functions that allow interacting with various interfaces provided by the systemd(1) service manager, as well as various other functions and constants useful for implementing services in general. https://www.freedesktop.org/software/systemd/man/latest/libsystemd.html https://www.freedesktop.org/software/systemd/man/latest/libs... E.g. the service readiness protocol is pretty trivial to implement yourself if you don't want to pull libsystemd as dep.
- logicprog 3y ago
  > kinda wish this was unpacked a bit more, why exactly is a service executable dynamically linking to a library without using any of its symbols or functions, because of systemd. If I recall correctly based on what I've read about this — I believe from the original mailing list post that noticed the vulnerability — it's because under certain circumstances in order to enable certain functionality you might want sshd to be able to talk to systemd, so distros often patch sshd with code to do that. But obviously you need a library to implement actually speaking system's protocol, and as it happens, the easiest way to do that is to include the entirety of libsystemd, since it has functions for doing that, even though there are at least two other libraries that implement just the communication functionality and are actually designed for non-systemd programs to use. The problem with that idea being that libsystemd, being a whole standard library for all systemd-related functionality that is probably mostly designed for use by programs in the tightly integrated systemd family and as a reference imenentation, also includes a lot of other code, including code that has to deal with compression that depends on liblzma, even though all of that is never used by sshd, because it only uses the small subsection of the library it needs.
  - o11c 3y ago
    > even though there are at least two other libraries that implement just the communication functionality and are actually designed for non-systemd programs to use. The important question is: if one of those libraries is used, and then something else pulls in `libsystemd`, will they conflict?
deskamess 3y ago
There really is an exception to every rule. "Do not attribute to malice that which can be adequately explained by neglect, ignorance or incompetence" and then you come across: > The stage 0 shell snippet looks at first glance like a plausible part of > the poorly readable autoconf/automake tooling.
- throwanem 3y ago
  None of those adequately explain these commits. This is no exception to Hanlon's razor.
  - deleted 3y ago
    [deleted]
  - francescovv 3y ago
    Hanlon's razor is useful to curb one's paranoia, but it is far from being a universal rule. In fact, malice and incompetence are not necessarily mutually exclusive. This very incident shows several instances where "Jia Tan" is being arguably incompetent, in addition to being clearly malicious: unintended breakage by adding extra space between "return" and "is_arch_extension_supported"; several redundant checks for `uname` == "Linux"; botched payload, so "test files" had to be replaced, with pretty fishy explanation; rather inefficient/slow GOT parsing, list goes on...
  - narrator 3y ago
    Hanlon's razor lets every single first degree murderer off the hook, or any other of the many crimes where you have to establish intent to convict.
    - throwanem 3y ago
      It is a rule-of-thumb heuristic, not a law. Even if this weren't the case, your claim would remain specious. "Establish intent" and "justify attribution to malice" are the same thing said two ways.
- jethro_tell 3y ago
eigenlicht 3y ago
Yes, the concluding paragraph is the most succinct (and fitting) here, there are better explanations as for the technical aspects. I know it's not related, but for the Germans around, this is just great: https://inka.de/angebot/ https://inka.de/angebot/ A community ISP. Last changed 02.05.2001, in case you missed it. ;) Millennium blues. I wasn't aware about the possibilities back then, but it's just before my time.
brenoguim 3y ago
> It does not plausibly pass for a typo because no typical editing glitch will leave a '.' character there. I would certainly attribute that to a typo if I was reviewing the code.
- AwaAwa 3y ago
  > The syntax error is a single period '.' as the first character on an otherwise empty line of C code. I believe the point is that given this context, it could/should not be construed as a typo.
  - dpkirchner 3y ago
    If you received this patch from someone you trusted, and noticed the period, would you call it a typo or would you assume it is malicious? Or is this just an argument over the use of the word "typo" instead of "mistake"? That wouldn't be very useful, IMO.
  - dvh 3y ago
    Very common if you copy paste (using mouse buffer) from midnight commander text editor or any other editor that highlights trailing spaces as dot.
  - lenerdenator 3y ago
    I take it they're not forcing contributors to lint code, then?
    - klyrs 3y ago
      Does your linter look for syntax errors in the content of strings? If so, how would you design a unit test meant to capture syntax errors?
    - tedunangst 3y ago
      Have you tried running lint on a configure.ac or CMakeLists.txt file?
  - 3y ago
ryanjshaw 3y ago
> the build-to-host.m4 macro package that ships as part of GNU gettext was replaced by a modified version that was copied into the release tarball Am I correct in concluding that this backdoor might have been picked up sooner if the practice of shipping tarballs of source releases was dropped in favour of running builds straight off a repo tag? I've setup and worked with many build pipelines in the .NET world over nearly 20 years and we manage fine without the concept of a source release tarball, but I realise C dev is more complex so I'm probably missing something?
- cryptonector 3y ago
  > Am I correct in concluding that this backdoor might have been picked up sooner if the practice of shipping tarballs of source releases was dropped in favour of running builds straight off a repo tag? I don't think so. The git repo did have everything needed. The release tarball thing was just an added benefit to the attackers, but not critical. IMO. > I've setup and worked with many build pipelines in the .NET world over nearly 20 years and we manage fine without the concept of a source release tarball, but I realise C dev is more complex so I'm probably missing something? .Net, but, portable .Net? The issue isn't the language but what things you need to detect presence of in the target host when building, and if you're doing systems programming you're likely to need to detect many such things, and if you're doing more high level programming (e.g., an HTTP app that uses a DB) then you're not likely to need to detect many such things. The real issue is the variance from standards (e.g., POSIX) and the fact that the standards (e.g., POSIX) are so far behind because they really just standardize the lowest common denominator.
  - edflsafoiewq 3y ago
    > I don't think so. The git repo did have everything needed. build-to-host.m4 is not checked into the repo. Without the modified build-to-host.m4, there is nothing to kick off the exploit.
  - ryanjshaw 3y ago
    > The release tarball thing was just an added benefit to the attackers, but not critical. IMO. What I mean is that Debian and Fedora presumably download the tarball and build from there, rather than cloning a repo and building from the repo. If they did the latter, it would have been much more difficult to manipulate the build process to inject the backdoor that lived inside the testcase binaries, because that change to the build scripts would be out in the open. But I could be mistaken...
cryptonector 3y ago
> Liblzma ends up dynamically linked to sshd because of a systemd-related extension added by many Linux packagers that pulls in liblzma as an unrelated dependency. Lots of dependencies -> bad. Reducing dependencies will sometimes necessitate open-coding things, which too can be bad. But what's interesting here is that if you're trying to find a good place to insert a backdoor, all you need to do is analyze the dependency graphs of targets of interest looking for ones which might be vulnerable to compromise via social engineering or other vectors. We should do our own such analysis looking for places that need to be watched better or removed.
DEADMINCE 3y ago
Kind of funny the developer dismisses Linux as a "Unix-like operating system".

107 points by signa11 3y ago

This one was much clearer to me: https://research.swtch.com/xz-script https://research.swtch.com/xz-script Discussion: https://news.ycombinator.com/item?id=39903685 https://news.ycombinator.com/item?id=39903685

PreInternet01 3y ago

> much of the emerging narrative about this backdoor that you can read all over the net is based on idle speculation ...which sums up the situation quite nicely. This was a pretty bad attack on a certain ecosystem. The ecosystem will recover, or not, regardless of your feelings. Unless you're in a position to truly make a difference, just sit back and enjoy the ride...

fmajid 3y ago
There is some protection to be had in using non-mainstream platforms (in my case, Alpine Linux, OpenBSD and Illumos, though I do have some Ubuntu or Pop-OS! laptops).

deleted 3y ago

[deleted]

waynesonfire 3y ago

> Liblzma ends up dynamically linked to sshd because of a systemd-related extension added by many Linux packagers that pulls in liblzma as an unrelated dependency kinda wish this was unpacked a bit more, why exactly is a service executable dynamically linking to a library without using any of its symbols or functions, because of systemd. and a follow up if some openbsd folks can comment. over the years i've read about various unique security capabilities in openbsd, it seems natural to ask, what kernel or OS capabilities does openbsd provide to thwart the stage 2 efforts for this class of injection techniques?

MrStonedOne 3y ago
[dead]
zokier 3y ago
Libsystemd is a grabbag utility library, and completely optional: > The libsystemd library provides functions that allow interacting with various interfaces provided by the systemd(1) service manager, as well as various other functions and constants useful for implementing services in general. https://www.freedesktop.org/software/systemd/man/latest/libsystemd.html https://www.freedesktop.org/software/systemd/man/latest/libs... E.g. the service readiness protocol is pretty trivial to implement yourself if you don't want to pull libsystemd as dep.
logicprog 3y ago
> kinda wish this was unpacked a bit more, why exactly is a service executable dynamically linking to a library without using any of its symbols or functions, because of systemd. If I recall correctly based on what I've read about this — I believe from the original mailing list post that noticed the vulnerability — it's because under certain circumstances in order to enable certain functionality you might want sshd to be able to talk to systemd, so distros often patch sshd with code to do that. But obviously you need a library to implement actually speaking system's protocol, and as it happens, the easiest way to do that is to include the entirety of libsystemd, since it has functions for doing that, even though there are at least two other libraries that implement just the communication functionality and are actually designed for non-systemd programs to use. The problem with that idea being that libsystemd, being a whole standard library for all systemd-related functionality that is probably mostly designed for use by programs in the tightly integrated systemd family and as a reference imenentation, also includes a lot of other code, including code that has to deal with compression that depends on liblzma, even though all of that is never used by sshd, because it only uses the small subsection of the library it needs.
- o11c 3y ago
  > even though there are at least two other libraries that implement just the communication functionality and are actually designed for non-systemd programs to use. The important question is: if one of those libraries is used, and then something else pulls in `libsystemd`, will they conflict?

deskamess 3y ago

There really is an exception to every rule. "Do not attribute to malice that which can be adequately explained by neglect, ignorance or incompetence" and then you come across: > The stage 0 shell snippet looks at first glance like a plausible part of > the poorly readable autoconf/automake tooling.

throwanem 3y ago
None of those adequately explain these commits. This is no exception to Hanlon's razor.
- deleted 3y ago
  [deleted]
- francescovv 3y ago
  Hanlon's razor is useful to curb one's paranoia, but it is far from being a universal rule. In fact, malice and incompetence are not necessarily mutually exclusive. This very incident shows several instances where "Jia Tan" is being arguably incompetent, in addition to being clearly malicious: unintended breakage by adding extra space between "return" and "is_arch_extension_supported"; several redundant checks for `uname` == "Linux"; botched payload, so "test files" had to be replaced, with pretty fishy explanation; rather inefficient/slow GOT parsing, list goes on...
- narrator 3y ago
  Hanlon's razor lets every single first degree murderer off the hook, or any other of the many crimes where you have to establish intent to convict.
  - throwanem 3y ago
    It is a rule-of-thumb heuristic, not a law. Even if this weren't the case, your claim would remain specious. "Establish intent" and "justify attribution to malice" are the same thing said two ways.
jethro_tell 3y ago

eigenlicht 3y ago

Yes, the concluding paragraph is the most succinct (and fitting) here, there are better explanations as for the technical aspects. I know it's not related, but for the Germans around, this is just great: https://inka.de/angebot/ https://inka.de/angebot/ A community ISP. Last changed 02.05.2001, in case you missed it. ;) Millennium blues. I wasn't aware about the possibilities back then, but it's just before my time.

brenoguim 3y ago

> It does not plausibly pass for a typo because no typical editing glitch will leave a '.' character there. I would certainly attribute that to a typo if I was reviewing the code.

AwaAwa 3y ago
> The syntax error is a single period '.' as the first character on an otherwise empty line of C code. I believe the point is that given this context, it could/should not be construed as a typo.
- dpkirchner 3y ago
  If you received this patch from someone you trusted, and noticed the period, would you call it a typo or would you assume it is malicious? Or is this just an argument over the use of the word "typo" instead of "mistake"? That wouldn't be very useful, IMO.
- dvh 3y ago
  Very common if you copy paste (using mouse buffer) from midnight commander text editor or any other editor that highlights trailing spaces as dot.
- lenerdenator 3y ago
  I take it they're not forcing contributors to lint code, then?
  - klyrs 3y ago
    Does your linter look for syntax errors in the content of strings? If so, how would you design a unit test meant to capture syntax errors?
  - tedunangst 3y ago
    Have you tried running lint on a configure.ac or CMakeLists.txt file?
- 3y ago

ryanjshaw 3y ago

> the build-to-host.m4 macro package that ships as part of GNU gettext was replaced by a modified version that was copied into the release tarball Am I correct in concluding that this backdoor might have been picked up sooner if the practice of shipping tarballs of source releases was dropped in favour of running builds straight off a repo tag? I've setup and worked with many build pipelines in the .NET world over nearly 20 years and we manage fine without the concept of a source release tarball, but I realise C dev is more complex so I'm probably missing something?

cryptonector 3y ago
> Am I correct in concluding that this backdoor might have been picked up sooner if the practice of shipping tarballs of source releases was dropped in favour of running builds straight off a repo tag? I don't think so. The git repo did have everything needed. The release tarball thing was just an added benefit to the attackers, but not critical. IMO. > I've setup and worked with many build pipelines in the .NET world over nearly 20 years and we manage fine without the concept of a source release tarball, but I realise C dev is more complex so I'm probably missing something? .Net, but, portable .Net? The issue isn't the language but what things you need to detect presence of in the target host when building, and if you're doing systems programming you're likely to need to detect many such things, and if you're doing more high level programming (e.g., an HTTP app that uses a DB) then you're not likely to need to detect many such things. The real issue is the variance from standards (e.g., POSIX) and the fact that the standards (e.g., POSIX) are so far behind because they really just standardize the lowest common denominator.
- edflsafoiewq 3y ago
  > I don't think so. The git repo did have everything needed. build-to-host.m4 is not checked into the repo. Without the modified build-to-host.m4, there is nothing to kick off the exploit.
- ryanjshaw 3y ago
  > The release tarball thing was just an added benefit to the attackers, but not critical. IMO. What I mean is that Debian and Fedora presumably download the tarball and build from there, rather than cloning a repo and building from the repo. If they did the latter, it would have been much more difficult to manipulate the build process to inject the backdoor that lived inside the testcase binaries, because that change to the build scripts would be out in the open. But I could be mistaken...

cryptonector 3y ago

> Liblzma ends up dynamically linked to sshd because of a systemd-related extension added by many Linux packagers that pulls in liblzma as an unrelated dependency. Lots of dependencies -> bad. Reducing dependencies will sometimes necessitate open-coding things, which too can be bad. But what's interesting here is that if you're trying to find a good place to insert a backdoor, all you need to do is analyze the dependency graphs of targets of interest looking for ones which might be vulnerable to compromise via social engineering or other vectors. We should do our own such analysis looking for places that need to be watched better or removed.

DEADMINCE 3y ago

Kind of funny the developer dismisses Linux as a "Unix-like operating system".