4 ms·

Genuine question: how do we know that Jia Tan is a bad actor and not simply an inept coder that introduced a bug?

by zxspectrum1982 3y ago

_zephyrus_ 3y ago
Please, take a look at how the exploit is crafted. It is not a bug in a piece of code.
sligor 3y ago
In what I understand: we don't know for sure, but the suspicions are so high that it more important to act quickly and revert the changes. The community will analyse later what the changes are exactly doing ( less urgent than reverting the changes )
- Sesse__ 3y ago
  We definitely know for sure this was inserted intentionally. What we don't know for sure is whether Jia Tan (whoever that might be, if it is even a single person) did it on purpose, or if they had their account compromised.
mr_mitm 3y ago
This attack was so sophisticated that I don't understand half of it after a first read. It's plainly obvious that it is not a simple typo or a blunder but very carefully crafted.
soneil 3y ago
First, the "bug" jumps through several well-crafted hurdles in order to hide itself from co-maintainers, package maintainers, and anyone else who might be looking. Hiding code in binary objects, hidden within in testcases and injected into the build under just-so conditions, would be incredibly difficult to justify with any other intent. Next, the affected codepath skips itself if it's not executed under just-so conditions. Specifically testing to see if it's being run interactively to avoid diagnosis, and testing to see if the executed binary is sshd. Lastly, what good reason could a compression library possibly have, for overriding encryption routines in sshd? It's a fair question, but it doesn't survive inspection at all. This is like finding someone in a ninja suit, wearing night vision equipment, creeping around a museum in the middle of the night, with the crown jewels in a bag slung over their shoulder - and asking yourself if they're perhaps a lost tourist.
- tetha 3y ago
  This is very much what I was thinking as well. This wasn't a casual allocation error, or some attempt to load data from a well-crafted archive to execute it or something like that. The payload already is pretty sophisticated and full of intent to both hide itself and to target sshd very specifically. And on top, there is another layer of pretty clever build infrastructure and manipulation - again with fairly clear intent to hide the inclusion of the payload. Not to take away from the discovery, but finding this was lucky. But then you have two actions with imo fairly clear intent to hide, deceive and target security infrastructure - and then some more mundane actions become weird as well. Deactivating parts in oss-fuzz is plausible under the right conditions, but in this light? Or being pushy to get this released into newer Fedora versions - again, might be understandable and boring in other circumstances. Staying in your example - suddenly it becomes strange that this person got lost in the museum two or three times over the last month. Or has been taking walks around the museum very often over the last weeks. Nothing illegal with those on their own, but like that?
  - zxspectrum1982 3y ago
    It's really scary, if you think about it. Cumulus Linux (originally by Cumulus Networks, which is now part of Nvidia) is a Debian-based Linux distribution used the data plane in datacenters: Dell, HPE, Mellanox, Lenovo and others let customers use Cumulus Linux. That means major datacenters running a potentially-compromised Linux distribution because of the xz attack. McAfee Linux used to be (it EOL'ed recently) a CentOS-based distribution (which in turn comes from Fedora) used for security appliances by McAfee. Back when the xz attack was initiated (Oct'21), McAfee Linux was in full support. Even more enterprise-grade appliances to be easily compromised. As a former Debian developer, I wonder if the "must use upstream tarball" approach Debian has (or at least, when I was a DD, just a handful of years ago) makes sense. It seems if built from the git repository, this xz attack would not have succeeded.
- zxspectrum1982 3y ago
  Wow, I see. I had no idea it was that sophisticated. Thanks for the explanation! PS: why the negatives to an honest question!???
mkl 3y ago
This has been on HN's front page all day, with >3500 upvotes, and it answers your question: https://www.openwall.com/lists/oss-security/2024/03/29/4 https://www.openwall.com/lists/oss-security/2024/03/29/4. The exploit is sophisticated and intentional.
ufo 3y ago
The malicious developer did not add bugs to xz, they used xz as an attack vector. They added malicious shell code to the xz build scripts, so that when you build and install xz it attempts to inject a backdoor into sshd. The malicious shell scripts were carefully hidden. They were compressed and added to the repository under the guise of "test files". Furthermore, the backdoor is only enabled in the release tarballs: the malicious actor has run the autoconf and then manually enabled the backdoor in the generated build scripts. If you clone the upstreaf repo and regenerate the buildscriots from scratch,the backdoor is not enabled.

Genuine question: how do we know that Jia Tan is a bad actor and not simply an inept coder that introduced a bug?

Please, take a look at how the exploit is crafted. It is not a bug in a piece of code.

In what I understand: we don't know for sure, but the suspicions are so high that it more important to act quickly and revert the changes. The community will analyse later what the changes are exactly doing ( less urgent than reverting the changes )

Sesse__ 3y ago
We definitely know for sure this was inserted intentionally. What we don't know for sure is whether Jia Tan (whoever that might be, if it is even a single person) did it on purpose, or if they had their account compromised.

mr_mitm 3y ago

This attack was so sophisticated that I don't understand half of it after a first read. It's plainly obvious that it is not a simple typo or a blunder but very carefully crafted.

soneil 3y ago

First, the "bug" jumps through several well-crafted hurdles in order to hide itself from co-maintainers, package maintainers, and anyone else who might be looking. Hiding code in binary objects, hidden within in testcases and injected into the build under just-so conditions, would be incredibly difficult to justify with any other intent. Next, the affected codepath skips itself if it's not executed under just-so conditions. Specifically testing to see if it's being run interactively to avoid diagnosis, and testing to see if the executed binary is sshd. Lastly, what good reason could a compression library possibly have, for overriding encryption routines in sshd? It's a fair question, but it doesn't survive inspection at all. This is like finding someone in a ninja suit, wearing night vision equipment, creeping around a museum in the middle of the night, with the crown jewels in a bag slung over their shoulder - and asking yourself if they're perhaps a lost tourist.

tetha 3y ago
This is very much what I was thinking as well. This wasn't a casual allocation error, or some attempt to load data from a well-crafted archive to execute it or something like that. The payload already is pretty sophisticated and full of intent to both hide itself and to target sshd very specifically. And on top, there is another layer of pretty clever build infrastructure and manipulation - again with fairly clear intent to hide the inclusion of the payload. Not to take away from the discovery, but finding this was lucky. But then you have two actions with imo fairly clear intent to hide, deceive and target security infrastructure - and then some more mundane actions become weird as well. Deactivating parts in oss-fuzz is plausible under the right conditions, but in this light? Or being pushy to get this released into newer Fedora versions - again, might be understandable and boring in other circumstances. Staying in your example - suddenly it becomes strange that this person got lost in the museum two or three times over the last month. Or has been taking walks around the museum very often over the last weeks. Nothing illegal with those on their own, but like that?
- zxspectrum1982 3y ago
  It's really scary, if you think about it. Cumulus Linux (originally by Cumulus Networks, which is now part of Nvidia) is a Debian-based Linux distribution used the data plane in datacenters: Dell, HPE, Mellanox, Lenovo and others let customers use Cumulus Linux. That means major datacenters running a potentially-compromised Linux distribution because of the xz attack. McAfee Linux used to be (it EOL'ed recently) a CentOS-based distribution (which in turn comes from Fedora) used for security appliances by McAfee. Back when the xz attack was initiated (Oct'21), McAfee Linux was in full support. Even more enterprise-grade appliances to be easily compromised. As a former Debian developer, I wonder if the "must use upstream tarball" approach Debian has (or at least, when I was a DD, just a handful of years ago) makes sense. It seems if built from the git repository, this xz attack would not have succeeded.
zxspectrum1982 3y ago
Wow, I see. I had no idea it was that sophisticated. Thanks for the explanation! PS: why the negatives to an honest question!???

mkl 3y ago

This has been on HN's front page all day, with >3500 upvotes, and it answers your question: https://www.openwall.com/lists/oss-security/2024/03/29/4 https://www.openwall.com/lists/oss-security/2024/03/29/4. The exploit is sophisticated and intentional.

ufo 3y ago

The malicious developer did not add bugs to xz, they used xz as an attack vector. They added malicious shell code to the xz build scripts, so that when you build and install xz it attempts to inject a backdoor into sshd. The malicious shell scripts were carefully hidden. They were compressed and added to the repository under the guise of "test files". Furthermore, the backdoor is only enabled in the release tarballs: the malicious actor has run the autoconf and then manually enabled the backdoor in the generated build scripts. If you clone the upstreaf repo and regenerate the buildscriots from scratch,the backdoor is not enabled.