4 ms·

I like the theory that actually, it wasn’t luck but was picked up on by detection tools of a large entity (Google / Microsoft / NSA / whatever), and they’re jus

I like the theory that actually, it wasn’t luck but was picked up on by detection tools of a large entity (Google / Microsoft / NSA / whatever), and they’re just presenting the story like this to keep their detection methods a secret. It’s what I would do.

jsmith99 3y ago
The attacker changed the projects contact details at oss fuzz (an automated detection tool). There’s an interesting discussion as to whether that would have picked up the vulnerability https://github.com/google/oss-fuzz/issues/11760 https://github.com/google/oss-fuzz/issues/11760
- meowface 3y ago
  That's a fascinating extra detail. They really tried to cover all their bases. There's some plausible evidence here that they may've tried to use alter egos to encourage Debian to update the package: https://twitter.com/f0wlsec/status/1773824841331740708 https://twitter.com/f0wlsec/status/1773824841331740708
- metzmanj 3y ago
  I work on oss-fuzz. I don't think it's plausible OSS-Fuzz could have found this. The backdoor required a build configuration that was not used in OSS-Fuzz. I'm guessing "Jia Tan" knew this and made changes to XZ's use of OSS-Fuzz for the purposes of cementing their position as the new maintainer of XZ, rather than out of worry OSS-Fuzz would find the backdoor as people have speculated.
  - Aloisius 3y ago
    How many oss-fuzz packages have a Dockerfile that runs apt-get install liblzma-dev first? Had this not been discovered, the backdoored version of xz could have eventually ended up in the ubuntu version oss-fuzz uses for its docker image - and linked into all those packages being tested as well. Except now there's an explanation if fuzzing starts to fail - honggfuzz uses -fsanitize which is incompatible with xz's use of ifunc, so any package that depends on it should rebuild xz from source with --disable-ifunc instead of using the binary package.
    - metzmanj 3y ago
      This is interesting, but do you think this would have aroused enough suspicion to find the backdoor (after every Ubuntu user was owned by it)? I don't see why this is the case. It wasn't a secret that ifuncs were being used in XZ. And if that's the case, it was sloppy of "Jia" to disable it in OSS-Fuzz and not do this: ``` __attribute__((__used__,__no_sanitize_address__)) ``` to the XZ source code to fix the false positive and turn off the compilation warning, no attention would have been drawn to this at all since no one would have to change their build script. With or without this PR, it's very unlikely OSS-Fuzz would have found the bug. OSS-Fuzz also happens to be on Ubuntu 20. I'm not very familiar with Ubuntu release cycles, but I think it would have been a very long time before backdoored packages made their way into Ubuntu 20.
est31 3y ago
I doubt that if Google detected it with some internal tool, they'd reach out to Microsoft to hide their contribution. It was reported by an MS engineer who happens to be involved in another OSS project. MS is doing business with the US intelligence community, for example there is the Skype story: First, rumors that NSA offers a lot of money for people who can break Skype's E2E encryption, then MS buys Skype, then MS changes Skype's client to not be E2E encrypted any more and to use MS servers instead of peer to peer, allowing undetectable wiretapping of arbitrary connections. But it's a quite credible story too that it was just a random discovery. Even if it was the NSA, why would they hide that capability. It doesn't take much to run a script to compare git state with uploaded source tarballs in distros like Debian (Debian has separate tarballs for the source and the source with Debian patches applied).
- anarazel 3y ago
  > It was reported by an MS engineer who happens to be involved in another OSS project. I view it as being OSS or postgresql dev that happens to work at microsoft. I've been doing the former for much longer (starting somewhere between 2005 and 2008, depending on how you count) than the latter (2019-12).
  - est31 3y ago
    Thanks for the explanation. Also thanks for catching this and protecting us all; I think in the end it's way more believable that you indeed found it on your own, above was just brainless blathering into the ether =). Lastly, thanks for your Postgres contributions.
  - iliane5 3y ago
    Just wanted to say thank you for your work and attention to detail, it's immensely valuable and we're all very grateful for it.
- bevekspldnw 3y ago
  “Even if it was the NSA, why would they hide that capability” Perhaps you’re not familiar with what NSA historically stood for: Never Say Anything.
edflsafoiewq 3y ago
They could announce it without revealing their detection method. I don't see what the extra indirection buys them.
- MOARDONGZPLZ 3y ago
  Shutting down 99.9% of speculation on how the vulnerability was found in the first place.
  - VMG 3y ago
    (1) obviously not (2) they could just say they found it during some routine dependency review or whatever
    - MOARDONGZPLZ 3y ago
      I mean, yes. I’ve read all the commentary I can get my hands on about this incident because it is fascinating and this is the first instance of some parallel construction theory of finding it I’ve seen. Second, maybe a routine dependency review is how they _actually_ found it but they don’t want future people like this focusing too much on that otherwise they may try to mitigate, whereas now they may focus on something inane like a 0.5 second increase in sshd’s load time or whatever.
meowface 3y ago
It's really interesting to think what might've happened if they could've implemented this with much less performance overhead. How long might it have lasted for? Years?
TacticalCoder 3y ago
> ... nd they’re just presenting the story like this to keep their detection methods a secret. It’s what I would do. Basically "parallel construction". It's very possible it's what happened. https://en.wikipedia.org/wiki/Parallel_construction https://en.wikipedia.org/wiki/Parallel_construction
takeda 3y ago
Sorry for that ugly comparison, but that explanation reminds me of the theories when covid started, that it was created by secret organization that is actually ruling the world. People love when there's some explanation that doesn't involve randomness, because with randomness looks like we don't have grasp on things. Google actually had tooling that was detecting it, but he disabled check that would show it. Google/Microsoft/NSA could just say they detected it with internal tooling and not disclose how exactly. Google and Microsoft would love to have credit.
deleted 3y ago
[deleted]
heavyset_go 3y ago
You're describing parallel construction and it is something that happens all of the time.
lyu07282 3y ago
While we are speculating, there was this case of university students trying to introduce malicious commits in the kernel in order to test open source to such attack vectors. Perhaps this was similar "research" by some students. https://www.cnx-software.com/2021/04/22/phd-students-willfully-committed-known-malicious-changes-to-mainline-linux/ https://www.cnx-software.com/2021/04/22/phd-students-willful...

jsmith99 3y ago

The attacker changed the projects contact details at oss fuzz (an automated detection tool). There’s an interesting discussion as to whether that would have picked up the vulnerability https://github.com/google/oss-fuzz/issues/11760 https://github.com/google/oss-fuzz/issues/11760

meowface 3y ago
That's a fascinating extra detail. They really tried to cover all their bases. There's some plausible evidence here that they may've tried to use alter egos to encourage Debian to update the package: https://twitter.com/f0wlsec/status/1773824841331740708 https://twitter.com/f0wlsec/status/1773824841331740708
metzmanj 3y ago
I work on oss-fuzz. I don't think it's plausible OSS-Fuzz could have found this. The backdoor required a build configuration that was not used in OSS-Fuzz. I'm guessing "Jia Tan" knew this and made changes to XZ's use of OSS-Fuzz for the purposes of cementing their position as the new maintainer of XZ, rather than out of worry OSS-Fuzz would find the backdoor as people have speculated.
- Aloisius 3y ago
  How many oss-fuzz packages have a Dockerfile that runs apt-get install liblzma-dev first? Had this not been discovered, the backdoored version of xz could have eventually ended up in the ubuntu version oss-fuzz uses for its docker image - and linked into all those packages being tested as well. Except now there's an explanation if fuzzing starts to fail - honggfuzz uses -fsanitize which is incompatible with xz's use of ifunc, so any package that depends on it should rebuild xz from source with --disable-ifunc instead of using the binary package.
  - metzmanj 3y ago
    This is interesting, but do you think this would have aroused enough suspicion to find the backdoor (after every Ubuntu user was owned by it)? I don't see why this is the case. It wasn't a secret that ifuncs were being used in XZ. And if that's the case, it was sloppy of "Jia" to disable it in OSS-Fuzz and not do this: ``` __attribute__((__used__,__no_sanitize_address__)) ``` to the XZ source code to fix the false positive and turn off the compilation warning, no attention would have been drawn to this at all since no one would have to change their build script. With or without this PR, it's very unlikely OSS-Fuzz would have found the bug. OSS-Fuzz also happens to be on Ubuntu 20. I'm not very familiar with Ubuntu release cycles, but I think it would have been a very long time before backdoored packages made their way into Ubuntu 20.

est31 3y ago

I doubt that if Google detected it with some internal tool, they'd reach out to Microsoft to hide their contribution. It was reported by an MS engineer who happens to be involved in another OSS project. MS is doing business with the US intelligence community, for example there is the Skype story: First, rumors that NSA offers a lot of money for people who can break Skype's E2E encryption, then MS buys Skype, then MS changes Skype's client to not be E2E encrypted any more and to use MS servers instead of peer to peer, allowing undetectable wiretapping of arbitrary connections. But it's a quite credible story too that it was just a random discovery. Even if it was the NSA, why would they hide that capability. It doesn't take much to run a script to compare git state with uploaded source tarballs in distros like Debian (Debian has separate tarballs for the source and the source with Debian patches applied).

anarazel 3y ago
> It was reported by an MS engineer who happens to be involved in another OSS project. I view it as being OSS or postgresql dev that happens to work at microsoft. I've been doing the former for much longer (starting somewhere between 2005 and 2008, depending on how you count) than the latter (2019-12).
- est31 3y ago
  Thanks for the explanation. Also thanks for catching this and protecting us all; I think in the end it's way more believable that you indeed found it on your own, above was just brainless blathering into the ether =). Lastly, thanks for your Postgres contributions.
- iliane5 3y ago
  Just wanted to say thank you for your work and attention to detail, it's immensely valuable and we're all very grateful for it.
bevekspldnw 3y ago
“Even if it was the NSA, why would they hide that capability” Perhaps you’re not familiar with what NSA historically stood for: Never Say Anything.

edflsafoiewq 3y ago

They could announce it without revealing their detection method. I don't see what the extra indirection buys them.

MOARDONGZPLZ 3y ago
Shutting down 99.9% of speculation on how the vulnerability was found in the first place.
- VMG 3y ago
  (1) obviously not (2) they could just say they found it during some routine dependency review or whatever
  - MOARDONGZPLZ 3y ago
    I mean, yes. I’ve read all the commentary I can get my hands on about this incident because it is fascinating and this is the first instance of some parallel construction theory of finding it I’ve seen. Second, maybe a routine dependency review is how they _actually_ found it but they don’t want future people like this focusing too much on that otherwise they may try to mitigate, whereas now they may focus on something inane like a 0.5 second increase in sshd’s load time or whatever.

meowface 3y ago

It's really interesting to think what might've happened if they could've implemented this with much less performance overhead. How long might it have lasted for? Years?

TacticalCoder 3y ago

> ... nd they’re just presenting the story like this to keep their detection methods a secret. It’s what I would do. Basically "parallel construction". It's very possible it's what happened. https://en.wikipedia.org/wiki/Parallel_construction https://en.wikipedia.org/wiki/Parallel_construction

takeda 3y ago

Sorry for that ugly comparison, but that explanation reminds me of the theories when covid started, that it was created by secret organization that is actually ruling the world. People love when there's some explanation that doesn't involve randomness, because with randomness looks like we don't have grasp on things. Google actually had tooling that was detecting it, but he disabled check that would show it. Google/Microsoft/NSA could just say they detected it with internal tooling and not disclose how exactly. Google and Microsoft would love to have credit.

deleted 3y ago

[deleted]

heavyset_go 3y ago

You're describing parallel construction and it is something that happens all of the time.

lyu07282 3y ago

While we are speculating, there was this case of university students trying to introduce malicious commits in the kernel in order to test open source to such attack vectors. Perhaps this was similar "research" by some students. https://www.cnx-software.com/2021/04/22/phd-students-willfully-committed-known-malicious-changes-to-mainline-linux/ https://www.cnx-software.com/2021/04/22/phd-students-willful...