5 ms·

For real, it's almost like a state-sponsored exploit. It's crafted and executed incredibly well, the performance issue feels like pure luck it got found.

jdewerd 3y ago
Yeah, we should probably expect that there are roughly 1/p(found) more of these lurking out there. Not a pleasant thought.
andersa 3y ago
It'd make total sense if it was. This way you get to have the backdoor without your enemies being able to use it against your own companies.
hnthrowaway0328 3y ago
Do we have a detailed technical analysis of the code? I read a few analysis but they all seem preliminary. It is very useful to learn from the code.
- coldpie 3y ago
  There's a few links down at the bottom of the OP to quite detailed analysis. From there you could join a Discord where discussion is ongoing.
  - hnthrowaway0328 3y ago
    Thanks coldpie.
stingraycharles 3y ago
I like the theory that actually, it wasn’t luck but was picked up on by detection tools of a large entity (Google / Microsoft / NSA / whatever), and they’re just presenting the story like this to keep their detection methods a secret. It’s what I would do.
- jsmith99 3y ago
  The attacker changed the projects contact details at oss fuzz (an automated detection tool). There’s an interesting discussion as to whether that would have picked up the vulnerability https://github.com/google/oss-fuzz/issues/11760 https://github.com/google/oss-fuzz/issues/11760
  - meowface 3y ago
    That's a fascinating extra detail. They really tried to cover all their bases. There's some plausible evidence here that they may've tried to use alter egos to encourage Debian to update the package: https://twitter.com/f0wlsec/status/1773824841331740708 https://twitter.com/f0wlsec/status/1773824841331740708
  - metzmanj 3y ago
    I work on oss-fuzz. I don't think it's plausible OSS-Fuzz could have found this. The backdoor required a build configuration that was not used in OSS-Fuzz. I'm guessing "Jia Tan" knew this and made changes to XZ's use of OSS-Fuzz for the purposes of cementing their position as the new maintainer of XZ, rather than out of worry OSS-Fuzz would find the backdoor as people have speculated.
    - Aloisius 3y ago
      How many oss-fuzz packages have a Dockerfile that runs apt-get install liblzma-dev first? Had this not been discovered, the backdoored version of xz could have eventually ended up in the ubuntu version oss-fuzz uses for its docker image - and linked into all those packages being tested as well. Except now there's an explanation if fuzzing starts to fail - honggfuzz uses -fsanitize which is incompatible with xz's use of ifunc, so any package that depends on it should rebuild xz from source with --disable-ifunc instead of using the binary package.
      - metzmanj 3y ago
        This is interesting, but do you think this would have aroused enough suspicion to find the backdoor (after every Ubuntu user was owned by it)? I don't see why this is the case. It wasn't a secret that ifuncs were being used in XZ. And if that's the case, it was sloppy of "Jia" to disable it in OSS-Fuzz and not do this: ``` __attribute__((__used__,__no_sanitize_address__)) ``` to the XZ source code to fix the false positive and turn off the compilation warning, no attention would have been drawn to this at all since no one would have to change their build script. With or without this PR, it's very unlikely OSS-Fuzz would have found the bug. OSS-Fuzz also happens to be on Ubuntu 20. I'm not very familiar with Ubuntu release cycles, but I think it would have been a very long time before backdoored packages made their way into Ubuntu 20.
webmaven 3y ago
Was the performance issue pure luck? Or was it a subtle bit of sabotage by someone inside the attacking group worried about the implications of the capability? If it had been successfully and secretly deployed, this is the sort of thing that could make your leaders much more comfortable with starting a "limited war". There are shades of "Setec Astronomy" here.
- papascrubs 3y ago
  Plot twist: It was a psyop to increase the scrutiny around OSS components. Kidding. Mostly... But given the amount of scrutiny folks are going to start putting into some supply chains... Probably cheaper to execute than most company's annual security awareness budgets cost.
- neodymiumphish 3y ago
  Considering how difficult it might be (and identifiable) to attempt direct exploitation of this without being sure your target is vulnerable, it’s plausible the performance issue allowed for an identifiable delay in attempts. This might be useful in determining whether to attempt the exploit, with an auto-skip if it received a response in less than N milliseconds.
lyu07282 3y ago
one question I still have is what exactly the performance issue was? I heard it might be related to enumeration of shared libraries, decoding of the scrambled strings[1], etc. anyone know for sure yet? one other point for investigation is if the code is similar to any other known implants? like the way it obfuscates strings, the way it detects debuggers, the way its setting up a vtable, there might be code fragments shared across projects. Which might give clues about its origin. [1] https://gist.github.com/q3k/af3d93b6a1f399de28fe194add452d01 https://gist.github.com/q3k/af3d93b6a1f399de28fe194add452d01
- saagarjha 3y ago
  It's the symbol lookup that's slow.
  - 400thecat 3y ago
    why does the backdoor rely on symbol lookup? is it because of the use of IFUNC ?
    - saagarjha 3y ago
      I believe it is looking for functions in OpenSSH that do key validation etc
timmytokyo 3y ago
I'm not sure why everyone is 100% sure this was a state-sponsored security breach. I agree that it's more likely than not state-sponsored, but I can imagine all sorts of other groups who would have an interest in something like this, organized crime in particular. Imagine how many banks or crypto wallets they could break into with a RCE this pervasive.
- blablabla123 3y ago
  Especially considering this introduced a 500ms waiting time. But surely this was quite a risky time investment, 2 years. How likely is it that this was the only attempt if this was done by a group? (And maybe there were failed attempts after trying to take over maintenance of other packages?) Maybe really a very well-funded cybercrime group that can afford such moonshot endeavours or a state group that doesn't completely know yet what it's doing or isn't that well equipped (anymore?). I'm definitely curious about analysis of attribution
- bevekspldnw 3y ago
  Motive and patience. Motive as you point out is shared by many parties. Typically its only state agencies that will fund an operation with uncertain pay off over long periods of time. That type of patience is expensive. Online criminals are beholden to changing market pressures and short term investment pressures like any other start up.
- deleted 3y ago
  [deleted]
- hackeraccount 3y ago
  I'm not 100% sure - it could have been a criminal(s) or even a single motivated actor. That said it's a lot of work - 2 years at least. It's an exploit that's so good that you'd have to use incredibly carefully - also because if/when it's discovered it's going to break everywhere. I've read descriptions about how the NSA (and presumably other such agencies) and they're really careful. The first job is to make sure the target doesn't get confirmation that they are in fact a target. The second is that they always cover their tracks so the target doesn't know they were a target. Criminals tend to do the first but almost never the second so a tool like this - while I'm sure they would love - isn't worth the amount of work it would take to develop. Again - I'm not 100% on this but ... 40% ? say 20% criminals, 10% lone wolf?
7373737373 3y ago
I read someone speculating that the performance issue was intentional, so infected machines could be easily identified by an internet wide scan without arousing further suspicicion. If this is or becomes a widespread method, then anti-malware groups should perhaps conduct these scans themselves.
- zarzavat 3y ago
  Very small differences in performance can be detected over the network as long as you have enough samples. Given that every port 22 is being hit by a gazillion attempts per day already, sample count shouldn’t be an issue. So if distinguishing infected machines was their intention they definitely over-egged it.
IshKebab 3y ago
I don't think it was executed incredibly well. There were definitely very clever aspects but they made multiple mistakes - triggering Valgrind, the performance issue, using a `.` to break the Landlock test, not giving the author a proper background identity. I guess you could also include the fact that they made it a very obvious back door rather than an exploitable bug, but that has the advantage of only letting you exploit it so it was probably an intentional trade-off. Just think how many back doors / intentional bugs there are that we don't know about because they didn't make any of these mistakes.
- richardfey 3y ago
  Maybe it's the first successful attempt of a state which nobody would right now suspect as capable of carrying this out. Everyone is looking at the big guys but a new player has entered the game.

jdewerd 3y ago

Yeah, we should probably expect that there are roughly 1/p(found) more of these lurking out there. Not a pleasant thought.

andersa 3y ago

It'd make total sense if it was. This way you get to have the backdoor without your enemies being able to use it against your own companies.

hnthrowaway0328 3y ago

Do we have a detailed technical analysis of the code? I read a few analysis but they all seem preliminary. It is very useful to learn from the code.

coldpie 3y ago
There's a few links down at the bottom of the OP to quite detailed analysis. From there you could join a Discord where discussion is ongoing.
- hnthrowaway0328 3y ago
  Thanks coldpie.

stingraycharles 3y ago

I like the theory that actually, it wasn’t luck but was picked up on by detection tools of a large entity (Google / Microsoft / NSA / whatever), and they’re just presenting the story like this to keep their detection methods a secret. It’s what I would do.

jsmith99 3y ago
The attacker changed the projects contact details at oss fuzz (an automated detection tool). There’s an interesting discussion as to whether that would have picked up the vulnerability https://github.com/google/oss-fuzz/issues/11760 https://github.com/google/oss-fuzz/issues/11760
- meowface 3y ago
  That's a fascinating extra detail. They really tried to cover all their bases. There's some plausible evidence here that they may've tried to use alter egos to encourage Debian to update the package: https://twitter.com/f0wlsec/status/1773824841331740708 https://twitter.com/f0wlsec/status/1773824841331740708
- metzmanj 3y ago
  I work on oss-fuzz. I don't think it's plausible OSS-Fuzz could have found this. The backdoor required a build configuration that was not used in OSS-Fuzz. I'm guessing "Jia Tan" knew this and made changes to XZ's use of OSS-Fuzz for the purposes of cementing their position as the new maintainer of XZ, rather than out of worry OSS-Fuzz would find the backdoor as people have speculated.
  - Aloisius 3y ago
    How many oss-fuzz packages have a Dockerfile that runs apt-get install liblzma-dev first? Had this not been discovered, the backdoored version of xz could have eventually ended up in the ubuntu version oss-fuzz uses for its docker image - and linked into all those packages being tested as well. Except now there's an explanation if fuzzing starts to fail - honggfuzz uses -fsanitize which is incompatible with xz's use of ifunc, so any package that depends on it should rebuild xz from source with --disable-ifunc instead of using the binary package.
    - metzmanj 3y ago
      This is interesting, but do you think this would have aroused enough suspicion to find the backdoor (after every Ubuntu user was owned by it)? I don't see why this is the case. It wasn't a secret that ifuncs were being used in XZ. And if that's the case, it was sloppy of "Jia" to disable it in OSS-Fuzz and not do this: ``` __attribute__((__used__,__no_sanitize_address__)) ``` to the XZ source code to fix the false positive and turn off the compilation warning, no attention would have been drawn to this at all since no one would have to change their build script. With or without this PR, it's very unlikely OSS-Fuzz would have found the bug. OSS-Fuzz also happens to be on Ubuntu 20. I'm not very familiar with Ubuntu release cycles, but I think it would have been a very long time before backdoored packages made their way into Ubuntu 20.

webmaven 3y ago

Was the performance issue pure luck? Or was it a subtle bit of sabotage by someone inside the attacking group worried about the implications of the capability? If it had been successfully and secretly deployed, this is the sort of thing that could make your leaders much more comfortable with starting a "limited war". There are shades of "Setec Astronomy" here.

papascrubs 3y ago
Plot twist: It was a psyop to increase the scrutiny around OSS components. Kidding. Mostly... But given the amount of scrutiny folks are going to start putting into some supply chains... Probably cheaper to execute than most company's annual security awareness budgets cost.
neodymiumphish 3y ago
Considering how difficult it might be (and identifiable) to attempt direct exploitation of this without being sure your target is vulnerable, it’s plausible the performance issue allowed for an identifiable delay in attempts. This might be useful in determining whether to attempt the exploit, with an auto-skip if it received a response in less than N milliseconds.

lyu07282 3y ago

one question I still have is what exactly the performance issue was? I heard it might be related to enumeration of shared libraries, decoding of the scrambled strings[1], etc. anyone know for sure yet? one other point for investigation is if the code is similar to any other known implants? like the way it obfuscates strings, the way it detects debuggers, the way its setting up a vtable, there might be code fragments shared across projects. Which might give clues about its origin. [1] https://gist.github.com/q3k/af3d93b6a1f399de28fe194add452d01 https://gist.github.com/q3k/af3d93b6a1f399de28fe194add452d01

saagarjha 3y ago
It's the symbol lookup that's slow.
- 400thecat 3y ago
  why does the backdoor rely on symbol lookup? is it because of the use of IFUNC ?
  - saagarjha 3y ago
    I believe it is looking for functions in OpenSSH that do key validation etc

timmytokyo 3y ago

I'm not sure why everyone is 100% sure this was a state-sponsored security breach. I agree that it's more likely than not state-sponsored, but I can imagine all sorts of other groups who would have an interest in something like this, organized crime in particular. Imagine how many banks or crypto wallets they could break into with a RCE this pervasive.

blablabla123 3y ago
Especially considering this introduced a 500ms waiting time. But surely this was quite a risky time investment, 2 years. How likely is it that this was the only attempt if this was done by a group? (And maybe there were failed attempts after trying to take over maintenance of other packages?) Maybe really a very well-funded cybercrime group that can afford such moonshot endeavours or a state group that doesn't completely know yet what it's doing or isn't that well equipped (anymore?). I'm definitely curious about analysis of attribution
bevekspldnw 3y ago
Motive and patience. Motive as you point out is shared by many parties. Typically its only state agencies that will fund an operation with uncertain pay off over long periods of time. That type of patience is expensive. Online criminals are beholden to changing market pressures and short term investment pressures like any other start up.
deleted 3y ago
[deleted]
hackeraccount 3y ago
I'm not 100% sure - it could have been a criminal(s) or even a single motivated actor. That said it's a lot of work - 2 years at least. It's an exploit that's so good that you'd have to use incredibly carefully - also because if/when it's discovered it's going to break everywhere. I've read descriptions about how the NSA (and presumably other such agencies) and they're really careful. The first job is to make sure the target doesn't get confirmation that they are in fact a target. The second is that they always cover their tracks so the target doesn't know they were a target. Criminals tend to do the first but almost never the second so a tool like this - while I'm sure they would love - isn't worth the amount of work it would take to develop. Again - I'm not 100% on this but ... 40% ? say 20% criminals, 10% lone wolf?

7373737373 3y ago

I read someone speculating that the performance issue was intentional, so infected machines could be easily identified by an internet wide scan without arousing further suspicicion. If this is or becomes a widespread method, then anti-malware groups should perhaps conduct these scans themselves.

zarzavat 3y ago
Very small differences in performance can be detected over the network as long as you have enough samples. Given that every port 22 is being hit by a gazillion attempts per day already, sample count shouldn’t be an issue. So if distinguishing infected machines was their intention they definitely over-egged it.

IshKebab 3y ago

I don't think it was executed incredibly well. There were definitely very clever aspects but they made multiple mistakes - triggering Valgrind, the performance issue, using a `.` to break the Landlock test, not giving the author a proper background identity. I guess you could also include the fact that they made it a very obvious back door rather than an exploitable bug, but that has the advantage of only letting you exploit it so it was probably an intentional trade-off. Just think how many back doors / intentional bugs there are that we don't know about because they didn't make any of these mistakes.

richardfey 3y ago
Maybe it's the first successful attempt of a state which nobody would right now suspect as capable of carrying this out. Everyone is looking at the big guys but a new player has entered the game.