12 ms·

XZ backdoor story – Initial analysis

406 points by kryster 3y ago

Animats 3y ago
Why is "hooking" allowed at all in non-debug code? The legit applications all involve tracing or profiling.
- rwmj 3y ago
  What does "non-debug" code mean in this context? Most distros compile everything with -g and then separate out the debug symbols into a different package, because you almost always want to be able to attach gdb to any process when something goes wrong (and no, recompiling at that point is a much worse idea, because it means installing compilers on your production system as well as not being able to debug what was actually running). It doesn't matter anyway, because once you're running inside a process you can do most anything you like. This was a convenient mechanism, but there are scores of ways to scan over the symbols and modify code inside the same process.
  - Animats 3y ago
    "You almost always want to be able to attach gdb to any process when something goes wrong". Now, this is why we can't have secure systems. Somehow, I've never needed to attach GDB to a running process. Start up programs under GDB, yes. Read crash dumps, yes. If you're using a debugger in a production system, something has gone horribly wrong.
    - troad 3y ago
      > If you're using a debugger in a production system, something has gone horribly wrong. That's right, hence the debugger.
    - rwmj 3y ago
      Or you want to capture accurate core dumps for a tool like ABRT. We (Red Hat) work very closely with customers to find out what is and isn't acceptable to be running on their production systems, including customers who store very sensitive data.
    - cozzyd 3y ago
      How do you debug rare hangs?
- saagarjha 3y ago
  Hooking of what?
  - Animats 3y ago
    Live-patching code so that a call to something becomes a call to something else.
    - saagarjha 3y ago
      How are you going to prevent this in general? At some point you’re going to make an indirect call.
andy_ppp 3y ago
Imagine all the back doors people haven’t discovered…
- ta1243 3y ago
  Especially in closed source software where nobody has an incentive to look, any performance hits will be ignored, code analysis tools can't be run (why bother - can't exactly sell it as a new feature)
throwaway67743 3y ago
[flagged]
- StressedDev 3y ago
  The analysis post was very good and did a great job of explaining how the attacker (hacker) obfuscated the backdoor.
- eclectic29 3y ago
  Did you create a throwaway account specifically to write a snarky comment here? Looks like this practice is becoming more common.
  - dredmorbius 3y ago
    It also seems to specifically violate HN guidelines: Throwaway accounts are ok for sensitive information, but please don't create accounts routinely. HN is a community—users should have an identity that others can relate to. <https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html> Snarky shallow dismissal ain't sensitive information. It's precisely the sort of behaviour which should be tied to, and reflect reputation of, a primary account. (Which can of course be pseudonymous, example myself.)
  - saagarjha 3y ago
    Their account seems too old for that.
- sho_hn 3y ago
  This one contained information I didn't encounter in the earlier ones linked from HN. The earlier ones mostly focused on the multi-step build and extraction process, this one has some detail on the inner workings of the library that gets extracted.
- Analemma_ 3y ago
  This is one of the most significant computer security events of all time. A genuine backdoor-- not an accidental vulnerability, a backdoor-- created by a very sophisticated, very-likely state-backed, actor who used a combination of social engineering and multiple clever points of indirection to get a vulnerability into a common library. There will be many more articles about it in the months and years to come, and I don't see what's wrong with that as long as they are providing new information.
vlovich123 3y ago
So if I'm understanding correctly, a useful hardening step would be to ensure that each dynamically linked library had it's own GOT & that the tables are marked as read-only once dynamic linking completes (i.e. you can't reach across dynamic boundaries to patch someone else's ifunc entries). That would improve the supply chain security of code getting linked in somewhere but never executed. EDIT: Or even better, perhaps ifunc should be implemented in a declarative fashion so that you can't just cause arbitrary code execution from each library you link against. That might be harder to implement at this point considering back compat, but probably is something that could be layered in over a longer time period (i.e. if you build any library with "declaratively linked ifunc" feature bit set, then the dynamic linker will force all linked libraries to have that feature flag or fail to launch).
- jerjerjer 3y ago
  From another angle: build systems. Currently building most libraries usually involves executing a series of highly complex byzantine scripts, requiring turing-complete environment. This gives attacker an endless attack surface, and when the build process is hijacked - well, the opportunities are there. Moving to a declarative build process with only a limited state machine as an executor would help. Requiring all source blobs being reproducible might also be something to think about.
  - deleted 3y ago
    [deleted]
  - kimixa 3y ago
    Having a "simple" declarative build definition won't help if the thing that interprets and executes that definition is shipped with the package, as said interpreter is likely orders of magnitude more complex and harder to personally review. As is what happened with the xz example - the "attack" was hidden in code that is autogenerated by autotools, not the build or configuration definitions. People put trust in distros and packagers to having something of a review chain - there's 0% chance you personally be an expert in everything executing on your workstation right now (outside of maybe toy systems). I'm not an expert in m4 or bash, but I hope enough experts are in the chain to get to my distro's package library are that such things are less likely. But that is all bypassed here. I think this particular approach is a one-off, as I know of no other build environment where it's expected to have the generated executable of the build system "helpfully" packaged in the tarball as a packaging step. If it is in some I'm not aware of, I hope that decision is being re-examined now.
dvsfish 3y ago
Great read - the lengths the attacker went to to not only hide it but also build the trust to be able to pull this off is incredible. So calculated. One can only imagine they felt a great deal of panic and despair as the security community lit up the issue, regardless of a near zero chance of repercussions. Very clever - those involved in catching this are, quietly, heroes
- jerjerjer 3y ago
  The use of trie structure to compare strings without hardcoding raw, searchable string values is very inspired. But yes, some black hat had a really really bad day: being so close to owning any ssh-enabled recently-updated box and when they can finally see a finish line it is just poof... gone in a day. It was two years of effort from at least one guy and more likely a team.
  - jumelles 3y ago
    Honestly? Good. Anyone doing this sort of work willingly deserves worse.
    - optimalsolver 3y ago
      That they were harassing and manipulating a lone, unthanked maintainer who had already told them he was dealing with mental issues makes them evil, IMO. The honorable thing for "Jia" to do after this epic failure is seppuku, or whatever his or her local equivalent is.
      - cqqxo4zV46cp 3y ago
        Nobody sees themselves as the bad guy, and that’s not the same as “some people are just fundamentally selfish”. There are definitely loads of people that’d feel like the end justifies the means. There are plenty of people for whom a workday involves doing far worse things for the world than cyberbullying one person, and will look you in the eye and justify it. Plenty of that stuff is socially acceptable in many many mainstream circles. “Being mean to a maintainer” is just one that this community is especially sensitive to, because it involves a highly personified victim that they can relate to.
deleted 3y ago
[deleted]
asveikau 3y ago
I saw the xz repo is back on GitHub, Lasse and a new contributor were cleaning it up. They removed ifunc support and they committed code to generate test files into the repo, so that they can do test files without blobs. It looks like they're doing good stuff.
- saagarjha 3y ago
  > a new contributor Jia Tan, is that you?
  - asveikau 3y ago
    Very funny. This one: https://github.com/tukaani-project/xz/commits?author=thesamesam https://github.com/tukaani-project/xz/commits?author=thesame... https://github.com/thesamesam https://github.com/thesamesam Seems like he's a Gentoo contributor.
    - asddubs 3y ago
      oh no, gentoo has been compromised
  - willsmith72 3y ago
    I know part of the fun of oss is anyone can contribute behind an anonymous github username, but is this a wake up call for the package owners? Do we need at least some form of basic identity verification (even in the form of an employer link, LinkedIn, or similar)?
    - some_furry 3y ago
      No. I love contributing as my fursona sometimes. I don't want to dox myself to do so. We shouldn't need to know who you are to be verify that your contribution is trustworthy. Code should stand on its own merits. Ideas like this tend to dovetail into "real name policies", which suck https://www.eff.org/deeplinks/2014/09/facebooks-real-name-policy-can-cause-real-world-harm-lgbtq-community https://www.eff.org/deeplinks/2014/09/facebooks-real-name-po...
m3kw9 3y ago
I’ve seen some google zero day vulnerability hack analysis and those looked out of this world but this hack must be one of the all time
matt3210 3y ago
I hope there aren’t any like this which went unnoticed :-(
ogurechny 3y ago
As an armchair specialist, I don't support the “owning each ssh server on the planet” reactions. Public networks have too many eyes potentially (not just the admins responsible for specific systems). Traffic to ssh server that corresponds to suspicious activity but is also absent from logs would raise the alarms quite soon. Hooking the build process also means that the main targets are places that are important enough to have their own siloed software repositories. Getting into two LTS systems was a bonus. I suppose that simpler exploits are available for popular systems (if they run regular complex applications instead of being a minimal empty box that securely does nothing). The number of modification/decryption layers is probably not arbitrary. Most likely, each was added to evade detection by one or another protection system you and I don't know about, but with which attacker is familiar.
- raggi 3y ago
  Can you point to an URL where a system has a documented or implemented configuration that correlates traffic to log entries in a manner that can be alarmed on without just going off all the time? Junk packets to port 22 arrive all the time.
  - ogurechny 3y ago
    I'm not talking about random packets. There is global and local traffic collection (after all, each pimple on the face of humanity wants to have own little NSA after a wonderful advertisement campaign, and lots of “engineers” are OK with getting paid for that). So when someone capable enough matches obvious abnormal ssh traffic to some intermediate victim with later unwanted behavior of that system, and ties it to the general activities of some known long term actor, that system is going to be studied most thoroughly. Using this backdoor over the internet would mean risking getting busted after each single use. On the contrary, inside high profile “secure” networks such covert management channel would worth the price even if it's single use (as last measure shutdown switch, for example). I suppose “smart” readers can be as easily manipulated to look in any wrong direction as “dumb” ones. All stories mention what each decryption and indirection step does, but they don't explain why exactly those steps were chosen, that this one hides from intrusion detection system A, and that one prevents collection of runtime activity artifacts by tracing system B, and so on. On one hand, it would be impolite to announce that you study, and know well, how to break your colleague's product, on the other hand, some of those systems might be secret if we talk about state level.
TheSeeker11 3y ago
I was rather amused to see Chris Titus announce that "XZ Backdoor is NOT that bad!" He seems to have done next to no research into this https://www.youtube.com/watch?v=s_WkMtEPsuE https://www.youtube.com/watch?v=s_WkMtEPsuE
- rwmj 3y ago
  Never heard of him before but after watching 10 seconds he seems like an idiot.
- Delk 3y ago
  Skimming through it, it sounds like the video is about 8 minutes to say that the vulnerability only affected test versions of Debian and Fedora. Which is true, and the number of actual running systems that the backdoor already made its way into is likely not massive -- especially in terms of production or otherwise critical systems. But that doesn't mean it wasn't bad. It's rather a disaster that was only avoided pretty much by accident.
levleontiev 3y ago
Sorry if I trigger anyone's paranoia, but 1. who told you that it's not a state-backed attack? 2. and if 1 is true, who told it, it's not Russia staying behind it? and if 2 is true, the KGB university alumni Kaspersky definitely is related to it.
- thrdbndndn 3y ago
  And it's relevant to the article because?
  - fodmap 3y ago
    I suppose they say that because the article has been written by Kaspersky, and it's hosted on a server under their control?
    - thrdbndndn 3y ago
      Even let's go all the way down and say the backdoor was directly written by Kaspersky themselves, does it change the value of the content (which is purely technical analysis) anyhow?
      - fodmap 3y ago
        I suppose parent comment don't discuss the quality of the article but the theoretical danger of connecting to a server under Kaspersky control. https://en.wikipedia.org/wiki/Kaspersky_bans_and_allegations_of_Russian_government_ties https://en.wikipedia.org/wiki/Kaspersky_bans_and_allegations...
noname120 3y ago
What proves us that tukaani and jiaT75 aren't the same person? Creating a new nickname and handing write access to it sounds like the perfect way to shift blame and claim that his trust was betrayed once the backdoor is found out.
- account42 3y ago
  First please prove that you are not working with the people behind the compromise and are just trying to blame the original maintainer in order to misdirect. Oh you can't? Too bad, off to gitmo with you.
buster 3y ago
What I am really curious about is what caused the initial delay in ssh which was what triggered the research into xz in the first place. Has someone figured that out?
- pja 3y ago
  It's doing an extra ECC operation on every connection, but that shouldn't take 500ms on a modern CPU. The people who have reverse engineered the code say it also requires the command message to be bound to the ssh host key as well, so if the host key is an RSA key then it might be doing an extra RSA decryption operation on every connection as well? That would probably do it.
  - tux3 3y ago
    I believe it's all the ELF parsing and disassembly in memory that happens on startup. The really went crazy with the home-made x86 disassembler and everything they do with it. Must have been super fun to write though! Someone clearly had a great time coming up with clever ideas for a backdoor.
    - pja 3y ago
      That only happens once at startup though? IIRC the original detection was because sshd was using a lot more CPU coping with the torrent of spam connections any sshd on the internet gets than it usually would.
      - tux3 3y ago
        sshd forks a new clean process for each connection, so this whole machinery happens each time you connect. Each connection is a full fork/exec, and the backdoor has to set itself up from scratch again, including all the parsing and hooking.
        rsc 3y ago
        I have looked in the sshd code at https://github.com/openssh/openssh-portable https://github.com/openssh/openssh-portable and I cannot find it forking and re-execing _itself_. It forks and execs other commands, of course, and it forks to handle new connections but does not re-exec in those paths that I can see. If some inetd-like program were listening on port 22 and fork+exec'ing sshd to handle each incoming connection, that would explain it. But on my systemd-based Linux system I see a long-running sshd that appears to be taking care of port 22. I do agree that it seems like the best explanation of the delay is that somehow sshd was being exec'ed per connection, but I haven't seen all the dots connected yet.
nonane 3y ago
Has anyone done an analysis of the bug in the backdoor that caused valgrind errors and slowness in SSH which eventually exposed it?
- lyu07282 3y ago
  Invalid memory write apparently: https://www.mail-archive.com/valgrind-users@lists.sourceforge.net/msg07546.html https://www.mail-archive.com/valgrind-users@lists.sourceforg... The valgrind "fix" was to disable ifunc which disabled the backdoor so the errors went away. The slowness derived from all the symbol and instruction lookups it did afaik
  - itomato 3y ago
    Oxide and Friends interviewed Andres Freund for their show on Wednesday where he discussed the discovery, including the slowness from all the symbol translations https://open.spotify.com/episode/0C04JlCpMFe07C1XCswbZ9?si=B85xmm_IT3OWzexvzu2i6w https://open.spotify.com/episode/0C04JlCpMFe07C1XCswbZ9?si=B...
    - Aachen 3y ago
      Not everyone has Spotify, better just link their own site where you can play it directly: https://oxide.computer/podcasts/oxide-and-friends/1843393 https://oxide.computer/podcasts/oxide-and-friends/1843393
      - itomato 3y ago
        Handier for sure, but not on my clipboard
lyu07282 3y ago
The authors show a great deal of knowledge of glibc internals, the sort of things you really have to be neck deep in the source code for. Lots of novel techniques in there. The custom elf parser and disassembler are so complex that I can't imagine they didn't use that code / or will never use that code for something else in the future. I wonder if this gets the serious investigation it deserves, but I doubt it.
johnwheeler 3y ago
Here’s a twist - with the amount of effort the attacker went through to evade detection in their scripts and code, this whole project could function as a diversion or at least fallback to one while multiple efforts go on simultaneously. How do you get one step ahead of that? Does the community focusing on SSHD impact some other part of the system as a whole? Some other technological aspect? Social? Tin foil hats are fun!
- AndyMcConachie 3y ago
  [flagged]
  - glitchcrab 3y ago
    Conversely, do you have evidence against any of it?
- mathverse 3y ago
  We dont need any tinfoil hats. FOSS projects were being backdoored even before they became a thing the difference is that this time it's state sponsored actors doing it. The old groups GOBBLES,ADM,ac1db1tch3z,~el8 were doing it, private "security researchers" like isec.pl were doing it. This time it's a problem because state actors are abusing the corporate capitalism that created this era of underpaid people working on foundational projects. The bad actors have unlimited resources for their objectives. That's basically what created the demand and ineption of groups like NSO,Zerodium etc.. Basically before that exploits and backdoors were worthless and hackers hoped to be sponsored or hired by companies like Qualys.
  - xhkkffbf 3y ago
    How did "corporate capitalism" create this situation? Some guy decided to volunteer his time to write some code. If anything, it might be said that corporate capitalism paid him well enough so he had the spare time? And I don't see what money has to do with any of this. There could be some well-paid programmer slipping backdoors into proprietary code just as easily. The salary or openness didn't affect the outcome. It's just as easy for salaried programmers to betray their bosses and their users.
usr1106 3y ago
As far as the 3 first stages are concerned that article adds little to what has been known for 2 weeks. Just a nice write-up with flow charts. But the analysis of the binary in that detail seems new to me. So how has the source code shown there been produced? Running a disassember, understanding what the code does and renaming everything to descriptive names? That looks quite like quite an achievement in 2 weeks.
- r721 3y ago
  >Authors GReAT >Global Research & Analysis Team, Kaspersky Lab https://securelist.com/author/great/ https://securelist.com/author/great/ The article is apparently authored by malware analysis team from Kaspersky Lab, so they are probably quite good at reverse engineering binaries.
  - usr1106 3y ago
    Sure. But that does not answer any questions about their tooling or methods. Considered a business secret? They present the readable C code of the binary payload like it has appeared just out of the blue.
    - saagarjha 3y ago
      They're using IDA Pro, with names that they presumably came up with themselves by analyzing what the code does.
- giantpotato 3y ago
  The tool used in those white screenshots is called IDA pro, a decompiler. https://hex-rays.com/ida-pro/ https://hex-rays.com/ida-pro/
robryk 3y ago
> The only variable inside its own table is yolAbejyiejuvnup=Evjtgvsh5okmkAvj, which in this context serves as a kill switch. This seems false? The trie contained also strings like `WAYLAND_DISPLAY=`, so IIUC the backdoor would remain dormant if someone tried to run sshd from something that looks like an interactive session.
phatfish 3y ago
It's a helpful collection of information on this topic. But Kaspersky... really? No one with any sense has that voluntary backdoor in their network.
schuyler2d 3y ago
Quite rightly most attention has been on the operational success of the backdoor and how it achieves it's goals. But I would love some deeper analysis of the mistakes and over-engineering parts. In the Bryan Cantril interview [1] Andrés suggests there are a lot of dumb things done because it was a kind of off-the-shelf backdooring piece that didn't necessarily know how it would be deployed. Eg the symbol table lookup that led him to investigate. Similarly, why were they chopping off 48 bytes using RC4 [2] So I'd love folks to talk about how (given more time/better team) this could have been better or where they messed up more [1] https://youtu.be/jg5F9UupL6I?si=gvXsYFXgagkGOMd4 https://youtu.be/jg5F9UupL6I?si=gvXsYFXgagkGOMd4 [2] https://twitter.com/matthew_d_green/status/1774472908020101420 https://twitter.com/matthew_d_green/status/17744729080201014...
HankB99 3y ago
I'm curious about the technical skills needed to pull this off. I'm not a maintainer and have never built a Debian package. It appears to me that it required a fairly deep knowledge of the package build pipeline to accomplish this. Is that well enough documented somewhere that someone with a technical background and desire to learn could pick it up? What's the possibility that this person (or team) has been an active participant in the Debian community? Likewise with the RH technology and team, if that differs.
- chuckadams 3y ago
  The exploit obviously took someone with lots of black hat experience to write. The trie-based string comparison trick is particularly clever, though even that is fairly simple compared to the stuff that comes out of the likes of the NSO group (check out the Wikipedia article for "Weird Machines", that's the sort of stuff they write). The build process of the exploit is also somewhat interesting, but what kicks it off is just an extra m4 file that autogen picks up and dutifully runs with its out of the box settings. That extra file was just added by hand to the dist tarballs, since the server was also owned by the attacker. It doesn't appear in the repo anywhere. Five minutes of reading autoconf docs and you too can create an autoconf trojan with ease. The various Jia Tan sockpuppets were actively pushing both Debian and Redhat to include the backdoored version of xz, so the answer to your last question is yes, they are in fact active participants in the community.
xg15 3y ago
> Unlike other supply chain attacks we have seen in Node.js, PyPI, FDroid, and the Linux Kernel that mostly consisted of atomic malicious patches, fake packages and typosquatted package names, this incident was a multi-stage operation that almost succeeded in compromising SSH servers on a global scale. I think this sentence actually understates what happened. What I find more frightening than the technical aspects of the backdoor is the amount and degree of "social engineering" involved: The backdoor was the final product, but inserting it was only possible because at that point the entire xz project had long been taken over by malicious actors ("Jia Tan" and friends) - after a yearlong psychological operation against the maintainer. All that without the maintainer or anyone else noticing anything. That's spy novel stuff - and if something like this is possible, it makes me wonder what else might be going on in other projects right now. (You can see the same mentality reflected in the backdoor code itself - I find it remarkable how much of the code is devoted to active misdirection. It's not just trying to look harmless, the code actively tries to build a "narrative" through commit messages, comments, variable names, choice of commands, etc, of what it's ostensibly doing while really doing something completely different - leading anyone who'd inspect the code first to doubt themselves and their understanding of the code, then suspect a bug and then, maybe, suspect malicious intent)
- npteljes 3y ago
  >leading anyone who'd inspect the code first to doubt themselves and their understanding of the code I completely agree. What seems incredible to me is that how much manipulation, thoughtfulness, patience, perseverance went into this. It's either a product of the obsession of someone, or a private security firm's or state actor's doing, who do these things to multiple projects as part of a 9-5 job.
- abkolan 3y ago
  Yes, this absolutely mind boggling. Not insinuating or suggesting anything, but if I could wear my tinfoil hat for a bit. I wonder if there was a significant psy-op conducted IRL so that the original author couldn't devote time and would eventually handover ownership to bad actors. I hope some three letter agencies are conducting a deeper investigation into this.
- 3y ago
rvba 3y ago
How can we know if not all other blobs in Linux are not spyware? I think there is a small de-blobbed kernel but it is unpopular

406 points by kryster 3y ago

Animats 3y ago

Why is "hooking" allowed at all in non-debug code? The legit applications all involve tracing or profiling.

rwmj 3y ago
What does "non-debug" code mean in this context? Most distros compile everything with -g and then separate out the debug symbols into a different package, because you almost always want to be able to attach gdb to any process when something goes wrong (and no, recompiling at that point is a much worse idea, because it means installing compilers on your production system as well as not being able to debug what was actually running). It doesn't matter anyway, because once you're running inside a process you can do most anything you like. This was a convenient mechanism, but there are scores of ways to scan over the symbols and modify code inside the same process.
- Animats 3y ago
  "You almost always want to be able to attach gdb to any process when something goes wrong". Now, this is why we can't have secure systems. Somehow, I've never needed to attach GDB to a running process. Start up programs under GDB, yes. Read crash dumps, yes. If you're using a debugger in a production system, something has gone horribly wrong.
  - troad 3y ago
    > If you're using a debugger in a production system, something has gone horribly wrong. That's right, hence the debugger.
  - rwmj 3y ago
    Or you want to capture accurate core dumps for a tool like ABRT. We (Red Hat) work very closely with customers to find out what is and isn't acceptable to be running on their production systems, including customers who store very sensitive data.
  - cozzyd 3y ago
    How do you debug rare hangs?
saagarjha 3y ago
Hooking of what?
- Animats 3y ago
  Live-patching code so that a call to something becomes a call to something else.
  - saagarjha 3y ago
    How are you going to prevent this in general? At some point you’re going to make an indirect call.

andy_ppp 3y ago

Imagine all the back doors people haven’t discovered…

ta1243 3y ago
Especially in closed source software where nobody has an incentive to look, any performance hits will be ignored, code analysis tools can't be run (why bother - can't exactly sell it as a new feature)

throwaway67743 3y ago

[flagged]

StressedDev 3y ago
The analysis post was very good and did a great job of explaining how the attacker (hacker) obfuscated the backdoor.
eclectic29 3y ago
Did you create a throwaway account specifically to write a snarky comment here? Looks like this practice is becoming more common.
- dredmorbius 3y ago
  It also seems to specifically violate HN guidelines: Throwaway accounts are ok for sensitive information, but please don't create accounts routinely. HN is a community—users should have an identity that others can relate to. <https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html> Snarky shallow dismissal ain't sensitive information. It's precisely the sort of behaviour which should be tied to, and reflect reputation of, a primary account. (Which can of course be pseudonymous, example myself.)
- saagarjha 3y ago
  Their account seems too old for that.
sho_hn 3y ago
This one contained information I didn't encounter in the earlier ones linked from HN. The earlier ones mostly focused on the multi-step build and extraction process, this one has some detail on the inner workings of the library that gets extracted.
Analemma_ 3y ago
This is one of the most significant computer security events of all time. A genuine backdoor-- not an accidental vulnerability, a backdoor-- created by a very sophisticated, very-likely state-backed, actor who used a combination of social engineering and multiple clever points of indirection to get a vulnerability into a common library. There will be many more articles about it in the months and years to come, and I don't see what's wrong with that as long as they are providing new information.

vlovich123 3y ago

So if I'm understanding correctly, a useful hardening step would be to ensure that each dynamically linked library had it's own GOT & that the tables are marked as read-only once dynamic linking completes (i.e. you can't reach across dynamic boundaries to patch someone else's ifunc entries). That would improve the supply chain security of code getting linked in somewhere but never executed. EDIT: Or even better, perhaps ifunc should be implemented in a declarative fashion so that you can't just cause arbitrary code execution from each library you link against. That might be harder to implement at this point considering back compat, but probably is something that could be layered in over a longer time period (i.e. if you build any library with "declaratively linked ifunc" feature bit set, then the dynamic linker will force all linked libraries to have that feature flag or fail to launch).

jerjerjer 3y ago
From another angle: build systems. Currently building most libraries usually involves executing a series of highly complex byzantine scripts, requiring turing-complete environment. This gives attacker an endless attack surface, and when the build process is hijacked - well, the opportunities are there. Moving to a declarative build process with only a limited state machine as an executor would help. Requiring all source blobs being reproducible might also be something to think about.
- deleted 3y ago
  [deleted]
- kimixa 3y ago
  Having a "simple" declarative build definition won't help if the thing that interprets and executes that definition is shipped with the package, as said interpreter is likely orders of magnitude more complex and harder to personally review. As is what happened with the xz example - the "attack" was hidden in code that is autogenerated by autotools, not the build or configuration definitions. People put trust in distros and packagers to having something of a review chain - there's 0% chance you personally be an expert in everything executing on your workstation right now (outside of maybe toy systems). I'm not an expert in m4 or bash, but I hope enough experts are in the chain to get to my distro's package library are that such things are less likely. But that is all bypassed here. I think this particular approach is a one-off, as I know of no other build environment where it's expected to have the generated executable of the build system "helpfully" packaged in the tarball as a packaging step. If it is in some I'm not aware of, I hope that decision is being re-examined now.

dvsfish 3y ago

Great read - the lengths the attacker went to to not only hide it but also build the trust to be able to pull this off is incredible. So calculated. One can only imagine they felt a great deal of panic and despair as the security community lit up the issue, regardless of a near zero chance of repercussions. Very clever - those involved in catching this are, quietly, heroes

jerjerjer 3y ago
The use of trie structure to compare strings without hardcoding raw, searchable string values is very inspired. But yes, some black hat had a really really bad day: being so close to owning any ssh-enabled recently-updated box and when they can finally see a finish line it is just poof... gone in a day. It was two years of effort from at least one guy and more likely a team.
- jumelles 3y ago
  Honestly? Good. Anyone doing this sort of work willingly deserves worse.
  - optimalsolver 3y ago
    That they were harassing and manipulating a lone, unthanked maintainer who had already told them he was dealing with mental issues makes them evil, IMO. The honorable thing for "Jia" to do after this epic failure is seppuku, or whatever his or her local equivalent is.
    - cqqxo4zV46cp 3y ago
      Nobody sees themselves as the bad guy, and that’s not the same as “some people are just fundamentally selfish”. There are definitely loads of people that’d feel like the end justifies the means. There are plenty of people for whom a workday involves doing far worse things for the world than cyberbullying one person, and will look you in the eye and justify it. Plenty of that stuff is socially acceptable in many many mainstream circles. “Being mean to a maintainer” is just one that this community is especially sensitive to, because it involves a highly personified victim that they can relate to.

deleted 3y ago

[deleted]

asveikau 3y ago

I saw the xz repo is back on GitHub, Lasse and a new contributor were cleaning it up. They removed ifunc support and they committed code to generate test files into the repo, so that they can do test files without blobs. It looks like they're doing good stuff.

saagarjha 3y ago
> a new contributor Jia Tan, is that you?
- asveikau 3y ago
  Very funny. This one: https://github.com/tukaani-project/xz/commits?author=thesamesam https://github.com/tukaani-project/xz/commits?author=thesame... https://github.com/thesamesam https://github.com/thesamesam Seems like he's a Gentoo contributor.
  - asddubs 3y ago
    oh no, gentoo has been compromised
- willsmith72 3y ago
  I know part of the fun of oss is anyone can contribute behind an anonymous github username, but is this a wake up call for the package owners? Do we need at least some form of basic identity verification (even in the form of an employer link, LinkedIn, or similar)?
  - some_furry 3y ago
    No. I love contributing as my fursona sometimes. I don't want to dox myself to do so. We shouldn't need to know who you are to be verify that your contribution is trustworthy. Code should stand on its own merits. Ideas like this tend to dovetail into "real name policies", which suck https://www.eff.org/deeplinks/2014/09/facebooks-real-name-policy-can-cause-real-world-harm-lgbtq-community https://www.eff.org/deeplinks/2014/09/facebooks-real-name-po...

m3kw9 3y ago

I’ve seen some google zero day vulnerability hack analysis and those looked out of this world but this hack must be one of the all time

matt3210 3y ago

I hope there aren’t any like this which went unnoticed :-(

ogurechny 3y ago

As an armchair specialist, I don't support the “owning each ssh server on the planet” reactions. Public networks have too many eyes potentially (not just the admins responsible for specific systems). Traffic to ssh server that corresponds to suspicious activity but is also absent from logs would raise the alarms quite soon. Hooking the build process also means that the main targets are places that are important enough to have their own siloed software repositories. Getting into two LTS systems was a bonus. I suppose that simpler exploits are available for popular systems (if they run regular complex applications instead of being a minimal empty box that securely does nothing). The number of modification/decryption layers is probably not arbitrary. Most likely, each was added to evade detection by one or another protection system you and I don't know about, but with which attacker is familiar.

raggi 3y ago
Can you point to an URL where a system has a documented or implemented configuration that correlates traffic to log entries in a manner that can be alarmed on without just going off all the time? Junk packets to port 22 arrive all the time.
- ogurechny 3y ago
  I'm not talking about random packets. There is global and local traffic collection (after all, each pimple on the face of humanity wants to have own little NSA after a wonderful advertisement campaign, and lots of “engineers” are OK with getting paid for that). So when someone capable enough matches obvious abnormal ssh traffic to some intermediate victim with later unwanted behavior of that system, and ties it to the general activities of some known long term actor, that system is going to be studied most thoroughly. Using this backdoor over the internet would mean risking getting busted after each single use. On the contrary, inside high profile “secure” networks such covert management channel would worth the price even if it's single use (as last measure shutdown switch, for example). I suppose “smart” readers can be as easily manipulated to look in any wrong direction as “dumb” ones. All stories mention what each decryption and indirection step does, but they don't explain why exactly those steps were chosen, that this one hides from intrusion detection system A, and that one prevents collection of runtime activity artifacts by tracing system B, and so on. On one hand, it would be impolite to announce that you study, and know well, how to break your colleague's product, on the other hand, some of those systems might be secret if we talk about state level.

TheSeeker11 3y ago

I was rather amused to see Chris Titus announce that "XZ Backdoor is NOT that bad!" He seems to have done next to no research into this https://www.youtube.com/watch?v=s_WkMtEPsuE https://www.youtube.com/watch?v=s_WkMtEPsuE

rwmj 3y ago
Never heard of him before but after watching 10 seconds he seems like an idiot.
Delk 3y ago
Skimming through it, it sounds like the video is about 8 minutes to say that the vulnerability only affected test versions of Debian and Fedora. Which is true, and the number of actual running systems that the backdoor already made its way into is likely not massive -- especially in terms of production or otherwise critical systems. But that doesn't mean it wasn't bad. It's rather a disaster that was only avoided pretty much by accident.

levleontiev 3y ago

Sorry if I trigger anyone's paranoia, but 1. who told you that it's not a state-backed attack? 2. and if 1 is true, who told it, it's not Russia staying behind it? and if 2 is true, the KGB university alumni Kaspersky definitely is related to it.

thrdbndndn 3y ago
And it's relevant to the article because?
- fodmap 3y ago
  I suppose they say that because the article has been written by Kaspersky, and it's hosted on a server under their control?
  - thrdbndndn 3y ago
    Even let's go all the way down and say the backdoor was directly written by Kaspersky themselves, does it change the value of the content (which is purely technical analysis) anyhow?
    - fodmap 3y ago
      I suppose parent comment don't discuss the quality of the article but the theoretical danger of connecting to a server under Kaspersky control. https://en.wikipedia.org/wiki/Kaspersky_bans_and_allegations_of_Russian_government_ties https://en.wikipedia.org/wiki/Kaspersky_bans_and_allegations...

noname120 3y ago

What proves us that tukaani and jiaT75 aren't the same person? Creating a new nickname and handing write access to it sounds like the perfect way to shift blame and claim that his trust was betrayed once the backdoor is found out.

account42 3y ago
First please prove that you are not working with the people behind the compromise and are just trying to blame the original maintainer in order to misdirect. Oh you can't? Too bad, off to gitmo with you.

buster 3y ago

What I am really curious about is what caused the initial delay in ssh which was what triggered the research into xz in the first place. Has someone figured that out?

pja 3y ago
It's doing an extra ECC operation on every connection, but that shouldn't take 500ms on a modern CPU. The people who have reverse engineered the code say it also requires the command message to be bound to the ssh host key as well, so if the host key is an RSA key then it might be doing an extra RSA decryption operation on every connection as well? That would probably do it.
- tux3 3y ago
  I believe it's all the ELF parsing and disassembly in memory that happens on startup. The really went crazy with the home-made x86 disassembler and everything they do with it. Must have been super fun to write though! Someone clearly had a great time coming up with clever ideas for a backdoor.
  - pja 3y ago
    That only happens once at startup though? IIRC the original detection was because sshd was using a lot more CPU coping with the torrent of spam connections any sshd on the internet gets than it usually would.
    - tux3 3y ago
      sshd forks a new clean process for each connection, so this whole machinery happens each time you connect. Each connection is a full fork/exec, and the backdoor has to set itself up from scratch again, including all the parsing and hooking.
      - rsc 3y ago
        I have looked in the sshd code at https://github.com/openssh/openssh-portable https://github.com/openssh/openssh-portable and I cannot find it forking and re-execing _itself_. It forks and execs other commands, of course, and it forks to handle new connections but does not re-exec in those paths that I can see. If some inetd-like program were listening on port 22 and fork+exec'ing sshd to handle each incoming connection, that would explain it. But on my systemd-based Linux system I see a long-running sshd that appears to be taking care of port 22. I do agree that it seems like the best explanation of the delay is that somehow sshd was being exec'ed per connection, but I haven't seen all the dots connected yet.

nonane 3y ago

Has anyone done an analysis of the bug in the backdoor that caused valgrind errors and slowness in SSH which eventually exposed it?

lyu07282 3y ago
Invalid memory write apparently: https://www.mail-archive.com/valgrind-users@lists.sourceforge.net/msg07546.html https://www.mail-archive.com/valgrind-users@lists.sourceforg... The valgrind "fix" was to disable ifunc which disabled the backdoor so the errors went away. The slowness derived from all the symbol and instruction lookups it did afaik
- itomato 3y ago
  Oxide and Friends interviewed Andres Freund for their show on Wednesday where he discussed the discovery, including the slowness from all the symbol translations https://open.spotify.com/episode/0C04JlCpMFe07C1XCswbZ9?si=B85xmm_IT3OWzexvzu2i6w https://open.spotify.com/episode/0C04JlCpMFe07C1XCswbZ9?si=B...
  - Aachen 3y ago
    Not everyone has Spotify, better just link their own site where you can play it directly: https://oxide.computer/podcasts/oxide-and-friends/1843393 https://oxide.computer/podcasts/oxide-and-friends/1843393
    - itomato 3y ago
      Handier for sure, but not on my clipboard

lyu07282 3y ago

The authors show a great deal of knowledge of glibc internals, the sort of things you really have to be neck deep in the source code for. Lots of novel techniques in there. The custom elf parser and disassembler are so complex that I can't imagine they didn't use that code / or will never use that code for something else in the future. I wonder if this gets the serious investigation it deserves, but I doubt it.

johnwheeler 3y ago

Here’s a twist - with the amount of effort the attacker went through to evade detection in their scripts and code, this whole project could function as a diversion or at least fallback to one while multiple efforts go on simultaneously. How do you get one step ahead of that? Does the community focusing on SSHD impact some other part of the system as a whole? Some other technological aspect? Social? Tin foil hats are fun!

AndyMcConachie 3y ago
[flagged]
- glitchcrab 3y ago
  Conversely, do you have evidence against any of it?
mathverse 3y ago
We dont need any tinfoil hats. FOSS projects were being backdoored even before they became a thing the difference is that this time it's state sponsored actors doing it. The old groups GOBBLES,ADM,ac1db1tch3z,~el8 were doing it, private "security researchers" like isec.pl were doing it. This time it's a problem because state actors are abusing the corporate capitalism that created this era of underpaid people working on foundational projects. The bad actors have unlimited resources for their objectives. That's basically what created the demand and ineption of groups like NSO,Zerodium etc.. Basically before that exploits and backdoors were worthless and hackers hoped to be sponsored or hired by companies like Qualys.
- xhkkffbf 3y ago
  How did "corporate capitalism" create this situation? Some guy decided to volunteer his time to write some code. If anything, it might be said that corporate capitalism paid him well enough so he had the spare time? And I don't see what money has to do with any of this. There could be some well-paid programmer slipping backdoors into proprietary code just as easily. The salary or openness didn't affect the outcome. It's just as easy for salaried programmers to betray their bosses and their users.

usr1106 3y ago

As far as the 3 first stages are concerned that article adds little to what has been known for 2 weeks. Just a nice write-up with flow charts. But the analysis of the binary in that detail seems new to me. So how has the source code shown there been produced? Running a disassember, understanding what the code does and renaming everything to descriptive names? That looks quite like quite an achievement in 2 weeks.

r721 3y ago
>Authors GReAT >Global Research & Analysis Team, Kaspersky Lab https://securelist.com/author/great/ https://securelist.com/author/great/ The article is apparently authored by malware analysis team from Kaspersky Lab, so they are probably quite good at reverse engineering binaries.
- usr1106 3y ago
  Sure. But that does not answer any questions about their tooling or methods. Considered a business secret? They present the readable C code of the binary payload like it has appeared just out of the blue.
  - saagarjha 3y ago
    They're using IDA Pro, with names that they presumably came up with themselves by analyzing what the code does.
giantpotato 3y ago
The tool used in those white screenshots is called IDA pro, a decompiler. https://hex-rays.com/ida-pro/ https://hex-rays.com/ida-pro/

robryk 3y ago

> The only variable inside its own table is yolAbejyiejuvnup=Evjtgvsh5okmkAvj, which in this context serves as a kill switch. This seems false? The trie contained also strings like `WAYLAND_DISPLAY=`, so IIUC the backdoor would remain dormant if someone tried to run sshd from something that looks like an interactive session.

phatfish 3y ago

It's a helpful collection of information on this topic. But Kaspersky... really? No one with any sense has that voluntary backdoor in their network.

schuyler2d 3y ago

Quite rightly most attention has been on the operational success of the backdoor and how it achieves it's goals. But I would love some deeper analysis of the mistakes and over-engineering parts. In the Bryan Cantril interview [1] Andrés suggests there are a lot of dumb things done because it was a kind of off-the-shelf backdooring piece that didn't necessarily know how it would be deployed. Eg the symbol table lookup that led him to investigate. Similarly, why were they chopping off 48 bytes using RC4 [2] So I'd love folks to talk about how (given more time/better team) this could have been better or where they messed up more [1] https://youtu.be/jg5F9UupL6I?si=gvXsYFXgagkGOMd4 https://youtu.be/jg5F9UupL6I?si=gvXsYFXgagkGOMd4 [2] https://twitter.com/matthew_d_green/status/1774472908020101420 https://twitter.com/matthew_d_green/status/17744729080201014...

HankB99 3y ago

I'm curious about the technical skills needed to pull this off. I'm not a maintainer and have never built a Debian package. It appears to me that it required a fairly deep knowledge of the package build pipeline to accomplish this. Is that well enough documented somewhere that someone with a technical background and desire to learn could pick it up? What's the possibility that this person (or team) has been an active participant in the Debian community? Likewise with the RH technology and team, if that differs.

chuckadams 3y ago
The exploit obviously took someone with lots of black hat experience to write. The trie-based string comparison trick is particularly clever, though even that is fairly simple compared to the stuff that comes out of the likes of the NSO group (check out the Wikipedia article for "Weird Machines", that's the sort of stuff they write). The build process of the exploit is also somewhat interesting, but what kicks it off is just an extra m4 file that autogen picks up and dutifully runs with its out of the box settings. That extra file was just added by hand to the dist tarballs, since the server was also owned by the attacker. It doesn't appear in the repo anywhere. Five minutes of reading autoconf docs and you too can create an autoconf trojan with ease. The various Jia Tan sockpuppets were actively pushing both Debian and Redhat to include the backdoored version of xz, so the answer to your last question is yes, they are in fact active participants in the community.

xg15 3y ago

> Unlike other supply chain attacks we have seen in Node.js, PyPI, FDroid, and the Linux Kernel that mostly consisted of atomic malicious patches, fake packages and typosquatted package names, this incident was a multi-stage operation that almost succeeded in compromising SSH servers on a global scale. I think this sentence actually understates what happened. What I find more frightening than the technical aspects of the backdoor is the amount and degree of "social engineering" involved: The backdoor was the final product, but inserting it was only possible because at that point the entire xz project had long been taken over by malicious actors ("Jia Tan" and friends) - after a yearlong psychological operation against the maintainer. All that without the maintainer or anyone else noticing anything. That's spy novel stuff - and if something like this is possible, it makes me wonder what else might be going on in other projects right now. (You can see the same mentality reflected in the backdoor code itself - I find it remarkable how much of the code is devoted to active misdirection. It's not just trying to look harmless, the code actively tries to build a "narrative" through commit messages, comments, variable names, choice of commands, etc, of what it's ostensibly doing while really doing something completely different - leading anyone who'd inspect the code first to doubt themselves and their understanding of the code, then suspect a bug and then, maybe, suspect malicious intent)

npteljes 3y ago
>leading anyone who'd inspect the code first to doubt themselves and their understanding of the code I completely agree. What seems incredible to me is that how much manipulation, thoughtfulness, patience, perseverance went into this. It's either a product of the obsession of someone, or a private security firm's or state actor's doing, who do these things to multiple projects as part of a 9-5 job.
abkolan 3y ago
Yes, this absolutely mind boggling. Not insinuating or suggesting anything, but if I could wear my tinfoil hat for a bit. I wonder if there was a significant psy-op conducted IRL so that the original author couldn't devote time and would eventually handover ownership to bad actors. I hope some three letter agencies are conducting a deeper investigation into this.
3y ago

rvba 3y ago

How can we know if not all other blobs in Linux are not spyware? I think there is a small de-blobbed kernel but it is unpopular