9 ms·

The weird thing about this one is how it seems super professional in some ways, and rather amateur in others. Professional in the sense of spending a long time

by ufmace 3y ago

The weird thing about this one is how it seems super professional in some ways, and rather amateur in others. Professional in the sense of spending a long time building up an identity that seemed trustworthy enough to be made maintainer of an important package, of probably involving multiple people in social manipulation attacks, of not leaking the true identity and source of the attack, and the sophistication and obfuscation techniques used. Yet also a bit amateur-ish in the bugs and performance regressions that slipped out into production versions. I'm not saying it's amateur-ish to have bugs. I'm saying, if this was developed by a highly competent state-sponsored organization, you'd think they would have developed the actual exploit and tested it heavily behind closed doors, fixing all of the bugs and ensuring there were no suspicion-creating performance regressions before any of it was submitted into a public project. If there was no performance regression, much higher chance this never would have been discovered at all.

darkclouds 3y ago
[dead]
braiamp 3y ago
There are thousands of ways that performance can be impacted. No matter how good you are at developing, there will be a workload that would have a performance hit. Phoronix has been several times reporting issues to the Linux kernel because performance regression with their test suite. Performance tests tend to take more time than correctness tests.
- ufmace 3y ago
  Not seeing that as a point. It's probably not possible to have no performance hit whatsoever when you're checking the exact nanosecond count of every little thing. But usually nobody is doing that. It shouldn't be hard to not cause a substantial enough performance regression in SSHD logins that somebody who wasn't already monitoring that would notice and decide to dig into what's going on. I'm not sure if it's been revealed yet what this thing actually does, but it seems like all it really needs to do is to check for some kind of special token in the auth request or check against another keypair.
  - deleted 3y ago
    [deleted]
heyoni 3y ago
I thought performance was actually fine? It only dragged when using valgrind, hence the rhetoric that it took some really unlikely circumstances for it to be detected that quickly.
- jwilk 3y ago
  No, it wasn't fine. From the advisory (https://news.ycombinator.com/item?id=39865810 https://news.ycombinator.com/item?id=39865810): > With the backdoored liblzma installed, logins via ssh become a lot slower.
  - heyoni 3y ago
    Yea you're right. 500ms vs 10ms on an older server. Was thrown off by this statement and thought only the perf/valgrind/gdb attachments were what really brought it to surface. > Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked. This appears to be part of some countermeasures to make analysis harder. Performance was much worse but not enough to actually nudge the author into digging into it until the random ssh logins started piling up: https://twitter.com/AndresFreundTec/status/1774190743776866374 https://twitter.com/AndresFreundTec/status/17741907437768663... https://nitter.poast.org/AndresFreundTec/status/1774190743776866374 https://nitter.poast.org/AndresFreundTec/status/177419074377...
  - olejorgenb 3y ago
    2-3x slower 0m0.299s -> 0m0.807s [1] [1] https://www.openwall.com/lists/oss-security/2024/03/29/4 https://www.openwall.com/lists/oss-security/2024/03/29/4
  - rstuart4133 3y ago
    It was noticed because of it's CPU usage. It seems most people think "that was lucky". It wasn't lucky, it was inevitable. It happened to be noticed by someone doing benchmarking, but there is another category of machines out there it would hit hard: low end VM's running an ssh server. I've had my low end VM's hit by exactly the same thing the postgres benchmarker saw, but on these box's it isn't just annoying. There are so many ssh logins happening the OOM killer comes out to play but these ssh logins are tiny so it tends to kill the real stuff you've got the box doing, like web serving. A 2 or 3 fold increase in CPU would suddenly bring a lot of VM to their knees. At least thousands of them. It would be noticed.
rgmerk 3y ago
Events can happen to anyone, even competent state-sponsored organisations. And intelligence agencies are sometimes rather less ruthlessly competent than imagined (Kremlin assisinations in the UK have been a comedy of errors [1]). Maybe another backdoor, or alternative access mechanism they were using, got closed and they wanted another one in a hurry. [1] https://en.wikipedia.org/wiki/Poisoning_of_Alexander_Litvinenko https://en.wikipedia.org/wiki/Poisoning_of_Alexander_Litvine...
- readyplayernull 3y ago
  And they are getting better at it: https://www.bbc.com/news/world-us-canada-68706317 https://www.bbc.com/news/world-us-canada-68706317
- cesarb 3y ago
  > Maybe another backdoor, or alternative access mechanism they were using, got closed and they wanted another one in a hurry. Or maybe the opportunity window for the mechanism this backdoor would use was closing. According to the timeline at https://research.swtch.com/xz-timeline https://research.swtch.com/xz-timeline there was a github comment from poettering at the end of January which implied that the relevant library would be changed soon to lazy load liblzma ("[...] Specifically the compression libs, i.e. libbz2, libz4 and suchlike at least. And also libpam. So expect more like this sooner or later."), as in fact happened a month later. The attacker had to get the backdoor into the targeted Linux distributions before the systemd change got into them. Of course, the attacker could instead take the loss and abandon the approach, but since they had written that amount of complex code, it probably felt hard to throw it all away.
- ifwinterco 3y ago
  Russia wants everyone to know when they do assassinations abroad, that's why they use obviously-Russian methods like polonium and novichok. It's designed to send a message to other Russian dissidents and remind them that they aren't safe
richin13 3y ago
I read somewhere that some recent changes in systems would've made the backdoor useless so they had to rush out, which caused them to be reckless and get discovered
- sho_hn 3y ago
  This refers to the fact that systemd was planning to drop the dependency on liblzma (the conpression library installed by xz), and instead dlopen it at runtime when needed. Not for security reasons, but to avoid pulling the libs into initramfs images. The backdoor relies on sshd being patched to depend on libsystemd to call sd_notify(), which several distros had done. OpenSSH has since merged a new patch upstream that implements similar logic to sd_notify() in sshd itself to allow distros to drop that patch. So the attack surface of both sshd and libsystemd has since shrunk a bit.
  - rav 3y ago
    > The backdoor relies on sshd being patched to depend on libsystemd to call sd_notify I remember when we added sd_notify support to our services at work, I was wondering why one would pull in libsystemd as a dependency for this. I mean, there's a pure-Python library [1] that basically boils down to: import os, socket def notify(state=b"READY=1"): sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) addr = os.getenv('NOTIFY_SOCKET') if addr[0] == '@': addr = '\0' + addr[1:] sock.connect(addr) sock.sendall(state) With proper error handling, that's about 50 lines of C code. I would vendor that into my application in a heartbeat. [1]: https://raw.githubusercontent.com/bb4242/sdnotify/master/sdnotify/__init__.py https://raw.githubusercontent.com/bb4242/sdnotify/master/sdn...
    - cesarb 3y ago
      > I was wondering why one would pull in libsystemd as a dependency for this. I mean, there's a pure-Python library [...] With proper error handling, that's about 50 lines of C code. There's also a pure C library (libsystemd itself) which already does all that, and you don't need to test all the error handling cases in your 50 lines of C code. It makes sense to use the battle-tested code, instead of writing your own.
dabbledabble 3y ago
It’s also possible that this could be a change in personnel. Maybe the one who earned trust and took over was no more working for them. And an amateur took over with tight deadlines that lead to this gaffe for them.
- versteegen 3y ago
  The abrupt change in time-of-day when commits occurred supports the theory that Jai Tan is more than one person: https://twitter.com/birchb0y/status/1773871381890924872 https://twitter.com/birchb0y/status/1773871381890924872
  - swid 3y ago
    The text near the box makes it sound like these are just the fixes - not adding the test files but updating them. At that point it would have been clear “the race is on” to avoid detection, so it’s not too surprising someone would work late to salvage the operation.
    - versteegen 3y ago
      Whoops, you're right. So this isn't really evidence of anything. Out of interest I looked up the other commit at that time of day visible in that graph, laying on the arrow. It's [1], which changes the git URL from git.tukaani.org to github.com. Of course, moving the project hosting to github was part of the attack. [1] https://git.tukaani.org/?p=xz.git;a=commitdiff;h=e1b1a9d6370b788bd6078952c6c201e12bc27cbf https://git.tukaani.org/?p=xz.git;a=commitdiff;h=e1b1a9d6370...
  - hinkley 3y ago
    If the mistakes align with the time of day change, perhaps the author had a distraction that pushed the hours and compromised judgement.
manquer 3y ago
Why do we assume the person building the trust is the attacker ? Is not possible the attacker simply took over the account of some one genuinely getting involved in the community either hacked or just with $5 wrench and then committed the malicious code ?
- ericpruitt 3y ago
  > Is not possible the attacker simply took over the account of some one genuinely getting involved in the community either hacked or just with $5 wrench and then committed the malicious code ? Given the behavior of the accounts that applied pressure on the original xz maintainer, this seems unlikely to me.
  - ilvez 3y ago
    Or they just bought the guy at one point, because I understood the malicious behaviour started quite recently.
    - timschmidt 3y ago
      Jia Tan's very first commit two years ago made suspicious changes to liblzma. See the timeline here: https://boehs.org/node/everything-i-know-about-the-xz-backdoor https://boehs.org/node/everything-i-know-about-the-xz-backdo...
tw04 3y ago
The fact that the guys developing the code weren't also simultaneously running valgrind and watching performance isn't hard to believe. They were targeting servers and appliances, how many servers and appliances do you know of that are running valgrind in their default image? Sure, in hindsight that's a "duh, why didn't we think of that" - but also it's not very hard at all to see why they didn't think of that. They were likely testing against the system images they were hoping to compromise, not joe-schmoe developer's custom image.
- dralley 3y ago
  In theory they should probably be testing against the CI pipelines of Debian and Fedora / CentOS, as that's the moat their backdoor has to cross.
  - tw04 3y ago
    They put code in to, in theory, avoid running in a development environment.
asveikau 3y ago
Modifying the sshd behavior without crashes seems by itself pretty difficult. I mean, conceptually it isn't hard, if you are in the same process and can patch various functions, but I think doing so and having it be "production ready" to ship in multiple linux distros all the time is a challenge. This thing wasn't around for very long but yet another thing to consider would be to keep it working across multiple versions of OpenSSH, libcrypto etc.
- nick238 3y ago
  I picture some division in [nation-state] where they're constantly creating personas, slowly working all sorts of languishing open source packages with few maintainers (this is the actual hard, very slow part), then once they have a bit of an in, they could recruit more technical expertise. The division is run by some evil genius who knows this could pay off big, but others are skeptical, so their resources are pretty minimal.
  - timschmidt 3y ago
    Moxie's reasons for disallowing Signal distribution via F-droid always rang a little flat to me ( https://github.com/signalapp/Signal-Android/issues/127 https://github.com/signalapp/Signal-Android/issues/127 ). Lots of chatter about the supposedly superior security model of Google Play Store, and as a result fewer eyes independently building and testing the Signal code base. Everyone is entitled to their opinions, but independent and reproducible builds seem like a net positive for everyone. Always struggled to understand releasing code as open source without taking advantage of the community's willingness to build and test. Looking at it in a new light after the XZ backdoor, and Jia Tan's interactions with other FOSS folk.
    - sethherr 3y ago
      He says the decision not to distribute prebuilt APKs is because: > if you aren't able to build TextSecure from source, you probably aren't capable of managing the risks associated with 3rd party sources. Which is a compelling argument from my perspective. I also think that people who can’t compile code should probably not root their phone.
est 3y ago
> super professional in some ways, and rather amateur in others so this has to be a coordinaed teamwork instead of a single hacker, right?
- bee_rider 3y ago
  It could be somebody who was just good at some things and not at other things.
k8svet 3y ago
(really tinfoil hatty - ) I almost wonder if it's misdirection?
- c6400sc 3y ago
  Or a whitehat who couldn't get attention another way?
  - account42 3y ago
    This hat is way too dirty to be white. Even if well intentioned (which is into fairy tale theory level of likelyness), the implementation is well beyond ethical.
orbital-decay 3y ago
You probably have too high expectations when you hear the "state-sponsored" part. Every large organization will inevitably end up like any other. They also have bureaucracy, deadlines, production cycle, poor communication between teams, the recent iOS "maybe-a-backdoor" story also shows that they don't always care about burning the vulnerabilities because they amassed a huge pile of them.
- Jasper_ 3y ago
  Eh. Take a look at other state-sponsored attackers. We know they have 0-days for iOS, we know they've been used, but even Apple doesn't know what they are since they are so good at hiding their tracks. I don't think a state-sponsored attack would upload their payload to the git repo and tarball for all to stare at after it's been found out, which only took about a month. NSO Group built a turing-complete VM out of a use-after-free exploit in some JBIG2 decompression code. Uploading a payload to the world wide web and calling it bad-3-corrupt_lzma2.xz is clownshoes by comparison. My best guess as to what this is is an amateur hacker ring, possibly funded by a ransomware group.
  - orbital-decay 3y ago
    That's why I mentioned the recent story. [0] [1] "Apple doesn't know" when the chain uses an internal backdoor in Apple hardware is a... stretch. And the chain gives the strong vibes of corporate-style development, with all its redundancy and mismatch between two parts. It's not alchemy, really. [0] https://securelist.com/operation-triangulation-the-last-hardware-mystery/111669/ https://securelist.com/operation-triangulation-the-last-hard... [1] https://news.ycombinator.com/item?id=38783112 https://news.ycombinator.com/item?id=38783112 - HN discussion
    - ThePowerOfFuet 3y ago
      Re the secret knock in the Apple silicon, a friend of mine once said "that's how you lose the NOBUS on a backdoor", and I think they were absolutely right. The one thing which most leads me to believe this was an intentional backdoor? The S-boxes.
pflenker 3y ago
We could be faced with a form of Survivorship Bias here[0]. I find that thought rather chilling. [0] https://en.wikipedia.org/wiki/Survivorship_bias https://en.wikipedia.org/wiki/Survivorship_bias
- gitaarik 3y ago
  Which would mean that we have all kinds of active backdoors in our systems without us knowing it. But wouldn't they be detected at some point by someone? Or would they be silently removed again after some time so the attackers are not revealed?
  - guenthert 3y ago
    Attacks on a given system or site can be expected to be removed (or auto-removed) after the operation ended, potentially w/o trace. But for supply-chain attacks there's always a history (if someone bothers to investigate).
  - threeseed 3y ago
    NSO Pegasus has existed for years without all of their exploits being detected. You have to remember there is a lot of code in our systems. And almost every developer is inadequately trained on how to write secure software.
- HarHarVeryFunny 3y ago
  There was a recent Dwarkesh Patel interview of Dario Amodei, CEO of Anthropic, who now have ex. national security people on their staff. He said that the assumption is that if a tech company has 10,000 or more employees then not only will you almost certainly have leakers, but there is also a high likelihood there is an actual spy among your staff. This is why they use need-to-know compartmentalism, etc. I wonder what our success rate is in identifying industrial spies? 50%?
rlt 3y ago
This doesn’t seem too surprising to me. Any moderately competent developer could gain maintainership of a huge percentage of open source projects if they’re being paid to focus solely on that goal… after all, they’re competing mostly against devs working on it part side or as a hobby. If this is state sponsored, they likely have similar programs in a large number of other projects.
bartimus 3y ago
What's also surprising is how quickly the community seems to be giving someone the benefit of the doubt. A compromised maintainer would probably exactly introduce a fake member joining the project to make certain commits. They might have a contact providing the sophisticated backdoor that they need to (amateurishly) implement.
renk 3y ago
That performance regression could also be a way to identify compromised systems.
- Rygian 3y ago
  If all systems are compromised you don't need to identify anymore.
  - renk 3y ago
    Well, for now the payload delivery relies on things like x86, glibc.. still enough.
_8j50 3y ago
This tracks with other nation state sponsored attack patterns. I've had that same reaction before. Most APTs are like this but some Chinese,US and Russian APTs are so well funded, every aspect of their attacks is impressive. Many hackers who work for nation states also have side gigs as crimeware/ransomgang members or actual pentesting jobs. Reminds me of apt3/boyusec: https://www.bleepingcomputer.com/news/security/chinese-government-contractor-identified-as-cyber-espionage-group-apt3/ https://www.bleepingcomputer.com/news/security/chinese-gover... It still boggles my mind that americans are against banning companies like huawei and bytedance. The MSS and PLA don't mess around.
- cycomanic 3y ago
  > It still boggles my mind that americans are against banning companies like huawei and bytedance. The MSS and PLA don't mess around. The problem is that many others would have as much reason to bann US companies. I mean the US has a much more extensive history of using their security apparatus both for intelligence and economic means even against their allies. Now if everyone bans everyone else we will let the world economy grind to a halt pretty quickly.
  - _8j50 3y ago
    China already bans US companies and this is an active malicious threat not a vague possibility of harm. You can jail mark zukerberg but you can't jail xi jiping
- afiori 3y ago
  For bytedance/TikTok the question is not really "should the US ban this company" but rather "should the government be able to ban any companies it wants (and also make it illegal for VPNs to allow US users to access relevant services/websites) without having to provide any substantial evidence? Which is a very different question in practice
  - _8j50 3y ago
    US companies? I'm with you, no! Foreign companies, absolutley. Even the suspicion of malicious abuse should be enough to ban a foreign company. Foreign persons and entities have no rights in the US and our government owes them as much explanation as they give us when they ban US companies on a whim.
varjag 3y ago
Different departments.
AndyMcConachie 3y ago
At this point we don't know who did this. It could have been a single really smart person, it could have been criminal, it could have been a state intelligence agency. We don't know shit.
cladopa 3y ago
You probably have seen lots of films deificating state-sponsored organisations. SSO are made by humans under real constrainsts in time, money, personal. It is almost impossible to make something perfect that nobody in the world could detect. In the world there are people with different backgrounds that could use techniques that you never accounted for. Maybe is a technique used in biology to study DNA, or for counting electrons in a microscope or the background radiation in astronomy. For example, I have seen strange people like that reverse engineer encrypted chips that were "impossible" to in a very easy way because nobody expected it. They spend 10million dollars protecting something and someone removed the protection using $10.
cykros 3y ago
States have spent decades if not centuries building up espionage apparatus. All of the testing and development that goes into avoiding bugs has a tendency to run counter to the goals of secrecy or speed. Secret, fast, well tested. Choose 2.
iforgotpassword 3y ago
My favorite theory is that Jia Tan is a troll. They tried some silly patches and were surprised they got accepted. What started as a little joke on the side because covid made you stay at home slowly spiraled into "I wonder how far I can push this?" Two years are enough to make yourself familiar with open ssh, ifuncs etc. Then you do silly things like "hey um I need to replace the bad test data with newly generated data, this time using a fixed seed so that they are reproducible", but you don't actually tell anyone the seed for the new data. Then you lol when that gets past the maintainers no questions asked. In the end they maybe just wanted to troll a coworker, like play some fart noises while they listen to music, and since they use Debian well, you better find a way to backdoor something in Debian to get into their machine. Like back in the day when sasser sabotaged half the internet and "security experts" said they have a plausible lead to Russia – which as is turned out was because said security experts ran strings on the binary and found Russian text – put there by the German teen who wrote sasser "for teh lulz".
swed420 3y ago
The tin foiler in me still suspects it could be Microsoft who planted it to make FOSS look bad.
setgree 3y ago
If Google could release Gemini with a straight face, is it so hard to believe that this shadowy org might fail in an even subtler way? Patrick McKenzie observes elsewhere that criminal orgs operate more or less like non-criminal ones — by committee and by consensus. By that light, it’s not so hard to fathom why this ship could have been sunk by performance degradation arising from feature bloat. Companies I’ve worked at have made more amateurish mistakes.
tdudhhu 3y ago
Sometimes I think it could be someone who was forced to embed the backdoor but was smart enough to make it detectable by others without raising suspicion by the entity that was forcing him.
ThePowerOfFuet 3y ago
On 2024-02-29, a PR was sent to stop linking liblzma into libsystemd [0]. Kevin Beaumont speculated [1] that "Jia Tan" saw this and immediately realized it would have neutered the backdoor and thus began to rush to avoid losing the very significant amount of work that went into this exploit; I think he's right. That rush caused the crashing seen in 5.6.0 and the lack of polish which could have eliminated or reduced the performance regressions which were the entire reason this was caught in the first place; they simply didn't have the time because the window had started to close and they didn't want all their work to be for nothing. [0]: https://github.com/systemd/systemd/pull/31550 https://github.com/systemd/systemd/pull/31550 [1]: https://doublepulsar.com/inside-the-failed-attempt-to-backdoor-ssh-globally-that-got-caught-by-chance-bbfe628fafdd https://doublepulsar.com/inside-the-failed-attempt-to-backdo...
ufo 3y ago
IIRC, according to Andres Freund the perf regression only happened in machines using the -fno-omit-frame-pointer setting, which was not the default at that point.
- anarazel 3y ago
  The -fno-omit-frame-pointer bit is separate from the slowdown. -fno-omit-frame-pointer lead to valgrind warnings, but no perceptible slowdown in most (including this) cases.
thih9 3y ago
For all we know earlier operations may have been high quality (and still undetected), this one for some reason may have been comparatively not that important and the actor decided to cut costs.
jcpham2 3y ago
The maintainer account (the identity) could have been sold to a third party. There are secondary markets [1] for this. [1] https://ogusers.gg/ https://ogusers.gg/ is the largest clear net marketplace for buying and selling usernames on popular sites
rzwitserloot 3y ago
I find this the most plausible explanation by far: * The highly professional outfit simply did not see teknoraver's commit to remove liblzma as standard dependency of systemd build scripts coming. * The race was on between their compromised code and that commit. They had to win it, with as large a window as possible. * This caused the amateuristic aspects: Haste. * The performance regression is __not__ big. It's lucky Andres caught it at all. It's also not necessarily all that simple to remove it. It's not simply a bug in a loop or some such. If I was the xz team I'd have enough faith in all the work that was done to give it high odds that they'd get there before discovery. That they'd have time; months, even. * The payload of the 'hack' contains fairly easy ways for the xz hackers to update the payload. They actually used it to remove a real issue where their hackery causes issues with valgrind that might lead to discovering it, and they also used it to release 5.6.1 which rewrites significant chunks; I've as yet not read, nor know of any analysis, as to why they changed so much. Point is, it's reasonable to think they had months, and therefore, months to find and fix issues that risk discovery. * I really can't spell this out enough: WE GOT REALLY LUCKY / ANDRES GOT LUCKY. 9 times out of 10 this wouldn't have been found in time. Extra info for those who don't know: https://github.com/systemd/systemd/commit/3fc72d54132151c131301fc7954e0b44cdd3c860 https://github.com/systemd/systemd/commit/3fc72d54132151c131... That's a commit that changes how liblzma is a dependency of systemd. Not because the author of this commit knew anything was wrong with it. But, pretty much entirely by accident (although removing deps was part of the point of that commit), almost entirely eliminates the value of all those 2 years of hard work. And that was with the finish line in sight for the xz hackers: On 24 feb 2024, the xz hackers release liblzma 5.6.0 which is the first fully operational compromised version. __12 days later systemd merges a commit that means it won't work__. So now the race is on. Can they get 5.6.0 integrated into stable releases of major OSes _before_ teknoraver's commit that removes liblzma's status as direct dep of systemd? I find it plausible that they knew about teknoraver's commit _just before_ Feb 24th 2024 (when liblzma v5.6.0 was released, the first backdoored release), and rushed to release ASAP, before doing the testing you describe. Buoyed by their efforts to add ways to update the payload which they indeed used - March 8th (after teknoraver's commit was accepted) it was used to fix the valgrind issue. So, no, I don't find this weird, and I don't think the amateurish aspects should be taken as some sort of indication that parts of the outfit were amateuristic. As long as it's plausible that the amateuristic aspects were simply due to time pressure, it sounds like a really bad idea to make assumptions in this regard.
emmelaich 3y ago
So, much like any other code base. Bit of a https://en.wikipedia.org/wiki/Curate%27s_egg https://en.wikipedia.org/wiki/Curate%27s_egg
treffer 3y ago
Well, there is a pretty logical explanation. Libsystemd was moving to a dlopen architecture for its dependencies. This means that the backdoor would not load as the sshd patch only used libsystemd for notify, which does not need liblzma at all. So they IMHO gave it a last shot. It's OK if it burns as it would be useless in 3 months (or even less). The collateral is the backdoor Binary, but given enough engineering power it will be irrelevant in 2-3 months, too.
- hackeraccount 3y ago
  I think this is probably the right answer. The only thing that makes me think this was amateurs/criminals instead of professionals is that I tend to think that professionals are more interested in post attack security. So if the gate was closing an amateur would say, "Act now! Let's get what we can!" A professional would say, "This is all going to come to light real soon - our exploit won't be read and there's a high chance of this all falling apart. Pull out everything, cover our tracks to the degree we can and find another opportunity to pull this off." But then again I also think on professionals would work an exploit that takes years. Criminals by their nature want a quick payout (If I had the patience for a long con I'd just get a job) a motivated individual amateurs (i.e. crazy people) rarely have a wide enough skill set.
- account42 3y ago
  This particular backdoor might have become useless anyway, but control of liblzma would have continued to be very valuable. Not only is it used in so many places (the embedded version even in the kernel), it also is a common part of the toolchain used to build everything else allowing for trusting trust style attacks.
fasa99 3y ago
My thought is they have 50-200 systems programmers where it's like "Hey George, I know you like to contribute to linux, have at it, we just need you to put a little pinhole here here and here" Then they have 5-20 security gurus / hackers who are like "Thanks George, I made this binary blob you need to insert, here's how you do it" So the systems developer job is to build a reputation over years as we see, the security guy's job is to handle the equation group heavily encrypted/obfuscated exploits. This would explain a mix of code quality / professionalism - multiple people, multiple roles. Of course the former "systems programmer" role need not be a full time employee. They good have motivated a white hat developer to change hats, either financially or through an offer they can't refuse.

by ufmace 3y ago

darkclouds 3y ago

[dead]

braiamp 3y ago

There are thousands of ways that performance can be impacted. No matter how good you are at developing, there will be a workload that would have a performance hit. Phoronix has been several times reporting issues to the Linux kernel because performance regression with their test suite. Performance tests tend to take more time than correctness tests.

ufmace 3y ago
Not seeing that as a point. It's probably not possible to have no performance hit whatsoever when you're checking the exact nanosecond count of every little thing. But usually nobody is doing that. It shouldn't be hard to not cause a substantial enough performance regression in SSHD logins that somebody who wasn't already monitoring that would notice and decide to dig into what's going on. I'm not sure if it's been revealed yet what this thing actually does, but it seems like all it really needs to do is to check for some kind of special token in the auth request or check against another keypair.
- deleted 3y ago
  [deleted]

heyoni 3y ago

I thought performance was actually fine? It only dragged when using valgrind, hence the rhetoric that it took some really unlikely circumstances for it to be detected that quickly.

jwilk 3y ago
No, it wasn't fine. From the advisory (https://news.ycombinator.com/item?id=39865810 https://news.ycombinator.com/item?id=39865810): > With the backdoored liblzma installed, logins via ssh become a lot slower.
- heyoni 3y ago
  Yea you're right. 500ms vs 10ms on an older server. Was thrown off by this statement and thought only the perf/valgrind/gdb attachments were what really brought it to surface. > Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked. This appears to be part of some countermeasures to make analysis harder. Performance was much worse but not enough to actually nudge the author into digging into it until the random ssh logins started piling up: https://twitter.com/AndresFreundTec/status/1774190743776866374 https://twitter.com/AndresFreundTec/status/17741907437768663... https://nitter.poast.org/AndresFreundTec/status/1774190743776866374 https://nitter.poast.org/AndresFreundTec/status/177419074377...
- olejorgenb 3y ago
  2-3x slower 0m0.299s -> 0m0.807s [1] [1] https://www.openwall.com/lists/oss-security/2024/03/29/4 https://www.openwall.com/lists/oss-security/2024/03/29/4
- rstuart4133 3y ago
  It was noticed because of it's CPU usage. It seems most people think "that was lucky". It wasn't lucky, it was inevitable. It happened to be noticed by someone doing benchmarking, but there is another category of machines out there it would hit hard: low end VM's running an ssh server. I've had my low end VM's hit by exactly the same thing the postgres benchmarker saw, but on these box's it isn't just annoying. There are so many ssh logins happening the OOM killer comes out to play but these ssh logins are tiny so it tends to kill the real stuff you've got the box doing, like web serving. A 2 or 3 fold increase in CPU would suddenly bring a lot of VM to their knees. At least thousands of them. It would be noticed.

rgmerk 3y ago

Events can happen to anyone, even competent state-sponsored organisations. And intelligence agencies are sometimes rather less ruthlessly competent than imagined (Kremlin assisinations in the UK have been a comedy of errors [1]). Maybe another backdoor, or alternative access mechanism they were using, got closed and they wanted another one in a hurry. [1] https://en.wikipedia.org/wiki/Poisoning_of_Alexander_Litvinenko https://en.wikipedia.org/wiki/Poisoning_of_Alexander_Litvine...

readyplayernull 3y ago
And they are getting better at it: https://www.bbc.com/news/world-us-canada-68706317 https://www.bbc.com/news/world-us-canada-68706317
cesarb 3y ago
> Maybe another backdoor, or alternative access mechanism they were using, got closed and they wanted another one in a hurry. Or maybe the opportunity window for the mechanism this backdoor would use was closing. According to the timeline at https://research.swtch.com/xz-timeline https://research.swtch.com/xz-timeline there was a github comment from poettering at the end of January which implied that the relevant library would be changed soon to lazy load liblzma ("[...] Specifically the compression libs, i.e. libbz2, libz4 and suchlike at least. And also libpam. So expect more like this sooner or later."), as in fact happened a month later. The attacker had to get the backdoor into the targeted Linux distributions before the systemd change got into them. Of course, the attacker could instead take the loss and abandon the approach, but since they had written that amount of complex code, it probably felt hard to throw it all away.
ifwinterco 3y ago
Russia wants everyone to know when they do assassinations abroad, that's why they use obviously-Russian methods like polonium and novichok. It's designed to send a message to other Russian dissidents and remind them that they aren't safe

richin13 3y ago

I read somewhere that some recent changes in systems would've made the backdoor useless so they had to rush out, which caused them to be reckless and get discovered

sho_hn 3y ago
This refers to the fact that systemd was planning to drop the dependency on liblzma (the conpression library installed by xz), and instead dlopen it at runtime when needed. Not for security reasons, but to avoid pulling the libs into initramfs images. The backdoor relies on sshd being patched to depend on libsystemd to call sd_notify(), which several distros had done. OpenSSH has since merged a new patch upstream that implements similar logic to sd_notify() in sshd itself to allow distros to drop that patch. So the attack surface of both sshd and libsystemd has since shrunk a bit.
- rav 3y ago
  > The backdoor relies on sshd being patched to depend on libsystemd to call sd_notify I remember when we added sd_notify support to our services at work, I was wondering why one would pull in libsystemd as a dependency for this. I mean, there's a pure-Python library [1] that basically boils down to: import os, socket def notify(state=b"READY=1"): sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) addr = os.getenv('NOTIFY_SOCKET') if addr[0] == '@': addr = '\0' + addr[1:] sock.connect(addr) sock.sendall(state) With proper error handling, that's about 50 lines of C code. I would vendor that into my application in a heartbeat. [1]: https://raw.githubusercontent.com/bb4242/sdnotify/master/sdnotify/__init__.py https://raw.githubusercontent.com/bb4242/sdnotify/master/sdn...
  - cesarb 3y ago
    > I was wondering why one would pull in libsystemd as a dependency for this. I mean, there's a pure-Python library [...] With proper error handling, that's about 50 lines of C code. There's also a pure C library (libsystemd itself) which already does all that, and you don't need to test all the error handling cases in your 50 lines of C code. It makes sense to use the battle-tested code, instead of writing your own.

dabbledabble 3y ago

It’s also possible that this could be a change in personnel. Maybe the one who earned trust and took over was no more working for them. And an amateur took over with tight deadlines that lead to this gaffe for them.

versteegen 3y ago
The abrupt change in time-of-day when commits occurred supports the theory that Jai Tan is more than one person: https://twitter.com/birchb0y/status/1773871381890924872 https://twitter.com/birchb0y/status/1773871381890924872
- swid 3y ago
  The text near the box makes it sound like these are just the fixes - not adding the test files but updating them. At that point it would have been clear “the race is on” to avoid detection, so it’s not too surprising someone would work late to salvage the operation.
  - versteegen 3y ago
    Whoops, you're right. So this isn't really evidence of anything. Out of interest I looked up the other commit at that time of day visible in that graph, laying on the arrow. It's [1], which changes the git URL from git.tukaani.org to github.com. Of course, moving the project hosting to github was part of the attack. [1] https://git.tukaani.org/?p=xz.git;a=commitdiff;h=e1b1a9d6370b788bd6078952c6c201e12bc27cbf https://git.tukaani.org/?p=xz.git;a=commitdiff;h=e1b1a9d6370...
- hinkley 3y ago
  If the mistakes align with the time of day change, perhaps the author had a distraction that pushed the hours and compromised judgement.

manquer 3y ago

Why do we assume the person building the trust is the attacker ? Is not possible the attacker simply took over the account of some one genuinely getting involved in the community either hacked or just with $5 wrench and then committed the malicious code ?

ericpruitt 3y ago
> Is not possible the attacker simply took over the account of some one genuinely getting involved in the community either hacked or just with $5 wrench and then committed the malicious code ? Given the behavior of the accounts that applied pressure on the original xz maintainer, this seems unlikely to me.
- ilvez 3y ago
  Or they just bought the guy at one point, because I understood the malicious behaviour started quite recently.
  - timschmidt 3y ago
    Jia Tan's very first commit two years ago made suspicious changes to liblzma. See the timeline here: https://boehs.org/node/everything-i-know-about-the-xz-backdoor https://boehs.org/node/everything-i-know-about-the-xz-backdo...

tw04 3y ago

The fact that the guys developing the code weren't also simultaneously running valgrind and watching performance isn't hard to believe. They were targeting servers and appliances, how many servers and appliances do you know of that are running valgrind in their default image? Sure, in hindsight that's a "duh, why didn't we think of that" - but also it's not very hard at all to see why they didn't think of that. They were likely testing against the system images they were hoping to compromise, not joe-schmoe developer's custom image.

dralley 3y ago
In theory they should probably be testing against the CI pipelines of Debian and Fedora / CentOS, as that's the moat their backdoor has to cross.
- tw04 3y ago
  They put code in to, in theory, avoid running in a development environment.

asveikau 3y ago

Modifying the sshd behavior without crashes seems by itself pretty difficult. I mean, conceptually it isn't hard, if you are in the same process and can patch various functions, but I think doing so and having it be "production ready" to ship in multiple linux distros all the time is a challenge. This thing wasn't around for very long but yet another thing to consider would be to keep it working across multiple versions of OpenSSH, libcrypto etc.

nick238 3y ago
I picture some division in [nation-state] where they're constantly creating personas, slowly working all sorts of languishing open source packages with few maintainers (this is the actual hard, very slow part), then once they have a bit of an in, they could recruit more technical expertise. The division is run by some evil genius who knows this could pay off big, but others are skeptical, so their resources are pretty minimal.
- timschmidt 3y ago
  Moxie's reasons for disallowing Signal distribution via F-droid always rang a little flat to me ( https://github.com/signalapp/Signal-Android/issues/127 https://github.com/signalapp/Signal-Android/issues/127 ). Lots of chatter about the supposedly superior security model of Google Play Store, and as a result fewer eyes independently building and testing the Signal code base. Everyone is entitled to their opinions, but independent and reproducible builds seem like a net positive for everyone. Always struggled to understand releasing code as open source without taking advantage of the community's willingness to build and test. Looking at it in a new light after the XZ backdoor, and Jia Tan's interactions with other FOSS folk.
  - sethherr 3y ago
    He says the decision not to distribute prebuilt APKs is because: > if you aren't able to build TextSecure from source, you probably aren't capable of managing the risks associated with 3rd party sources. Which is a compelling argument from my perspective. I also think that people who can’t compile code should probably not root their phone.

est 3y ago

> super professional in some ways, and rather amateur in others so this has to be a coordinaed teamwork instead of a single hacker, right?

bee_rider 3y ago
It could be somebody who was just good at some things and not at other things.

k8svet 3y ago

(really tinfoil hatty - ) I almost wonder if it's misdirection?

c6400sc 3y ago
Or a whitehat who couldn't get attention another way?
- account42 3y ago
  This hat is way too dirty to be white. Even if well intentioned (which is into fairy tale theory level of likelyness), the implementation is well beyond ethical.

orbital-decay 3y ago

You probably have too high expectations when you hear the "state-sponsored" part. Every large organization will inevitably end up like any other. They also have bureaucracy, deadlines, production cycle, poor communication between teams, the recent iOS "maybe-a-backdoor" story also shows that they don't always care about burning the vulnerabilities because they amassed a huge pile of them.

Jasper_ 3y ago
Eh. Take a look at other state-sponsored attackers. We know they have 0-days for iOS, we know they've been used, but even Apple doesn't know what they are since they are so good at hiding their tracks. I don't think a state-sponsored attack would upload their payload to the git repo and tarball for all to stare at after it's been found out, which only took about a month. NSO Group built a turing-complete VM out of a use-after-free exploit in some JBIG2 decompression code. Uploading a payload to the world wide web and calling it bad-3-corrupt_lzma2.xz is clownshoes by comparison. My best guess as to what this is is an amateur hacker ring, possibly funded by a ransomware group.
- orbital-decay 3y ago
  That's why I mentioned the recent story. [0] [1] "Apple doesn't know" when the chain uses an internal backdoor in Apple hardware is a... stretch. And the chain gives the strong vibes of corporate-style development, with all its redundancy and mismatch between two parts. It's not alchemy, really. [0] https://securelist.com/operation-triangulation-the-last-hardware-mystery/111669/ https://securelist.com/operation-triangulation-the-last-hard... [1] https://news.ycombinator.com/item?id=38783112 https://news.ycombinator.com/item?id=38783112 - HN discussion
  - ThePowerOfFuet 3y ago
    Re the secret knock in the Apple silicon, a friend of mine once said "that's how you lose the NOBUS on a backdoor", and I think they were absolutely right. The one thing which most leads me to believe this was an intentional backdoor? The S-boxes.

pflenker 3y ago

We could be faced with a form of Survivorship Bias here[0]. I find that thought rather chilling. [0] https://en.wikipedia.org/wiki/Survivorship_bias https://en.wikipedia.org/wiki/Survivorship_bias

gitaarik 3y ago
Which would mean that we have all kinds of active backdoors in our systems without us knowing it. But wouldn't they be detected at some point by someone? Or would they be silently removed again after some time so the attackers are not revealed?
- guenthert 3y ago
  Attacks on a given system or site can be expected to be removed (or auto-removed) after the operation ended, potentially w/o trace. But for supply-chain attacks there's always a history (if someone bothers to investigate).
- threeseed 3y ago
  NSO Pegasus has existed for years without all of their exploits being detected. You have to remember there is a lot of code in our systems. And almost every developer is inadequately trained on how to write secure software.
HarHarVeryFunny 3y ago
There was a recent Dwarkesh Patel interview of Dario Amodei, CEO of Anthropic, who now have ex. national security people on their staff. He said that the assumption is that if a tech company has 10,000 or more employees then not only will you almost certainly have leakers, but there is also a high likelihood there is an actual spy among your staff. This is why they use need-to-know compartmentalism, etc. I wonder what our success rate is in identifying industrial spies? 50%?

rlt 3y ago

This doesn’t seem too surprising to me. Any moderately competent developer could gain maintainership of a huge percentage of open source projects if they’re being paid to focus solely on that goal… after all, they’re competing mostly against devs working on it part side or as a hobby. If this is state sponsored, they likely have similar programs in a large number of other projects.

bartimus 3y ago

What's also surprising is how quickly the community seems to be giving someone the benefit of the doubt. A compromised maintainer would probably exactly introduce a fake member joining the project to make certain commits. They might have a contact providing the sophisticated backdoor that they need to (amateurishly) implement.

renk 3y ago

That performance regression could also be a way to identify compromised systems.

Rygian 3y ago
If all systems are compromised you don't need to identify anymore.
- renk 3y ago
  Well, for now the payload delivery relies on things like x86, glibc.. still enough.

_8j50 3y ago

This tracks with other nation state sponsored attack patterns. I've had that same reaction before. Most APTs are like this but some Chinese,US and Russian APTs are so well funded, every aspect of their attacks is impressive. Many hackers who work for nation states also have side gigs as crimeware/ransomgang members or actual pentesting jobs. Reminds me of apt3/boyusec: https://www.bleepingcomputer.com/news/security/chinese-government-contractor-identified-as-cyber-espionage-group-apt3/ https://www.bleepingcomputer.com/news/security/chinese-gover... It still boggles my mind that americans are against banning companies like huawei and bytedance. The MSS and PLA don't mess around.

cycomanic 3y ago
> It still boggles my mind that americans are against banning companies like huawei and bytedance. The MSS and PLA don't mess around. The problem is that many others would have as much reason to bann US companies. I mean the US has a much more extensive history of using their security apparatus both for intelligence and economic means even against their allies. Now if everyone bans everyone else we will let the world economy grind to a halt pretty quickly.
- _8j50 3y ago
  China already bans US companies and this is an active malicious threat not a vague possibility of harm. You can jail mark zukerberg but you can't jail xi jiping
afiori 3y ago
For bytedance/TikTok the question is not really "should the US ban this company" but rather "should the government be able to ban any companies it wants (and also make it illegal for VPNs to allow US users to access relevant services/websites) without having to provide any substantial evidence? Which is a very different question in practice
- _8j50 3y ago
  US companies? I'm with you, no! Foreign companies, absolutley. Even the suspicion of malicious abuse should be enough to ban a foreign company. Foreign persons and entities have no rights in the US and our government owes them as much explanation as they give us when they ban US companies on a whim.

varjag 3y ago

Different departments.

AndyMcConachie 3y ago

At this point we don't know who did this. It could have been a single really smart person, it could have been criminal, it could have been a state intelligence agency. We don't know shit.

cladopa 3y ago

You probably have seen lots of films deificating state-sponsored organisations. SSO are made by humans under real constrainsts in time, money, personal. It is almost impossible to make something perfect that nobody in the world could detect. In the world there are people with different backgrounds that could use techniques that you never accounted for. Maybe is a technique used in biology to study DNA, or for counting electrons in a microscope or the background radiation in astronomy. For example, I have seen strange people like that reverse engineer encrypted chips that were "impossible" to in a very easy way because nobody expected it. They spend 10million dollars protecting something and someone removed the protection using $10.

cykros 3y ago

States have spent decades if not centuries building up espionage apparatus. All of the testing and development that goes into avoiding bugs has a tendency to run counter to the goals of secrecy or speed. Secret, fast, well tested. Choose 2.

iforgotpassword 3y ago

My favorite theory is that Jia Tan is a troll. They tried some silly patches and were surprised they got accepted. What started as a little joke on the side because covid made you stay at home slowly spiraled into "I wonder how far I can push this?" Two years are enough to make yourself familiar with open ssh, ifuncs etc. Then you do silly things like "hey um I need to replace the bad test data with newly generated data, this time using a fixed seed so that they are reproducible", but you don't actually tell anyone the seed for the new data. Then you lol when that gets past the maintainers no questions asked. In the end they maybe just wanted to troll a coworker, like play some fart noises while they listen to music, and since they use Debian well, you better find a way to backdoor something in Debian to get into their machine. Like back in the day when sasser sabotaged half the internet and "security experts" said they have a plausible lead to Russia – which as is turned out was because said security experts ran strings on the binary and found Russian text – put there by the German teen who wrote sasser "for teh lulz".

swed420 3y ago

The tin foiler in me still suspects it could be Microsoft who planted it to make FOSS look bad.

setgree 3y ago

If Google could release Gemini with a straight face, is it so hard to believe that this shadowy org might fail in an even subtler way? Patrick McKenzie observes elsewhere that criminal orgs operate more or less like non-criminal ones — by committee and by consensus. By that light, it’s not so hard to fathom why this ship could have been sunk by performance degradation arising from feature bloat. Companies I’ve worked at have made more amateurish mistakes.

tdudhhu 3y ago

Sometimes I think it could be someone who was forced to embed the backdoor but was smart enough to make it detectable by others without raising suspicion by the entity that was forcing him.

ThePowerOfFuet 3y ago

On 2024-02-29, a PR was sent to stop linking liblzma into libsystemd [0]. Kevin Beaumont speculated [1] that "Jia Tan" saw this and immediately realized it would have neutered the backdoor and thus began to rush to avoid losing the very significant amount of work that went into this exploit; I think he's right. That rush caused the crashing seen in 5.6.0 and the lack of polish which could have eliminated or reduced the performance regressions which were the entire reason this was caught in the first place; they simply didn't have the time because the window had started to close and they didn't want all their work to be for nothing. [0]: https://github.com/systemd/systemd/pull/31550 https://github.com/systemd/systemd/pull/31550 [1]: https://doublepulsar.com/inside-the-failed-attempt-to-backdoor-ssh-globally-that-got-caught-by-chance-bbfe628fafdd https://doublepulsar.com/inside-the-failed-attempt-to-backdo...

ufo 3y ago

IIRC, according to Andres Freund the perf regression only happened in machines using the -fno-omit-frame-pointer setting, which was not the default at that point.

anarazel 3y ago
The -fno-omit-frame-pointer bit is separate from the slowdown. -fno-omit-frame-pointer lead to valgrind warnings, but no perceptible slowdown in most (including this) cases.

thih9 3y ago

For all we know earlier operations may have been high quality (and still undetected), this one for some reason may have been comparatively not that important and the actor decided to cut costs.

jcpham2 3y ago

The maintainer account (the identity) could have been sold to a third party. There are secondary markets [1] for this. [1] https://ogusers.gg/ https://ogusers.gg/ is the largest clear net marketplace for buying and selling usernames on popular sites

rzwitserloot 3y ago

I find this the most plausible explanation by far: * The highly professional outfit simply did not see teknoraver's commit to remove liblzma as standard dependency of systemd build scripts coming. * The race was on between their compromised code and that commit. They had to win it, with as large a window as possible. * This caused the amateuristic aspects: Haste. * The performance regression is __not__ big. It's lucky Andres caught it at all. It's also not necessarily all that simple to remove it. It's not simply a bug in a loop or some such. If I was the xz team I'd have enough faith in all the work that was done to give it high odds that they'd get there before discovery. That they'd have time; months, even. * The payload of the 'hack' contains fairly easy ways for the xz hackers to update the payload. They actually used it to remove a real issue where their hackery causes issues with valgrind that might lead to discovering it, and they also used it to release 5.6.1 which rewrites significant chunks; I've as yet not read, nor know of any analysis, as to why they changed so much. Point is, it's reasonable to think they had months, and therefore, months to find and fix issues that risk discovery. * I really can't spell this out enough: WE GOT REALLY LUCKY / ANDRES GOT LUCKY. 9 times out of 10 this wouldn't have been found in time. Extra info for those who don't know: https://github.com/systemd/systemd/commit/3fc72d54132151c131301fc7954e0b44cdd3c860 https://github.com/systemd/systemd/commit/3fc72d54132151c131... That's a commit that changes how liblzma is a dependency of systemd. Not because the author of this commit knew anything was wrong with it. But, pretty much entirely by accident (although removing deps was part of the point of that commit), almost entirely eliminates the value of all those 2 years of hard work. And that was with the finish line in sight for the xz hackers: On 24 feb 2024, the xz hackers release liblzma 5.6.0 which is the first fully operational compromised version. __12 days later systemd merges a commit that means it won't work__. So now the race is on. Can they get 5.6.0 integrated into stable releases of major OSes _before_ teknoraver's commit that removes liblzma's status as direct dep of systemd? I find it plausible that they knew about teknoraver's commit _just before_ Feb 24th 2024 (when liblzma v5.6.0 was released, the first backdoored release), and rushed to release ASAP, before doing the testing you describe. Buoyed by their efforts to add ways to update the payload which they indeed used - March 8th (after teknoraver's commit was accepted) it was used to fix the valgrind issue. So, no, I don't find this weird, and I don't think the amateurish aspects should be taken as some sort of indication that parts of the outfit were amateuristic. As long as it's plausible that the amateuristic aspects were simply due to time pressure, it sounds like a really bad idea to make assumptions in this regard.

emmelaich 3y ago

So, much like any other code base. Bit of a https://en.wikipedia.org/wiki/Curate%27s_egg https://en.wikipedia.org/wiki/Curate%27s_egg

treffer 3y ago

Well, there is a pretty logical explanation. Libsystemd was moving to a dlopen architecture for its dependencies. This means that the backdoor would not load as the sshd patch only used libsystemd for notify, which does not need liblzma at all. So they IMHO gave it a last shot. It's OK if it burns as it would be useless in 3 months (or even less). The collateral is the backdoor Binary, but given enough engineering power it will be irrelevant in 2-3 months, too.

hackeraccount 3y ago
I think this is probably the right answer. The only thing that makes me think this was amateurs/criminals instead of professionals is that I tend to think that professionals are more interested in post attack security. So if the gate was closing an amateur would say, "Act now! Let's get what we can!" A professional would say, "This is all going to come to light real soon - our exploit won't be read and there's a high chance of this all falling apart. Pull out everything, cover our tracks to the degree we can and find another opportunity to pull this off." But then again I also think on professionals would work an exploit that takes years. Criminals by their nature want a quick payout (If I had the patience for a long con I'd just get a job) a motivated individual amateurs (i.e. crazy people) rarely have a wide enough skill set.
account42 3y ago
This particular backdoor might have become useless anyway, but control of liblzma would have continued to be very valuable. Not only is it used in so many places (the embedded version even in the kernel), it also is a common part of the toolchain used to build everything else allowing for trusting trust style attacks.

fasa99 3y ago

My thought is they have 50-200 systems programmers where it's like "Hey George, I know you like to contribute to linux, have at it, we just need you to put a little pinhole here here and here" Then they have 5-20 security gurus / hackers who are like "Thanks George, I made this binary blob you need to insert, here's how you do it" So the systems developer job is to build a reputation over years as we see, the security guy's job is to handle the equation group heavily encrypted/obfuscated exploits. This would explain a mix of code quality / professionalism - multiple people, multiple roles. Of course the former "systems programmer" role need not be a full time employee. They good have motivated a white hat developer to change hats, either financially or through an offer they can't refuse.