9 ms·
Great read - the lengths the attacker went to to not only hide it but also build the trust to be able to pull this off is incredible. So calculated. One can on
by dvsfish 2y ago
Great read - the lengths the attacker went to to not only hide it but also build the trust to be able to pull this off is incredible. So calculated.
One can only imagine they felt a great deal of panic and despair as the security community lit up the issue, regardless of a near zero chance of repercussions.
Very clever - those involved in catching this are, quietly, heroes
- jerjerjer 2y agoThe use of trie structure to compare strings without hardcoding raw, searchable string values is very inspired. But yes, some black hat had a really really bad day: being so close to owning any ssh-enabled recently-updated box and when they can finally see a finish line it is just poof... gone in a day. It was two years of effort from at least one guy and more likely a team.
- jumelles 2y agoHonestly? Good. Anyone doing this sort of work willingly deserves worse.
- optimalsolver 2y agoThat they were harassing and manipulating a lone, unthanked maintainer who had already told them he was dealing with mental issues makes them evil, IMO. The honorable thing for "Jia" to do after this epic failure is seppuku, or whatever his or her local equivalent is.
- cqqxo4zV46cp 2y agoNobody sees themselves as the bad guy, and that’s not the same as “some people are just fundamentally selfish”. There are definitely loads of people that’d feel like the end justifies the means. There are plenty of people for whom a workday involves doing far worse things for the world than cyberbullying one person, and will look you in the eye and justify it. Plenty of that stuff is socially acceptable in many many mainstream circles. “Being mean to a maintainer” is just one that this community is especially sensitive to, because it involves a highly personified victim that they can relate to.
- optimalsolver 2y agoThese maintainers add vast amounts of value to the modern world, though most of the people that benefit indirectly from their work can't really conceive of what it is they do. People like "Jia" are pure parasites. It's one of the best cases of "why we can't have nice things" I've ever seen.
- qweqwe14 2y agoYeah, they add vast amounts of "value", including (accidentally) reinforcing the status-quo. It would definitely be interesting to see what would happen if the attack wasn't noticed, but instead people focus their interest on attacking Jia Tan because "wow, that guy is one hell of an asshole, it sure is a great thing that he failed!". Whether or not this attack was the rare one that failed out of many similar ones is largely irrelevant to people. Quick, discuss this one particular case where we noticed it, news flash and all. > People like "Jia" are pure parasites They are "parasites" because they don't do what they are "supposed to"? That's pretty crazy. I guess every person that's doing what matches their interests is somehow a bad person/parasite/etc. Or is that only if what they do is forbidden by some rule book? Do you see what I'm getting at here? "pls sir dont create linux backdoar"
- optimalsolver 2y ago>I guess every person that's doing what matches their interests is somehow a bad person/parasite/etc If they inflict harm on their fellow humans in order to do so, then literally yes. There's a reason why a society will collapse if you have too many of this type of person.
- qweqwe14 2y agoHFT is also harmful, and so are the majority of startups that don't do anything actually useful and just take VC money. Those are just a few examples off the top of my head. What I'm saying is that there are a lot more technically legal ways to profit that harm society, some of them more nefarious than what Jia Tan did. Doing things that are bad for the society in a fucked up society seems justifiable. It doesn't necessarily make you a bad person. People just have a more averse reaction to things that are obviously bad, even if in practice there are way worse things that initially seem innocuous and are actually legal to do. That's just the textbook example of hypocrisy.
- lenerdenator 2y agoI think it goes beyond just this particular attempt. I bet they were looking for other targets, targets that are widely-used but maintained by resource-deficient projects. They wanted to do maximum damage with the fewest possible supply chain attacks. The fact that many projects are now searching deep and wide through their commit histories, looking at just who they're taking code from, beginning to develop frameworks for attack mitigation and remediation... an entire type of previously very-promising attack is completely burned. This has been a massive defeat. And it happened all by chance.
- fauigerzigerk 2y ago>an entire type of previously very-promising attack is completely burned. I fear it's not just the attack that is burned. If new contributors have to be distrusted and/or go through some sort of vetting that isn't based on the merit of their contributions, that is a terrible blow to the entire open source movement. The threshold for young coders without a great deal of history and without a network of contacts to become contributors to important open source projects has just gone up massively.
- Brian_K_White 2y agoNah. Hardly going to change anything for the worse, only for the better. The bar is still trivial. The worst part is that it will be difficult to be anonymous. There are a lot of valid, even life & death reasons for anonymity, and that will now be much harder. So the loss is all the contributors who don't dare let their government or employer see them helping the wrong projects, or in some cases don't want to disclose that they are any good at coding or particularly interested in security etc.
- cqqxo4zV46cp 2y ago“GitHub Verified”.
- nonrandomstring 2y ago
- issafram 2y agoIt was most likely done by a government. Not just one hacker.
- gizmo686 2y agoGovernments are made up of people. There is likely at least 1 real person (if not more) for whom this attack has been the entirety of their professional life for years.
- deleted 2y ago[deleted]
- Brian_K_White 2y agoIf so, then they were paid for those hours and are perfectly whole right now. Maybe their career takes a dent but I do not weep for whoever this was no matter which government they worked for, unless it was so bad they were actually a slave and forced to do this against their will. The government who paid them just lost the investment, but who cares about them? It's only a good thing if this does not get a reputation for being a good invesment. If it was a criminal, then the same as the government they lost the investment but again that's only a good thing.
- cqqxo4zV46cp 2y agoAre you being argumentative for the sake of it? The statement was that someone had a really bad day. The implication that people are driven by money and nothing else, or even that they have no right to feel like they had a bad day if they were paid, is absurd. Nobody is saying that you should be sympathetic. It’s just an interesting comment: An interesting thing to think about. A worthwhile contribution to the conversation. This was someone’s very bad ending to a very long project.
- Brian_K_White 2y agoOk, true. It was a bad day.
- KeplerBoy 2y agoHow much effort was it really? Yes, they worked on it over 2 years, but I guess it wasn't more effort than a few hours every other week (apart from engineering the actual exploit). After all putting in a full time effort as an unpaid contributor would be suspicious in itself. Assuming they work 40 hours a week and are doing this in a team (presumably every major player has such a team or is scrambling to get one now), one must expect many potentially infiltrated projects out there.
- NamTaf 2y ago“Apart from engineering” is doing some seriously heavy lifting here. Writing the actual code is likely just an afterthought by comparison. The engineering - identification of a target weakness, the design of the exploit chain, etc. - is overwhelmingly going to have been the lion’s share of the effort.
- wjholden 2y agoI agree, I had never heard of using a trie as a means of hiding strings. I'm not familiar with any programming language that provides tries in a standardized way. They're not so hard to code, so I wonder if this will become a trend in future malware.
- Terr_ 2y ago> The use of trie structure to compare strings without hardcoding raw, searchable string values is very inspired. Aside, this is one of my grumpy-old-dude opinions when it comes to regular non-malicious code: Source "greppability" is a virtue, one that must be balanced against "don't repeat yourself" and "compile time checking", etc. Some examples off the top of my head: 1. One import/alias per line, no wildcards. 2. If you have to concatenate a literal string across multiple lines, try to ensure the break isn't inside an interesting substring someone would search for. 3. If you write a compile-time constant like foo=4*6*9, the comments around it should also contain the result. Sure, someone might screw up and forget to keep the comment in-sync, but it can also be invaluable when someone else is tearing their hair out trying to find out why the production logs are saying nothing but "Illegal frump: 216". Don't get me wrong, I love a good IDE, and many will offer "Find Usages" or "Find Literal"... but it doesn't always work all the time, for all languages in the project at once, for everyone on a team who might use other tools, or during a PR review through a web-browser, etc.
- ncruces 2y agoIf you have to concatenate a literal string across multiple lines, try to ensure the break isn't inside an interesting substring someone would search for. This is called semantic line breaks and is also important to get (admittedly more naive) diffs sensible. https://sembr.org/ https://sembr.org/
- btown 2y agoAlong the lines of source greppability, at some point someone will try to replace successive a.foo_x, a.foo_y, and a.foo_z accesses with a loop over foo_{suffix}, thus breaking any attempt to grep for foo_y. That someone may be me! This should almost always be struck down in code review wherever possible. Being a bit more DRY isn’t worth it!
- kubanczyk 2y ago4. Avoid complex regexes. E.g. prefer '>>>>>' over '>{5}'
- Terr_ 2y ago
- jlmorton 2y ago> It was two years of effort from at least one guy and more likely a team. You think these guys have standups?
- Dalewyn 2y ago>One can only imagine they felt a great deal of panic and despair as the security community lit up the issue, Jia Tan mook: "Sir, they found us out." Jia Tan officer: "What's the sitrep, are they coming for us? Are they getting our operatives in the field?" Jia Tan mook: "No sir, they're just reverting and blacklisting our accounts and releases. Muttering something about 'uptime'." Jia Tan officer: "Shit, we're dealing with sysadmins."
- flomo 2y ago--
- Dalewyn 2y agoIf a government agency tasked with national security started caring about a critical home project by Nebraskaman only and long after Nebraskaman collapsed, that country needs a new government agency.
- deleted 2y ago[deleted]
- arrowsmith 2y agoI imagine that government security agencies are mainly thinking "shit, why didn't we think to do that?"
- deleted 2y ago[deleted]
- jessekv 2y agohttps://xkcd.com/705/ https://xkcd.com/705/
- inkyoto 2y agoThe fundamental question remains: considering the in-depth knowledge of low-level (a small scale disassembler), system programming and the practical application of data structures (i.e. using a trie for all string operations to avoid the use of string constants), was it a state sponsored attack, or it was a one man show who did it for personal gain? Attempting to backdoor nearly every single Linux instance of sshd goes beyond the definition of just being brazen.
- codezero 2y agoThat nobody is talking about the person's IP addresses (the xz project was hosted on a personal server of the maintainer), or any details about their actions indicates to me it was a state actor and the original xz maintainer is cooperating with law enforcement to uncover their actions.
- rwmj 2y agoCISA is investigating it.
- cdelsolar 2y agoI thought it was on GitHub pages?
- codezero 2y agoBefore pages the project was hosted on the server of the original maintainer.
- willvarfar 2y agoWhilst the ip addresses and email headers etc should be examined meticulously, in the distant hope that they lead somewhere, the chances are that they won't. Very basic opsec.
- miohtama 2y agoBased on working hours Israeli is likely https://rheaeve.substack.com/p/xz-backdoor-times-damned-times-and https://rheaeve.substack.com/p/xz-backdoor-times-damned-time... There are other countries on the same time zone, but only Israel has history of using and selling hacks of this level.
- codezero 2y agoor they are totally nonplussed because they are not even a single individual, and a state actor who is doing this on 10-20 other projects for the past 5-10 years which haven't got the same attention. This is their job, being compromised is always a risk.
- redox99 2y agoIt doesn't matter if it's a team working for a state actor. You'd still be pissed your 2 year project failed so close to the finish line.
- codezero 2y agoNot if this is one of a few dozen or few hundred similar ongoing operations. The risk is always there, they have to expect some amount of failure. Open source software is constantly being probed for vulnerabilities in every way possible, from random commits to high level members of committees for standards. Every operation is one in a grab bag of disposable and deniable efforts. I also am a little biased and assume being burned in state-sponsored acts is similar to the no-blame culture of breaking infrastructure in tech :) because by all accounts this compromise was extremely well done, until it wasn't. Also, we can't be sure the compromise wasn't intentionally telegraphed to cause some other action (using a different library) on purpose.
- HPsquared 2y agoThey might even get a kick out of people discussing how clever their hack was. A rare bit of public acknowledgement, in a way.
- mimimi31 2y ago>Not if this is one of a few dozen or few hundred similar ongoing operations. The risk is always there, they have to expect some amount of failure. That actually makes me think it's not happening at a larger scale, since we'd likely have heard of at least a few similarly elaborate cases being uncovered by now. If not during the attempt itself, then at least at some later point in time. Either almost all of these operations remain undetected, because they are even more sophisticated and much of the world's software ecosystem has been secretly compromised for years or there aren't actually that many such operations.
- aaron695 2y ago[dead]
- macrolime 2y agoI wonder then, whether the attacker has already started over with a new open source project, or found something else to do. Or was xz only one of multiple targets where the overall plan didn't actually fail and an updated backdoor, without the performance issues, will soon be injected again through some other project.
- peddling-brink 2y agoStarted over? They’ve certainly been cultivating other unrelated sock puppet accounts that they’ve switched over to.
- Brian_K_White 2y agoI thougt I read several years ago, like 10 or 15 even, that they could fingerprint people by communication patterns. Like how you can sometimes recognize an author just by the way they write, just by the text itself. Surely that would be something the current ais could do 100x better? So I wonder if there isn't a business there in scanning anonymous posts to associate them with others to deanonymize them all? You could obviously use an ai to rewrite your text, but I bet there is something there that can still be correlated. Simply avoiding using favorite words and phrases doesn't change the essence of whatever you're saying. I think the concept of sockpuppet is about to get complicated. Really it probably already is. If I can imagine something, someone else has probably already been doing it and countering it, and countering that for a decade.
- HPsquared 2y agoThat's a big part of how they caught the Unabomber. Unusual spellings of words and ways of saying things. His brother apparently recognised some of these tics. https://theconversation.com/how-the-unabombers-unique-linguistic-fingerprints-led-to-his-capture-207681#:~:text=For%20example%2C%20the%20Unabomber%20used,they%20were%20never%20widely%20adopted https://theconversation.com/how-the-unabombers-unique-lingui....
- rvnx 2y ago
- grumpyprole 2y agoThose who caught it were indeed very clever. But the attacker did make mistakes, particularly the Valgrind issues and large performance regression. That ultimately is what raised suspicions.
- glandium 2y agoThe large performance regression was a feature of the string obfuscation.
- grumpyprole 2y agoInteresting! So perhaps not really effective obfuscation after all!
- hiisukun 2y ago"Large" is doing quite a bit of work here: the engineer who found it had disabled turbo boost and was, by at least a little serendipity, quite performance focused at the time rather than being busy on other elements of their release target.
- account42 2y agoA 0.5s additional delay added to SSH is HUGE. No way this would not have been noticed sooner or later.
- INTPenis 2y agoThis is why I think it all came falling down due to the teknoraver patch to libsystemd to replace lzma with libarchive. Because anyone has the resources to test this locally and find the issues, but they just didn't have the time anymore.
- cnvogel 2y agoHere’s what I don’t get: why the many layers of obfuscation in the build phase? (I understand why in the binary linked into ssh.) Once the first stage, extracting a shell script from one of the „test“ data blobs, has been found it was clear to everybody that something fishy is going on. It’s inconceivable that I’ve would have found the first stage and just given up, but then it was „only“ a matter of tedious shell reversing… They could easily done without the „striping“ or „awk RC4“, but that must have complicated their internal testing and development quite a bit.
- almostnormal 2y ago> Here’s what I don’t get: why the many layers of obfuscation in the build phase? For a one of its kind deployment it would probably not matter. However, deploying to multiple targets using the same basic approach would allow all of them to be found once one was discovered. With some mildly confsing but different scripting for each target systematic detection of others becomes more difficult.
- makomk 2y agoReduces the attack area through which this could be found, I expect. Without all the obfuscation someone might spot suspicious data in the test data or at some other stage, but this basically forces them to find the single line of suspicious shell script and follow the trail to find the rest of the stuff added to the build process.
- cesarb 2y ago> It’s inconceivable that I’ve would have found the first stage and just given up But what you were looking at might not be the first stage. You might be looking at the modified Makefile. You might be looking at the object files generated during the build. You might be looking at the build logs. You might be investigating a linking failure. The reason for so many layers of obfuscation, is that the attacker had no idea at which layer the good guys would start looking; at each point, they tried to hide in the noise of the corresponding build system step. In the end, this was caught not at the build steps, but at the runtime injection steps; in a bit of poetic justice, all this obfuscation work caused so much slowdown that the obfuscation itself made it more visible. As tvtropes would say, this was a "Revealing Cover-Up" (https://tvtropes.org/pmwiki/pmwiki.php/Main/RevealingCoverup https://tvtropes.org/pmwiki/pmwiki.php/Main/RevealingCoverup) (warning: tvtropes can be addictive)
- deleted 2y ago[deleted]