9 ms·
Debian on xz-utils: revert to version that does not contain changes by bad actor
- jgb1984 3y agoMaybe it's time to move away from xz which has been criticized in the past as a not great format, has had maintainer issues for years, and now turns out to be infiltrated with security problems. Moving to zstd seems worthwhile.
- angiosperm 3y agoBest would be to disable anything in sshd that can perform compression or decompression. There is no value in that capability, and way too much risk.
- lifthrasiir 3y agoAgreed on the SSH compression, but it would be not relevant to this particular vulnerability because liblzma was never used in the SSH compression anyway.
- deleted 3y ago[deleted]
- cpach 3y agoI don’t really understand the connection between liblzma and sshd, it had something do with systemd, or…?
- lifthrasiir 3y agoApparently liblzma in the compromised tarball would search for sshd functions and replace them with their own versions. This normally doesn't happen for the reason I've described, but many distros tend to compile sshd with a plugin that makes use of libsystemd, which depends on liblzma and therefore sshd will be linked with liblzma in that case. Liblzma could have done anything else it wanted, so the limited reach of this vulnerability is actually a better scenario...
- dboreham 3y agoYes.
- cesarb 3y agoThe connection is that sshd calls code in libsystemd when running as a systemd service, to notify systemd that it finished starting. An unrelated functionality in libsystemd (my guess is that it's related to parsing journald logs, since they can use LZMA compression) uses the lzma library, so libsystemd is linked against it. That is, the chain is: sshd is linked against libsystemd, which is linked against liblzma. When loading the sshd executable, the dynamic linker will load all these libraries. The backdoor was added to initialization code in liblzma (which runs whenever the dynamic linker loads that library). So that malicious code runs before sshd starts executing, and manipulates the dynamic linker so that it will do something (which is still being analyzed AFAIK) when some specific functions are called by the sshd executable.
- angiosperm 3y agoWow, thank you! So it is wrong for such a fundamental service as systemd to depend on such complicated processes ... just as the anti-systemd people said. Except of course shellscripts and /bin programs are just as complicated, when you check.
- begueradj 3y agoWe could say something similar about other tools. Even something like OpenSSL had a severe security issue (Heartbleed) discovered on 2014 and which was introduced 2 years earlier by the maintainer himself through a Git commit which occurred, if I remember, on the middle of the night of 31.12.2011 or 01.01.2012 (which lead some people to speculate whether that was intentional since "everyone" is usually celebrating with his/her family at that time of the year)
- Maxious 3y agoPeople did indeed move to LibreSSL and s2n-tls after that one incident
- cpach 3y agoAnd later on crypto/tls, etc.
- liquidpele 3y agoThe maintainer merged the patch, but the code change came from someone else IIRC.
- pmontra 3y agoIt's more or less what the current last comment on the Debian thread is wondering about "Is xz-utils going to be maintained? Will we want to keep in the archive an unmaintained low-level library - low-level as in, susceptible of getting pulled as a dependency in lots of places - and rely on it for components such as dpkg?"
- jl6 3y agoThe format is fine, and zstd would be a downgrade in terms of compression ratio (which is the main reason xz is chosen for its current use cases). The lesson to learn here is not about making technical improvements, it’s to do something about improving the way we manage and maintain core open source infrastructure. Ref XKCD 2347.
- lifthrasiir 3y agoNo, xz was chosen mainly because zstandard didn't exist at that time [1]. Zstandard does almost completely replace those "expensive" compression formats by providing a comparable compression ratio but being much faster. [1] https://lists.debian.org/debian-devel-announce/2011/08/msg00001.html https://lists.debian.org/debian-devel-announce/2011/08/msg00...
- jl6 3y ago"almost" and "comparable", but xz (lzma) still compresses at a higher ratio, which is still a highly desirable feature. If the decision was being made today, a good faith debate could be had over whether the distro packaging use case benefits more from faster decompression or smaller file size, but I suspect that, globally, the average PC can decompress xz faster than the average broadband connection can download it.
- lifthrasiir 3y agoI will assume that the average effective broadband connection is about 20 MB/s, and both xz and zstd decompressions are faster than the download under this assumption. But there are lesser known compression algorithms that are more efficient in terms of compression ratio for this target decompression speed, like zpaq [1], which are rarely considered as alternatives. So multiple factors exist here and I believe zstd excels at pretty much every other factor compared to xz, making it a very likely choice. [1] https://mattmahoney.net/dc/zpaq.html https://mattmahoney.net/dc/zpaq.html
- pgeorgi 3y agoThe format is fine, except for https://www.nongnu.org/lzip/xz_inadequate.html https://www.nongnu.org/lzip/xz_inadequate.html
- LeoPanthera 3y agoMigrating to zstd instead of Lzip seems an odd decision. Presumably xz was chosen to begin with because of its high level of compression, migrating to zstd would make every single package considerably larger. Lzip uses the same compression algorithm as xz, but is otherwise a simpler and better-designed format, without any of xz's bizarre countainer-inside-container design decisions.
- Sakos 3y agoIt looks like Lzip only has a single maintainer and there's no public repository to track commits or contributors (at least I couldn't find any). I don't think it's a viable option.
- fabian2k 3y agoI think it makes sense to use a commonly used format here, the maintenance aspects are very important as this incident has shown. And I don't think the tradeoffs between compressed size and speed are entirely obvious. Trading off slightly larger files for easier maintenance would seem like a defensible choice to me.
- arp242 3y agoIt doesn't even have a git/mercurial/CVS repo; just .tar.gz on some random website. And installation instructions are: tar -xf lzip[version].tar.gz cd lzip[version] && ./configure && make check The entire thing seems entirely the work by one person. Yeah, this is going to massively improve things. I foresee no plausible way we can gave a repeat if we switch to this lzip thing. Absolutely watertight. We learned from past events. Problem solved.
- rurban 3y agoAbsolutely not. Extecatly this recipe brought us this desaster. The backdoor was in the released tar only, the sources were fine, because such a malicious m4 would have been detected. The compressed test bin containing the backdoor was in git, but such unobvious things do get attention.
- zxspectrum1982 3y agoGenuine question: how do we know that Jia Tan is a bad actor and not simply an inept coder that introduced a bug?
- _zephyrus_ 3y agoPlease, take a look at how the exploit is crafted. It is not a bug in a piece of code.
- sligor 3y agoIn what I understand: we don't know for sure, but the suspicions are so high that it more important to act quickly and revert the changes. The community will analyse later what the changes are exactly doing ( less urgent than reverting the changes )
- Sesse__ 3y agoWe definitely know for sure this was inserted intentionally. What we don't know for sure is whether Jia Tan (whoever that might be, if it is even a single person) did it on purpose, or if they had their account compromised.
- mr_mitm 3y agoThis attack was so sophisticated that I don't understand half of it after a first read. It's plainly obvious that it is not a simple typo or a blunder but very carefully crafted.
- soneil 3y agoFirst, the "bug" jumps through several well-crafted hurdles in order to hide itself from co-maintainers, package maintainers, and anyone else who might be looking. Hiding code in binary objects, hidden within in testcases and injected into the build under just-so conditions, would be incredibly difficult to justify with any other intent. Next, the affected codepath skips itself if it's not executed under just-so conditions. Specifically testing to see if it's being run interactively to avoid diagnosis, and testing to see if the executed binary is sshd. Lastly, what good reason could a compression library possibly have, for overriding encryption routines in sshd? It's a fair question, but it doesn't survive inspection at all. This is like finding someone in a ninja suit, wearing night vision equipment, creeping around a museum in the middle of the night, with the crown jewels in a bag slung over their shoulder - and asking yourself if they're perhaps a lost tourist.
- toxik 3y agoWhen is Google, Amazon, Facebook, the DARPA, the EU - ANYONE going to step up to the plate and fund developer time for such obviously critical infrastructure? Why could a single Chinese guy undermine the entire system with so little effort?
- ddalex 3y agoHow do you know it is a guy, and it is Chinese?
- toxik 3y agoI take it you haven’t looked into the details of this, there is no proof but there is strong evidence in name choice, language use and the MO.
- jeltz 3y agoName choice I disagree with. The attacker used at least Chinese, Indian and Scandinavian names when working on this. It is possible we can find out something by doing a statistical analysis of the language but as far as I know no such analysis has been done. Edit: A Russian sounding alias was also used.
- kijin 3y agoMy current hypothesis is that the attacker(s) deliberately chose a Chinese-sounding name to make people jump to conclusions. A sophisticated state actor would be no stranger to making their actions look like somebody else's. Or maybe they actually are Chinese, and want me to believe the sentence above. :)
- lodovic 3y agoIndeed - seeing the level of sophistication and long term thinking behind this attack, it seems everything about it was planned meticulously, including usernames, attack vectors, communication with the community, and when to commit the parts of the exploit. There's probably a full team of people behind it and attributing it to China because of the username may be too quick.
- perihelions 3y agoDebian-stable users, once again, remain unaffected by $currentDumpsterFire. (Experts: how do you weigh the risks of the two choices against each other? Running bleeding-edge upgrades brings in these fresh security holes before anyone's noticed them. But, running "stable" software can mean running vulnerable software that upstream has already patched. Not every vulnerability is recognized as a vulnerability, and patched as a security update; many "ordinary" bugs have security consequences which aren't obvious to recognize. Which choice is less worse?)
- kijin 3y agoPackages in "stable" versions get bugfixes, too. How often obviously depends on the maintainer, but anything marked as a security issue upstream tends to be patched fairly quickly. On the whole, I would assume that someone who maintains a Debian or RHEL package is better qualified to assess the severity of a bug than I am. The bigger problem is that xz hasn't had a proper maintainer for the better part of a decade.
- perihelions 3y agoAn example that worried me was this Linux kernel bug [0], which in hindsight turned out to be a severe vulnerability, but didn't get a Debian-stable security update for three months. The vulnerability aspect wasn't obvious in the initial reports. Is this common? Is it more common in software that has lower profile than the Linux kernel—software with fewer maintainers and fewer eyeballs, and fewer security-focused people flagging the right issues in the right ways to downstream? [0] https://news.ycombinator.com/item?id=28926426 https://news.ycombinator.com/item?id=28926426
- cyanydeez 3y agoIf everyone is running LTS, then discovery just runs to LTS. the reason $currentDumpsterFire is identified is only because it's being used. Cannon fodder exists for a purpose.
- kijin 3y ago
- stephc_int13 3y agoThe big lesson from this is that we maybe should spend a bit more time hardening the build process instead of rewriting everything in Rust. Also, I am a strong believer of strength by diversity, especially for anything that is safety-critical.
- lifthrasiir 3y agoOr do both, because it also turns out that Rust build process is a real joy compared to that of C/C++ [1]. [1] Except when there are C/C++ dependencies :-)
- uecker 3y agoIt is a joy but makes it far too easy to pull in dependencies. And then Build.rs also runs. From a supply chain security perspective, Rust scares me.
- lifthrasiir 3y agoI'm fully supportive to sandbox build.rs for that reason, but the same concern applies to pretty much everything in your computer so I'm not worried as much. In the other words, that risk is currently constant with respect to the choice of language.
- armitron 3y agoIt's not constant, most C projects don't have anywhere near the number of library dependencies seen in your typical cargo project. From a supply chain attack perspective, Rust and Node promote ecosystems that are total disasters.
- lifthrasiir 3y agoMost software projects are not written in C. Conversely C would be used for projects where C would be less annoying than alternatives, and that condition largely precludes projects with many required dependencies. And every language-level package manager, not just that of Rust and Node, would be a disaster from the supply chain attack perspective---it can even be said that C is suffering from the huge amount of supply chain attacks in spite of the lack of good language-level package manager.
- cpach 3y agoMain thread here: https://news.ycombinator.com/item?id=39865810 https://news.ycombinator.com/item?id=39865810
- sampo 3y agoA really long con would be to first identify some component than already has a naturally occurring, so far unknown hard-to-find vulnerability, and then slowly add this kind of injected vulnerability to it. If the first vulnerability is eventually accidentally fixed by some normal refactoring, and if the second vulnerability if eventually found, everyone will revert back to the old version and now you can again use the first vulnerability.
- influx 3y agoThis is the second time Devin changes to ssh have led to vulnerabilities.
- cpach 3y ago?
- fragmede 3y agoCVE-2008-0166 It was really bad. https://github.com/g0tmi1k/debian-ssh https://github.com/g0tmi1k/debian-ssh
- deleted 3y ago[deleted]
- cpach 3y agoYes I know that one. But I don’t really see a conection still. The xz backdoor was not introduced by Debian, it was introduced by a bad faith actor via the xz project (aka liblzma). Not only Debian was affected, Fedora was too. AFAIK there is no evidence that CVE-2008-0166 was done deliberately, whereas with the xz backdoor, there seems to be lots of signs of deliberate intent.
- armitron 3y agoThe backdoor was possible because Debian introduced a dependency into sshd, libsystemd, which in turn linked to liblzma. Official sshd does not have a systemd dependency in the same way that official sshd back in 2008 did not suffer from a predictable PRNG. Debian is also 100% at fault here and I hope this slap in the face makes them reconsider their poor security practices. Amateur package maintainers should not be overstepping their job (packaging) and enabling disastrous vulnerabilities in security-critical software.
- cpach 3y agoYou have a point there. For the record, Fedora was affected too.
- avsteele 3y agoIt would appear that no one knows the RL identity of the bad actor? Or even vouches for them? It doesn't seem wise to accept contributions to projects which may have security implications from anons. Code contributions clearly can't or aren't independently validated.
- thewavelength 3y agoPlease read this email [0] from the original author of xz. Try take his perspective in his described situation. I also recommend this summary [1]. [0] https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.html https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.h... [1] https://boehs.org/node/everything-i-know-about-the-xz-backdoor https://boehs.org/node/everything-i-know-about-the-xz-backdo...
- avsteele 3y agoI am not assigning any personal blame or making an accusation of bad judgement on the maintainer. Taking contributions from anons appears to be common. I am suggesting that should change.
- lifthrasiir 3y agoContributions from anons can and in most cases will be verified by maintainers. The actual problem here is appointing them as maintainers. EDIT: To be clear, this problem is not directed to the original xz maintainer, but more about how to prevent or reduce such appointments in the first place.
- boomboomsubban 3y agoWhat do you propose? What would constitute a meaningful verification of identity that wouldn't create a near insurmountable hurdle for newcomers or those unable to travel?
- redbar0n 3y agoPasskeys?
- nialv7 3y agoGood question was asked by debian devs: what's the future for xz now? I feel it is not really worth it to vet 2 years' worth of changes by a malicious actor, who had been actively trying to make any exploits add by them as hard to discover as possible, when good alternatives, like zstd, exist.
- deleted 3y ago[deleted]
- lostmsu 3y agoWindows might also be affected: they just expanded built-in explorer archive format support with libarchive, which depends on one of affected libraries AFAIU: https://news.ycombinator.com/item?id=36052169 https://news.ycombinator.com/item?id=36052169