11 ms·

> openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depen

• capitainenemo 3y ago
  FWIW, I did a quick check on a Devuan system. The sshd in Devuan does link to a libsystemd stub - this is to cut down on their maintenance of upstream packages. However that stub does not link to lzma.
  • cf100clunk 3y ago
    On an MX Linux (non-systemd Debian-derived distro) box I ran ldd on /sbin/ssh and also ran: [EDIT: this string gives cleaner results:] lsof -w -P -T -p $(pgrep sshd)|grep mem and saw liblzma in the results of both, so there is some sort of similar trickery going on.
    • capitainenemo 3y ago
      Huh. That's rather surprising. Do you know how MX Linux handles systemd? Devuan does that shimming of upstream. Do they perhaps just try to leave out certain packages? Anyway. I did not see lzma in the results on Devuan running a process check (just in case). I did see it on a Debian.
      • cf100clunk 3y ago
        It turns out MX uses a package called systemd-shim that seems to be the Debian one: $aptitude show systemd-shim Package: systemd-shim Version: 10-6 State: installed Automatically installed: no Priority: extra Section: admin Maintainer: Debian QA Group <packages@qa.debian.org> Architecture: amd64 Uncompressed Size: 82.9 k Depends: libc6 (>= 2.34), libglib2.0-0 (>= 2.39.4), cgmanager (>= 0.32) Suggests: pm-utils Conflicts: systemd-shim:i386 Breaks: systemd (< 209), systemd:i386 (< 209) Description: shim for systemd This package emulates the systemd function that are required to run the systemd helpers without using the init service
• delroth 3y ago
  > The systemd notification protocol could have been as simple as just writing a newline to a pipe It basically is. libsystemd links to liblzma for other features not related to notifications. (The protocol is that systemd passes the path to a unix socket in the `NOTIFY_SOCKET` env variable, and the daemon writes "READY=1" into it.)
  • agwa 3y ago
    Is that protocol documented/stable? For whatever reason, daemons are choosing to link to libsystemd instead of implementing it themselves. It doesn't matter that libsystemd links to liblzma for other reasons. It's still in the address space of any daemon that is using libsystemd for the notification protocol.
    • wickberg 3y ago
      I know Golang has their own implementation of sd_notify(). For Slurm, I looked at what a PITA pulling libsystemd into our autoconf tooling would be, stumbled on the Golang implementation, and realized it's trivial to implement directly.
      • pkaye 3y ago
        Can me point me to the Golang implementation? Is it a standard package?
        • yencabulator 3y ago
          Most likely https://github.com/coreos/go-systemd https://github.com/coreos/go-systemd
      • tripflag 3y ago
        indeed; it should be trivial in any language. Here's python: https://github.com/9001/copyparty/blob/a080759a03ef5c0a6b06c697f37ae4e00812739f/copyparty/svchub.py#L1019-L1038 https://github.com/9001/copyparty/blob/a080759a03ef5c0a6b06c...
• fullstop 3y ago
  Also thanks to Debian for modifying openssh.
  • cassianoleal 3y ago
    You're not wrong. Had Debian not patched it in this way, OP might have never found it, leaving all other distros who do the same vulnerable. Note that OP found this in Debian sid as well, which means it's highly unlikely this issue will find its way into any Debian stable systems.
    • fullstop 3y ago
      Right, the systemd notification framework is very simple and I've used it in my projects. I didn't even know that libsystemd provided an implementation. My Arch system was not vulnerable because openssh was not linked to xz. IMO every single commit from JiaT75 should be reviewed and maybe even rolled back, as they have obliterated their trust. edit: https://github.com/google/oss-fuzz/pull/10667 https://github.com/google/oss-fuzz/pull/10667 Even this might be nefarious.
      • gopher_space 3y ago
        > the systemd notification framework is very simple and I've used it in my projects Have you come across an outline or graph of systemd that you really like, or maybe a good example of a minimal setup?
    • SAI_Peregrinus 3y ago
      If they hadn't been modifying SSH their users would never have been hit by this backdoor. Of course if it is actually intended to target SSH on Debian systems, the attacker would likely have picked a different dependency. But adding dependencies like Debian did here means that those dependencies aren't getting reviewed by the original authors. For security-critical software like OpenSSH such unaudited dependencies are prime targets for attacks like this.
      • 3y ago
• Jasper_ 3y ago
  That is all the protocol is. From https://www.freedesktop.org/software/systemd/man/latest/sd_notify.html https://www.freedesktop.org/software/systemd/man/latest/sd_n...: > These functions send a single datagram with the state string as payload to the socket referenced in the $NOTIFY_SOCKET environment variable. The simplest implementation (pseudocode, no error handling, not guaranteed to compile), is something like: const char *addrstr = getenv("NOTIFY_SOCKET"); if (addrstr) { int fd = socket(AF_UNIX, SOCK_DGRAM, 0); struct sockaddr_un addr = { .sun_family = AF_UNIX }; strncpy(addr.sun_path, sizeof(addr.sun_path), addrstr); connect(fd, (struct sockaddr*) &addr); write(fd, "READY=1"); close(fd); }
  • Repulsion9513 3y ago
    Whoops, you forgot `vsock:`, `@`, `SO_PASSCRED` (I think)... oh and where is that example provided? But yep that's all the protocol is for sure (and forever)!
  • throwaway71271 3y ago
    goddamnit leftpad got us too :)
  • iforgotpassword 3y ago
    This is what I did for a daemon I'm maintaining. Type=notify support was requested but I'm really allergic to adding new libs to a project until they really do some heavy lifting and add enough value. I was pleasantly surprised the protocol was that simple and implemented it myself. I think systemd should just provide a simple standalone reference implementation and encourage people to copy it into their project directly. (But maybe they already do, I did that almost a decade ago IIRC when the feature was relatively new.)
    • deleted 3y ago
      [deleted]
  • deleted 3y ago
    [deleted]
• b112 3y ago
  One of the objections that many people do not understand, is that systemd adds complexity. Unnecessary complexity. Boats full, loads full, mountains full of complexity. Yes, there are things delivered with that complexity. However, as an example, sysvinit is maybe, oh, 20k lines of code including binaries, heck including all core init scripts. What's systemd? 2M lines? It was >1M lines 4+ years ago. For an init system, a thing that is to be the core of stability, security, and most importantly glacial, stable change -- that is absurdly complex. It's exceedingly over engineered. And so you get cases like this. And cases like that, and that over there, and that case over there too. All which could not exist, if systemd didn't try to overengineer, over complicate everything. Ah well. I'm still waiting for someone to basically fork systemd, remove all the fluff (udev, ntp, dns, timers, restart code, specialized logging, on and on and on), and just end up with systemd compatible service files. But not yet. So... well, oh well.
  • gavinhoward 3y ago
    I have a design in the works to do just this. The problem? It's on the backburner because I don't think I could find a business model to make money from it. I don't think offering support for a price would work, for example.
    • adr1an 3y ago
      What about sponsors? Actually, now I have the idea of a platform similar to Kickstarter but for software development, and with just sponsors. It wouldn't work, sure... Except in some cases. Like when things like this happen...
      • gavinhoward 3y ago
        Sponsors are fickle, unfortunately, and they tend to remove "donations" when money gets tight. If I am considered a full vendor, though, and a vendor for a critical piece of software, they might keep me around.
    • iforgotpassword 3y ago
• bennyhill 3y ago
  > so now security-critical daemons like openssh have additional dependencies like liblzma Systemd itself seems security-critical to me. Would removing other dependencies on libsystemd really make a secure system where systemd was compromised through its library?
  • agwa 3y ago
    1. systemd (at least the PID 1 part) does not talk to the network, so a remotely-accessible backdoor would need to be more complex (and thus more likely to be detected) than a backdoor that can be loaded into a listening daemon like openssh. 2. You can run Debian systems without systemd as PID 1, but you're still stuck with libsystemd because so many daemons now link with it.
    • capitainenemo 3y ago
      .. well, you can use a shim package as devuan did.
    • chasil 3y ago
      > systemd... does not talk to the network... Socket activation and the NFS automounter appear to. If I run "netstat -ap" I see pid 1 listening on enabled units. Edit: tinysshd is specifically launched this way. Edit2: there is also substantial criticism of xz on technical grounds. https://www.nongnu.org/lzip/xz_inadequate.html https://www.nongnu.org/lzip/xz_inadequate.html
• poettering 3y ago
  Uh. systemd documents the protocol at various places and the protocol is trivial: a single text datagram sent to am AF_UNIX socket whose path you get via the NOTIFY_SOCKET. That's trivial to implement for any one with some basic unix programming knowledge. And i tell pretty much anyone who wants to listen that they should just implement the proto on their own if thats rhe only reason for a libsystemd dep otherwise. In particular non-C environments really should do their own native impl and not botjer wrapping libsystemd just for this. But let me stress two other things: Libselinux pulls in liblzma too and gets linked into tons more programs than libsystemd. And will end up in sshd too (at the very least via libpam/pam_selinux). And most of the really big distros tend do support selinux at least to some level. Hence systemd or not, sshd remains vulnerable by this specific attack. With that in mind libsystemd git dropped the dep on liblzma actually, all compressors are now dlopen deps and thus only pulled in when needed.
  • o11c 3y ago
    Deferring the load of the library often just makes things harder to analyze, not necessarily more secure. I imagine many of the comments quoting `ldd` are wrongly forgetting about `dlopen`. (I really wish there were a way to link such that the library isn't actually loaded but it still shows in the metadata, so you can get the performance benefits of doing less work but can still analyze the dependency DAG easily)
    • poettering 3y ago
      It would make things more secure in this specific backdooring case, since sshd only calls a single function of libsystemd (sd_notify) and that one would not trigger the dlopen of liblzma, hence the specific path chosen by the backdoor would not work (unless libselinux fucks it up fter all, see other comments) Dlopen has drawbacks but also major benefits. We decided the benefits relatively clearly outweigh the drawbacks, but of course people may disagree. I have proposed a mechanism before, that would expose the list of libs we potentially load via dlopen into an ELF section or ELF note. This could be consumed by things such as packagae managers (for auto-dep generation) and ldd. However there was no interest in getting this landed from anyone else, so I dropped it. Note that there are various cases where people use dlopen not on hardcoded lib names, but dynamically configured ones, where this would not help. I.e. things like glibc nss or pam or anything else plugin based. But in particular pam kinda matters since that tends to be loaded into almost any kind of security relavant software, including sshd.
• mardifoufs 3y ago
  What? I don't get it? Isn't it on Debian if they modified the package to do something like this? Why would you blame systemd for maintainers doing something that upstream has never required or recommended?
• deleted 3y ago
  [deleted]
• hnald 3y ago
  It's unfortunate that the anti-systemd party lost the war... years ago. But I don't blame systemd, Lennart Pottering or the fanboys (though it would have been so much better if the guy never worked in open source or wasn't such a prolific programmer). I blame Debian and its community for succumbing to this assault on Unix philosophy (again, years ago).
  • 1attice 3y ago
    Sometimes things evolve in ways that make us feel a little obsolete. I've been learning NixOS for a few years now, and it would have been impossible without systemd. It's one heck of a learning curve, but when you get to the other side, you know something of great power and value. Certain kinds of complexity adds 'land' (eg. systemd) that can become 'real estate' (eg. NixOS), which in turn hopes to become 'land' for the next innovation, and so forth. Whether this happens or not (whether it's the right kind of complexity) is really hard to assess up-front, and probably impossible without knowing the complex new technology in question very well. (And by then you have the bias of depending, in part, yourself on the success of the new tech, as you've committed significant resources to mastering it, so good luck on convincing skeptical newcomers!) It's almost like a sort of event horizon -- once you know a complex new technology well enough to see whether or not it's useful, the conflict-of-interest makes your opinion unreliable to outsiders! Nevertheless, the assessment process itself, while difficult to get right, is worth getting better at. It's easy for impatience and the sensation of what I've taken to calling 'daunt' -- that intrinsic recoil that the mind has from absorbing a large amounts of information whose use case is not immediately relevant -- to dissuade one from exploring. But then, one never discovers new 'land', and one never builds new real estate! [ Aside: This is why I'm a little skeptical of the current rebellion against frontend frameworks. Certainly some of them, like tailwind, are clearly adding fetters to an otherwise powerful browser stack. But others, like Svelte, and to some extent, even React, bring significant benefits. The rebellion has this vibe like, well, users _should_ prefer more simply-built interfaces, and if they don't, well, they just have bad taste. What would be more humble would be to let the marketplace (e.g. consumers) decide what is preferable, and then build that. ]
• shirro 3y ago
  The notify protocol isn't much more complicated than that. From memory you send a string to a unix socket. I have written both systemd notify and listenfd in a few languages for little experiments and it is hard to imagine how the protocols could be simpler. Looking at most popular projects these days they are a mass of dependencies and I think very few of them can be properly audited and verified by the projects that use them. Rust and Go might be more memory safe than C but look at the number of cargo or go modules in most projects. I have mostly stopped using node/npm on my systems.
• esbranson 3y ago
  Not a programmer, but couldn't the distribution's sshd patches for systemd (and all other distro patches for privileged daemons) use static includes? Wouldn't that have only pulled in the simple client-side communication API? Would that have defeated this vector? Would it be doable?