9 ms·

Ubuntu 24.04 (and Debian) removed libsystemd from SSH server dependencies

117 points by 3v1n0 2y ago

3v1n0 2y ago
As per the well known XZ-utils backdoor, we decided to take a step further and drop libsystemd dependency altogether by implementing the missing bits with few small patches (one upstream, the other to be forwarded).
- aborsy 2y ago
  A back door in the same library is not likely. It drew attention and many looked into it. What could be done to prevent supply chain attacks more broadly?
  - wrayjustin 2y ago
    One way to reduce supply chain risks would be to reduce the supply chain. Less dependencies results in less risk. For example, dropping the libsystemd dependency.
  - bdd8f1df777b 2y ago
    > A back door in the same library is not likely. But libsystemd is not linked to xz only. By removing it, sshd is free of many other potential risks.
    - aborsy 2y ago
      Correct, I misread. Still the question remains: what technology could be implemented to mitigate this type of attack (beyond sshd)? For example, Linux sandboxing is poor, and SeLinux is not usually enforced.
      - fsflover 2y ago
        See: Qubes OS.
      - vimax 2y ago
        Disable calling functions in transitive dependencies, force them to be direct dependencies. Why should sshd be allowed to call an xz function directly without xz being an immediate dependency. I'm not sure what all that would entail with the ifunc stuff, but I remember encountering a glibc linking change moving from Red Hat 6 to RH7 that did something similar and broke the build process for some legacy code.
      - hulitu 2y ago
        > what technology could be implemented to mitigate this type of attack (beyond sshd)? UNIX ? (do one thing and do it well). When your program links with 20 libraries, i have i very hard time to believe that security is one of your goals.
    - metalspoon 2y ago
      I pointed out this weakness the other day on the internet. I got attacked by open source software armies.
      - trust_bt_verify 2y ago
        Thanks for sharing.
      - ranger_danger 2y ago
        FOSS zealots are not always security experts.
      - ratg13 2y ago
        Maybe it was because you weren't pointing out anything new? There was a pull request to stop linking liblzma into libsystemd a month before the backdoor was found https://github.com/systemd/systemd/pull/31550 https://github.com/systemd/systemd/pull/31550 This was likely one of many things that pushed the attackers to work faster, and forced them into making mistakes.
        metalspoon 2y ago
        No. They couldn't get along with the idea that some open source software packages should also sometimes be cut away, just like all other packages. They felt threatened by the idea that open source maintenance can ever go wrong and started attacking me. They argued closed source was worse. That was not my point at all. I was not raising a weakness of open source. I was just pointing out that linking to libsystemd had that kind of problem.
      - flohofwoe 2y ago
        Please don't throw despicable systemd zealots and honorable open source zealots into the same bucket ;)
        ikekkdcjkfke 2y ago
        Plot twist: systemd zealots were all sockpuppets
  - dmitrygr 2y ago
    > What could be done to prevent supply chain attacks more broadly? The UNIX philosophy was right all along - each tool does one simple thing.
    - growse 2y ago
      I can't tell if this is sarcasm or not. Most things we want to do are necessarily complex, so dogmatically adhering to "one simple thing" necessarily drives you towards a towering heap of composed dependencies. If you want to get rid of the supply chain, you want everything specifically to be non-composable, so that everything has to be reinvented from scratch for that specific solution.
      - dmitrygr 2y ago
        Try that supply chain attack on dropbear running as ssh server on devuan. Simple utility on a simple system made of simple utilities
  - 3v1n0 2y ago
    Indeed it's not. But this implies removing dependencies on various libraries, and keep a such important process small is already relevant, despite the fact that it will load PAM libraries, making it quite still prone to issues.
- HackerThemAll 2y ago
  Bravo. On BSD somehow OpenSSH doesn't require systemd, but on Linux it did. Magic.
  - jbverschoor 2y ago
    Why would it require even systemd?
    - angiosperm 2y ago
      Systemd manages daemons. Sshd is a daemon. It knows when sshd has finished starting up, in case something else is waiting on that, and if it dies, and whether it needs to be shut down. If systemd didn't do it, something else would. That is not to say systemd isn't a hypertrophied pig, but it does do important work.
  - VancouverMan 2y ago
    As a Debian user since the 1990s, this situation has once again made me thankful that I've moved important systems I'm responsible for away from Debian and over to OpenBSD and FreeBSD whenever possible. Even if Debian wasn't perfect before systemd was introduced, at least I knew there was a very high probability that I could trust it to function well. That stopped being the case after systemd was introduced. I've had far too many problems caused by systemd, to the point where all trust I had in newer versions of Debian has been lost. Initially, I thought that maybe the problem was with me. But as I investigated the issues I was having with systemd, I'd see so many other bug reports, forum postings, mailing list postings, IRC logs, blog articles, and other online communications from people who were also having many other problems with systemd. Debian offered a much better user experience before it switched to systemd, and a much worse user experience since.
    - TimeBearingDown 2y ago
      It’s interesting you say that? It runs completely counter to my experience coming from the “BSD-style” rc.d init Arch used to use, and migrating to systemd. Any specific issues? I didn’t see any. No offense. One factor may be that Arch prioritizes not patching upstream - helped save them from targeting here, and it doesn’t go overboard with default configs, which I’ve long appreciated. Not to distro-war, I’m very grateful for Debian. My background is finding Linux in the mid-00s and breaking many SuSE, Ubuntu, and one or two Debian systems before finding something I could understand, repair, and maintain in 2008 Arch. systemd accentuated its ability to stay relevant with enterprise Linux, made it even easier to package for, and has been a useful tool in diagnosing service issues and managing bad software for me. I’m not sure how often it’s posted here but Benno Rice formerly of FreeBSD Core Team has an excellent and amusing discussion of systemd’s technical merits. https://youtu.be/o_AIw9bGogo https://youtu.be/o_AIw9bGogo
mrjin 2y ago
I thought Ubuntu is a downstream of Debian, did I take it wrong?
- mistrial9 2y ago
  the relationship shifts, many parts are unclear.. especially after MSFT took direct interest in Ubuntu, with adding custom partition types (edit msft-reserved, msft-data) to the OS installer, and new code to require signed encryption keys to the UEFI boot; keys registered or directly issued by MSFT.
  - k8svet 2y ago
    > MSFT_PRIVATE The only evidence I can find of this... Is another comment from you. (Googled: '"MSFT_PRIVATE" partition'. One result.)
    - mistrial9 2y ago
      OK - try an LUbuntu Install disk for 22.04; run the installer, after Welcome,Location,Keyboard in the Partition task, create a new Partition Table; make a new disk partition, see the list of available partition types, choose ext4; see FLAGS [apple-tv-recovery,bios-grub,boot,diag,hidden,hpservice,lba,legacy-boot,lvm,msft-data,msft-reserved,palo,prep,raid,root,swap]
      - k8svet 2y ago
        Those are standard partition types that have existed for decades.
      - bravetraveler 2y ago
        That's from 'parted', the disk utility - not Canonical, Ubuntu, or even Microsoft Best I can tell support for the 'res' type has existed at least since 2006; search for 'msft': https://git.savannah.gnu.org/gitweb/?p=parted.git;a=commitdiff;h=232dbda https://git.savannah.gnu.org/gitweb/?p=parted.git;a=commitdi... Then in 2012 it was... an individual and Red Hat who introduced the 'msftdata' flag: https://git.savannah.gnu.org/gitweb/?p=parted.git;a=commitdiff;h=e6536360 https://git.savannah.gnu.org/gitweb/?p=parted.git;a=commitdi... I don't get the insinuation, it conflates things that span at least a decade. It sucks that Microsoft is where they are in secure boot... but one can manage their own policies/keys. Support is a good thing, nobody is forced to use secure boot or things signed by Microsoft.
noplacelikehome 2y ago
libsystem is such a bizarre abstraction covering far too much surface area. The name alone is a code smell. Why is the same library used for both internal service management and for services implementing on-demand launches and notifications?
- tomohawk 2y ago
  That is the systemd way, and what many of us have pointed out for years as a major risk of the approach.
  - klysm 2y ago
    I have never seen anybody point to it as a security risk before this happened. Would be happy to see a reference of somebody saying that prior to the xz event
    - angiosperm 2y ago
      Report is that shortly before the hole was reported, a PR had been posted requesting to remove the dependency on xz.
  - noplacelikehome 2y ago
    This reads as though your objection is to the scope of systemd rather than its implementation detail, which isn’t where my objection lies. I have nothing against the service management stack also addressing common principles like logging and on-demand starts a la inetd, but the notion that applications should link against a component of the service manager which is also used by the service manager boggles my tiny mind.
    - andrewshadura 2y ago
      libsystemd is not being used by systemd itself.
- exabrial 2y ago
TomJansen 2y ago
If you requested this 5 years ago every would think you were crazy... Same as the suckless people. They were right after all.
- angiosperm 2y ago
  Seems like the code they added in place of the call to the huge libsystemd should be in a little library that all the other programs that need to perform systemd_notify may use. Seems like systemd should ship that library, or split out the part of their massive (789981 bytes in the text section) library that does just that bit. And probably split out other parts.
- logicprog 2y ago
  > If you requested this 5 years ago every would think you were crazy... That's a straw man. Nobody was ever pushing for people to import all of libsystemd just to use the communication protocol with systemd, that protocol was designed to be very easy and simple to implement on your own precisely so you didn't have to depend on anything else, libsystemd just happened to provide an implementation too, and somebody was lazy and imported that instead, but I seriously doubt that's what anyone thought was best practice. Contrary to popular belief, systemd isn't about linking gigantic binaries and libraries together into a giant blob like everyone things (e.g. the standard nonsense line from detractors that e.g. `systemd-resolved` is "part of systemd" as in "part of the same binary", which it isn't), but about just letting programs talk to each other, so that you can get reliable, featureful integration on your desktop instead of everything being a half-working mess of shims and ad hoc communication, and providing a centralized service that can consistently and from first principles solve certain tasks, so that you don't have to have every single daemon reimplementing their own, or a central implementation that's a big pile of preprocessed shell scripts, spinlocks, edge cases, and bullshit. > Same as the suckless people. They were right after all. Right about what? Right in myopically judging software quality by "lines of code" and setting nonsensical arbitrary line limits, and as a consequence confusing a (poor) map for the territory, because while, yes, "few lines of code" can make software good in some ways (less buggy, etc), it can also make it quite bad in others (less featureful, doesn't handle edge cases, brittle, annoying to use), or just be completely unrelated? Or finding every possible way to vrware software that most definitely "sucks more" for the vast majority of users that don't randomly happen to perfectly align with it, software that sucks so bad people have to maintain patch lists to make it useable, with all the inherent problems with maintainability and stability over time patches incur? Suckless is a cargo cult of tradition following a fundamentalist interpretation of its holy texts, refusing to innovate and actually make computing better, stubbornly sticking to a model left unchanged since the 70s, which wasn't even that great back then, and proud of it.

117 points by 3v1n0 2y ago

As per the well known XZ-utils backdoor, we decided to take a step further and drop libsystemd dependency altogether by implementing the missing bits with few small patches (one upstream, the other to be forwarded).

aborsy 2y ago
A back door in the same library is not likely. It drew attention and many looked into it. What could be done to prevent supply chain attacks more broadly?
- wrayjustin 2y ago
  One way to reduce supply chain risks would be to reduce the supply chain. Less dependencies results in less risk. For example, dropping the libsystemd dependency.
- bdd8f1df777b 2y ago
  > A back door in the same library is not likely. But libsystemd is not linked to xz only. By removing it, sshd is free of many other potential risks.
  - aborsy 2y ago
    Correct, I misread. Still the question remains: what technology could be implemented to mitigate this type of attack (beyond sshd)? For example, Linux sandboxing is poor, and SeLinux is not usually enforced.
    - fsflover 2y ago
      See: Qubes OS.
    - vimax 2y ago
      Disable calling functions in transitive dependencies, force them to be direct dependencies. Why should sshd be allowed to call an xz function directly without xz being an immediate dependency. I'm not sure what all that would entail with the ifunc stuff, but I remember encountering a glibc linking change moving from Red Hat 6 to RH7 that did something similar and broke the build process for some legacy code.
    - hulitu 2y ago
      > what technology could be implemented to mitigate this type of attack (beyond sshd)? UNIX ? (do one thing and do it well). When your program links with 20 libraries, i have i very hard time to believe that security is one of your goals.
  - metalspoon 2y ago
    I pointed out this weakness the other day on the internet. I got attacked by open source software armies.
    - trust_bt_verify 2y ago
      Thanks for sharing.
    - ranger_danger 2y ago
      FOSS zealots are not always security experts.
    - ratg13 2y ago
      Maybe it was because you weren't pointing out anything new? There was a pull request to stop linking liblzma into libsystemd a month before the backdoor was found https://github.com/systemd/systemd/pull/31550 https://github.com/systemd/systemd/pull/31550 This was likely one of many things that pushed the attackers to work faster, and forced them into making mistakes.
      - metalspoon 2y ago
        No. They couldn't get along with the idea that some open source software packages should also sometimes be cut away, just like all other packages. They felt threatened by the idea that open source maintenance can ever go wrong and started attacking me. They argued closed source was worse. That was not my point at all. I was not raising a weakness of open source. I was just pointing out that linking to libsystemd had that kind of problem.
    - flohofwoe 2y ago
      Please don't throw despicable systemd zealots and honorable open source zealots into the same bucket ;)
      - ikekkdcjkfke 2y ago
        Plot twist: systemd zealots were all sockpuppets
- dmitrygr 2y ago
  > What could be done to prevent supply chain attacks more broadly? The UNIX philosophy was right all along - each tool does one simple thing.
  - growse 2y ago
    I can't tell if this is sarcasm or not. Most things we want to do are necessarily complex, so dogmatically adhering to "one simple thing" necessarily drives you towards a towering heap of composed dependencies. If you want to get rid of the supply chain, you want everything specifically to be non-composable, so that everything has to be reinvented from scratch for that specific solution.
    - dmitrygr 2y ago
      Try that supply chain attack on dropbear running as ssh server on devuan. Simple utility on a simple system made of simple utilities
- 3v1n0 2y ago
  Indeed it's not. But this implies removing dependencies on various libraries, and keep a such important process small is already relevant, despite the fact that it will load PAM libraries, making it quite still prone to issues.
HackerThemAll 2y ago
Bravo. On BSD somehow OpenSSH doesn't require systemd, but on Linux it did. Magic.
- jbverschoor 2y ago
  Why would it require even systemd?
  - angiosperm 2y ago
    Systemd manages daemons. Sshd is a daemon. It knows when sshd has finished starting up, in case something else is waiting on that, and if it dies, and whether it needs to be shut down. If systemd didn't do it, something else would. That is not to say systemd isn't a hypertrophied pig, but it does do important work.
- VancouverMan 2y ago
  As a Debian user since the 1990s, this situation has once again made me thankful that I've moved important systems I'm responsible for away from Debian and over to OpenBSD and FreeBSD whenever possible. Even if Debian wasn't perfect before systemd was introduced, at least I knew there was a very high probability that I could trust it to function well. That stopped being the case after systemd was introduced. I've had far too many problems caused by systemd, to the point where all trust I had in newer versions of Debian has been lost. Initially, I thought that maybe the problem was with me. But as I investigated the issues I was having with systemd, I'd see so many other bug reports, forum postings, mailing list postings, IRC logs, blog articles, and other online communications from people who were also having many other problems with systemd. Debian offered a much better user experience before it switched to systemd, and a much worse user experience since.
  - TimeBearingDown 2y ago
    It’s interesting you say that? It runs completely counter to my experience coming from the “BSD-style” rc.d init Arch used to use, and migrating to systemd. Any specific issues? I didn’t see any. No offense. One factor may be that Arch prioritizes not patching upstream - helped save them from targeting here, and it doesn’t go overboard with default configs, which I’ve long appreciated. Not to distro-war, I’m very grateful for Debian. My background is finding Linux in the mid-00s and breaking many SuSE, Ubuntu, and one or two Debian systems before finding something I could understand, repair, and maintain in 2008 Arch. systemd accentuated its ability to stay relevant with enterprise Linux, made it even easier to package for, and has been a useful tool in diagnosing service issues and managing bad software for me. I’m not sure how often it’s posted here but Benno Rice formerly of FreeBSD Core Team has an excellent and amusing discussion of systemd’s technical merits. https://youtu.be/o_AIw9bGogo https://youtu.be/o_AIw9bGogo

mrjin 2y ago

I thought Ubuntu is a downstream of Debian, did I take it wrong?

mistrial9 2y ago
the relationship shifts, many parts are unclear.. especially after MSFT took direct interest in Ubuntu, with adding custom partition types (edit msft-reserved, msft-data) to the OS installer, and new code to require signed encryption keys to the UEFI boot; keys registered or directly issued by MSFT.
- k8svet 2y ago
  > MSFT_PRIVATE The only evidence I can find of this... Is another comment from you. (Googled: '"MSFT_PRIVATE" partition'. One result.)
  - mistrial9 2y ago
    OK - try an LUbuntu Install disk for 22.04; run the installer, after Welcome,Location,Keyboard in the Partition task, create a new Partition Table; make a new disk partition, see the list of available partition types, choose ext4; see FLAGS [apple-tv-recovery,bios-grub,boot,diag,hidden,hpservice,lba,legacy-boot,lvm,msft-data,msft-reserved,palo,prep,raid,root,swap]
    - k8svet 2y ago
      Those are standard partition types that have existed for decades.
    - bravetraveler 2y ago
      That's from 'parted', the disk utility - not Canonical, Ubuntu, or even Microsoft Best I can tell support for the 'res' type has existed at least since 2006; search for 'msft': https://git.savannah.gnu.org/gitweb/?p=parted.git;a=commitdiff;h=232dbda https://git.savannah.gnu.org/gitweb/?p=parted.git;a=commitdi... Then in 2012 it was... an individual and Red Hat who introduced the 'msftdata' flag: https://git.savannah.gnu.org/gitweb/?p=parted.git;a=commitdiff;h=e6536360 https://git.savannah.gnu.org/gitweb/?p=parted.git;a=commitdi... I don't get the insinuation, it conflates things that span at least a decade. It sucks that Microsoft is where they are in secure boot... but one can manage their own policies/keys. Support is a good thing, nobody is forced to use secure boot or things signed by Microsoft.

noplacelikehome 2y ago

libsystem is such a bizarre abstraction covering far too much surface area. The name alone is a code smell. Why is the same library used for both internal service management and for services implementing on-demand launches and notifications?

tomohawk 2y ago
That is the systemd way, and what many of us have pointed out for years as a major risk of the approach.
- klysm 2y ago
  I have never seen anybody point to it as a security risk before this happened. Would be happy to see a reference of somebody saying that prior to the xz event
  - angiosperm 2y ago
    Report is that shortly before the hole was reported, a PR had been posted requesting to remove the dependency on xz.
- noplacelikehome 2y ago
  This reads as though your objection is to the scope of systemd rather than its implementation detail, which isn’t where my objection lies. I have nothing against the service management stack also addressing common principles like logging and on-demand starts a la inetd, but the notion that applications should link against a component of the service manager which is also used by the service manager boggles my tiny mind.
  - andrewshadura 2y ago
    libsystemd is not being used by systemd itself.
exabrial 2y ago

TomJansen 2y ago

If you requested this 5 years ago every would think you were crazy... Same as the suckless people. They were right after all.

angiosperm 2y ago
Seems like the code they added in place of the call to the huge libsystemd should be in a little library that all the other programs that need to perform systemd_notify may use. Seems like systemd should ship that library, or split out the part of their massive (789981 bytes in the text section) library that does just that bit. And probably split out other parts.
logicprog 2y ago
> If you requested this 5 years ago every would think you were crazy... That's a straw man. Nobody was ever pushing for people to import all of libsystemd just to use the communication protocol with systemd, that protocol was designed to be very easy and simple to implement on your own precisely so you didn't have to depend on anything else, libsystemd just happened to provide an implementation too, and somebody was lazy and imported that instead, but I seriously doubt that's what anyone thought was best practice. Contrary to popular belief, systemd isn't about linking gigantic binaries and libraries together into a giant blob like everyone things (e.g. the standard nonsense line from detractors that e.g. `systemd-resolved` is "part of systemd" as in "part of the same binary", which it isn't), but about just letting programs talk to each other, so that you can get reliable, featureful integration on your desktop instead of everything being a half-working mess of shims and ad hoc communication, and providing a centralized service that can consistently and from first principles solve certain tasks, so that you don't have to have every single daemon reimplementing their own, or a central implementation that's a big pile of preprocessed shell scripts, spinlocks, edge cases, and bullshit. > Same as the suckless people. They were right after all. Right about what? Right in myopically judging software quality by "lines of code" and setting nonsensical arbitrary line limits, and as a consequence confusing a (poor) map for the territory, because while, yes, "few lines of code" can make software good in some ways (less buggy, etc), it can also make it quite bad in others (less featureful, doesn't handle edge cases, brittle, annoying to use), or just be completely unrelated? Or finding every possible way to vrware software that most definitely "sucks more" for the vast majority of users that don't randomly happen to perfectly align with it, software that sucks so bad people have to maintain patch lists to make it useable, with all the inherent problems with maintainability and stability over time patches incur? Suckless is a cargo cult of tradition following a fundamentalist interpretation of its holy texts, refusing to innovate and actually make computing better, stubbornly sticking to a model left unchanged since the 70s, which wasn't even that great back then, and proud of it.