7 ms·

Supply Chain Attacks on Linux Distributions – Fedora Pagure

216 points by akyuu 2y ago

ufo 2y ago
I've got to say, Git resignifying -- and requiring --end-of-options instead is bonkers
- musicale 2y ago
  The pathway of untrusted/malicious input -> trusted command line argument seems to be a common problem, and one that could possibly be mitigated by better type/taint checking. It looks like there is some prior work in this area, but it hasn't resulted in commonly available implementations (even something basic like a type/taint checking version of exec() etc. on one side and getopt() etc. on the other.)
  - PhilipRoman 2y ago
    I could've sworn I remember something about bash and glibc cooperating to indicate which arguments come from expanded variables but I cannot find anything on the internet or in the sources. Either I'm going insane or it was an unmerged proposal.
    - BobbyTables2 2y ago
      Seems like it would be quite painful to do that in C without heavy refactoring. Maybe a Rust alternative ?
      - yjftsjthsd-h 2y ago
        Why would that be painful in C? The code is already handling all the pieces
        pluto_modadic 2y ago
        Painful to write vs painful to audit for correctness. Writing code that you barely understand is a surefire recipe to have /no clue/ how to debug said code (because debugging it will be more complex). (let alone audit it for correctness/security).
        yjftsjthsd-h 2y ago
        That just sounds like you don't believe it's ever possible to change existing C code, which... is a position you can argue, but I'm pretty sure that bash and glibc are actively developed to the point where I wouldn't personally commit to that position.
        ForOldHack 2y ago
        Insert obligatory ref to "on trusting trust" what is the compiler doing? Has the pre-processor been vetted? This is looking more and more like hardware solutions are becoming more attractive.
        yjftsjthsd-h 2y ago
        1. I don't see why you need to solve Trusting Trust to make libc and the shell more robust. 2. If we are worried about Trusting Trust, then Rust is worse; at least C has the wide range of compilers needed for diverse double-compiling and as of https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/ https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-... we arguably have a working solution. Rust only has a single compiler, and that compiler is used to build itself, making it the poster child for Trusting Trust targets.
        steveklabnik 2y ago
        Rust has a second compiler, mrustc, written in C++ and that is able to bit-reproduce rustc. This has been the case for a few years now.
    - jwilk 2y ago
      It was a thing back in 1996/1997: https://sourceware.org/cgit/glibc/commit/?id=bf079e19f50d64aa5e05b5e1 https://sourceware.org/cgit/glibc/commit/?id=bf079e19f50d64a...
      - PhilipRoman 2y ago
        Yeah this is exactly what I was thinking of, thanks for finding it.
      - musicale 2y ago
        > "disabled later" because "it caused problems" ;-(
- pests 2y ago
  Sadly due to legacy. —- disambiguates revisions and paths in some commands so another option was needed.
  - immibis 2y ago
    Just write all commands with structured I/O instead, like Powershell. Now I want an operating system where everything is a YANG model...
superb_dev 2y ago
> We can’t change git’s shell to /sbin/nologin or /bin/false, or users wouldn’t be able to connect over SSH. Git actually has a solution for this! I don’t know if it would work with the custom python stuff going on, but you can set the login shell to `git-shell`
- jbverschoor 2y ago
  Or just use git over https. And heck, if that's such a big problem, switch to a vcs that you can properly manage
  - xenophonf 2y ago
    You shouldn't use Git over HTTPS. With SSH, you can use a hardware authenticator that requires both proof of ownership (i.e., the unlock PIN) and proof of possession (i.e., physical touch) out of the box. That's technically possible over HTTPS, of course, but I have yet to see a Git server that works that way.
    - jbverschoor 2y ago
      Why not? Your password manager or passkeys do the same thing. Heck, just true the public key as the authentication header. Or use authentication certificates. If ssh is such a big problem, use something else
      - rwteueru 2y ago
        [flagged]
        arccy 2y ago
        credential helpers: like the github cli: https://cli.github.com/manual/gh_auth_setup-git https://cli.github.com/manual/gh_auth_setup-git or git-credential-manager: https://github.com/git-ecosystem/git-credential-manager https://github.com/git-ecosystem/git-credential-manager
blueflow 2y ago
Because bash for some goddamn reason loads the bashrc for interactive shells AND when started by sshd, regardless of whether the shell is interactive or an tty is present. Bash (and only bash) literally has a special case for sshd to enable this kind of exploit. As a result of this, git and rsync wont work at all if the bashrc on the remote machine writes any data to stdout. Like setting a window title. To work around that, every bashrc on this earth needs a case switch to return early to avoid this specific bug.
- yjftsjthsd-h 2y ago
  Wait, ssh doesn't let you specify the remote command without running it through the user's shell? That seems like a deficiency too IMHO.
  - nurettin 2y ago
    It is simpler to export paths locally, so the remote doesn't have to know your file/folder structure.
    - yjftsjthsd-h 2y ago
      Sure, it's convenient to have PATH, but why not have an optional way to say "hey, ssh, run /usr/bin/rsync on the remote with the following arguments, directly and without a shell please"? Equivalent to a Dockerfile containing cmd foo vs cmd ["/bin/foo"] (IIRC, at least - been a minute since I needed to do this.)
      - pabs3 2y ago
        See the OpenSSH bug above.
  - kpcyrd 2y ago
    It's a limitation in the ssh protocol. I wish they would fix it, but I'm not holding my breath. Trying to do anything about it would be a compatibility nightmare. If you need to pass data through ssh you're better off doing it through stdin.
curtisszmania 2y ago
[dead]
INTPenis 2y ago
>In addition, this is a self-service application, in the sense that anyone can create a Fedora contributor account and gain authenticated access to various services. I legitimately wanted to get a package into Fedora a few years ago, a service that did not exist already, and I couldn't get past the fact that they require new contributor accounts to be sponsored by someone already a contributor. I was unable to secure sponsorship by anyone and just gave up.
- pluto_modadic 2y ago
  this could be a useful mechanism actually... shame it didn't work out
  - INTPenis 2y ago
    Agreed, I have been a RHEL ecosystem user for 11 years now. My experience in the Fedora community was actually comforting to me.
    - ForOldHack 2y ago
      Now that is cool.
  - ForOldHack 2y ago
    Then you have someone to blame for sponsoring an exploit vs some unsponsored person who identified the same exploit patched it, and no one bothered to check either way. I guess I got lucky never trusting Mint. I did trust Slackware, because I knew someone who trusted Pat V. I guess I also need to rub shoulders with Linux maintainers vs people like Marc Andressen. I did meet RMS, but he has no personality. Woz? Wolfram? Anna V? I met the SUSE people, and one of the RedHat maintainers, who gave me a tee-shirt, a poster, two bumper stickers, and the latest distribution of RedHat. It was easier a decade ago... Despite the amount of brilliance here, again, we never have had a single meet and greet.
TZubiri 2y ago
"Let's use red hat's products but let's not get them from red hat to save some bucks" This is the risk you take
cadamsdotcom 2y ago
Nice writeup! Thinking generally it seems something like the xz/lzma vulnerability could be snuck in by 1-2 nefarious people colluding across packaging and package producing, especially if we are talking about nation-state actors who can afford to look legit for years before being trusted to work without oversight - then when no one is watching, sneak in a backdoor. I feel we are in a very innocent age and will look wistfully back at the days we trusted our anonymous open source brethren. On macOS I think about this every time I “brew install”, and every time oh-my-zsh auto-updates. Do Linux users think about this?
- tremon 2y ago
  I think about this every time I install software. As a citizen of a small European nation, 100% of the software I use is under control of a foreign government, and I trust none of them. At least with open source software, there is a better chance of nefarious changes being detected by at least one of the parties building and packaging the software. With proprietary software, even that small level of assurance is not available.
- mmh0000 2y ago
  Think about it all the time. But, I find it overwhelming and shutdown. I'll just enumerate my random thoughts * If you're worried about nations, remember that they can, quite literally, send ninjas-in-attack-helicopters at you. * Most nations already have "laws" that "require" you to provide passwords/data access on demand[4ab]. * If you use any "cloud", there's a high likelihood that they're already backdoored, either through legal means (i.e. through National Security Letters) or "not so legal, but who's going to stop them?" means such as just plain old hacking[1] * All consumer CPUs already have builtin backdoors that can't really (but kind of, but who really know if it's effective) be disabled[2abc] * Most printers print secret codes on printed documents that link back to the printer[3] * I have no control over device firmware and some important drivers. I really don't know what my network card firmware is doing when I'm not using it and it has DMA to my system RAM. I "need" nvidia proprietary drivers to have a decent experience, no idea what they actually do. * Nearly every piece of software includes some form of "Analytics" or "Telemetry", which often doesn't actually turn off when you click the stupid opt-out button. [1] https://www.npr.org/sections/thetwo-way/2013/10/30/241855353/report-nsa-has-broken-into-google-and-yahoo-data-centers https://www.npr.org/sections/thetwo-way/2013/10/30/241855353... [2a] https://en.wikipedia.org/wiki/Intel_Management_Engine https://en.wikipedia.org/wiki/Intel_Management_Engine [2b] https://en.wikipedia.org/wiki/AMD_Platform_Security_Processor https://en.wikipedia.org/wiki/AMD_Platform_Security_Processo... [2c] https://en.wikipedia.org/wiki/ARM_architecture_family#TrustZone https://en.wikipedia.org/wiki/ARM_architecture_family#TrustZ... [3] https://en.wikipedia.org/wiki/Printer_tracking_dots https://en.wikipedia.org/wiki/Printer_tracking_dots [4a] https://en.wikipedia.org/wiki/Key_disclosure_law https://en.wikipedia.org/wiki/Key_disclosure_law [4b] https://www.eff.org/wp/digital-privacy-us-border-2017 https://www.eff.org/wp/digital-privacy-us-border-2017
b112 2y ago
Does anyone else have the title overlay taking up 2/7ths of the top of the screen?
lrvick 2y ago
For those looking for alternatives to the status quo on Linux supply chain security, check out [Stageˣ]. It is 100% deterministic, hermetic, and reproducible. Also it is full source bootstrapped from 180 bytes of human-auditable machine code, all released artifacts are multi-party reproduced/reviewed/signed, and it is fully container-native all the way down "FROM scratch" making it easy to pin dependency hashes to reproduce your own projects with it. I started it after years of unsuccessful pleading with existing distros to stop giving ultimate solitary trust to -any- maintainers or sysadmins involved in the project. https://codeberg.org/stagex/stagex https://codeberg.org/stagex/stagex
- noman-land 2y ago
  Pardon my naivete but I've heard Nix described in many of the same terms. What are the differences and similarities between Stageˣ and Nix/NixOS?
  - lrvick 2y ago
    I am obviously pretty biased having authored a failed RFC to Nix to mandate signing, and having founded StageX, but here goes. Unlike StageX, Nix has a wikipedia-style low friction approach to packaging allowing a large community to maintain a huge number of packages which are signed by a central trusted party, while being -mostly- reproducible. It relies on a custom language and toolchain for packaging. Nix is designed for hobbyists seeking to reproduce their own workstations. Unlike Nix, StageX is 100% reproducible and all changes are independently signed by authors and reviewers, then artifacts are reproduced and signed by multiple distributed maintainers. It is also full source bootstrapped, and is built on the broadly compatible OCI standard for packaging. StageX is designed for production threat models where only compiler and infrastructure toolchains are needed and trusting any single person is unacceptable. One maintainer even uses Nix as a workstation to contribute to StageX. They have fundamentally different goals and use cases.
- green7ea 2y ago
  This project looks amazing — I didn't think bootstrapping like this was possible. Kudos on the project :-). This might displace chainguard as my goto docker images :-).
guillem_lefait 2y ago
NoLimitSecu, French cybersecurity podcast, released an episode yesterday with the authors: https://www.nolimitsecu.fr/compromission-de-distributions-linux/ https://www.nolimitsecu.fr/compromission-de-distributions-li... It was amazing to hear that they chose the weakest path, argument injection and were able to found a vector in two weeks twice (fedora + opensuse).

216 points by akyuu 2y ago

ufo 2y ago

I've got to say, Git resignifying -- and requiring --end-of-options instead is bonkers

musicale 2y ago
The pathway of untrusted/malicious input -> trusted command line argument seems to be a common problem, and one that could possibly be mitigated by better type/taint checking. It looks like there is some prior work in this area, but it hasn't resulted in commonly available implementations (even something basic like a type/taint checking version of exec() etc. on one side and getopt() etc. on the other.)
- PhilipRoman 2y ago
  I could've sworn I remember something about bash and glibc cooperating to indicate which arguments come from expanded variables but I cannot find anything on the internet or in the sources. Either I'm going insane or it was an unmerged proposal.
  - BobbyTables2 2y ago
    Seems like it would be quite painful to do that in C without heavy refactoring. Maybe a Rust alternative ?
    - yjftsjthsd-h 2y ago
      Why would that be painful in C? The code is already handling all the pieces
      - pluto_modadic 2y ago
        Painful to write vs painful to audit for correctness. Writing code that you barely understand is a surefire recipe to have /no clue/ how to debug said code (because debugging it will be more complex). (let alone audit it for correctness/security).
        yjftsjthsd-h 2y ago
        That just sounds like you don't believe it's ever possible to change existing C code, which... is a position you can argue, but I'm pretty sure that bash and glibc are actively developed to the point where I wouldn't personally commit to that position.
      - ForOldHack 2y ago
        Insert obligatory ref to "on trusting trust" what is the compiler doing? Has the pre-processor been vetted? This is looking more and more like hardware solutions are becoming more attractive.
        yjftsjthsd-h 2y ago
        1. I don't see why you need to solve Trusting Trust to make libc and the shell more robust. 2. If we are worried about Trusting Trust, then Rust is worse; at least C has the wide range of compilers needed for diverse double-compiling and as of https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/ https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-... we arguably have a working solution. Rust only has a single compiler, and that compiler is used to build itself, making it the poster child for Trusting Trust targets.
        steveklabnik 2y ago
        Rust has a second compiler, mrustc, written in C++ and that is able to bit-reproduce rustc. This has been the case for a few years now.
  - jwilk 2y ago
    It was a thing back in 1996/1997: https://sourceware.org/cgit/glibc/commit/?id=bf079e19f50d64aa5e05b5e1 https://sourceware.org/cgit/glibc/commit/?id=bf079e19f50d64a...
    - PhilipRoman 2y ago
      Yeah this is exactly what I was thinking of, thanks for finding it.
    - musicale 2y ago
      > "disabled later" because "it caused problems" ;-(
pests 2y ago
Sadly due to legacy. —- disambiguates revisions and paths in some commands so another option was needed.
- immibis 2y ago
  Just write all commands with structured I/O instead, like Powershell. Now I want an operating system where everything is a YANG model...

superb_dev 2y ago

> We can’t change git’s shell to /sbin/nologin or /bin/false, or users wouldn’t be able to connect over SSH. Git actually has a solution for this! I don’t know if it would work with the custom python stuff going on, but you can set the login shell to `git-shell`

jbverschoor 2y ago
Or just use git over https. And heck, if that's such a big problem, switch to a vcs that you can properly manage
- xenophonf 2y ago
  You shouldn't use Git over HTTPS. With SSH, you can use a hardware authenticator that requires both proof of ownership (i.e., the unlock PIN) and proof of possession (i.e., physical touch) out of the box. That's technically possible over HTTPS, of course, but I have yet to see a Git server that works that way.
  - jbverschoor 2y ago
    Why not? Your password manager or passkeys do the same thing. Heck, just true the public key as the authentication header. Or use authentication certificates. If ssh is such a big problem, use something else
    - rwteueru 2y ago
      [flagged]
      - arccy 2y ago
        credential helpers: like the github cli: https://cli.github.com/manual/gh_auth_setup-git https://cli.github.com/manual/gh_auth_setup-git or git-credential-manager: https://github.com/git-ecosystem/git-credential-manager https://github.com/git-ecosystem/git-credential-manager

blueflow 2y ago

Because bash for some goddamn reason loads the bashrc for interactive shells AND when started by sshd, regardless of whether the shell is interactive or an tty is present. Bash (and only bash) literally has a special case for sshd to enable this kind of exploit. As a result of this, git and rsync wont work at all if the bashrc on the remote machine writes any data to stdout. Like setting a window title. To work around that, every bashrc on this earth needs a case switch to return early to avoid this specific bug.

yjftsjthsd-h 2y ago
Wait, ssh doesn't let you specify the remote command without running it through the user's shell? That seems like a deficiency too IMHO.
- nurettin 2y ago
  It is simpler to export paths locally, so the remote doesn't have to know your file/folder structure.
  - yjftsjthsd-h 2y ago
    Sure, it's convenient to have PATH, but why not have an optional way to say "hey, ssh, run /usr/bin/rsync on the remote with the following arguments, directly and without a shell please"? Equivalent to a Dockerfile containing cmd foo vs cmd ["/bin/foo"] (IIRC, at least - been a minute since I needed to do this.)
    - pabs3 2y ago
      See the OpenSSH bug above.
- kpcyrd 2y ago
  It's a limitation in the ssh protocol. I wish they would fix it, but I'm not holding my breath. Trying to do anything about it would be a compatibility nightmare. If you need to pass data through ssh you're better off doing it through stdin.

curtisszmania 2y ago

[dead]

INTPenis 2y ago

>In addition, this is a self-service application, in the sense that anyone can create a Fedora contributor account and gain authenticated access to various services. I legitimately wanted to get a package into Fedora a few years ago, a service that did not exist already, and I couldn't get past the fact that they require new contributor accounts to be sponsored by someone already a contributor. I was unable to secure sponsorship by anyone and just gave up.

pluto_modadic 2y ago
this could be a useful mechanism actually... shame it didn't work out
- INTPenis 2y ago
  Agreed, I have been a RHEL ecosystem user for 11 years now. My experience in the Fedora community was actually comforting to me.
  - ForOldHack 2y ago
    Now that is cool.
- ForOldHack 2y ago
  Then you have someone to blame for sponsoring an exploit vs some unsponsored person who identified the same exploit patched it, and no one bothered to check either way. I guess I got lucky never trusting Mint. I did trust Slackware, because I knew someone who trusted Pat V. I guess I also need to rub shoulders with Linux maintainers vs people like Marc Andressen. I did meet RMS, but he has no personality. Woz? Wolfram? Anna V? I met the SUSE people, and one of the RedHat maintainers, who gave me a tee-shirt, a poster, two bumper stickers, and the latest distribution of RedHat. It was easier a decade ago... Despite the amount of brilliance here, again, we never have had a single meet and greet.

TZubiri 2y ago

"Let's use red hat's products but let's not get them from red hat to save some bucks" This is the risk you take

cadamsdotcom 2y ago

Nice writeup! Thinking generally it seems something like the xz/lzma vulnerability could be snuck in by 1-2 nefarious people colluding across packaging and package producing, especially if we are talking about nation-state actors who can afford to look legit for years before being trusted to work without oversight - then when no one is watching, sneak in a backdoor. I feel we are in a very innocent age and will look wistfully back at the days we trusted our anonymous open source brethren. On macOS I think about this every time I “brew install”, and every time oh-my-zsh auto-updates. Do Linux users think about this?

tremon 2y ago
I think about this every time I install software. As a citizen of a small European nation, 100% of the software I use is under control of a foreign government, and I trust none of them. At least with open source software, there is a better chance of nefarious changes being detected by at least one of the parties building and packaging the software. With proprietary software, even that small level of assurance is not available.
mmh0000 2y ago
Think about it all the time. But, I find it overwhelming and shutdown. I'll just enumerate my random thoughts * If you're worried about nations, remember that they can, quite literally, send ninjas-in-attack-helicopters at you. * Most nations already have "laws" that "require" you to provide passwords/data access on demand[4ab]. * If you use any "cloud", there's a high likelihood that they're already backdoored, either through legal means (i.e. through National Security Letters) or "not so legal, but who's going to stop them?" means such as just plain old hacking[1] * All consumer CPUs already have builtin backdoors that can't really (but kind of, but who really know if it's effective) be disabled[2abc] * Most printers print secret codes on printed documents that link back to the printer[3] * I have no control over device firmware and some important drivers. I really don't know what my network card firmware is doing when I'm not using it and it has DMA to my system RAM. I "need" nvidia proprietary drivers to have a decent experience, no idea what they actually do. * Nearly every piece of software includes some form of "Analytics" or "Telemetry", which often doesn't actually turn off when you click the stupid opt-out button. [1] https://www.npr.org/sections/thetwo-way/2013/10/30/241855353/report-nsa-has-broken-into-google-and-yahoo-data-centers https://www.npr.org/sections/thetwo-way/2013/10/30/241855353... [2a] https://en.wikipedia.org/wiki/Intel_Management_Engine https://en.wikipedia.org/wiki/Intel_Management_Engine [2b] https://en.wikipedia.org/wiki/AMD_Platform_Security_Processor https://en.wikipedia.org/wiki/AMD_Platform_Security_Processo... [2c] https://en.wikipedia.org/wiki/ARM_architecture_family#TrustZone https://en.wikipedia.org/wiki/ARM_architecture_family#TrustZ... [3] https://en.wikipedia.org/wiki/Printer_tracking_dots https://en.wikipedia.org/wiki/Printer_tracking_dots [4a] https://en.wikipedia.org/wiki/Key_disclosure_law https://en.wikipedia.org/wiki/Key_disclosure_law [4b] https://www.eff.org/wp/digital-privacy-us-border-2017 https://www.eff.org/wp/digital-privacy-us-border-2017

b112 2y ago

Does anyone else have the title overlay taking up 2/7ths of the top of the screen?

lrvick 2y ago

For those looking for alternatives to the status quo on Linux supply chain security, check out [Stageˣ]. It is 100% deterministic, hermetic, and reproducible. Also it is full source bootstrapped from 180 bytes of human-auditable machine code, all released artifacts are multi-party reproduced/reviewed/signed, and it is fully container-native all the way down "FROM scratch" making it easy to pin dependency hashes to reproduce your own projects with it. I started it after years of unsuccessful pleading with existing distros to stop giving ultimate solitary trust to -any- maintainers or sysadmins involved in the project. https://codeberg.org/stagex/stagex https://codeberg.org/stagex/stagex

noman-land 2y ago
Pardon my naivete but I've heard Nix described in many of the same terms. What are the differences and similarities between Stageˣ and Nix/NixOS?
- lrvick 2y ago
  I am obviously pretty biased having authored a failed RFC to Nix to mandate signing, and having founded StageX, but here goes. Unlike StageX, Nix has a wikipedia-style low friction approach to packaging allowing a large community to maintain a huge number of packages which are signed by a central trusted party, while being -mostly- reproducible. It relies on a custom language and toolchain for packaging. Nix is designed for hobbyists seeking to reproduce their own workstations. Unlike Nix, StageX is 100% reproducible and all changes are independently signed by authors and reviewers, then artifacts are reproduced and signed by multiple distributed maintainers. It is also full source bootstrapped, and is built on the broadly compatible OCI standard for packaging. StageX is designed for production threat models where only compiler and infrastructure toolchains are needed and trusting any single person is unacceptable. One maintainer even uses Nix as a workstation to contribute to StageX. They have fundamentally different goals and use cases.
green7ea 2y ago
This project looks amazing — I didn't think bootstrapping like this was possible. Kudos on the project :-). This might displace chainguard as my goto docker images :-).

guillem_lefait 2y ago

NoLimitSecu, French cybersecurity podcast, released an episode yesterday with the authors: https://www.nolimitsecu.fr/compromission-de-distributions-linux/ https://www.nolimitsecu.fr/compromission-de-distributions-li... It was amazing to hear that they chose the weakest path, argument injection and were able to found a vector in two weeks twice (fedora + opensuse).