7 ms·

The cpio trailer problem (2018)

52 points by tjalfi 6y ago

avian 6y ago
> I assume that cpio is very rarely used these days. From Wikipedia: > The use of cpio by the RPM Package Manager, in the initramfs program of Linux kernel 2.6, and in Apple's Installer (pax) make cpio an important archiving tool.
- marcan_42 6y ago
  All of which are administrative formats that do not include user data. I can't think of a way that this "feature" could be abused in those contexts to cause a security problem.
  - sp332 6y ago
    If, say, an anti-virus program extracts an RPM's contents and scans each file separately, then it might miss files smuggled in that would be readable by malware with a custom cpio extractor.
- rwmj 6y ago
  RPM hasn't used true cpio for a while [since 2013] because cpio didn't support > 4GB files. It now uses its own format which it evolved from cpio.
  - tyingq 6y ago
    Seems they still use the same logic though: https://github.com/rpm-software-management/rpm/blob/853c48ba6468ce1a516621a2fa6d1fc51e4f7410/lib/cpio.c#L49 https://github.com/rpm-software-management/rpm/blob/853c48ba...
    - rwmj 6y ago
      That doesn't actually matter though does it, because RPM is no longer using cpio, it's now using a custom format that happens to once have been related to cpio. What matters is what RPM does, not what cpio does. In this case it's not possible to create such an archive using rpmbuild as all files added to RPMs much start with "/". I just tried it and got: RPM build errors: File must begin with "/": TRAILER!!! You cannot hack on your own rpmbuild because you wouldn't be able to make a signed RPM (as those can only be built through the Koji build system which doesn't have your hacked rpmbuild), so you couldn't distribute that RPM to Fedora or Red Hat Enterprise Linux. In other words this is essentially a non-problem for RPM.
      - tyingq 6y ago
        The comparison is just to the filename, not including the path. You may be right, but that check doesn't prove it. Edit: Thanks for the reply! Apparently they are including the path.
        rwmj 6y ago
        I can create an RPM containing a filename "TRAILER!!!" just fine: $ rpm -qlp /home/rjones/d/fedora/virt-what/master/x86_64/virt-what-1.20-1.fc33.x86_64.rpm /TRAILER!!! /usr/lib/.build-id /usr/lib/.build-id/59 /usr/lib/.build-id/59/59c1cb95689b7babd5a6cb1b5bc58f12e20b3a /usr/libexec/virt-what-cpuid-helper /usr/sbin/virt-what /usr/share/doc/virt-what /usr/share/doc/virt-what/COPYING /usr/share/doc/virt-what/README /usr/share/man/man1/virt-what.1.gz Furthermore I installed and uninstalled it with no ill effects, and the file was created and removed in /. So whatever problem cpio may have with this file does not affect the RPM-not-quite-cpio format.
- philsnow 6y ago
  I saw cpio archives in a .pkg file on mac as recently as last week so yeah very much still in use.
saagarjha 6y ago
TL;DR a lot of formats have “sentinel” values that are little more than “nobody sane would ever pick this value” and we end up in a conundrum when the obvious ambiguities arise.
- swiley 6y ago
  They could have just used a special character and escaped names containing it.
- raverbashing 6y ago
  It's not an issue with sentinel values, but an issue with not escaping your inputs I don't believe tar "falls for" such a stupid trick. Confusing metadata with actual data is a classic security blunder tl;dr: cpio has amateurish levels of input validation/handling
  - masklinn 6y ago
    > tl;dr: cpio is an amateur project TBF cpio dates back to v7 unix, some time before considerations of security really became a thing.
    - raverbashing 6y ago
      The question is not so much why it was like that, the question is why nobody fixed this yet I like the Unix environment, but I can understand why stuff like systemd take hold, because the legacy is painful. And any Unix environment has this weird mix of tools where the most popular ones got better over time (tar, etc) then you have the weird ones that didn't evolve so much like cpio
      - cyphar 6y ago
        A bit of shameless self-promotion, I wrote a blog post some time ago which goes through the history of tar[1]. Tar definitely has evolved through the Unix wars, but I wouldn't say the final result is really that great. While we do tend to reinvent tools for reinvention's sake, I do think tar really does deserve a redesign based on modern constraints (rather than the constraints of tape archives). Zip archives are markedly better designed. Also, cpio did evolve at least somewhat. In fact POSIX has a separate archive tool called "pax" (which supports both tar and cpio) which was meant to act as a truce between the tar and cpio factions of Unix vendors. Nobody uses it, as far as I know. [1]: https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar
qalmakka 6y ago
I never really understood why tar "won" and cpio "lost" the archive wars. Both do pretty much the same thing, thus I think it was just a matter of personal preference? I started using *NIX around ~2006 and nobody was using cpio already then, I only had the chance of learning of its existence when I first had to extract an RPM.
- chungy 6y ago
  tar is far easier to use, I think.
- jan6 6y ago
  maybe it was preference at first, some people finding it annoying to deal with cpio's file list, and it built up to the modern stuff like builtin integration with compressors (probably coinciding with tar.gz becoming the de facto *nix archive)? I personally find xz and cpio both cool, despite both being barely used ;)
  - jhoechtl 6y ago
    XZ used to be the compression format for many Linux packages in many distros (at least Arch based) and has only lately been replaced by ZStd. How do you come to the conclusion that xz is barely used?
    - ComputerGuru 6y ago
      I definitely would not say zstd has replaced xz, but it is on its way too. I'm not sure it is even the best replacement because the needs of packages differ from the strengths of zstd. Zstd is designed to optimize for both compression and decompression (i.e. It does neither spectacularly but has very good/pareto trade offs) while packages are very much written once and read many times. If xz weren't so memory intensive to decompress it would be a good choice (ie it's not a good choice), but I feel like there is still a better option out there that approaches the problem from a "optimize for size at the expense of heavy compression costs" and zstd is not it
- raverbashing 6y ago
rwmj 6y ago
The Linux kernel will keep going after the "TRAILER!!!" entry in the initramfs. This is useful in the context of initramfs because you can concatenate them together - a trivial way to add more files to the init ramdisk or even overwrite existing ones. You can even concat gz-compressed cpio initramfs files which causes a similar problem finding the uncompressed size because the last 4 bytes will only show the size of the last gzip section (try with gzip -l).
teddyh 6y ago
That “TRAILER!!!” string should be added to the Big List of Naughty Strings¹ (discussed here previously²). 1. https://github.com/minimaxir/big-list-of-naughty-strings https://github.com/minimaxir/big-list-of-naughty-strings 2. https://news.ycombinator.com/item?id=13406119 https://news.ycombinator.com/item?id=13406119
tyingq 6y ago
Interesting. I wonder why they didn't just use a value that's not valid for a filename. TRAILER/// might have worked for Unix.
appleflaxen 6y ago
Is there a name for this general class of problems? Where you have an accidental parsing/scoping error because the encoding format is ambiguous, or doesn't take into account a specific configuration of the message contents?
- giovannibajo1 6y ago
  I would classify this as using an in-band signal rather than an out of band signal. Twitter used to have this same problems ages ago. I don’t remember the details but it was something like DMs being standard twitter messages with “/dm” at the beginning.

52 points by tjalfi 6y ago

> I assume that cpio is very rarely used these days. From Wikipedia: > The use of cpio by the RPM Package Manager, in the initramfs program of Linux kernel 2.6, and in Apple's Installer (pax) make cpio an important archiving tool.

marcan_42 6y ago
All of which are administrative formats that do not include user data. I can't think of a way that this "feature" could be abused in those contexts to cause a security problem.
- sp332 6y ago
  If, say, an anti-virus program extracts an RPM's contents and scans each file separately, then it might miss files smuggled in that would be readable by malware with a custom cpio extractor.
rwmj 6y ago
RPM hasn't used true cpio for a while [since 2013] because cpio didn't support > 4GB files. It now uses its own format which it evolved from cpio.
- tyingq 6y ago
  Seems they still use the same logic though: https://github.com/rpm-software-management/rpm/blob/853c48ba6468ce1a516621a2fa6d1fc51e4f7410/lib/cpio.c#L49 https://github.com/rpm-software-management/rpm/blob/853c48ba...
  - rwmj 6y ago
    That doesn't actually matter though does it, because RPM is no longer using cpio, it's now using a custom format that happens to once have been related to cpio. What matters is what RPM does, not what cpio does. In this case it's not possible to create such an archive using rpmbuild as all files added to RPMs much start with "/". I just tried it and got: RPM build errors: File must begin with "/": TRAILER!!! You cannot hack on your own rpmbuild because you wouldn't be able to make a signed RPM (as those can only be built through the Koji build system which doesn't have your hacked rpmbuild), so you couldn't distribute that RPM to Fedora or Red Hat Enterprise Linux. In other words this is essentially a non-problem for RPM.
    - tyingq 6y ago
      The comparison is just to the filename, not including the path. You may be right, but that check doesn't prove it. Edit: Thanks for the reply! Apparently they are including the path.
      - rwmj 6y ago
        I can create an RPM containing a filename "TRAILER!!!" just fine: $ rpm -qlp /home/rjones/d/fedora/virt-what/master/x86_64/virt-what-1.20-1.fc33.x86_64.rpm /TRAILER!!! /usr/lib/.build-id /usr/lib/.build-id/59 /usr/lib/.build-id/59/59c1cb95689b7babd5a6cb1b5bc58f12e20b3a /usr/libexec/virt-what-cpuid-helper /usr/sbin/virt-what /usr/share/doc/virt-what /usr/share/doc/virt-what/COPYING /usr/share/doc/virt-what/README /usr/share/man/man1/virt-what.1.gz Furthermore I installed and uninstalled it with no ill effects, and the file was created and removed in /. So whatever problem cpio may have with this file does not affect the RPM-not-quite-cpio format.
philsnow 6y ago
I saw cpio archives in a .pkg file on mac as recently as last week so yeah very much still in use.

saagarjha 6y ago

TL;DR a lot of formats have “sentinel” values that are little more than “nobody sane would ever pick this value” and we end up in a conundrum when the obvious ambiguities arise.

swiley 6y ago
They could have just used a special character and escaped names containing it.
raverbashing 6y ago
It's not an issue with sentinel values, but an issue with not escaping your inputs I don't believe tar "falls for" such a stupid trick. Confusing metadata with actual data is a classic security blunder tl;dr: cpio has amateurish levels of input validation/handling
- masklinn 6y ago
  > tl;dr: cpio is an amateur project TBF cpio dates back to v7 unix, some time before considerations of security really became a thing.
  - raverbashing 6y ago
    The question is not so much why it was like that, the question is why nobody fixed this yet I like the Unix environment, but I can understand why stuff like systemd take hold, because the legacy is painful. And any Unix environment has this weird mix of tools where the most popular ones got better over time (tar, etc) then you have the weird ones that didn't evolve so much like cpio
    - cyphar 6y ago
      A bit of shameless self-promotion, I wrote a blog post some time ago which goes through the history of tar[1]. Tar definitely has evolved through the Unix wars, but I wouldn't say the final result is really that great. While we do tend to reinvent tools for reinvention's sake, I do think tar really does deserve a redesign based on modern constraints (rather than the constraints of tape archives). Zip archives are markedly better designed. Also, cpio did evolve at least somewhat. In fact POSIX has a separate archive tool called "pax" (which supports both tar and cpio) which was meant to act as a truce between the tar and cpio factions of Unix vendors. Nobody uses it, as far as I know. [1]: https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar

qalmakka 6y ago

I never really understood why tar "won" and cpio "lost" the archive wars. Both do pretty much the same thing, thus I think it was just a matter of personal preference? I started using *NIX around ~2006 and nobody was using cpio already then, I only had the chance of learning of its existence when I first had to extract an RPM.

chungy 6y ago
tar is far easier to use, I think.
jan6 6y ago
maybe it was preference at first, some people finding it annoying to deal with cpio's file list, and it built up to the modern stuff like builtin integration with compressors (probably coinciding with tar.gz becoming the de facto *nix archive)? I personally find xz and cpio both cool, despite both being barely used ;)
- jhoechtl 6y ago
  XZ used to be the compression format for many Linux packages in many distros (at least Arch based) and has only lately been replaced by ZStd. How do you come to the conclusion that xz is barely used?
  - ComputerGuru 6y ago
    I definitely would not say zstd has replaced xz, but it is on its way too. I'm not sure it is even the best replacement because the needs of packages differ from the strengths of zstd. Zstd is designed to optimize for both compression and decompression (i.e. It does neither spectacularly but has very good/pareto trade offs) while packages are very much written once and read many times. If xz weren't so memory intensive to decompress it would be a good choice (ie it's not a good choice), but I feel like there is still a better option out there that approaches the problem from a "optimize for size at the expense of heavy compression costs" and zstd is not it
raverbashing 6y ago

rwmj 6y ago

The Linux kernel will keep going after the "TRAILER!!!" entry in the initramfs. This is useful in the context of initramfs because you can concatenate them together - a trivial way to add more files to the init ramdisk or even overwrite existing ones. You can even concat gz-compressed cpio initramfs files which causes a similar problem finding the uncompressed size because the last 4 bytes will only show the size of the last gzip section (try with gzip -l).

teddyh 6y ago

That “TRAILER!!!” string should be added to the Big List of Naughty Strings¹ (discussed here previously²). 1. https://github.com/minimaxir/big-list-of-naughty-strings https://github.com/minimaxir/big-list-of-naughty-strings 2. https://news.ycombinator.com/item?id=13406119 https://news.ycombinator.com/item?id=13406119

tyingq 6y ago

Interesting. I wonder why they didn't just use a value that's not valid for a filename. TRAILER/// might have worked for Unix.

appleflaxen 6y ago

Is there a name for this general class of problems? Where you have an accidental parsing/scoping error because the encoding format is ambiguous, or doesn't take into account a specific configuration of the message contents?

giovannibajo1 6y ago
I would classify this as using an in-band signal rather than an out of band signal. Twitter used to have this same problems ages ago. I don’t remember the details but it was something like DMs being standard twitter messages with “/dm” at the beginning.