3 ms·

Does someone know if using grep on a binary file is somehow defined by POSIX? At a glance, I couldn't find a reference on the grep page: https://pubs.opengrou

Does someone know if using grep on a binary file is somehow defined by POSIX? At a glance, I couldn't find a reference on the grep page: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/grep.html#top https://pubs.opengroup.org/onlinepubs/9699919799/utilities/g...

jwilk 7y ago
It isn't. The page says: > The input files shall be text files. "Text file" is defined in https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_403 https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1... : > A file that contains characters organized into zero or more lines. The lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in length, including the <newline> character.
- arendtio 7y ago
  Thank you :-) Interesting, especially the part about the LINE_MAX. Even though it kinda makes sense, I would never have thought that having a very long line makes a file a non-text file when all characters are 'normal' characters.
  - bawolff 7y ago
    Id kind of like that. Grepping in a directory also containing minified js (with ten thousand character long lines) is a pain
- seppel 7y ago
  So, what does trigger grep to regard a file as binary/non-text? For me, it seems these are not only NUL characters. And I also think that this would not be a discussion if it was just about NUL. There is something else that grep does not like (escape characters? Invalid utf8? Wrong codepage? I don't know), but this makes the binary detection annoying.
  - jwilk 7y ago
    I haven't looked at the GNU grep source, but yes, invalid UTF-8 is one of the reasons for treating file as binary: $ printf 'a\222b' | LC_ALL=en_US.UTF-8 grep a Binary file (standard input) matches This is POSIX-compliant, because "character" is defined as: > A sequence of one or more bytes representing a single graphic symbol or control code. https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_87 https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1...

jwilk 7y ago

It isn't. The page says: > The input files shall be text files. "Text file" is defined in https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_403 https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1... : > A file that contains characters organized into zero or more lines. The lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in length, including the <newline> character.

arendtio 7y ago
Thank you :-) Interesting, especially the part about the LINE_MAX. Even though it kinda makes sense, I would never have thought that having a very long line makes a file a non-text file when all characters are 'normal' characters.
- bawolff 7y ago
  Id kind of like that. Grepping in a directory also containing minified js (with ten thousand character long lines) is a pain
seppel 7y ago
So, what does trigger grep to regard a file as binary/non-text? For me, it seems these are not only NUL characters. And I also think that this would not be a discussion if it was just about NUL. There is something else that grep does not like (escape characters? Invalid utf8? Wrong codepage? I don't know), but this makes the binary detection annoying.
- jwilk 7y ago
  I haven't looked at the GNU grep source, but yes, invalid UTF-8 is one of the reasons for treating file as binary: $ printf 'a\222b' | LC_ALL=en_US.UTF-8 grep a Binary file (standard input) matches This is POSIX-compliant, because "character" is defined as: > A sequence of one or more bytes representing a single graphic symbol or control code. https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_87 https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1...