6 ms·

I just looked through the GNU grep history to see when it suddenly started being able to decide halfway through a file that it is binary after all; this is sinc

by gpvos 7y ago

I just looked through the GNU grep history to see when it suddenly started being able to decide halfway through a file that it is binary after all; this is since 16 September 2014, so fairly recently. Before that, it just checked the first few kilobytes to decide, and didn't change its opinion afterwards. To me, this is a very nonintuitive change.

iforgotpassword 7y ago
Oh yes. I was bitten by that when I piped around some tool's output that eventually went through grep. The tool's output was actually text but for some reason there was a null byte at the end which nobody noticed before. The fun part was that the way the data got chunked through the pipes was not deterministic, so sometimes you got the desired output from grep, but other times just "binary file matches", even when the raw output from the first tool was identical in both runs. That took quite some head scratching to figure out.
gpvos 7y ago
Why is it that in recent years, with this and the more recent ls quoting fiasco, maintainers of longstanding UNIX utilities suddenly got the urge to fix what isn't broken?
- chubot 7y ago
  What's the 'ls quoting fiasco'? Actually I recently found that coreutils and ls behave fairly well with funny filenames: Here is an invalid utf-8 byte and then a valid utf-8 sequence $ x=$'\xce\xce\xbc' $ touch "$x" You can list it: $ ls ?μ And here 'ls' does better than other tools that display filenames. It shows the invalid byte and then keeps decoding with error recovery: $ ls --escape \316μ However GNU stat (which I think is also in coreutils) does something similar, but weirdly messed up: $ stat * File: ''$'\316''μ' (it looks like it's outputting a valid shell string, except with extra quotes) ----- Most command line tools are not aware of stuff like this. For example you can touch "x$ANSI_TERMINAL_CODES" and if you do "bash x??" or "python x??", then your terminal will change color because of the escape codes printed back to the terminal. I just changed Oil to use a well-defined format I called QSN (quoted string notation): http://www.oilshell.org/blog/2020/04/release-0.8.pre4.html#the-highlights http://www.oilshell.org/blog/2020/04/release-0.8.pre4.html#t... It adapts Rust's string literal syntax to express arbitrary byte strings precisely and losslessly. (JSON can't express arbitrary byte strings.) The QSN encoder does UTF-8 decoding with a specific error recovery mechanism. So it's basically like what ls and stat do, but it's more precise. (If anyone is interested in QSN, please contact me. I think it's more generally useful in a lot of places. It's something we already do but it's precise like JSON.)
  - boring_twenties 7y ago
    GNU ls prints the filenames already-quoted as of late: % touch foo % touch 'bar baz' % ls 'bar baz' foo
  - _jal 7y ago
    They broke it in 2016. https://www.gnu.org/software/coreutils/quotes.html https://www.gnu.org/software/coreutils/quotes.html At least with Gnu, you can recompile your own, non-broken version, which is the only saving grace of these stupid, trendy changes.
    - Anthony-G 7y ago
      Not broken at all. I really don’t understand the hate for this change. I deal with a lot of filenames with spaces and think this change is a great improvement for listing such files. With this change it’s much easier to see where one filename ends and the other begins. Before this change, I had to use the `-1` option to ensure that each filename was listed on a line by itself. Now the filename listings are much more readable and it takes less cognitive effort to take it all in. The way it handles filenames with ASCII apostrophes/single quotes works particularly well (wraps the filename in double quotes instead of single quotes) and makes it very easy to copy and paste filenames to and from the terminal. Best of all, this change only applies when standard output is a TTY device so this does not break any shell scripts (even though parsing `ls` is a bad idea in any case) and is still compliant with the POSIX specification[1] which states that “If the output is to a terminal, the format is implementation-defined”. 1. https://pubs.opengroup.org/onlinepubs/9699919799/utilities/ls.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/l...
      - deleted 7y ago
        [deleted]
    - Delk 7y ago
      It would seem that calling ls with the -N switch would disable the quoting. Might be easier to just have ls aliased to "ls -N" rather than recompiling. Unless the -N switch also does something else you don't like.
- knolax 7y ago
  A google search for 'ls quoting fiasco' shows this thread as the top result. Could you explain what it is?
  - smegma2 7y ago
    There was a change in ls which caused it to quote file names with spaces or special characters instead of (I think?) escaping them. Some people were upset because they considered this a breaking change. I'll just add that ls only does this if it's running interactively so I wouldn't expect it to break scripts.
    - ansible 7y ago
      I'm a little surprised people are using ls in scripts. I'm normally using find, with the mill separator to deal with spaces in filenames.
      - ansible 7y ago
        ... null separator ... I thought I proof-read that sentence when I wrote it...
- lazyjones 7y ago
  Because of the prevalence of UTF-8 and its consequences/pitfalls?
  - gpvos 7y ago
    That's the best explanation I can think of too.
- TheGoddessInari 7y ago
  "GNU's Not Unix!"
jwilk 7y ago
> this is since 16 September 2014 Can you be more specific? I don't see any commmit or release on that date.
- gpvos 7y ago
  I see 10. https://git.savannah.gnu.org/cgit/grep.git/log/?ofs=550 https://git.savannah.gnu.org/cgit/grep.git/log/?ofs=550 (over time, this link will start to point at the wrong date)
  - jwilk 7y ago
    My bad, I looked only at author dates. (But then I missed some commits anyway?!) I guess you had this commit in mind: https://git.savannah.gnu.org/cgit/grep.git/commit/?id=cd36abd46c5e0768606979ea75a51732062f5624 https://git.savannah.gnu.org/cgit/grep.git/commit/?id=cd36ab...

by gpvos 7y ago

iforgotpassword 7y ago

Oh yes. I was bitten by that when I piped around some tool's output that eventually went through grep. The tool's output was actually text but for some reason there was a null byte at the end which nobody noticed before. The fun part was that the way the data got chunked through the pipes was not deterministic, so sometimes you got the desired output from grep, but other times just "binary file matches", even when the raw output from the first tool was identical in both runs. That took quite some head scratching to figure out.

gpvos 7y ago

Why is it that in recent years, with this and the more recent ls quoting fiasco, maintainers of longstanding UNIX utilities suddenly got the urge to fix what isn't broken?

chubot 7y ago
What's the 'ls quoting fiasco'? Actually I recently found that coreutils and ls behave fairly well with funny filenames: Here is an invalid utf-8 byte and then a valid utf-8 sequence $ x=$'\xce\xce\xbc' $ touch "$x" You can list it: $ ls ?μ And here 'ls' does better than other tools that display filenames. It shows the invalid byte and then keeps decoding with error recovery: $ ls --escape \316μ However GNU stat (which I think is also in coreutils) does something similar, but weirdly messed up: $ stat * File: ''$'\316''μ' (it looks like it's outputting a valid shell string, except with extra quotes) ----- Most command line tools are not aware of stuff like this. For example you can touch "x$ANSI_TERMINAL_CODES" and if you do "bash x??" or "python x??", then your terminal will change color because of the escape codes printed back to the terminal. I just changed Oil to use a well-defined format I called QSN (quoted string notation): http://www.oilshell.org/blog/2020/04/release-0.8.pre4.html#the-highlights http://www.oilshell.org/blog/2020/04/release-0.8.pre4.html#t... It adapts Rust's string literal syntax to express arbitrary byte strings precisely and losslessly. (JSON can't express arbitrary byte strings.) The QSN encoder does UTF-8 decoding with a specific error recovery mechanism. So it's basically like what ls and stat do, but it's more precise. (If anyone is interested in QSN, please contact me. I think it's more generally useful in a lot of places. It's something we already do but it's precise like JSON.)
- boring_twenties 7y ago
  GNU ls prints the filenames already-quoted as of late: % touch foo % touch 'bar baz' % ls 'bar baz' foo
- _jal 7y ago
  They broke it in 2016. https://www.gnu.org/software/coreutils/quotes.html https://www.gnu.org/software/coreutils/quotes.html At least with Gnu, you can recompile your own, non-broken version, which is the only saving grace of these stupid, trendy changes.
  - Anthony-G 7y ago
    Not broken at all. I really don’t understand the hate for this change. I deal with a lot of filenames with spaces and think this change is a great improvement for listing such files. With this change it’s much easier to see where one filename ends and the other begins. Before this change, I had to use the `-1` option to ensure that each filename was listed on a line by itself. Now the filename listings are much more readable and it takes less cognitive effort to take it all in. The way it handles filenames with ASCII apostrophes/single quotes works particularly well (wraps the filename in double quotes instead of single quotes) and makes it very easy to copy and paste filenames to and from the terminal. Best of all, this change only applies when standard output is a TTY device so this does not break any shell scripts (even though parsing `ls` is a bad idea in any case) and is still compliant with the POSIX specification[1] which states that “If the output is to a terminal, the format is implementation-defined”. 1. https://pubs.opengroup.org/onlinepubs/9699919799/utilities/ls.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/l...
    - deleted 7y ago
      [deleted]
  - Delk 7y ago
    It would seem that calling ls with the -N switch would disable the quoting. Might be easier to just have ls aliased to "ls -N" rather than recompiling. Unless the -N switch also does something else you don't like.
knolax 7y ago
A google search for 'ls quoting fiasco' shows this thread as the top result. Could you explain what it is?
- smegma2 7y ago
  There was a change in ls which caused it to quote file names with spaces or special characters instead of (I think?) escaping them. Some people were upset because they considered this a breaking change. I'll just add that ls only does this if it's running interactively so I wouldn't expect it to break scripts.
  - ansible 7y ago
    I'm a little surprised people are using ls in scripts. I'm normally using find, with the mill separator to deal with spaces in filenames.
    - ansible 7y ago
      ... null separator ... I thought I proof-read that sentence when I wrote it...
lazyjones 7y ago
Because of the prevalence of UTF-8 and its consequences/pitfalls?
- gpvos 7y ago
  That's the best explanation I can think of too.
TheGoddessInari 7y ago
"GNU's Not Unix!"

jwilk 7y ago

> this is since 16 September 2014 Can you be more specific? I don't see any commmit or release on that date.

gpvos 7y ago
I see 10. https://git.savannah.gnu.org/cgit/grep.git/log/?ofs=550 https://git.savannah.gnu.org/cgit/grep.git/log/?ofs=550 (over time, this link will start to point at the wrong date)
- jwilk 7y ago
  My bad, I looked only at author dates. (But then I missed some commits anyway?!) I guess you had this commit in mind: https://git.savannah.gnu.org/cgit/grep.git/commit/?id=cd36abd46c5e0768606979ea75a51732062f5624 https://git.savannah.gnu.org/cgit/grep.git/commit/?id=cd36ab...