4 ms·

> Is there any legitimate use for filenames with newlines? IMHO no, but they can exist, so you need to handle them without blowing up. Also, even spaces are c

> Is there any legitimate use for filenames with newlines? IMHO no, but they can exist, so you need to handle them without blowing up. Also, even spaces are considered delimiters here, which is why it's bad form to parse the output of ls. $ touch "foo bar baz" $ for f in `ls`; do echo $f; done foo bar baz # always use double quotes, though they aren't needed here $ for f in *; do echo "$f"; done foo bar baz At least the OS guarantees you won't run into NUL though.

unqueued 2y ago
There is a pretty good syntax for dealing with nasty filenames, if you must: ANSI-C quoting[1]. If you have to output in a shellscript in this format, use printf %q from man printf: %q ARGUMENT is printed in a format that can be reused as shell input, escaping non-printable characters with the proposed POSIX $'' syntax. It is just $'<nasty ansi-c escaped chars>' $ touch $'\nHello\tWorld\n' $ ls One thing I do like about a filesystem that fully supports POSIX filenames is that at the end of the day a filesystem is supposed to represent data. I think it is totally sensible to exclude certain characters, but that it should be done higher up in the stack if possible. Or have a flag that is set at mount time. Perhaps even by subvolume/dataset. One thing I haven't seen mentioned is that POSIX filenames are so permissive that they allow you to have bytes as filenames that are invalid UTF-8. That's why the popular ncdu[2] program does NOT use json as it's file format, although most think it does. It's actually json but with raw POSIX bytes in filename fields, which is outside of the official json spec. That does not stop folks from using json tools to parse ncdu output though. Another standard that is also very permissive with filenames is git. When I started exploring new ways to encode data into a git repo, it was only natural that I encountered issues with limitations of filesystems that I would check out in. Try cloning this repo, and see if you are able to check it out: https://github.com/benibela/nasty-files https://github.com/benibela/nasty-files It is amazing how many things it breaks. If you are writing software that deals with git filenames or POSIX filenames (that includes things like parsing a zip file footer), you can not rely on your standard json encoding function, because the input may contain invalid utf-8. So you may need to do extra encoding/filtering. [1]: https://www.gnu.org/s/bash/manual/html_node/ANSI_002dC-Quoting.html https://www.gnu.org/s/bash/manual/html_node/ANSI_002dC-Quoti... [2]: https://dev.yorhel.nl/ncdu/jsonfmt https://dev.yorhel.nl/ncdu/jsonfmt
kstrauser 2y ago
I’m not in a place where I can easily check. What happens there if the file name contains a quote?
- chuckadams 2y ago
  It's fine, the content of an expanded variable isn't parsed further: $ touch "foo \"bar baz"; for f in *; do echo "$f"; done foo "bar baz # quotes don't affect it either $ touch "foo \"bar baz"; for f in *; do echo $f; done foo "bar baz Though once you start passing args with quotes to other scripts, things get ugly. Rule of thumb is to always pass with "$@", and if that isn't enough to preserve quoting for whatever use case, write them out to a tempfile instead, or don't use a shell script for it in the first place.
  - kstrauser 2y ago
    What about in the case of for f in `ls`; do echo "$f"; done Same behavior, for the same reason?
    - chuckadams 2y ago
      The quotes are preserved, but backquote expansion fills the argument list using any whitespace as a delimiter. $ for f in `ls`; do echo "$f"; done foo "bar baz If you absolutely must parse ls (let's assume it's some other script that outputs items with spaces) and the output can contain spaces, you have a few options: $ ls | while read f; do echo "$f"; done foo "bar baz # parens keep the IFS change isolated to a subshell $ (IFS="\n"; for f in `ls`; do echo "$f"; done) foo "bar baz But if your filenames contain newlines, you'll really want to stick with the glob expansion, or output custom delimiters and set IFS to that.
      - ChoHag 2y ago
        > If you absolutely must parse ls ... stop and rethink your options. You may be able to get away with parsing the first columns of ls -l but even then a pathologically named file could make itself look like a line of ls output. It's simply not possible in all cases. If you can constrain your input then you may be able to make use of it but in the general case, that's why xargs and find grew a -0 option. Or glob.

unqueued 2y ago

There is a pretty good syntax for dealing with nasty filenames, if you must: ANSI-C quoting[1]. If you have to output in a shellscript in this format, use printf %q from man printf: %q ARGUMENT is printed in a format that can be reused as shell input, escaping non-printable characters with the proposed POSIX $'' syntax. It is just $'<nasty ansi-c escaped chars>' $ touch $'\nHello\tWorld\n' $ ls One thing I do like about a filesystem that fully supports POSIX filenames is that at the end of the day a filesystem is supposed to represent data. I think it is totally sensible to exclude certain characters, but that it should be done higher up in the stack if possible. Or have a flag that is set at mount time. Perhaps even by subvolume/dataset. One thing I haven't seen mentioned is that POSIX filenames are so permissive that they allow you to have bytes as filenames that are invalid UTF-8. That's why the popular ncdu[2] program does NOT use json as it's file format, although most think it does. It's actually json but with raw POSIX bytes in filename fields, which is outside of the official json spec. That does not stop folks from using json tools to parse ncdu output though. Another standard that is also very permissive with filenames is git. When I started exploring new ways to encode data into a git repo, it was only natural that I encountered issues with limitations of filesystems that I would check out in. Try cloning this repo, and see if you are able to check it out: https://github.com/benibela/nasty-files https://github.com/benibela/nasty-files It is amazing how many things it breaks. If you are writing software that deals with git filenames or POSIX filenames (that includes things like parsing a zip file footer), you can not rely on your standard json encoding function, because the input may contain invalid utf-8. So you may need to do extra encoding/filtering. [1]: https://www.gnu.org/s/bash/manual/html_node/ANSI_002dC-Quoting.html https://www.gnu.org/s/bash/manual/html_node/ANSI_002dC-Quoti... [2]: https://dev.yorhel.nl/ncdu/jsonfmt https://dev.yorhel.nl/ncdu/jsonfmt

kstrauser 2y ago

I’m not in a place where I can easily check. What happens there if the file name contains a quote?

chuckadams 2y ago
It's fine, the content of an expanded variable isn't parsed further: $ touch "foo \"bar baz"; for f in *; do echo "$f"; done foo "bar baz # quotes don't affect it either $ touch "foo \"bar baz"; for f in *; do echo $f; done foo "bar baz Though once you start passing args with quotes to other scripts, things get ugly. Rule of thumb is to always pass with "$@", and if that isn't enough to preserve quoting for whatever use case, write them out to a tempfile instead, or don't use a shell script for it in the first place.
- kstrauser 2y ago
  What about in the case of for f in `ls`; do echo "$f"; done Same behavior, for the same reason?
  - chuckadams 2y ago
    The quotes are preserved, but backquote expansion fills the argument list using any whitespace as a delimiter. $ for f in `ls`; do echo "$f"; done foo "bar baz If you absolutely must parse ls (let's assume it's some other script that outputs items with spaces) and the output can contain spaces, you have a few options: $ ls | while read f; do echo "$f"; done foo "bar baz # parens keep the IFS change isolated to a subshell $ (IFS="\n"; for f in `ls`; do echo "$f"; done) foo "bar baz But if your filenames contain newlines, you'll really want to stick with the glob expansion, or output custom delimiters and set IFS to that.
    - ChoHag 2y ago
      > If you absolutely must parse ls ... stop and rethink your options. You may be able to get away with parsing the first columns of ls -l but even then a pathologically named file could make itself look like a line of ls output. It's simply not possible in all cases. If you can constrain your input then you may be able to make use of it but in the general case, that's why xargs and find grew a -0 option. Or glob.