5 ms·

On a similar note, I sometimes think about how newline characters are allowed in filenames, and how that can break simple... for each $filename in `ls` lo

On a similar note, I sometimes think about how newline characters are allowed in filenames, and how that can break simple... for each $filename in `ls` loops -- because in many contexts, UNIX treats newlines as a delimiter. Is there any legitimate use for filenames with newlines?

xxpor 2y ago
this is why things like `find -print0` exist, which is IMO the easiest way to handle this robustly.
Joker_vD 2y ago
Sticky notes on the desktop :) Who needs data storage when you can store it all in the metadata?
IsTom 2y ago
You can also create files named e.g. '--help' (if you're not particularly malicious) and with globbing it'll cause e.g. 'ls *' to print help.
- jasonjayr 2y ago
  touch -- '-f ..' (If you want to lay an evil trap) Remember that in most option parsing libraries, putting '--' in your arguments stops option parsing, so you can safely run: rm -- '-f ..'
bityard 2y ago
Well, knowing how to deal with wacky input and corner cases are a requirement of learning ANY programming language. Bourne-style shells are no exception. Your example has illegal syntax, but the biggest issue is that you should never parse the output of ls. The shell has built-in globbing. This is how you would loop over all entries (files, dirs, symlinks, etc) in the current directory without getting tripped up by whitespace: for e in *; do echo "got: $e"; done
- dzaima 2y ago
  But, of course, this wouldn't be shell if that didn't have footguns more; namely, it breaks if ran in an empty directory (giving a literal "got: *"), and excludes the arbitrary set of files whose name begins with ".".
- Taikonerd 2y ago
  > knowing how to deal with wacky input and corner cases are a requirement of learning ANY programming language. In general, I agree. But if there's a corner case that occasionally breaks naive code but otherwise doesn't do anything, then I'm going to think, "maybe we should just remove that corner case."
  - bell-cot 2y ago
    Replace "maybe" with "OBVIOUSLY". Keeping useless-but-hazardous "features" in any language is as idiotic as keeping a heap of oily rags in the furniture factory warehouse.
  - zokier 2y ago
    David Wheeler has been complaining (and suggesting fixes) about this for a long time: https://dwheeler.com/essays/fixing-unix-linux-filenames.html https://dwheeler.com/essays/fixing-unix-linux-filenames.html safename LSM https://lwn.net/Articles/686789/ https://lwn.net/Articles/686789/
    - Taikonerd 2y ago
      Thank you, I had wondered if there was something like safename.
chuckadams 2y ago
> Is there any legitimate use for filenames with newlines? IMHO no, but they can exist, so you need to handle them without blowing up. Also, even spaces are considered delimiters here, which is why it's bad form to parse the output of ls. $ touch "foo bar baz" $ for f in `ls`; do echo $f; done foo bar baz # always use double quotes, though they aren't needed here $ for f in *; do echo "$f"; done foo bar baz At least the OS guarantees you won't run into NUL though.
- unqueued 2y ago
  There is a pretty good syntax for dealing with nasty filenames, if you must: ANSI-C quoting[1]. If you have to output in a shellscript in this format, use printf %q from man printf: %q ARGUMENT is printed in a format that can be reused as shell input, escaping non-printable characters with the proposed POSIX $'' syntax. It is just $'<nasty ansi-c escaped chars>' $ touch $'\nHello\tWorld\n' $ ls One thing I do like about a filesystem that fully supports POSIX filenames is that at the end of the day a filesystem is supposed to represent data. I think it is totally sensible to exclude certain characters, but that it should be done higher up in the stack if possible. Or have a flag that is set at mount time. Perhaps even by subvolume/dataset. One thing I haven't seen mentioned is that POSIX filenames are so permissive that they allow you to have bytes as filenames that are invalid UTF-8. That's why the popular ncdu[2] program does NOT use json as it's file format, although most think it does. It's actually json but with raw POSIX bytes in filename fields, which is outside of the official json spec. That does not stop folks from using json tools to parse ncdu output though. Another standard that is also very permissive with filenames is git. When I started exploring new ways to encode data into a git repo, it was only natural that I encountered issues with limitations of filesystems that I would check out in. Try cloning this repo, and see if you are able to check it out: https://github.com/benibela/nasty-files https://github.com/benibela/nasty-files It is amazing how many things it breaks. If you are writing software that deals with git filenames or POSIX filenames (that includes things like parsing a zip file footer), you can not rely on your standard json encoding function, because the input may contain invalid utf-8. So you may need to do extra encoding/filtering. [1]: https://www.gnu.org/s/bash/manual/html_node/ANSI_002dC-Quoting.html https://www.gnu.org/s/bash/manual/html_node/ANSI_002dC-Quoti... [2]: https://dev.yorhel.nl/ncdu/jsonfmt https://dev.yorhel.nl/ncdu/jsonfmt
fragmede 2y ago
A GUI file browser will display the filename with a newline in it as a new line (and an icon above it) so as to be asthetically pleasing.

xxpor 2y ago

this is why things like `find -print0` exist, which is IMO the easiest way to handle this robustly.

Joker_vD 2y ago

Sticky notes on the desktop :) Who needs data storage when you can store it all in the metadata?

IsTom 2y ago

You can also create files named e.g. '--help' (if you're not particularly malicious) and with globbing it'll cause e.g. 'ls *' to print help.

jasonjayr 2y ago
touch -- '-f ..' (If you want to lay an evil trap) Remember that in most option parsing libraries, putting '--' in your arguments stops option parsing, so you can safely run: rm -- '-f ..'

bityard 2y ago

Well, knowing how to deal with wacky input and corner cases are a requirement of learning ANY programming language. Bourne-style shells are no exception. Your example has illegal syntax, but the biggest issue is that you should never parse the output of ls. The shell has built-in globbing. This is how you would loop over all entries (files, dirs, symlinks, etc) in the current directory without getting tripped up by whitespace: for e in *; do echo "got: $e"; done

dzaima 2y ago
But, of course, this wouldn't be shell if that didn't have footguns more; namely, it breaks if ran in an empty directory (giving a literal "got: *"), and excludes the arbitrary set of files whose name begins with ".".
Taikonerd 2y ago
> knowing how to deal with wacky input and corner cases are a requirement of learning ANY programming language. In general, I agree. But if there's a corner case that occasionally breaks naive code but otherwise doesn't do anything, then I'm going to think, "maybe we should just remove that corner case."
- bell-cot 2y ago
  Replace "maybe" with "OBVIOUSLY". Keeping useless-but-hazardous "features" in any language is as idiotic as keeping a heap of oily rags in the furniture factory warehouse.
- zokier 2y ago
  David Wheeler has been complaining (and suggesting fixes) about this for a long time: https://dwheeler.com/essays/fixing-unix-linux-filenames.html https://dwheeler.com/essays/fixing-unix-linux-filenames.html safename LSM https://lwn.net/Articles/686789/ https://lwn.net/Articles/686789/
  - Taikonerd 2y ago
    Thank you, I had wondered if there was something like safename.

chuckadams 2y ago

> Is there any legitimate use for filenames with newlines? IMHO no, but they can exist, so you need to handle them without blowing up. Also, even spaces are considered delimiters here, which is why it's bad form to parse the output of ls. $ touch "foo bar baz" $ for f in `ls`; do echo $f; done foo bar baz # always use double quotes, though they aren't needed here $ for f in *; do echo "$f"; done foo bar baz At least the OS guarantees you won't run into NUL though.

unqueued 2y ago
There is a pretty good syntax for dealing with nasty filenames, if you must: ANSI-C quoting[1]. If you have to output in a shellscript in this format, use printf %q from man printf: %q ARGUMENT is printed in a format that can be reused as shell input, escaping non-printable characters with the proposed POSIX $'' syntax. It is just $'<nasty ansi-c escaped chars>' $ touch $'\nHello\tWorld\n' $ ls One thing I do like about a filesystem that fully supports POSIX filenames is that at the end of the day a filesystem is supposed to represent data. I think it is totally sensible to exclude certain characters, but that it should be done higher up in the stack if possible. Or have a flag that is set at mount time. Perhaps even by subvolume/dataset. One thing I haven't seen mentioned is that POSIX filenames are so permissive that they allow you to have bytes as filenames that are invalid UTF-8. That's why the popular ncdu[2] program does NOT use json as it's file format, although most think it does. It's actually json but with raw POSIX bytes in filename fields, which is outside of the official json spec. That does not stop folks from using json tools to parse ncdu output though. Another standard that is also very permissive with filenames is git. When I started exploring new ways to encode data into a git repo, it was only natural that I encountered issues with limitations of filesystems that I would check out in. Try cloning this repo, and see if you are able to check it out: https://github.com/benibela/nasty-files https://github.com/benibela/nasty-files It is amazing how many things it breaks. If you are writing software that deals with git filenames or POSIX filenames (that includes things like parsing a zip file footer), you can not rely on your standard json encoding function, because the input may contain invalid utf-8. So you may need to do extra encoding/filtering. [1]: https://www.gnu.org/s/bash/manual/html_node/ANSI_002dC-Quoting.html https://www.gnu.org/s/bash/manual/html_node/ANSI_002dC-Quoti... [2]: https://dev.yorhel.nl/ncdu/jsonfmt https://dev.yorhel.nl/ncdu/jsonfmt

fragmede 2y ago

A GUI file browser will display the filename with a newline in it as a new line (and an icon above it) so as to be asthetically pleasing.