7 ms·

Rather unfortunately, I happen to have a handful of files on my machine with newlines in them (the filenames were programmatically generated from a summary of t

Rather unfortunately, I happen to have a handful of files on my machine with newlines in them (the filenames were programmatically generated from a summary of their contents). I loathe the possibility that my shell tools are going to suddenly crash when confronted with these weird files, rather than just producing some slightly silly output. I wish we'd standardized the behaviour of just escaping such characters as `\n/\r` or `^J/^M`...

nasretdinov 2y ago
The thing is, it's hard to predict what would happen to those scripts regardless... E.g. try naming your files "-rf" and see how many things break :)
- redserk 2y ago
  If one really wanted to embrace chaos, introduce this as a new team file naming standard for "risk finding" files ;)
- tetha 2y ago
  I do enjoy "ls *; touch -- -lisah; ls *" as a fun little brainteaser for those uninitiated to this behavior.
- ykonstant 2y ago
  A correct script will have no problems with "-rf" or any other file name. I have (and recommend script writers make their own) a directory hierarchy of "dangerous" file names to test scripts. For example, it contains a directory where all file and subdirectory names are in unary, consisting only of repetitions of the newline character. A correct script should be able to enumerate, access and modify files in there without issue.
- nneonneo 2y ago
  export TMPDIR=" / " to surprise the next person or script to do "rm -rf $TMPDIR/foo"...
ykonstant 2y ago
In academia, I get (and used to create) pdfs with names like: "On the number of associative foobars of degree blah - Johnson and Anderson.pdf" all the time. It is very convenient for non-technical academics to have a descriptive file name, and to be able to see it entirely in the navigator they use newlines.
- ykonstant 2y ago
  I am interested in hearing the rationale for downvotes explicitly. I am describing a reality that exists and must be taken into account. Why are you downvoting?
- oneeyedpigeon 2y ago
  Oh god. I already get upset enough by spaces in a file name, although I realise that fight is basically lost now!
  - enriquto 2y ago
    As a fellow spaces-in-filenames-hater, the fight is not lost. We are on the brink of winning it; it's just a mount option away! While we cannot avoid that people hit the spacebar when writing a filename on a gui, this does not mean at all that the resulting filename itself need contain a plain space character. Those spaces can and should be transparently translated to non-breaking space characters at some point. Maybe by the gui itself, or more robustly by the filesystem. This would make everybody happy: gui users and naive shell script writers.
    - poincaredisk 2y ago
      >Those spaces can and should be transparently translated to non-breaking space characters at some point Why? This just introduces more complexity and interoperability headaches for seemingly no reason.
      - enriquto 2y ago
        > Why? In order to preserve the sacrosanct simplicity of naive shell scripts. Seems like a very noble goal to me. The only unexpexted compexity arises when you want to deal with filenames having mixed spaces and nbsps. But I'd say that people who do that had it coming.
jraph 2y ago
They did the right thing for this: make the tools fail on file creation, but not on existing files. I guess it's still advisable to rename those files, I don't know how things like cp, mv or rsync will behave when copying such files in the future.
- ykonstant 2y ago
  If your file system allows them, be careful with symlinks though!
  - jraph 2y ago
    Why, specifically? I'm convinced we will need to be careful with symbolic links related to new line characters in filenames, but I'm curious of which specific aspect you had in mind.
    - ykonstant 2y ago
      Oh, nothing specific to newlines. Just, when you rename files to fix newlines, you need to check if they break symlinks pointing to them. For instance, I had project folders for my individual research projects. In order to have a central repository of resources and not have copies of multi-megabyte pdfs in each folder, I put all referenced papers in a single directory and symlinked them for each project that needed them. Later, I wanted to rename the papers to remove newlines. The symlinks complicated this process quite a bit!
      - jraph 2y ago
        Ah, right, indeed :-)
- nneonneo 2y ago
  No, they did not do the right thing: > the following utilities are now either encouraged to error out if they are to create a filename that contains a newline, and/or encouraged to error out if they are *about to print a pathname that contains a newline* in a context where newlines may be used as a separator It then proceeds to list a bunch of utilities including diff, file, find, grep, head, du, etc., none of which create files directly. These utilities could be updated to reject newlines in file paths if they're going to print in a "newline delimited" form - but for some of these utilities, that's the only available form.
stouset 2y ago
Dude, just fix the filenames.
funcDropShadow 2y ago
Of course, there is an xkcd for that: https://xkcd.com/1172/ https://xkcd.com/1172/

nasretdinov 2y ago

The thing is, it's hard to predict what would happen to those scripts regardless... E.g. try naming your files "-rf" and see how many things break :)

redserk 2y ago
If one really wanted to embrace chaos, introduce this as a new team file naming standard for "risk finding" files ;)
tetha 2y ago
I do enjoy "ls *; touch -- -lisah; ls *" as a fun little brainteaser for those uninitiated to this behavior.
ykonstant 2y ago
A correct script will have no problems with "-rf" or any other file name. I have (and recommend script writers make their own) a directory hierarchy of "dangerous" file names to test scripts. For example, it contains a directory where all file and subdirectory names are in unary, consisting only of repetitions of the newline character. A correct script should be able to enumerate, access and modify files in there without issue.
nneonneo 2y ago
export TMPDIR=" / " to surprise the next person or script to do "rm -rf $TMPDIR/foo"...

ykonstant 2y ago

In academia, I get (and used to create) pdfs with names like: "On the number of associative foobars of degree blah - Johnson and Anderson.pdf" all the time. It is very convenient for non-technical academics to have a descriptive file name, and to be able to see it entirely in the navigator they use newlines.

ykonstant 2y ago
I am interested in hearing the rationale for downvotes explicitly. I am describing a reality that exists and must be taken into account. Why are you downvoting?
oneeyedpigeon 2y ago
Oh god. I already get upset enough by spaces in a file name, although I realise that fight is basically lost now!
- enriquto 2y ago
  As a fellow spaces-in-filenames-hater, the fight is not lost. We are on the brink of winning it; it's just a mount option away! While we cannot avoid that people hit the spacebar when writing a filename on a gui, this does not mean at all that the resulting filename itself need contain a plain space character. Those spaces can and should be transparently translated to non-breaking space characters at some point. Maybe by the gui itself, or more robustly by the filesystem. This would make everybody happy: gui users and naive shell script writers.
  - poincaredisk 2y ago
    >Those spaces can and should be transparently translated to non-breaking space characters at some point Why? This just introduces more complexity and interoperability headaches for seemingly no reason.
    - enriquto 2y ago
      > Why? In order to preserve the sacrosanct simplicity of naive shell scripts. Seems like a very noble goal to me. The only unexpexted compexity arises when you want to deal with filenames having mixed spaces and nbsps. But I'd say that people who do that had it coming.

jraph 2y ago

They did the right thing for this: make the tools fail on file creation, but not on existing files. I guess it's still advisable to rename those files, I don't know how things like cp, mv or rsync will behave when copying such files in the future.

ykonstant 2y ago
If your file system allows them, be careful with symlinks though!
- jraph 2y ago
  Why, specifically? I'm convinced we will need to be careful with symbolic links related to new line characters in filenames, but I'm curious of which specific aspect you had in mind.
  - ykonstant 2y ago
    Oh, nothing specific to newlines. Just, when you rename files to fix newlines, you need to check if they break symlinks pointing to them. For instance, I had project folders for my individual research projects. In order to have a central repository of resources and not have copies of multi-megabyte pdfs in each folder, I put all referenced papers in a single directory and symlinked them for each project that needed them. Later, I wanted to rename the papers to remove newlines. The symlinks complicated this process quite a bit!
    - jraph 2y ago
      Ah, right, indeed :-)
nneonneo 2y ago
No, they did not do the right thing: > the following utilities are now either encouraged to error out if they are to create a filename that contains a newline, and/or encouraged to error out if they are *about to print a pathname that contains a newline* in a context where newlines may be used as a separator It then proceeds to list a bunch of utilities including diff, file, find, grep, head, du, etc., none of which create files directly. These utilities could be updated to reject newlines in file paths if they're going to print in a "newline delimited" form - but for some of these utilities, that's the only available form.

stouset 2y ago

Dude, just fix the filenames.

funcDropShadow 2y ago

Of course, there is an xkcd for that: https://xkcd.com/1172/ https://xkcd.com/1172/