22 ms·

> We’ve established that, yes, pathnames can include newlines. We have not established why they can do that. After some deliberation, the Austin Group could not

> We’ve established that, yes, pathnames can include newlines. We have not established why they can do that. After some deliberation, the Austin Group could not find a single use-case for newlines in pathnames besides breaking naive scripts. Wouldn’t it be nice if the naive scripts were just correct now? Ok, that might be a bit much all at once. We’re heading there though! Oh my god. This makes me so happy. This is the most lovely think I've read in the world of computing since the unix gods decided that newlines were to be a single character. The philosophy underlying the sentence "Wouldn’t it be nice if the naive scripts were just correct now?" is incredibly positive. We are surrounded by arrogant jerks who break old code by aggressively enforcing stricter compliance of some stupid rules. But here come these posix heros who do the exact opposite: make old code correct! There is hope in mankind after all.

ezoe 2y ago
Don't assume UTF-8 is the only character encoding used in the wild. There are character encoding with leading bytes not easily detectable like UTF-8.
- arghwhat 2y ago
  In 2024, if you don't get the correct result decoding a text as UTF-8, the bug is the text, not the decoding. And luckily, adoption of UTF-8 in the past 30+ years have gone will enough that you don't need to worry. Caveats for cursed hardware standards demanding two-byte encodings like USB.
  - poincaredisk 2y ago
    I hope you're happy in your ivory tower, but I personally work with a lot of files with other encoding, most often that weird utf16 (Windows), sometimes also legacy files with different ANSI encoding. Declaring "my decoder is fine, it's the text that is buggy" is not going to score a lot of points with my boss and clients.
    - zelphirkalt 2y ago
      Shouldn't hurt to tell clients to right their weird proprietary software originated encodings though.
    - arghwhat 2y ago
      The only valid reason for still having files stored in legacy ANSI encodings is that their only use is input to software that has not been maintained for ~30 years and cannot be updated. That's fine because they're just binary inputs in a closed ecosystem that no one touches. But if they are supposed to be treated as text, then yes it's the text that's buggy - they should just be converted to UTF-8 once and have the originals thrown away. UTF-16 is something that Microsoft has cursed us with by inserting it into specifications (like USB) so that we cannot get rid of it, even if it never made any sense what so ever. But those are in effect explicit protocols with a hard contract, very different from something where you would "assume an encoding".
  - 1oooqooq 2y ago
    why people assume utf8 had only know locale encoding still? you're probably guilty of the sin you preach and is showing wrongly decoded utf8 and don't even know.
anal_reactor 2y ago
It's a bandaid on a wider problem: the design of Unix shell is bonkers and the whole thing should be deleted. Why? Because I haven't seen any other tool ever have so many pitfalls. Take n random languages and m random developers and tell them to loop over a string array and print its contents, and count how many correct programs you get on average per language. There will be easy languages, then difficult languages, then a huge gap, then Unix shell because in your random sample you managed to get one guy who has PhD in bash.
- blueflow 2y ago
  Someone needs to come up with a interactive shell first, one that is comparable in usability. Then we can think about replacing the unix shell. I tried both python and lua interactively, but they are a pain when it comes to handling files. You have to type much more to get the same things done.
  - anal_reactor 2y ago
    The bigger issue is the sheer momentum of Unix shell. Even if you come up with an alternative that is better by every objectively measurable metric, it's still going to be a monumental task to have it packages with commonly used distros. Kinda like the "why can't the US switch to the metric system" problem.
    - blueflow 2y ago
      People already use different shells, mksh, fish, and so on. With fish there is a non-posix shell in wide use.
      - oguz-ismail 2y ago
        >wide use Five people around the globe isn't wide use.
        blueflow 2y ago
nneonneo 2y ago
Rather unfortunately, I happen to have a handful of files on my machine with newlines in them (the filenames were programmatically generated from a summary of their contents). I loathe the possibility that my shell tools are going to suddenly crash when confronted with these weird files, rather than just producing some slightly silly output. I wish we'd standardized the behaviour of just escaping such characters as `\n/\r` or `^J/^M`...
- nasretdinov 2y ago
  The thing is, it's hard to predict what would happen to those scripts regardless... E.g. try naming your files "-rf" and see how many things break :)
  - redserk 2y ago
    If one really wanted to embrace chaos, introduce this as a new team file naming standard for "risk finding" files ;)
  - tetha 2y ago
    I do enjoy "ls *; touch -- -lisah; ls *" as a fun little brainteaser for those uninitiated to this behavior.
  - ykonstant 2y ago
    A correct script will have no problems with "-rf" or any other file name. I have (and recommend script writers make their own) a directory hierarchy of "dangerous" file names to test scripts. For example, it contains a directory where all file and subdirectory names are in unary, consisting only of repetitions of the newline character. A correct script should be able to enumerate, access and modify files in there without issue.
  - nneonneo 2y ago
    export TMPDIR=" / " to surprise the next person or script to do "rm -rf $TMPDIR/foo"...
- ykonstant 2y ago
account42 2y ago
Their proposed solution is not compatible with reality though where POSIX does not get to define what kind of files exist on filesystems you need to work with. All they did is introduce new error cases in C programs while not actually fixing anything for shell scripts. If anything, it's going to result in more exploits as people write shell scripts with the assumption that newlines cannot appear in filenames.
- quotemstr 2y ago
  In the real world, nobody writes shell scripts that handle newlines in filenames.
  - account42 2y ago
    I do. Single files are handled with quotes around arguments just fine. For lists of files you need to use NUL as a separator. That's not really hard to do once you are aware of the problem but ergonomics could be better - which is something useful that POSIX could change.
    - quotemstr 2y ago
      Most shell scripts that correctly handle newlines do so by accident.
      - account42 2y ago
        Because they don't deal with lists of files so this whole issue doesn't apply? Yes. Even more reason to just fix the shell scripts where this matters instead of trying to impose additional restrictions on filesystems with many decades of history.
zokier 2y ago
But they did not make old code correct. Filenames are still allowed to contain newlines. Shell scripts still need to be prepared to deal with that. Nothing really changed, they just added a feel-good half-measure.
- quotemstr 2y ago
  It's a step in the right direction. You have to understand that for decades a vocal group of Unix die-hards has opposed any limitations whatsoever on the bytewise content of file names. The newline restriction in this latest version of POSIX may be modest, but it represents a dam breaking. When (obviously) the sky doesn't fall, the next version of POSIX will have a lot more filename cleanup.
  - rixed 2y ago
    Next step is to forbid newlines from file content itself, to fix conforment json parsers ?
- janderland 2y ago
  This is pretty standard for a human run system. Gotta make the human feel good about an idea before they can do said idea. If you’re not familiar with humans, there are several manuals available online.
hwc 2y ago
Now do that with all whitespace!

ezoe 2y ago

Don't assume UTF-8 is the only character encoding used in the wild. There are character encoding with leading bytes not easily detectable like UTF-8.

arghwhat 2y ago
In 2024, if you don't get the correct result decoding a text as UTF-8, the bug is the text, not the decoding. And luckily, adoption of UTF-8 in the past 30+ years have gone will enough that you don't need to worry. Caveats for cursed hardware standards demanding two-byte encodings like USB.
- poincaredisk 2y ago
  I hope you're happy in your ivory tower, but I personally work with a lot of files with other encoding, most often that weird utf16 (Windows), sometimes also legacy files with different ANSI encoding. Declaring "my decoder is fine, it's the text that is buggy" is not going to score a lot of points with my boss and clients.
  - zelphirkalt 2y ago
    Shouldn't hurt to tell clients to right their weird proprietary software originated encodings though.
  - arghwhat 2y ago
    The only valid reason for still having files stored in legacy ANSI encodings is that their only use is input to software that has not been maintained for ~30 years and cannot be updated. That's fine because they're just binary inputs in a closed ecosystem that no one touches. But if they are supposed to be treated as text, then yes it's the text that's buggy - they should just be converted to UTF-8 once and have the originals thrown away. UTF-16 is something that Microsoft has cursed us with by inserting it into specifications (like USB) so that we cannot get rid of it, even if it never made any sense what so ever. But those are in effect explicit protocols with a hard contract, very different from something where you would "assume an encoding".
- 1oooqooq 2y ago
  why people assume utf8 had only know locale encoding still? you're probably guilty of the sin you preach and is showing wrongly decoded utf8 and don't even know.

anal_reactor 2y ago

It's a bandaid on a wider problem: the design of Unix shell is bonkers and the whole thing should be deleted. Why? Because I haven't seen any other tool ever have so many pitfalls. Take n random languages and m random developers and tell them to loop over a string array and print its contents, and count how many correct programs you get on average per language. There will be easy languages, then difficult languages, then a huge gap, then Unix shell because in your random sample you managed to get one guy who has PhD in bash.

blueflow 2y ago
Someone needs to come up with a interactive shell first, one that is comparable in usability. Then we can think about replacing the unix shell. I tried both python and lua interactively, but they are a pain when it comes to handling files. You have to type much more to get the same things done.
- anal_reactor 2y ago
  The bigger issue is the sheer momentum of Unix shell. Even if you come up with an alternative that is better by every objectively measurable metric, it's still going to be a monumental task to have it packages with commonly used distros. Kinda like the "why can't the US switch to the metric system" problem.
  - blueflow 2y ago
    People already use different shells, mksh, fish, and so on. With fish there is a non-posix shell in wide use.
    - oguz-ismail 2y ago
      >wide use Five people around the globe isn't wide use.
      - blueflow 2y ago

nneonneo 2y ago

Rather unfortunately, I happen to have a handful of files on my machine with newlines in them (the filenames were programmatically generated from a summary of their contents). I loathe the possibility that my shell tools are going to suddenly crash when confronted with these weird files, rather than just producing some slightly silly output. I wish we'd standardized the behaviour of just escaping such characters as `\n/\r` or `^J/^M`...

nasretdinov 2y ago
The thing is, it's hard to predict what would happen to those scripts regardless... E.g. try naming your files "-rf" and see how many things break :)
- redserk 2y ago
  If one really wanted to embrace chaos, introduce this as a new team file naming standard for "risk finding" files ;)
- tetha 2y ago
  I do enjoy "ls *; touch -- -lisah; ls *" as a fun little brainteaser for those uninitiated to this behavior.
- ykonstant 2y ago
  A correct script will have no problems with "-rf" or any other file name. I have (and recommend script writers make their own) a directory hierarchy of "dangerous" file names to test scripts. For example, it contains a directory where all file and subdirectory names are in unary, consisting only of repetitions of the newline character. A correct script should be able to enumerate, access and modify files in there without issue.
- nneonneo 2y ago
  export TMPDIR=" / " to surprise the next person or script to do "rm -rf $TMPDIR/foo"...
ykonstant 2y ago

account42 2y ago

Their proposed solution is not compatible with reality though where POSIX does not get to define what kind of files exist on filesystems you need to work with. All they did is introduce new error cases in C programs while not actually fixing anything for shell scripts. If anything, it's going to result in more exploits as people write shell scripts with the assumption that newlines cannot appear in filenames.

quotemstr 2y ago
In the real world, nobody writes shell scripts that handle newlines in filenames.
- account42 2y ago
  I do. Single files are handled with quotes around arguments just fine. For lists of files you need to use NUL as a separator. That's not really hard to do once you are aware of the problem but ergonomics could be better - which is something useful that POSIX could change.
  - quotemstr 2y ago
    Most shell scripts that correctly handle newlines do so by accident.
    - account42 2y ago
      Because they don't deal with lists of files so this whole issue doesn't apply? Yes. Even more reason to just fix the shell scripts where this matters instead of trying to impose additional restrictions on filesystems with many decades of history.

zokier 2y ago

But they did not make old code correct. Filenames are still allowed to contain newlines. Shell scripts still need to be prepared to deal with that. Nothing really changed, they just added a feel-good half-measure.

quotemstr 2y ago
It's a step in the right direction. You have to understand that for decades a vocal group of Unix die-hards has opposed any limitations whatsoever on the bytewise content of file names. The newline restriction in this latest version of POSIX may be modest, but it represents a dam breaking. When (obviously) the sky doesn't fall, the next version of POSIX will have a lot more filename cleanup.
- rixed 2y ago
  Next step is to forbid newlines from file content itself, to fix conforment json parsers ?
janderland 2y ago
This is pretty standard for a human run system. Gotta make the human feel good about an idea before they can do said idea. If you’re not familiar with humans, there are several manuals available online.

hwc 2y ago

Now do that with all whitespace!