3 ms·

> Why would "the path name you entered is not valid Unicode/UTF-8" be a good error message, if the path actually exists on the system? Some programs have to co

> Why would "the path name you entered is not valid Unicode/UTF-8" be a good error message, if the path actually exists on the system? Some programs have to cope with arbitrary file names. (So yes, cp shouldn't require filenames to be UTF-8.) The vast majority don't. I maintain a Rust crate called camino [1], the readme for which outlines the general philosophy. The fact is that simply enforcing that the file paths you deal with are always UTF-8 greatly simplifies a lot of code. > Also, what does "jank" mean here? What do you gain by treating file names as Unicode instead of byte sequences, for the majority of programs that don't even need to display the name, except perhaps in logs? If you ever have filenames in a text file, how do you match them up with filenames on disk? If you try to support the full space of filenames that can possibly exist on a platform, there is no general, cross-platform solution for doing so. (This is known as the "makefile problem", and if you find the right wiki page you'll see a large table exploring all the possibilities and their tradeoffs.) And if you start pulling that thread, you'll unravel a very large number of problems trying to handle non-Unicode output in reality. But you can simply cut the knot by restricting filenames to Unicode, and most programs should do that. For example, I work at Oxide. Why would any of our services want any internal filenames to be anything but UTF-8 (or really ASCII)? Trying to support weird filenames is unnecessary complexity. So we just use camino. [1] https://crates.io/crates/camino/ https://crates.io/crates/camino/

lesuorac 2y ago
Why serde1 and proptest1 as opposed to say just serde and proptest or serde-camino and proptest-camino?
- sunshowers 2y ago
  Because there can be a serde 2 or proptest 2 in the future, but camino's API surface is relatively small and pretty rigid so there will never be a camino 2. If camino's MSRV was more modern (1.60 I think?) Rust I'd remove the `serde` and `proptest` features entirely via the `dep:` syntax. (Come to think of it, it may be worth bumping the MSRV for that! Would want to look at some data, and maybe in a few months -- camino deliberately has an ancient MSRV as a foundational crate.)
  - lesuorac 2y ago
    So if Serde goes from 1.0.202 to 2.0.202 you're going to have both a serde1 feature and a serde2 feature? Is this just an oxide style guide thing to include a major version in the feature name?
    - sunshowers 2y ago
      Yes, we'd have both serde1 and serde2 features. Published this library years before I started at Oxide.
tsimionescu 2y ago
I would say loads of Unix tools have a much bigger problem with files that contain whitespace and especially newlines then they do with the character encoding. Similarly, you can usefully process a lot of file types if you can safely assume only the encoding of special characters for that format, like {",[ and newline for JSON. This is why I don't get what you mean by "text file" in this context. Obviously it's hard, if not impossible, to meaningfully interpret any part of a random text file as a file name, regardless of encoding. But if you have a text file in some known structured format, it shouldn't be a significant problem at all, as long as you know the encoding of those special characters and have some basic conventions. In particular, the agreement could be that the filenames will be represented as raw bytes except for format-specific escapes (like escaping " in JSON or > in XML), then the file name part need not even fully match the intended encoding of the rest of the file. It's true though that it's not very easy to work with a byte array that has different encodings in different parts. On the other hand, I fully agree that it's a good idea to restrict things to simple sunsets of characters if you can get away with it. I just don't think that restring to "all of Unicode" is particularly useful. Restricting to a subset of ASCII or even just to the BMP does have meaningful advantages, if it's an option for a particular domain.
- sunshowers 2y ago
  Your proposed scheme works on Unix, but importantly, not on Windows. This is exactly the makefile problem.
- sunshowers 2y ago
  > Restricting to a subset of ASCII or even just to the BMP does have meaningful advantages, if it's an option for a particular domain. This is definitely appropriate in some cases, but for Rust specifically gets in the way a lot. For example, &camino::Utf8Path and &str have transparent conversions both ways, in a way that users rely on heavily (passing in a string into a function that takes an AsRef<Path> or AsRef<Utf8Path> is extremely common). If you introduced, say, AsciiPath, there would be a lot more friction -- you couldn't just pass in an arbitrary string and treat it as an AsciiPath. Again, Schelling point -- without prior coordination you can assume that folks are using strings.

lesuorac 2y ago

Why serde1 and proptest1 as opposed to say just serde and proptest or serde-camino and proptest-camino?

sunshowers 2y ago
Because there can be a serde 2 or proptest 2 in the future, but camino's API surface is relatively small and pretty rigid so there will never be a camino 2. If camino's MSRV was more modern (1.60 I think?) Rust I'd remove the `serde` and `proptest` features entirely via the `dep:` syntax. (Come to think of it, it may be worth bumping the MSRV for that! Would want to look at some data, and maybe in a few months -- camino deliberately has an ancient MSRV as a foundational crate.)
- lesuorac 2y ago
  So if Serde goes from 1.0.202 to 2.0.202 you're going to have both a serde1 feature and a serde2 feature? Is this just an oxide style guide thing to include a major version in the feature name?
  - sunshowers 2y ago
    Yes, we'd have both serde1 and serde2 features. Published this library years before I started at Oxide.

tsimionescu 2y ago

I would say loads of Unix tools have a much bigger problem with files that contain whitespace and especially newlines then they do with the character encoding. Similarly, you can usefully process a lot of file types if you can safely assume only the encoding of special characters for that format, like {",[ and newline for JSON. This is why I don't get what you mean by "text file" in this context. Obviously it's hard, if not impossible, to meaningfully interpret any part of a random text file as a file name, regardless of encoding. But if you have a text file in some known structured format, it shouldn't be a significant problem at all, as long as you know the encoding of those special characters and have some basic conventions. In particular, the agreement could be that the filenames will be represented as raw bytes except for format-specific escapes (like escaping " in JSON or > in XML), then the file name part need not even fully match the intended encoding of the rest of the file. It's true though that it's not very easy to work with a byte array that has different encodings in different parts. On the other hand, I fully agree that it's a good idea to restrict things to simple sunsets of characters if you can get away with it. I just don't think that restring to "all of Unicode" is particularly useful. Restricting to a subset of ASCII or even just to the BMP does have meaningful advantages, if it's an option for a particular domain.

sunshowers 2y ago
Your proposed scheme works on Unix, but importantly, not on Windows. This is exactly the makefile problem.
sunshowers 2y ago
> Restricting to a subset of ASCII or even just to the BMP does have meaningful advantages, if it's an option for a particular domain. This is definitely appropriate in some cases, but for Rust specifically gets in the way a lot. For example, &camino::Utf8Path and &str have transparent conversions both ways, in a way that users rely on heavily (passing in a string into a function that takes an AsRef<Path> or AsRef<Utf8Path> is extremely common). If you introduced, say, AsciiPath, there would be a lot more friction -- you couldn't just pass in an arbitrary string and treat it as an AsciiPath. Again, Schelling point -- without prior coordination you can assume that folks are using strings.