3 ms·

This article seems to assume paths will be valid UTF-8, which isn't true on either Linux certainly, and Windows as far as I know. of course we could say "paths

This article seems to assume paths will be valid UTF-8, which isn't true on either Linux certainly, and Windows as far as I know. of course we could say "paths must be valid UTF-8 for this program to work" (quite a few Rust programs do require this, as they store paths in standard Rust strings, which themselves must be valid UTF-8), but if your concern is dodgy paths breaking things, you probably need to check for that somewhere?

tialaramex 3y ago
Yes, in Windows the actual implementation requires that paths consist of sequences of 16-bit values, and those values have some constraints, but the constraints don't include requiring that they're valid UTF-16, and so they aren't necessarily valid Unicode text, and so there may be no UTF-8 representation. In Unix they're just bytes and the bytes may not be UTF-8. In Rust terms what Windows is doing is basically [u16] and what Unix does is basically [u8] and neither of these is necessarily meaningful human text. Internally Rust's OsString is probably like the hack in this blog post, all valid UTF-8 is just stored as UTF-8 which means everything else must be using the byte values which aren't needed in UTF-8. But Rust is explicit that this is opaque and not guaranteed to stay the same across compiler or library versions.
- account42 3y ago
  Afaik Rust (currently) uses WTF-8 [0] to store Windows (WTF-16) strings which is a very useful encoding if you need to deal with such strings in your programs written in other languages as well. The conversion is essentially the same as UTF-16 -> to UTF-8 except you interpret unmatched surrogate pairs as (reserved) Unicode values with the same value and encode those to UTF-8 as you would any other Unicode value. So this doesn't use exactly the same trick as TFA - instead of using invalid UTF-8 encodings it uses "normal" UTF-8 encodings of invalid Unicode values (specifically exactly those that were reserved when extending UCS-2 to UTF-16). Or in other words, WTF-16 <-> WTF-8 conversion is the same as UTF-16 <-> UTF-8 conversion but without (some of) the error handling. [0] https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/
- ekimekim 3y ago
  What unix does is closer to [NonZeroU8] which is actually very helpful as can be seen in shell commands like find -print0 | xargs -0, since a NUL character is guarenteed to not be part of the actual arguments. You could do the same here, but I suspect the program in question does not support NUL characters in their strings for the same reason unix doesn't (because that's what C does).
  - tialaramex 3y ago
    A good observation, I'd forgotten NonZeroU8 is a standard library type. I suspect (but haven't verified) Windows likewise requires [NonZeroU16]
- toast0 3y ago
  > In Unix they're just bytes and the bytes may not be UTF-8. Depends on the Unix. I believe MacOS enforces unicode or at least does some form of unicode normalization.
- BobbyTables2 3y ago
  Even Python assumes Unicode, unless one passes “bytes” strings for filenames. I was recently astounded when small Python script I whipped up to hash and compare binary file content died with a Unicode related exception — from the filename itself! (Walking a directory using “bytes” paths fixed it)

tialaramex 3y ago

Yes, in Windows the actual implementation requires that paths consist of sequences of 16-bit values, and those values have some constraints, but the constraints don't include requiring that they're valid UTF-16, and so they aren't necessarily valid Unicode text, and so there may be no UTF-8 representation. In Unix they're just bytes and the bytes may not be UTF-8. In Rust terms what Windows is doing is basically [u16] and what Unix does is basically [u8] and neither of these is necessarily meaningful human text. Internally Rust's OsString is probably like the hack in this blog post, all valid UTF-8 is just stored as UTF-8 which means everything else must be using the byte values which aren't needed in UTF-8. But Rust is explicit that this is opaque and not guaranteed to stay the same across compiler or library versions.

account42 3y ago
Afaik Rust (currently) uses WTF-8 [0] to store Windows (WTF-16) strings which is a very useful encoding if you need to deal with such strings in your programs written in other languages as well. The conversion is essentially the same as UTF-16 -> to UTF-8 except you interpret unmatched surrogate pairs as (reserved) Unicode values with the same value and encode those to UTF-8 as you would any other Unicode value. So this doesn't use exactly the same trick as TFA - instead of using invalid UTF-8 encodings it uses "normal" UTF-8 encodings of invalid Unicode values (specifically exactly those that were reserved when extending UCS-2 to UTF-16). Or in other words, WTF-16 <-> WTF-8 conversion is the same as UTF-16 <-> UTF-8 conversion but without (some of) the error handling. [0] https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/
ekimekim 3y ago
What unix does is closer to [NonZeroU8] which is actually very helpful as can be seen in shell commands like find -print0 | xargs -0, since a NUL character is guarenteed to not be part of the actual arguments. You could do the same here, but I suspect the program in question does not support NUL characters in their strings for the same reason unix doesn't (because that's what C does).
- tialaramex 3y ago
  A good observation, I'd forgotten NonZeroU8 is a standard library type. I suspect (but haven't verified) Windows likewise requires [NonZeroU16]
toast0 3y ago
> In Unix they're just bytes and the bytes may not be UTF-8. Depends on the Unix. I believe MacOS enforces unicode or at least does some form of unicode normalization.
BobbyTables2 3y ago
Even Python assumes Unicode, unless one passes “bytes” strings for filenames. I was recently astounded when small Python script I whipped up to hash and compare binary file content died with a Unicode related exception — from the filename itself! (Walking a directory using “bytes” paths fixed it)