3 ms·
Rust uses a solution similar to his original proposal: OsString/OsStr: https://doc.rust-lang.org/std/ffi/struct.OsString.html https://doc.rust-lang.org/std/ffi/
by mmastrac 4y ago
Rust uses a solution similar to his original proposal: OsString/OsStr: https://doc.rust-lang.org/std/ffi/struct.OsString.html https://doc.rust-lang.org/std/ffi/struct.OsString.html
I wonder if unicode needs a "undecodable byte escape" sequence for these raw sequences that cannot be mapped into the string space, allowing for a guaranteed round-trip between the OS and Unicode. For this to work, you'd need a guarantee that all codepage/unicode mappings are consistent and 1:1 which might not be possible.
Paths are tricky because they act as both an identifier and a handle, and the string encoding of the identifier and handle are _close but not exactly the same_ when unicode is in the mix. In the ASCII world, you can assume that this is the case.
- LegionMammal978 4y agoPython already has the "surrogateescape" error handler [0] that performs something similar to what you described: undecodable bytes are translated into unpaired U+DC80 to U+DCFF surrogates. Of course, this isn't standardized in any way, but I've found it useful myself for smuggling raw pathnames through Java. [0] https://peps.python.org/pep-0383/ https://peps.python.org/pep-0383/
- mmastrac 4y agoThanks for this reference! It's exactly what I was thinking.
- pvillano 4y agowrt "undecidable byte escape" I think that wouldn't solve the problems with filenames. Files would have to enter Python land escaped which would mean my file really named \x00 would become \\x00 You can't escape only the broken filenames, because a valid unicode filename could contain an escape sequence, which would incorrectly be unescaped when returning to the OS I think you really do need a non-str type that can be silently converted to str 99% of the time