10 ms·

The History and rationale of the Python 3 Unicode model for the operating system

77 points by goranmoomin 4y ago

zgrxat 4y ago
The Unicode model is complex and overengineered. Like many Python features, it requires permanent marketing until people believe it is good. It provides a steady income stream for those paid to "work" on Python. The right way would have been utf-8, which is the only reasonable representation. Indexing could have been broken early in Python 3 or outsourced to a special type.
- StefanKarpinski 4y ago
  Absolutely right. Deprecating direct string indexing would have been the right move. Require writing `str.chars()` to get something that lets you slice by Unicode characters (i.e. code points); provide `str.graphemes()` and `str.grapheme_clusters()` to get something that lets you slice by graphemes and grapheme clusters, respectively. Cache an index structure that lets you do that kind of indexing efficiently once you've asked for it the first time. Provide an API to clear the caches. Not allowing strings to represent invalid Unicode is also a huge mistake (and essentially forced by the representation strategy that they adopted). It forces any programmer who wants to robustly handle potentially invalid string data to use byte vectors instead. Which is exactly what they did with OS paths, but that's far from the only place you can get invalid strings. You can get invalid strings almost anywhere! Worse, since it's incredibly inconvenient to work with byte vectors when you want to do stringlike stuff, no one does it unless forced to, so this design choice effectively guarantees that all Python code that works with strings will blow up if it encounters anything invalid—which is a very common occurrence. If only there was a type that behaves like a string and supports all the handy string operations but which handles invalid data gracefully. Then you could write robust string code conveniently. But at that point, you should just make that the standard string type! This isn't hypothetical, it's exactly how Burnt Sushi's bstr type [1] works in Rust and how the standard String type works in Julia. [1] https://github.com/BurntSushi/bstr https://github.com/BurntSushi/bstr
  - Jasper_ 4y ago
    It's worth noting that Python str's are sequences of code points, not scalar values. This was a truly horrendous mistake made mostly out of ignorance, but now they rely upon it in surrogateescape to hide "invalid" data, so... I have ranted for long hours go friends about the insanity of Python 3's text model before. It's mostly the blind leading the blind.
    - 323 4y ago
      I've seen this happen multiple times in Python, where they implement a feature without a thorough survey of what other languages are doing. Then when it turns out the new feature has a major broken part, they fix that, but again, without looking how it's done in other places. asyncio is a major example.
  - Spivak 4y ago
    > It forces any programmer who wants to robustly handle potentially invalid string data to use byte vectors instead. Is this not the only thing you can really do? If strings could hold invalid unicode then they effectively become bytes and you now have to be wary of every possible string. I would rather Python just do away with strings entirely in the integration points with the OS and make you either keep them as opaque bytes or decode them. > behaves like a string and supports all the handy string operations but which handles invalid data gracefully Unless Guido was feeling particularly practical I can't imagine this ever making it into the stdlib because the choice of "gracefully" is somewhat arbitrary and application dependent.
  - wk_end 4y ago
    Deprecating direct string indexing might have been the "right" move in some sense but it's hard to imagine it ever happening in Python; it's such a natural and frequently-used operation in the language that you would've broken an enormous amount of code. Because indexing strings is syntactically identical to indexing lists, and because there's no* real static typing, there'd be no* good and robust way to automate the conversion. Like, the changes Python 3 made were honestly pretty subtle, and nearly 15 years later there's still people reluctant to upgrade. If they broke string indexing I'm pretty sure Python 3 adoption would make Perl 6 uptake look impressive.
- mistrial9 4y ago
  > requires permanent marketing LOL this is great, so true
- froh 4y ago
  utf-8 needs 24bit per glyph for the logographic unicode scripts, while utf-16 needs 16 bits. So if you're into selling silicon or don't care about these languages, or you only care about heavily mixed content (like html), then yes, utf-8 would be The Right Way (TM).
- zokier 4y ago
  Utf8 would not help with the issue in the article in any way.
  - StefanKarpinski 4y ago
    It's not at all obvious how it helps, but it does. First, why is Python unable to represent invalid path names as strings? Because internally it converts strings from UTF-8, UTF-16, or any other encoding, to a fixed-with array of decoded Unicode code points. The width of integer used to represent code points is determined by the largest code point in the string: if the string is ASCII, it can use a byte (uint8) per character; if the string is non-ASCII but all BMP, then it can use a uint16 per character; otherwise it has to use uint32 per character. Why does Python do all this? So that you can have O(1) character indexing. If you gave up on that, you wouldn't need to convert the string at all, you could just leave it as (potentially invalid) UTF-8 data. Suppose you get an invalid path on UNIX where paths are UTF-8 by convention? What does Python do with this string? It can't convert it to an array of code points because invalid UTF-8 doesn't correspond to a code point (well, it can if it's just illegal, not malformed, but in general, we have to consider completely malformed strings that don't even follow the basic UTF-8 format). So Python is stuck: it can only replace the invalid data with something like the Unicode replacement character. But then you can't do anything useful with that because it's not the correct name of the path you're trying to work with. How does using UTF-8 to represent strings help? Because you can represent invalid strings: just leave them as-is and don't try to decode them unless you have to. Sure, you can't decode them as code points, but that's actually a pretty unusual thing to do. If someone asks for decoding, _then_ you can give an error. What about Windows where paths are UTF-16 by convention? You can convert them to WTF-8 and everything works out. (Described in way more detail here: https://news.ycombinator.com/item?id=33984308 https://news.ycombinator.com/item?id=33984308).
    - masklinn 4y ago
      > How does using UTF-8 to represent strings help? Because you can represent invalid strings: just leave them as-is and don't try to decode them unless you have to. That’s not UTF8. That’s a bag’o bytes which might be UTF8. Very different thing. > Sure, you can't decode them as code points, but that's actually a pretty unusual thing to do. It’s not, any unicode-aware text processing does it implicitly. This means any such processing has to either perform its own validation that the input is valid, or it may fly off the rails entirely if fed nonsense. This also increases risks if security issues, either outright UBs, or the ability to smuggle payloads through overlong encoding.
- jerf 4y ago
  Do you mean the right way would be for Python to use utf-8 for file names, or for the filesystem? I'm not sure how "utf-8" is the solution in this case for the problem. The problem here is that file names aren't any kind of Unicode. You can have a utf-8 filename right next to a utf-16 file name right next to a Shift-JIS filename right next to one that's just binary gibberish that isn't a legal string in any encoding (that has any restrictions at all). They're just bytes with no indication of what coding scheme they are at all. You can't solve this problem with any Unicode solution. (Even the one that tries to wrap bad bytes in a recoverable way, whose name escapes me and I can't recall if it's actually a standard or not, would still make me nervous. You really need code that just treats the filenames as opaque bytes, and only later at display time tries to figure out if they're UTF-8 or not.) It doesn't matter what Unicode model Python used. That may solve Python's other problems, sure. But filenames would always be some sort of problem. Even if you do just treat them as bytes, all that really does is make your potential errors deterministic, which is still better than a crash probably, but the filesystem's lack of encoding standards is a fundamental problem that can't be solved by higher code levels.
  - StefanKarpinski 4y ago
    On UNIX, paths are UTF-8 by convention, but not forced to be valid. Treating paths as UTF-8 works very well as long as you hadn't also make the mistake of requiring your UTF-8 strings to be valid (which Python did, unfortunately). On Windows, paths are UTF-16 by convention, but also not forced to be valid. However, invalid UTF-16 can be faithfully converted to WTF-8 and converted back losslessly, so you can translate Windows paths to WTF-16 and everything Just Works™ [1]. There aren't any operating systems I'm aware of where paths are actually Shift-JIS by convention, so that seems like a non-issue. Using "UTF-8 by convention" strings works on all modern OSes. [1] Ok, here's why the WTF-8 thing works so well. If we write WTF-16 for potentially invalid UTF-16 (just arbitrary sequences of 16-bit code units), then the mapping between WTF-16 and WTF-8 space is a bijection because it's losslessly round-trippable. But more importantly, this WTF-8/16 bijection is also a homomorphism with respect to pretty much any string operation you can think of. For example `utf16_concat(a, b) == utf8_concat(wtf8(a), wtf8(b))` for arbitrary UTF-16 strings a and b. Similar identities hold for other string operations like searching for substrings or splitting on specific strings.
mmastrac 4y ago
Rust uses a solution similar to his original proposal: OsString/OsStr: https://doc.rust-lang.org/std/ffi/struct.OsString.html https://doc.rust-lang.org/std/ffi/struct.OsString.html I wonder if unicode needs a "undecodable byte escape" sequence for these raw sequences that cannot be mapped into the string space, allowing for a guaranteed round-trip between the OS and Unicode. For this to work, you'd need a guarantee that all codepage/unicode mappings are consistent and 1:1 which might not be possible. Paths are tricky because they act as both an identifier and a handle, and the string encoding of the identifier and handle are _close but not exactly the same_ when unicode is in the mix. In the ASCII world, you can assume that this is the case.
- LegionMammal978 4y ago
  Python already has the "surrogateescape" error handler [0] that performs something similar to what you described: undecodable bytes are translated into unpaired U+DC80 to U+DCFF surrogates. Of course, this isn't standardized in any way, but I've found it useful myself for smuggling raw pathnames through Java. [0] https://peps.python.org/pep-0383/ https://peps.python.org/pep-0383/
  - mmastrac 4y ago
    Thanks for this reference! It's exactly what I was thinking.
- pvillano 4y ago
  wrt "undecidable byte escape" I think that wouldn't solve the problems with filenames. Files would have to enter Python land escaped which would mean my file really named \x00 would become \\x00 You can't escape only the broken filenames, because a valid unicode filename could contain an escape sequence, which would incorrectly be unescaped when returning to the OS I think you really do need a non-str type that can be silently converted to str 99% of the time
cabirum 4y ago
Filenames come from multiple filesystems (and not just filesystems!), with different encodings and such. Both ntfs and ext3 can use arbitrary bytes as names, not representable in utf8. (Contrary to popular belief, ntfs works with any byte sequences except 0x0000, and may contain invalid utf16 code points) The only sane way to deal with them all is to use bytes and bytes only (Or use a sane language with encoding-agnostic string type; see: golang) Other solutions introduce loss of data.
- chungy 4y ago
  NTFS disallows 0x002F as well; it is possible to use the entire set even from Windows (installing Interix, for instance). Windows NT was designed around UCS-2, the file system didn't have to do much but be a dumb bag of bytes for file names, character interpretation was on upper layers. You can mostly pretend it's UTF-16, until it's not. Something fun on file systems: ZFS lets you set "utf8only=on" which will guarantee no file names will ever be possible that are not valid UTF-8. I use it on all my systems.
  - jcranmer 4y ago
    > Something fun on file systems: ZFS lets you set "utf8only=on" which will guarantee no file names will ever be possible that are not valid UTF-8. I use it on all my systems. This is the sort of option that should be available for all filesystems, and distros should default new installs to turning it on.
    - deckard1 4y ago
      if we have a time machine then yes. In fact, it shouldn't even be an option. UTF-8 period. Always. But alas, we do not and someone will plug in a memory card or USB drive with some FAT formatted drive and your nice universe crumbles before you.
      - chungy 4y ago
        This is why "convmv" exists :) and also why my /tmp is a normal tmpfs where file names can be any byte sequence. I can, for instance, extract random archives there, use convmv to make them UTF-8 (if not already), and then it can go on my ZFS file systems.
PaulHoule 4y ago
One time my job was standardizing a machine learning model training system in Python so you could develop models on a Mac or PC and then train them with large data sets on a DGX-1 for production. Over the course of a few months I developed quite a list of configuration problems that could break Python scripts and some of the worse involved character encoding. In particular, in many Pythons, writing invalid Unicode in a print statement could crash your Python. It's one thing to say "don't write invalid Unicode in a print" but if you are sucking in a large number of 3rd party libraries you can't control their use of print. It turned out many customers had CSV files with invalid Unicode sequences so we couldn't control the input. People would say "just use docker" but the team was always finding and making docker images with strangely configured Pythons. (If anything Docker seemed to make it easier and faster to use and install misconfigured software, not make it easier to get things under control.) It's good news that Python, like Java, is defaulting to UTF-8 instead of whatever locale it gets from the OS because frequently this locale is not just "not UTF-8" but something really bizarre.
- nitwit005 4y ago
  > If anything Docker seemed to make it easier and faster to use and install misconfigured software, not make it easier to get things under control. Hadn't thought about it that way, but I have seen the same thing.
  - PaulHoule 4y ago
    My take is that you can control your environment with Docker and discipline but discipline alone gets you far. Circa 2004 I was maintaining about 80 web sites for a large academic library and not much late 200 web sites for a prolific web consulting firm, plus at least 10 sites on my own account. I came to the conclusion that you could host as many sites with the same software that you like one one machine so long as you can configure: (i) where files are in the filesystem, (ii) how to connect to the relational database, (iii) cryptographic keys. If the server was Apache there was always an Apache configuration file, there would usually be one for the actual application. In a complicated case there would be some templating system that writes all the configuration files. With that kind of system I could frequently get a site up 'n running in a few minutes and there was no difficultly with dev, test, and prod instances or "clon e customer A's site to make a site for customer B". Today it is not that different though the server might be nginx and a system today might have a non-relational database and might have more than one of them and there are probably some API keys. When docker came along me immediate response was "ho hum" and when I tried it first I was stuck behind a slow internet connection docker seemed to be a way to download 3GB of files to make a 3 kilobyte change. I've been at an office with a gigabit connection and it is not so bad but still not so good: it seems fast from an "ops" perspective but from a "dev" perspective it is slow.
sakras 4y ago
I’m a little confused, how can a file name be non-decodable? A file with that name exists, so someone somewhere knows how to decode it. Why wouldn’t Python just always use the same encoding as the OS it’s running on? Is this some locale-related thing?
- CorrectHorseBat 4y ago
  Unix allows any byte in a filename except 0x0 (ASCII Nul) and 0x2f (ASCII '/'), anything else is allowed. It doesn't have to be decodable to text.
- jjtheblunt 4y ago
  Does trying to decode, on Unix, a Windows file path (not filename) like "drive:blah" freak it out as an example of non-decodable?
  - johannes1234321 4y ago
    That's a valid file name. Non-decodable are byte sequences which are invalid UTF-8 byte sequences (if UTF-8 is expected)
- StefanKarpinski 4y ago
  Unfortunately, neither UNIX nor Windows require path names to be valid Unicode. UNIX interprets them as “UTF-8 by convention” and Windows as “UTF-16 by convention” but both actually allow arbitrary sequences of code units. It would be nice if this didn’t actually occur, but alas, it does, and if you’re writing general purpose utilities that work with files, you don’t want them to simply crash when this happens.
- heisenzombie 4y ago
  It's filesystem-dependent, but a lot of filesystems treat filenames as just an arbitrary sequence of bytes. Most OSes sort of hide this from you, but there's nothing stopping you doing weird stuff like: > touch `echo 00: DEADBEEF | xxd -r` > ls 'ޭ'$'\276\357'
- 4y ago
zzzeek 4y ago
SQLAlchemy creator here. No single improvement in Python has made my life more dramatically easier than the introduction and standardization of strings as Unicode. The basic reason is because it clearly placed the burden of encoding / decoding on the *outside* edges of a program; right when data is coming in, or right when data is going out. If you want the world to consider the data between those to be a "a string", you have to do that for us. For me, it meant that every PEP-249 DBAPI finally took on the role of handling database encodings fully and completely. Enormous reams of arbitrary string encode/decode logic in SQLAlchemy, having to guess its way around all the different decisions every DBAPI made about this, gone. If I get a string from the DBAPI, it's decoded. If I have a string, I can send it out, DBAPI handles it. If it contains characters that aren't appropriate for the database's encoding, it raises. Great! I send the users off to the manual for their database, they aren't complaining to me. Total lifesaver. I see a lot of odd complaints here which seem to be people wanting non-textual bytestrings to do things that make sense for text; if someone has a use case like that, there can be libraries that do the particular non-textual-but-text-like manipulations they need, however, these are not the "standard" uses for textual strings in Python. The standard use for textual strings is...to represent text! If I have data that's bytes, then I use bytes! These threads always seem to fill up with the vanishingly small number of people that have issues with this. Just like the anti-ORM threads. All the while the vast majority of devs are just getting work done.
- Alir3z4 4y ago
  Unicode errors were one the most painful and annoying issues I had. Regardless of working with xml files, parsing text or just dealing with anything text related. When upgrading to Python 3, they just vanished. Anti ORM people probably don't know what ORM is or confused about their usage and benefit they bring. Once you're dealing with several tables and relations and lots of aggregation and annotations, raw sql is the one way ticket to insanity. By the way, thank you very much for making SQLAlchemy. You saved millions of hours of developer time and god knows how many SQL bugs have been prevented by using SQLAlchemy
  - kstrauser 4y ago

77 points by goranmoomin 4y ago

The Unicode model is complex and overengineered. Like many Python features, it requires permanent marketing until people believe it is good. It provides a steady income stream for those paid to "work" on Python. The right way would have been utf-8, which is the only reasonable representation. Indexing could have been broken early in Python 3 or outsourced to a special type.

StefanKarpinski 4y ago
Absolutely right. Deprecating direct string indexing would have been the right move. Require writing `str.chars()` to get something that lets you slice by Unicode characters (i.e. code points); provide `str.graphemes()` and `str.grapheme_clusters()` to get something that lets you slice by graphemes and grapheme clusters, respectively. Cache an index structure that lets you do that kind of indexing efficiently once you've asked for it the first time. Provide an API to clear the caches. Not allowing strings to represent invalid Unicode is also a huge mistake (and essentially forced by the representation strategy that they adopted). It forces any programmer who wants to robustly handle potentially invalid string data to use byte vectors instead. Which is exactly what they did with OS paths, but that's far from the only place you can get invalid strings. You can get invalid strings almost anywhere! Worse, since it's incredibly inconvenient to work with byte vectors when you want to do stringlike stuff, no one does it unless forced to, so this design choice effectively guarantees that all Python code that works with strings will blow up if it encounters anything invalid—which is a very common occurrence. If only there was a type that behaves like a string and supports all the handy string operations but which handles invalid data gracefully. Then you could write robust string code conveniently. But at that point, you should just make that the standard string type! This isn't hypothetical, it's exactly how Burnt Sushi's bstr type [1] works in Rust and how the standard String type works in Julia. [1] https://github.com/BurntSushi/bstr https://github.com/BurntSushi/bstr
- Jasper_ 4y ago
  It's worth noting that Python str's are sequences of code points, not scalar values. This was a truly horrendous mistake made mostly out of ignorance, but now they rely upon it in surrogateescape to hide "invalid" data, so... I have ranted for long hours go friends about the insanity of Python 3's text model before. It's mostly the blind leading the blind.
  - 323 4y ago
    I've seen this happen multiple times in Python, where they implement a feature without a thorough survey of what other languages are doing. Then when it turns out the new feature has a major broken part, they fix that, but again, without looking how it's done in other places. asyncio is a major example.
- Spivak 4y ago
  > It forces any programmer who wants to robustly handle potentially invalid string data to use byte vectors instead. Is this not the only thing you can really do? If strings could hold invalid unicode then they effectively become bytes and you now have to be wary of every possible string. I would rather Python just do away with strings entirely in the integration points with the OS and make you either keep them as opaque bytes or decode them. > behaves like a string and supports all the handy string operations but which handles invalid data gracefully Unless Guido was feeling particularly practical I can't imagine this ever making it into the stdlib because the choice of "gracefully" is somewhat arbitrary and application dependent.
- wk_end 4y ago
  Deprecating direct string indexing might have been the "right" move in some sense but it's hard to imagine it ever happening in Python; it's such a natural and frequently-used operation in the language that you would've broken an enormous amount of code. Because indexing strings is syntactically identical to indexing lists, and because there's no* real static typing, there'd be no* good and robust way to automate the conversion. Like, the changes Python 3 made were honestly pretty subtle, and nearly 15 years later there's still people reluctant to upgrade. If they broke string indexing I'm pretty sure Python 3 adoption would make Perl 6 uptake look impressive.
mistrial9 4y ago
> requires permanent marketing LOL this is great, so true
froh 4y ago
utf-8 needs 24bit per glyph for the logographic unicode scripts, while utf-16 needs 16 bits. So if you're into selling silicon or don't care about these languages, or you only care about heavily mixed content (like html), then yes, utf-8 would be The Right Way (TM).
zokier 4y ago
Utf8 would not help with the issue in the article in any way.
- StefanKarpinski 4y ago
  It's not at all obvious how it helps, but it does. First, why is Python unable to represent invalid path names as strings? Because internally it converts strings from UTF-8, UTF-16, or any other encoding, to a fixed-with array of decoded Unicode code points. The width of integer used to represent code points is determined by the largest code point in the string: if the string is ASCII, it can use a byte (uint8) per character; if the string is non-ASCII but all BMP, then it can use a uint16 per character; otherwise it has to use uint32 per character. Why does Python do all this? So that you can have O(1) character indexing. If you gave up on that, you wouldn't need to convert the string at all, you could just leave it as (potentially invalid) UTF-8 data. Suppose you get an invalid path on UNIX where paths are UTF-8 by convention? What does Python do with this string? It can't convert it to an array of code points because invalid UTF-8 doesn't correspond to a code point (well, it can if it's just illegal, not malformed, but in general, we have to consider completely malformed strings that don't even follow the basic UTF-8 format). So Python is stuck: it can only replace the invalid data with something like the Unicode replacement character. But then you can't do anything useful with that because it's not the correct name of the path you're trying to work with. How does using UTF-8 to represent strings help? Because you can represent invalid strings: just leave them as-is and don't try to decode them unless you have to. Sure, you can't decode them as code points, but that's actually a pretty unusual thing to do. If someone asks for decoding, _then_ you can give an error. What about Windows where paths are UTF-16 by convention? You can convert them to WTF-8 and everything works out. (Described in way more detail here: https://news.ycombinator.com/item?id=33984308 https://news.ycombinator.com/item?id=33984308).
  - masklinn 4y ago
    > How does using UTF-8 to represent strings help? Because you can represent invalid strings: just leave them as-is and don't try to decode them unless you have to. That’s not UTF8. That’s a bag’o bytes which might be UTF8. Very different thing. > Sure, you can't decode them as code points, but that's actually a pretty unusual thing to do. It’s not, any unicode-aware text processing does it implicitly. This means any such processing has to either perform its own validation that the input is valid, or it may fly off the rails entirely if fed nonsense. This also increases risks if security issues, either outright UBs, or the ability to smuggle payloads through overlong encoding.
jerf 4y ago
Do you mean the right way would be for Python to use utf-8 for file names, or for the filesystem? I'm not sure how "utf-8" is the solution in this case for the problem. The problem here is that file names aren't any kind of Unicode. You can have a utf-8 filename right next to a utf-16 file name right next to a Shift-JIS filename right next to one that's just binary gibberish that isn't a legal string in any encoding (that has any restrictions at all). They're just bytes with no indication of what coding scheme they are at all. You can't solve this problem with any Unicode solution. (Even the one that tries to wrap bad bytes in a recoverable way, whose name escapes me and I can't recall if it's actually a standard or not, would still make me nervous. You really need code that just treats the filenames as opaque bytes, and only later at display time tries to figure out if they're UTF-8 or not.) It doesn't matter what Unicode model Python used. That may solve Python's other problems, sure. But filenames would always be some sort of problem. Even if you do just treat them as bytes, all that really does is make your potential errors deterministic, which is still better than a crash probably, but the filesystem's lack of encoding standards is a fundamental problem that can't be solved by higher code levels.
- StefanKarpinski 4y ago
  On UNIX, paths are UTF-8 by convention, but not forced to be valid. Treating paths as UTF-8 works very well as long as you hadn't also make the mistake of requiring your UTF-8 strings to be valid (which Python did, unfortunately). On Windows, paths are UTF-16 by convention, but also not forced to be valid. However, invalid UTF-16 can be faithfully converted to WTF-8 and converted back losslessly, so you can translate Windows paths to WTF-16 and everything Just Works™ [1]. There aren't any operating systems I'm aware of where paths are actually Shift-JIS by convention, so that seems like a non-issue. Using "UTF-8 by convention" strings works on all modern OSes. [1] Ok, here's why the WTF-8 thing works so well. If we write WTF-16 for potentially invalid UTF-16 (just arbitrary sequences of 16-bit code units), then the mapping between WTF-16 and WTF-8 space is a bijection because it's losslessly round-trippable. But more importantly, this WTF-8/16 bijection is also a homomorphism with respect to pretty much any string operation you can think of. For example `utf16_concat(a, b) == utf8_concat(wtf8(a), wtf8(b))` for arbitrary UTF-16 strings a and b. Similar identities hold for other string operations like searching for substrings or splitting on specific strings.

mmastrac 4y ago

Rust uses a solution similar to his original proposal: OsString/OsStr: https://doc.rust-lang.org/std/ffi/struct.OsString.html https://doc.rust-lang.org/std/ffi/struct.OsString.html I wonder if unicode needs a "undecodable byte escape" sequence for these raw sequences that cannot be mapped into the string space, allowing for a guaranteed round-trip between the OS and Unicode. For this to work, you'd need a guarantee that all codepage/unicode mappings are consistent and 1:1 which might not be possible. Paths are tricky because they act as both an identifier and a handle, and the string encoding of the identifier and handle are _close but not exactly the same_ when unicode is in the mix. In the ASCII world, you can assume that this is the case.

LegionMammal978 4y ago
Python already has the "surrogateescape" error handler [0] that performs something similar to what you described: undecodable bytes are translated into unpaired U+DC80 to U+DCFF surrogates. Of course, this isn't standardized in any way, but I've found it useful myself for smuggling raw pathnames through Java. [0] https://peps.python.org/pep-0383/ https://peps.python.org/pep-0383/
- mmastrac 4y ago
  Thanks for this reference! It's exactly what I was thinking.
pvillano 4y ago
wrt "undecidable byte escape" I think that wouldn't solve the problems with filenames. Files would have to enter Python land escaped which would mean my file really named \x00 would become \\x00 You can't escape only the broken filenames, because a valid unicode filename could contain an escape sequence, which would incorrectly be unescaped when returning to the OS I think you really do need a non-str type that can be silently converted to str 99% of the time

cabirum 4y ago

Filenames come from multiple filesystems (and not just filesystems!), with different encodings and such. Both ntfs and ext3 can use arbitrary bytes as names, not representable in utf8. (Contrary to popular belief, ntfs works with any byte sequences except 0x0000, and may contain invalid utf16 code points) The only sane way to deal with them all is to use bytes and bytes only (Or use a sane language with encoding-agnostic string type; see: golang) Other solutions introduce loss of data.

chungy 4y ago
NTFS disallows 0x002F as well; it is possible to use the entire set even from Windows (installing Interix, for instance). Windows NT was designed around UCS-2, the file system didn't have to do much but be a dumb bag of bytes for file names, character interpretation was on upper layers. You can mostly pretend it's UTF-16, until it's not. Something fun on file systems: ZFS lets you set "utf8only=on" which will guarantee no file names will ever be possible that are not valid UTF-8. I use it on all my systems.
- jcranmer 4y ago
  > Something fun on file systems: ZFS lets you set "utf8only=on" which will guarantee no file names will ever be possible that are not valid UTF-8. I use it on all my systems. This is the sort of option that should be available for all filesystems, and distros should default new installs to turning it on.
  - deckard1 4y ago
    if we have a time machine then yes. In fact, it shouldn't even be an option. UTF-8 period. Always. But alas, we do not and someone will plug in a memory card or USB drive with some FAT formatted drive and your nice universe crumbles before you.
    - chungy 4y ago
      This is why "convmv" exists :) and also why my /tmp is a normal tmpfs where file names can be any byte sequence. I can, for instance, extract random archives there, use convmv to make them UTF-8 (if not already), and then it can go on my ZFS file systems.

PaulHoule 4y ago

One time my job was standardizing a machine learning model training system in Python so you could develop models on a Mac or PC and then train them with large data sets on a DGX-1 for production. Over the course of a few months I developed quite a list of configuration problems that could break Python scripts and some of the worse involved character encoding. In particular, in many Pythons, writing invalid Unicode in a print statement could crash your Python. It's one thing to say "don't write invalid Unicode in a print" but if you are sucking in a large number of 3rd party libraries you can't control their use of print. It turned out many customers had CSV files with invalid Unicode sequences so we couldn't control the input. People would say "just use docker" but the team was always finding and making docker images with strangely configured Pythons. (If anything Docker seemed to make it easier and faster to use and install misconfigured software, not make it easier to get things under control.) It's good news that Python, like Java, is defaulting to UTF-8 instead of whatever locale it gets from the OS because frequently this locale is not just "not UTF-8" but something really bizarre.

nitwit005 4y ago
> If anything Docker seemed to make it easier and faster to use and install misconfigured software, not make it easier to get things under control. Hadn't thought about it that way, but I have seen the same thing.
- PaulHoule 4y ago
  My take is that you can control your environment with Docker and discipline but discipline alone gets you far. Circa 2004 I was maintaining about 80 web sites for a large academic library and not much late 200 web sites for a prolific web consulting firm, plus at least 10 sites on my own account. I came to the conclusion that you could host as many sites with the same software that you like one one machine so long as you can configure: (i) where files are in the filesystem, (ii) how to connect to the relational database, (iii) cryptographic keys. If the server was Apache there was always an Apache configuration file, there would usually be one for the actual application. In a complicated case there would be some templating system that writes all the configuration files. With that kind of system I could frequently get a site up 'n running in a few minutes and there was no difficultly with dev, test, and prod instances or "clon e customer A's site to make a site for customer B". Today it is not that different though the server might be nginx and a system today might have a non-relational database and might have more than one of them and there are probably some API keys. When docker came along me immediate response was "ho hum" and when I tried it first I was stuck behind a slow internet connection docker seemed to be a way to download 3GB of files to make a 3 kilobyte change. I've been at an office with a gigabit connection and it is not so bad but still not so good: it seems fast from an "ops" perspective but from a "dev" perspective it is slow.

sakras 4y ago

I’m a little confused, how can a file name be non-decodable? A file with that name exists, so someone somewhere knows how to decode it. Why wouldn’t Python just always use the same encoding as the OS it’s running on? Is this some locale-related thing?

CorrectHorseBat 4y ago
Unix allows any byte in a filename except 0x0 (ASCII Nul) and 0x2f (ASCII '/'), anything else is allowed. It doesn't have to be decodable to text.
jjtheblunt 4y ago
Does trying to decode, on Unix, a Windows file path (not filename) like "drive:blah" freak it out as an example of non-decodable?
- johannes1234321 4y ago
  That's a valid file name. Non-decodable are byte sequences which are invalid UTF-8 byte sequences (if UTF-8 is expected)
StefanKarpinski 4y ago
Unfortunately, neither UNIX nor Windows require path names to be valid Unicode. UNIX interprets them as “UTF-8 by convention” and Windows as “UTF-16 by convention” but both actually allow arbitrary sequences of code units. It would be nice if this didn’t actually occur, but alas, it does, and if you’re writing general purpose utilities that work with files, you don’t want them to simply crash when this happens.
heisenzombie 4y ago
It's filesystem-dependent, but a lot of filesystems treat filenames as just an arbitrary sequence of bytes. Most OSes sort of hide this from you, but there's nothing stopping you doing weird stuff like: > touch `echo 00: DEADBEEF | xxd -r` > ls 'ޭ'$'\276\357'
4y ago

zzzeek 4y ago

SQLAlchemy creator here. No single improvement in Python has made my life more dramatically easier than the introduction and standardization of strings as Unicode. The basic reason is because it clearly placed the burden of encoding / decoding on the *outside* edges of a program; right when data is coming in, or right when data is going out. If you want the world to consider the data between those to be a "a string", you have to do that for us. For me, it meant that every PEP-249 DBAPI finally took on the role of handling database encodings fully and completely. Enormous reams of arbitrary string encode/decode logic in SQLAlchemy, having to guess its way around all the different decisions every DBAPI made about this, gone. If I get a string from the DBAPI, it's decoded. If I have a string, I can send it out, DBAPI handles it. If it contains characters that aren't appropriate for the database's encoding, it raises. Great! I send the users off to the manual for their database, they aren't complaining to me. Total lifesaver. I see a lot of odd complaints here which seem to be people wanting non-textual bytestrings to do things that make sense for text; if someone has a use case like that, there can be libraries that do the particular non-textual-but-text-like manipulations they need, however, these are not the "standard" uses for textual strings in Python. The standard use for textual strings is...to represent text! If I have data that's bytes, then I use bytes! These threads always seem to fill up with the vanishingly small number of people that have issues with this. Just like the anti-ORM threads. All the while the vast majority of devs are just getting work done.

Alir3z4 4y ago
Unicode errors were one the most painful and annoying issues I had. Regardless of working with xml files, parsing text or just dealing with anything text related. When upgrading to Python 3, they just vanished. Anti ORM people probably don't know what ORM is or confused about their usage and benefit they bring. Once you're dealing with several tables and relations and lots of aggregation and annotations, raw sql is the one way ticket to insanity. By the way, thank you very much for making SQLAlchemy. You saved millions of hours of developer time and god knows how many SQL bugs have been prevented by using SQLAlchemy
- kstrauser 4y ago