2 ms·

So, my favorite thought process from the Python 2 to 3 transition comes from staring deeply into the hubris of the Python 3000 effort itself as wanting to insis

by saurik 6y ago

So, my favorite thought process from the Python 2 to 3 transition comes from staring deeply into the hubris of the Python 3000 effort itself as wanting to insist "everything is Unicode"; on the one hand they were trying to shame developers using Python 2 for using it incorrectly (even though it was totally possible to handle Unicode correctly with that stack), yet on the other they were pumping out ridiculous unavoidable security SNAFUs in their API surface by assuming that filenames on disk are UTF-8 :/. To tell another story of the importance of reading the--in this case, not even long!--specification: on Windows, filenames are stored as UCS-2 (as it all predates UTF-16), and is most likely most correctly modeled as an arbitrary sequence of non-0 16-bit values; meanwhile, on Unix systems, a filename is pretty much just an arbitrary sequence of non-0 8-bit values (with some weirdness around the slash character... the API surface doesn't support it really but sometimes the filesystem does). The premise that filenames have a display encoding mostly comes from the software rendering the strings and is then defined in terms of a bunch of hacks involving "locales", and isn't a property of the underlying API or storage semantics. You can't assume in either case that they are canonical Unicode representations, and so storing them as "strings" in your program is super wrong and can lead to anything from data storage mistakes to security vulnerabilities. The Python 3 API filesystem API was thereby so bad that it would do things like just skip over files during directory traversal that didn't map to its attempt to decode stuff as Unicode. Like: if you wanted to hide a file from a Python 3 program, you could just make it invalid Unicode :/. (I realized people reading this comment might find a reference on this fun, and so found this seemingly-comprehensive postmortem. I haven't read it, but it at least looks really good ;P.) https://vstinner.github.io/python30-listdir-undecodable-filenames.html https://vstinner.github.io/python30-listdir-undecodable-file... I thereby find the whole Python 3 debacle extra demoralizing as it felt like a bunch of people who didn't truly understand how hard the problem they were dealing with was--and how much of the world actually is binary, and how often (when dealing with things like filesystems and network protocols) parsing binary sequences as pseudo-strings is "correct"--both sometimes-correctly insisting everyone was wrong while themselves failing to build something even minimally useful until like, Python 3.4 or so; and, in the end, I feel like they just ended up creating a "new mess" by having filenames now encoded as "Unicode strings"... but with weird exceptions for allowing round-trip behaviors of strings that couldn't be decoded :(. The reality is that there are a number of things that developers work with which are pseudo-strings, and languages should provide good tooling for working with them. We see the same issue come up with HTTP headers and filenames: things that are actually binary but have string-like structure. There is a reason why both C++ and Python 2 are often looked at fondly by low-level developers for providing low-level string representations, even while they end up encouraging broken behavior higher up the stack. I haven't looked enough at Rust's standard library to see if they truly nailed this yet, but maybe one day we will have a language that encourages the right behavior for these mechanisms that are neither strictly string not strictly binary.

pansa2 6y ago
> Python 3000 effort itself as wanting to insist "everything is Unicode" How does Python 3 handle these HTTP headers and other things that are purely ASCII? Does it treat them as type `bytes` instead of (Unicode) `str`?
- saurik 6y ago
  I just did some digging, and the Python 3 built-in HTTP stuff seems to represent HTTP headers using a reasonably-comprehensive mechanism that is shared with MIME parsing and the various IMF semantics, and so I am pretty sure you end up getting back an object of type email.header.Header (but I might have misunderstood the documentation just now... I mostly gave up on Python when they moved to Python 3 so I haven't experienced this first hand). https://docs.python.org/3/library/email.header.html#email.header.Header https://docs.python.org/3/library/email.header.html#email.he... This actually looks like a really solid implementation--like, I am super impressed by this (though watch it suck... everything sucks ;P)--as it doesn't assume any ability to flatten the result! That said, I am also deeply concerned that this might pre-decode something that should actually be in ASCII all the way down from MIME (which, notably, is complex enough of a format to include support for embedded comments). I kind of want to do some auditing here, to make sure there aren't some parser discrepancy vulnerabilities in common Python HTTP frameworks ;P. But like, I think this object represents a sequence of strings each with its own encoding, allowing it to "Q" encode the result exactly as specified (and correctly round trip something it parsed, which still does feel a tad dangerous, and maybe the API lets you avoid it in those cases). It then has an __str__ implementation (to let you use it as a string) that explicitly claims to return an "approximation" of the header (which may or may not be a safe thing to use in different circumstances; but like, that in some sense isn't the API's fault?...).
Measter 6y ago
Rust is a bit notorious for its string-related complexity, and that's for the reasons you talked about. The most common form of string you'll see is `String`[0], or its view type `str`[1], which are UTF-8 encoded. These have most of the string functions you'd expect to have. Of course, as you mentioned, operating systems don't all use UTF-8, which brings us to `OsString`[2], and its view type `OsStr`[3]. This is the data type that string data from the OS is represented as. Both are basically bundles of bytes, with no specific encoding. You'll note that these are in the FFI namespace. You then have `PathBuf`[4] and its view type `Path`[5], which are wrappers around `OsString` and `OsStr` respectively, and provide path-specific functionality. All the file system related APIs take anything that implements `AsRef<Path>`, which is runtime-cost-free for all of these types. There's not many functions that take an `OsStr`, but those that do take it in the form `AsRef<OsStr>`, which like `Path` is runtime-cost-free for all these types. None of the APIs take `PathBuf` or `OsString`. Return values are typically `PathBuf` for file system APIs, and `OsString` for other APIs. The only exceptions I know of are the functions for getting environment variables and command line arguments, which also provide functions that return `String` because a common use case would be converting them anyway. [0] https://doc.rust-lang.org/std/string/struct.String.html https://doc.rust-lang.org/std/string/struct.String.html [1] https://doc.rust-lang.org/std/primitive.str.html https://doc.rust-lang.org/std/primitive.str.html [2] https://doc.rust-lang.org/std/ffi/struct.OsString.html https://doc.rust-lang.org/std/ffi/struct.OsString.html [3] https://doc.rust-lang.org/std/ffi/struct.OsStr.html https://doc.rust-lang.org/std/ffi/struct.OsStr.html [4] https://doc.rust-lang.org/std/path/struct.PathBuf.html https://doc.rust-lang.org/std/path/struct.PathBuf.html [5] https://doc.rust-lang.org/std/path/struct.Path.html https://doc.rust-lang.org/std/path/struct.Path.html

by saurik 6y ago

pansa2 6y ago

> Python 3000 effort itself as wanting to insist "everything is Unicode" How does Python 3 handle these HTTP headers and other things that are purely ASCII? Does it treat them as type `bytes` instead of (Unicode) `str`?

saurik 6y ago
I just did some digging, and the Python 3 built-in HTTP stuff seems to represent HTTP headers using a reasonably-comprehensive mechanism that is shared with MIME parsing and the various IMF semantics, and so I am pretty sure you end up getting back an object of type email.header.Header (but I might have misunderstood the documentation just now... I mostly gave up on Python when they moved to Python 3 so I haven't experienced this first hand). https://docs.python.org/3/library/email.header.html#email.header.Header https://docs.python.org/3/library/email.header.html#email.he... This actually looks like a really solid implementation--like, I am super impressed by this (though watch it suck... everything sucks ;P)--as it doesn't assume any ability to flatten the result! That said, I am also deeply concerned that this might pre-decode something that should actually be in ASCII all the way down from MIME (which, notably, is complex enough of a format to include support for embedded comments). I kind of want to do some auditing here, to make sure there aren't some parser discrepancy vulnerabilities in common Python HTTP frameworks ;P. But like, I think this object represents a sequence of strings each with its own encoding, allowing it to "Q" encode the result exactly as specified (and correctly round trip something it parsed, which still does feel a tad dangerous, and maybe the API lets you avoid it in those cases). It then has an __str__ implementation (to let you use it as a string) that explicitly claims to return an "approximation" of the header (which may or may not be a safe thing to use in different circumstances; but like, that in some sense isn't the API's fault?...).

Measter 6y ago

Rust is a bit notorious for its string-related complexity, and that's for the reasons you talked about. The most common form of string you'll see is `String`[0], or its view type `str`[1], which are UTF-8 encoded. These have most of the string functions you'd expect to have. Of course, as you mentioned, operating systems don't all use UTF-8, which brings us to `OsString`[2], and its view type `OsStr`[3]. This is the data type that string data from the OS is represented as. Both are basically bundles of bytes, with no specific encoding. You'll note that these are in the FFI namespace. You then have `PathBuf`[4] and its view type `Path`[5], which are wrappers around `OsString` and `OsStr` respectively, and provide path-specific functionality. All the file system related APIs take anything that implements `AsRef<Path>`, which is runtime-cost-free for all of these types. There's not many functions that take an `OsStr`, but those that do take it in the form `AsRef<OsStr>`, which like `Path` is runtime-cost-free for all these types. None of the APIs take `PathBuf` or `OsString`. Return values are typically `PathBuf` for file system APIs, and `OsString` for other APIs. The only exceptions I know of are the functions for getting environment variables and command line arguments, which also provide functions that return `String` because a common use case would be converting them anyway. [0] https://doc.rust-lang.org/std/string/struct.String.html https://doc.rust-lang.org/std/string/struct.String.html [1] https://doc.rust-lang.org/std/primitive.str.html https://doc.rust-lang.org/std/primitive.str.html [2] https://doc.rust-lang.org/std/ffi/struct.OsString.html https://doc.rust-lang.org/std/ffi/struct.OsString.html [3] https://doc.rust-lang.org/std/ffi/struct.OsStr.html https://doc.rust-lang.org/std/ffi/struct.OsStr.html [4] https://doc.rust-lang.org/std/path/struct.PathBuf.html https://doc.rust-lang.org/std/path/struct.PathBuf.html [5] https://doc.rust-lang.org/std/path/struct.Path.html https://doc.rust-lang.org/std/path/struct.Path.html