4 ms·

Every time someone creates a new programming language, the world builds yet another pile of broken HTTP implementations--clients and servers--because everyone r

by saurik 6y ago

Every time someone creates a new programming language, the world builds yet another pile of broken HTTP implementations--clients and servers--because everyone refuses to read the specifications, thinking "none of this is hard". I remember dealing with some early Ruby web servers, 15 years ago, and was just shocked at how broken they were despite being used by everyone in production! In fact, there is a lot of nuance to HTTP, and as a civilization we seemed doomed to reliving the same mistakes over and over again every few years when the entire stack gets rewritten from scratch :/. So, I thereby was neither surprised to read that the Go HTTP server did this obviously-wrong thing... NOR was I particularly surprised to read the person complaining about it also didn't seem to understand the correct behavior, either (though I do appreciate this is a bit "unfair"--I mean, it's a tweet ;P--but I think still makes for an interesting point here) :/. I have dealt with so many broken HTTP clients and servers over the years written by people who probably also never read the manual that came with their microwave because "documentation is for idiots" or whatever, that I have just become demoralized about it :(. A few months ago, I was complaining about this, with a short list of stuff everyone tended to get wrong, and this HTTP header field encoding issue was on my shortlist :(. https://news.ycombinator.com/item?id=24835617 https://news.ycombinator.com/item?id=24835617 > FWIW, I have encountered tons of issues in peoples' "this is easy, right?" implementations of 100 Continue (including in some at-the-time major Ruby HTTP servers), non-ASCII encoding in HTTP headers (which has a ton of subtle rules, and just throwing around UTF-8 is not correct), and chunked Transfer-Encoding (though I can't remember what was wrong); further, people often don't even realize HTTP trailers exist at all, which is a bit infuriating :/. (Note, though, that I am still 100% on the side of the anti-HTTP/2 camp in this thread.) The core problem is that somehow people think they can implement a spec without even, you know, reading the spec, which is so far past hubris as to have gone through incompetence into obvious negligence. AFAIK, HTTP header fields are (in their generality) specified to be read as binary octets... but then are restricted to using ISO-8859-1 with the premise you are supposed to run them through a MIME parser? Like, some fields have strict formats, so you can't do the full decoding until you know what you are working with--and so I would argue an HTTP library should give you the raw binary data (though decoding them through ISO-8859-1 first isn't supposed to be "wrong")--but for fields with user-displayable text (including HTTP status messages!) you can seriously embed "Q" encoded strings (of all things) if you want, a la RFC 2047. You can't assume a field is restricted to ASCII too early in the parsing process. https://tools.ietf.org/html/rfc2047 https://tools.ietf.org/html/rfc2047

baq 6y ago
reading this i'm getting flashbacks from python 2 -> 3 migration and all those people complaining that they're now getting Unicode*Errors thrown at them. the fact is, none of this stuff is simple! you can pretend it is until a corner case bites you in the behind, suddenly the tower of abstraction falls apart and you're left picking up the parts in four different string encoding formats. (see the abysmal C which at least doesn't pretend anything at all, C++ which doesn't require you to know all this stuff and then it just blows up if you make a particularly wrong misstep and most recently Rust, which will call you names for using the wrong type in the wrong place, in turn greatly confusing people who say strings are easy).
- wizzwizz4 6y ago
  In this case, Python and Rust's behaviours are correct. You only get type errors / Unicode* exceptions when you're making wrong assumptions; if you get them, and don't understand why, you're writing buggy code to do things you don't understand and should be thankful for the errors letting you know there's a problem.
- saurik 6y ago
  So, my favorite thought process from the Python 2 to 3 transition comes from staring deeply into the hubris of the Python 3000 effort itself as wanting to insist "everything is Unicode"; on the one hand they were trying to shame developers using Python 2 for using it incorrectly (even though it was totally possible to handle Unicode correctly with that stack), yet on the other they were pumping out ridiculous unavoidable security SNAFUs in their API surface by assuming that filenames on disk are UTF-8 :/. To tell another story of the importance of reading the--in this case, not even long!--specification: on Windows, filenames are stored as UCS-2 (as it all predates UTF-16), and is most likely most correctly modeled as an arbitrary sequence of non-0 16-bit values; meanwhile, on Unix systems, a filename is pretty much just an arbitrary sequence of non-0 8-bit values (with some weirdness around the slash character... the API surface doesn't support it really but sometimes the filesystem does). The premise that filenames have a display encoding mostly comes from the software rendering the strings and is then defined in terms of a bunch of hacks involving "locales", and isn't a property of the underlying API or storage semantics. You can't assume in either case that they are canonical Unicode representations, and so storing them as "strings" in your program is super wrong and can lead to anything from data storage mistakes to security vulnerabilities. The Python 3 API filesystem API was thereby so bad that it would do things like just skip over files during directory traversal that didn't map to its attempt to decode stuff as Unicode. Like: if you wanted to hide a file from a Python 3 program, you could just make it invalid Unicode :/. (I realized people reading this comment might find a reference on this fun, and so found this seemingly-comprehensive postmortem. I haven't read it, but it at least looks really good ;P.) https://vstinner.github.io/python30-listdir-undecodable-filenames.html https://vstinner.github.io/python30-listdir-undecodable-file... I thereby find the whole Python 3 debacle extra demoralizing as it felt like a bunch of people who didn't truly understand how hard the problem they were dealing with was--and how much of the world actually is binary, and how often (when dealing with things like filesystems and network protocols) parsing binary sequences as pseudo-strings is "correct"--both sometimes-correctly insisting everyone was wrong while themselves failing to build something even minimally useful until like, Python 3.4 or so; and, in the end, I feel like they just ended up creating a "new mess" by having filenames now encoded as "Unicode strings"... but with weird exceptions for allowing round-trip behaviors of strings that couldn't be decoded :(. The reality is that there are a number of things that developers work with which are pseudo-strings, and languages should provide good tooling for working with them. We see the same issue come up with HTTP headers and filenames: things that are actually binary but have string-like structure. There is a reason why both C++ and Python 2 are often looked at fondly by low-level developers for providing low-level string representations, even while they end up encouraging broken behavior higher up the stack. I haven't looked enough at Rust's standard library to see if they truly nailed this yet, but maybe one day we will have a language that encourages the right behavior for these mechanisms that are neither strictly string not strictly binary.

by saurik 6y ago

baq 6y ago

reading this i'm getting flashbacks from python 2 -> 3 migration and all those people complaining that they're now getting Unicode*Errors thrown at them. the fact is, none of this stuff is simple! you can pretend it is until a corner case bites you in the behind, suddenly the tower of abstraction falls apart and you're left picking up the parts in four different string encoding formats. (see the abysmal C which at least doesn't pretend anything at all, C++ which doesn't require you to know all this stuff and then it just blows up if you make a particularly wrong misstep and most recently Rust, which will call you names for using the wrong type in the wrong place, in turn greatly confusing people who say strings are easy).

wizzwizz4 6y ago
In this case, Python and Rust's behaviours are correct. You only get type errors / Unicode* exceptions when you're making wrong assumptions; if you get them, and don't understand why, you're writing buggy code to do things you don't understand and should be thankful for the errors letting you know there's a problem.
saurik 6y ago
So, my favorite thought process from the Python 2 to 3 transition comes from staring deeply into the hubris of the Python 3000 effort itself as wanting to insist "everything is Unicode"; on the one hand they were trying to shame developers using Python 2 for using it incorrectly (even though it was totally possible to handle Unicode correctly with that stack), yet on the other they were pumping out ridiculous unavoidable security SNAFUs in their API surface by assuming that filenames on disk are UTF-8 :/. To tell another story of the importance of reading the--in this case, not even long!--specification: on Windows, filenames are stored as UCS-2 (as it all predates UTF-16), and is most likely most correctly modeled as an arbitrary sequence of non-0 16-bit values; meanwhile, on Unix systems, a filename is pretty much just an arbitrary sequence of non-0 8-bit values (with some weirdness around the slash character... the API surface doesn't support it really but sometimes the filesystem does). The premise that filenames have a display encoding mostly comes from the software rendering the strings and is then defined in terms of a bunch of hacks involving "locales", and isn't a property of the underlying API or storage semantics. You can't assume in either case that they are canonical Unicode representations, and so storing them as "strings" in your program is super wrong and can lead to anything from data storage mistakes to security vulnerabilities. The Python 3 API filesystem API was thereby so bad that it would do things like just skip over files during directory traversal that didn't map to its attempt to decode stuff as Unicode. Like: if you wanted to hide a file from a Python 3 program, you could just make it invalid Unicode :/. (I realized people reading this comment might find a reference on this fun, and so found this seemingly-comprehensive postmortem. I haven't read it, but it at least looks really good ;P.) https://vstinner.github.io/python30-listdir-undecodable-filenames.html https://vstinner.github.io/python30-listdir-undecodable-file... I thereby find the whole Python 3 debacle extra demoralizing as it felt like a bunch of people who didn't truly understand how hard the problem they were dealing with was--and how much of the world actually is binary, and how often (when dealing with things like filesystems and network protocols) parsing binary sequences as pseudo-strings is "correct"--both sometimes-correctly insisting everyone was wrong while themselves failing to build something even minimally useful until like, Python 3.4 or so; and, in the end, I feel like they just ended up creating a "new mess" by having filenames now encoded as "Unicode strings"... but with weird exceptions for allowing round-trip behaviors of strings that couldn't be decoded :(. The reality is that there are a number of things that developers work with which are pseudo-strings, and languages should provide good tooling for working with them. We see the same issue come up with HTTP headers and filenames: things that are actually binary but have string-like structure. There is a reason why both C++ and Python 2 are often looked at fondly by low-level developers for providing low-level string representations, even while they end up encouraging broken behavior higher up the stack. I haven't looked enough at Rust's standard library to see if they truly nailed this yet, but maybe one day we will have a language that encourages the right behavior for these mechanisms that are neither strictly string not strictly binary.