3 ms·

> Python 3000 effort itself as wanting to insist "everything is Unicode" How does Python 3 handle these HTTP headers and other things that are purely ASCII? Do

by pansa2 6y ago

> Python 3000 effort itself as wanting to insist "everything is Unicode" How does Python 3 handle these HTTP headers and other things that are purely ASCII? Does it treat them as type `bytes` instead of (Unicode) `str`?

saurik 6y ago
I just did some digging, and the Python 3 built-in HTTP stuff seems to represent HTTP headers using a reasonably-comprehensive mechanism that is shared with MIME parsing and the various IMF semantics, and so I am pretty sure you end up getting back an object of type email.header.Header (but I might have misunderstood the documentation just now... I mostly gave up on Python when they moved to Python 3 so I haven't experienced this first hand). https://docs.python.org/3/library/email.header.html#email.header.Header https://docs.python.org/3/library/email.header.html#email.he... This actually looks like a really solid implementation--like, I am super impressed by this (though watch it suck... everything sucks ;P)--as it doesn't assume any ability to flatten the result! That said, I am also deeply concerned that this might pre-decode something that should actually be in ASCII all the way down from MIME (which, notably, is complex enough of a format to include support for embedded comments). I kind of want to do some auditing here, to make sure there aren't some parser discrepancy vulnerabilities in common Python HTTP frameworks ;P. But like, I think this object represents a sequence of strings each with its own encoding, allowing it to "Q" encode the result exactly as specified (and correctly round trip something it parsed, which still does feel a tad dangerous, and maybe the API lets you avoid it in those cases). It then has an __str__ implementation (to let you use it as a string) that explicitly claims to return an "approximation" of the header (which may or may not be a safe thing to use in different circumstances; but like, that in some sense isn't the API's fault?...).

by pansa2 6y ago

saurik 6y ago

I just did some digging, and the Python 3 built-in HTTP stuff seems to represent HTTP headers using a reasonably-comprehensive mechanism that is shared with MIME parsing and the various IMF semantics, and so I am pretty sure you end up getting back an object of type email.header.Header (but I might have misunderstood the documentation just now... I mostly gave up on Python when they moved to Python 3 so I haven't experienced this first hand). https://docs.python.org/3/library/email.header.html#email.header.Header https://docs.python.org/3/library/email.header.html#email.he... This actually looks like a really solid implementation--like, I am super impressed by this (though watch it suck... everything sucks ;P)--as it doesn't assume any ability to flatten the result! That said, I am also deeply concerned that this might pre-decode something that should actually be in ASCII all the way down from MIME (which, notably, is complex enough of a format to include support for embedded comments). I kind of want to do some auditing here, to make sure there aren't some parser discrepancy vulnerabilities in common Python HTTP frameworks ;P. But like, I think this object represents a sequence of strings each with its own encoding, allowing it to "Q" encode the result exactly as specified (and correctly round trip something it parsed, which still does feel a tad dangerous, and maybe the API lets you avoid it in those cases). It then has an __str__ implementation (to let you use it as a string) that explicitly claims to return an "approximation" of the header (which may or may not be a safe thing to use in different circumstances; but like, that in some sense isn't the API's fault?...).