4 ms·
The "right way" in every single language is to follow the encoding sniffing algorithm: https://html.spec.whatwg.org/multipage/syntax.html#encoding-sniffing-algo
by masklinn 10y ago
The "right way" in every single language is to follow the encoding sniffing algorithm: https://html.spec.whatwg.org/multipage/syntax.html#encoding-sniffing-algorithm https://html.spec.whatwg.org/multipage/syntax.html#encoding-....
You may want to note the following:
1. implementing the entire encoding-sniffing algorithm for a basic example is a bit extreme
2. the very first step of the encoding-sniffing algorithm is "if the use has explicitly provided an encoding, use it", which is essentially what the example does
> To me it looks like by insisting on lossy byte/string conversion of all I/O the language painted itself into a corner
That makes literally no sense. If you want to "fetch an URL with text content into a string" — emphasis on string, not bag of bytes, if you want a bag of bytes you can skip the whole decoding thing in Python it's not necessary — there's no other way than to decode it, which means you need to assert or discover its encoding, for all you know the document could be in Big5.
- haikuginger 10y agoSeriously. It's easy to "fetch a URL with text content into a string" with Python 2 right now _because Python 2 assumes that all bytes are ASCII_.
- throwawayish 10y agoWhich is something you can do in Python 3 as well, if you really want the broken behaviour of Python 2 or the behaviour of Py2 is "ok for me".
- TheDong 10y agoNo, it's not! It's easy to fetch a url's content into a bag of bytes. That's it. If the content of the page happens to contain an emoji (which hey, more and more do) or even a friggin' unicode double-quote mark, then you're no longer fetching it into a valid string, if you take string to mean faithful textual representation. You've got all the bytes there, but you can't interpret it correctly. This is a real problem that has caused real pain for me with various tools written in python2
- rini17 10y ago> emphasis on string, not bag of bytes, This is the crux of the issue. You run around reminding everyone of this, yet you are fine with "read bag of 100 bytes, then go merrily utf-8 decode them" as canonical example in official documentation.