4 ms·

So, for example what is The Py3k Right Way to fetch an URL with text content into a string? The example https://docs.python.org/3/library/urllib.request.html#ex

• masklinn 10y ago
  The "right way" in every single language is to follow the encoding sniffing algorithm: https://html.spec.whatwg.org/multipage/syntax.html#encoding-sniffing-algorithm https://html.spec.whatwg.org/multipage/syntax.html#encoding-.... You may want to note the following: 1. implementing the entire encoding-sniffing algorithm for a basic example is a bit extreme 2. the very first step of the encoding-sniffing algorithm is "if the use has explicitly provided an encoding, use it", which is essentially what the example does > To me it looks like by insisting on lossy byte/string conversion of all I/O the language painted itself into a corner That makes literally no sense. If you want to "fetch an URL with text content into a string" — emphasis on string, not bag of bytes, if you want a bag of bytes you can skip the whole decoding thing in Python it's not necessary — there's no other way than to decode it, which means you need to assert or discover its encoding, for all you know the document could be in Big5.
  • haikuginger 10y ago
    Seriously. It's easy to "fetch a URL with text content into a string" with Python 2 right now _because Python 2 assumes that all bytes are ASCII_.
    • throwawayish 10y ago
      Which is something you can do in Python 3 as well, if you really want the broken behaviour of Python 2 or the behaviour of Py2 is "ok for me".
    • TheDong 10y ago
      No, it's not! It's easy to fetch a url's content into a bag of bytes. That's it. If the content of the page happens to contain an emoji (which hey, more and more do) or even a friggin' unicode double-quote mark, then you're no longer fetching it into a valid string, if you take string to mean faithful textual representation. You've got all the bytes there, but you can't interpret it correctly. This is a real problem that has caused real pain for me with various tools written in python2
  • rini17 10y ago
    > emphasis on string, not bag of bytes, This is the crux of the issue. You run around reminding everyone of this, yet you are fine with "read bag of 100 bytes, then go merrily utf-8 decode them" as canonical example in official documentation.
• detaro 10y ago
  What is text content? You have to either know, find from the metadata or blindly assume an encoding from the byte string HTTP gives you. In the "simplest" case Python3 only forces you to explicitly write that assumption in your code, Python2 just made one for you.
  • rini17 10y ago
    Very well, now how do I put down that assumption? Is "read 100 bytes, utf8-decode them" right example?
    • detaro 10y ago
      The example probably does the truncation this way to just show the effect of decode() on the console, but if you really want a truncated version you should either decode everything and then truncate or use a StreamReader to only decode the first X characters. The docs probably should make that issue clear, so yes, the example there is badly choosen. Another real-life answer probably is "don't use urllib directly", since sadly the trend seems to go away from stdlib libraries for these things. E.g. Requests will take the encoding from the HTTP headers if available, and allows you to manually specify it as well if you want.