6 ms·

> Unicode support was actually an anti-feature for most existing code. If you're writing a simple script you prefer 'garbage-in, garbage-out' unicode rather tha

• pmontra 7y ago
  > In python 3 it always blows up when you mix bytes with text so you can catch the issue early on. This is definitely the case. I've been wrestling with bytes and strings all the time during the port of a Django application to Python 3 for a costumer. I can see myself encoding and decoding response bodies and JSON for the time being. For reasons I didn't investigate I don't have to do that with projects in Ruby and Elixir. It seems everything is a string there and yet they work.
  • CaptainMarvel 7y ago
    I’ve worked in a variety of Django codebases, and the last time I had trouble with string encoding/decoding was with Python 2. Since moving to Python 3, I have rarely needed to manually encode or decode, and I genuinely can't remember the last time I did. Perhaps there’s something about a port that requires encoding/decoding bytes/strings?
    • takeda 7y ago
      The encoding/decoding is heavy in codebases that have to run on Python 2 and Python 3 at the same time, and authors are worried about handling unicode correctly on python 2. Ironically when your python 2 app doesn't care about unicode, the porting to python 3 is actually much easier.
  • takeda 7y ago
    you don't have to do these things in python 3 either, your problem was that you had python 2 code that was already broken and you are started adding encode/decode to fix it, typically making the problem worse. If you write code in python 3 from the start you rarely need to use encode() and decode(). Typically what you always want is a text not bytes. Exception to it might be places where you want to serialize like IO (network or files, although even files are converted on the fly unless you open file in a binary mode).
    • pmontra 7y ago
      The problem are external APIs returning whatever they want no matter what they should return. The world is messy. Example, I just had to write this return urllib.request .urlopen(url, timeout=60) .read() .decode("utf-8", errors="backslashreplace") (probably not valid code because of the newlines but you'll forgive me) Then I use that string in a regexp, etc. This is the only language where I have to explicitly deal with encodings at such low level. I don't feel like I want to use it for my pet projects.
      • takeda 7y ago
        Why is that bad? The result returned from an URL is always binary. In certain situations it could be text but it doesn't have to be. If the result was an image and you would want to convert the data to image, if it was sound file same, you should think of text as another distinguished type. Of course urllib could have method text() that would do such conversion, but then urllib is not requests. It never was user friendly. Edit: personally I use aiohttp, the interface is much nicer: https://aiohttp.readthedocs.io/en/stable/client_reference.html#aiohttp.ClientResponse.text https://aiohttp.readthedocs.io/en/stable/client_reference.ht... if I can't use asyncio then would use requests.
• some_random 7y ago
  >Actually that's the behavior of python 2, it works fine, until you send invalid characters then it blows up. We're talking about simple scripts, the solution is to not send in invalid characters.
  • takeda 7y ago
    even in very simple scripts you don't get invalid characters until you actually get them.
• dyingkneepad 7y ago
  > Actually that's the behavior of python 2, it works fine, until you send invalid characters then it blows up. > In python 3 it always blows up when you mix bytes with text so you can catch the issue early on. Sometimes you don't care about weird characters being print as weird things. In python 2 it works fine: you receive garbage, you pass garbage. In python 3 it shuts down your application with a backtrace. Dealing with this was one of my first Python experiences and it was very frustrating, because I realized that simply using #!/usr/bin/python2 would solve my problem but people wanted python3 just because it was fancier. So we played a lot of whack-a-mole to make it not explode regardless of the input. And the documentation was particularly horrible regarding that, not even the experienced pythoners knew how to deal with it properly.
  • takeda 7y ago
    Those issues are common when you're having python 2 code that uses unicode datatype and you have a task to migrate it to python 3. You run your python 2 code on python 3 and it fails, most people at that point will place encode() or decode() in place where you have a failure. When the correct fix would be to place encode/decode at I/O boundary (writing to files (and in python 3 even that is not needed if you open files in text mode), network etc). Ironically a python 2 code that doesn't use unicode is easier to port. When you program in python 3 from the start it's very rare to need encode/decode strings. You only do that if you are working on I/O level. > And the documentation was particularly horrible regarding that, not even the experienced pythoners knew how to deal with it properly. Because it's not really python specific knowledge. It's really about understanding what the unicode is, what bytes are, and when to use each. The general practice is to keep everything you do as text, and do the conversion only when doing I/O. You should think of unicode/text as as a representation of a text, as you think of a picture or sound. Similarly to image and audio text can be encoded as bytes. Once it is bytes it can be transmitted over network or written to a file etc. If you're reading the data, you need to decode it back to the text. This is what Python 3 is doing: - by default all string is of type str, which is unicode - bytes are meant for binary data - you can open files in text and binary mode, if you open in text the encoding is happening for you - socket communication - here if you need to convert string to bytes and back Python 2 is a tire fire in this area: - text is bytes - text also can be unicode (so two ways to represent the same thing) - binary data can also be text - I/O accepts text/bytes, no conversion happening - a lot (most? all?) stdlib is actually expecting string/bytes as input and output - cherry on top is that python2 also implicitly converts between unicode and string so you can do crazy thing like my_string.encode().encode() or my_string.decode() So now you get a python 2 code, where someone wanted to be correct (it is actually quite hard to do it, mainly because of the implicit conversion) so the existing code will have plenty of encode() and decode() because some functions now expect str some unicode. At different functions you might then have bytes or unicode as a string. Now you take such code and try to move it to python 3, which no longer has implicit conversion and will throw an error when it expected text and got bytes and vice versa. The str now is unicode, unicode type no longer exists and bytes is now not the same thing as str. So your code now blows up. Most people see an error so they add encode() or decode() often trying which one works (like what you were removing) when the proper fix would be actually removing encodes() and decodes() in other places of the code. It's quite difficult task when your code base is big, so this is why Guido put a lot of effort with type annotations, mypy. One of its benefits supposed to help with these issues.
• josefx 7y ago
  > Actually that's the behavior of python 2, it works fine, until you send invalid characters then it blows up. Not always. As far as I can tell writing garbage bytes to various APIs works fine unless they explicitly try to handle encoding issues. First time I noticed encoding issues in my code was when writing an xml structure failed on windows, all because of an umlaut in an error message I couldn't care less about. The solution was to simply kill any non ascii character in the string, not a nice or clean solution but the issue wasn't worth more effort. > In python 3 it always blows up when you mix bytes with text so you can catch the issue early on. That is nice if your job involves dealing with unicode issues. My job doesn't, any time I have to deal with it despite that is time wasted.
  • c-cube 7y ago
    What kind of text do you have to process at your job, that you never meet any unicode in it? Nowadays unicode is everywhere, especially with emojis. Even a simple IRC bot needs to handle that.
    • josefx 7y ago
      I deal with file formats that like plain text files and zip do not specify an encoding and have different encodings depending on where they come from. I think the generic approach is to guess, which means trying encodings until one successfully converts unknown input to garbage unicode resulting in output that is both wrong and different from the original input. Most of the time I can just treat the text contents as byte array, with a few exceptions that are specified to use ascii compatible names. So you can throw in your emoji and they might not correctly show up on the xml logging metadata I write, because I don't care. But they will end up in the processed file the same way they came in instead of <?> or some random Chinese or Japanese symbol that the guessing algorithm thought appropriate.
      • takeda 7y ago
        In that case you should be opening files in binary mode "b", then you will be operating in bytes. Also, there's no guessing happening in this instance. A locale configured in your environment variable are used if you open files using text mode.
• tedunangst 7y ago
  Doesn't always blow up. Notably b"key" and "key" are now distinct dictionary keys, and both can coexist in the same dict. Is the absence of an optional key a fatal error? No, the program runs, and just does the wrong thing, or fails to copy the right value to the next stage, or whatever. Fun to debug.
  • takeda 7y ago
    To get b'key' and 'key' in a dictionary in python 3 you really need to try hard. The only reasonable scenario I can think of is when you are porting python 2 code to python 3 and play with .decode() and .encode().
• rstuart4133 7y ago
  > Actually that's the behavior of python 2, it works fine, until you send invalid characters then it blows up. Not that I've seen. Example of where Python 3 has rained shit on my parade: I wrote a program that backs up files for Linux. It works fine in python 2, but in python 3 you rapidly learn you must treat filenames as bytes otherwise your backup program blows up on valid Linux filenames. It's not just decoding errors, it's worse. Because Unicode doesn't have a unique encoding for each string, so the round trip (binary -> string -> binary) is not guaranteed to get you the same binary. If you make the mistake of using that route (which Python3 does by default) then one day Python3 will tell you can't open a file you os.listdir() microseconds ago and can clearly see is still there. Later, you get some sort of error when handling one of those filenames, so you sys.stderr.write('%s: this file has an error' % (filename,)). That worked in python2 just fine, but in python3 generates crappy looking error messages even for good filenames. You can't try to decode the filename to a string because it might generate a coding error. This works: sys.write('b%b: this file has an error' % (filename,)), but then you find you've inserted other strings into error messages and soon the only "sane" thing to do is to to convert every string in your program to bytes. Other solutions like sys.write('%s: this file has an error' % (filename.decode(errors='ignore'),)) but corrupt the filename the user sees, are verbose, and worst of all if you forget it isn't caught by unit tests but still will cause your program to blow up in rare instances. I realise that for people who live in a land of clearly delineated text and binary, such as the django user posting here, these issues never arise and the clear delineation between text and bytes is a bonus. But people who use python2 as a better bash scripting language than bash don't live in that world. For them python2 was a better scripting language than bash, but is being being depreciated in favour of python3 that's actually more fragile than bash for their use case. (That's a pretty impressive "accomplishment".) Perhaps they will go to back to Perl or something, because it stands Python3 isn't a good replacement.
  • Enginerrrd 7y ago
    >For them python2 was a better scripting language than bash This! IMO Python 2 has better usability for prototyping and thinking and doing things on the fly. Python 3 also often seems to have deprecated the functions I want to use in favor of those that are more cumbersome and take more keystrokes. More explicit sure, but less fluid.