11 ms·
IMO one of the reasons for all the angst is that .encode() and .decode() are so ambiguous and unintuitive which makes them incredibly confusing to use. Which d
by andrewstuart 11y ago
IMO one of the reasons for all the angst is that .encode() and .decode() are so ambiguous and unintuitive which makes them incredibly confusing to use. Which direction are you converting? From what to what? The whole unicode thing is hard enough to understand without Python's encoding and decoding functions adding to the mystery. I still have to refer to the documentation to make sure I'm encoding or decoding as expected.
I think there would have been much less of a problem if encode and decode were far more obvious, unambiguous and intuitive to use. Probably without there being two functions.
Still a problem of course today.
- cjauvin 11y agoI know it's a deeply subjective and personal thing unlikely to help anyone other than me, but I've always successfully relied on the following mnemonic for this particular problem: * unicode -> encode, u -> e (vowels) * str -> decode, s -> d (consonants)
- deleted 11y ago[deleted]
- andrewstuart 11y agoThere should be one obvious way to do it.
- davidism 11y ago> Although that way may not be obvious at first unless you're Dutch. There is one obvious way: strings are encoded to bytes, bytes are decoded to strings.
- jes5199 11y agoI think that's backwards
- rspeer 11y agoIt's not backwards. I think that reveals that the names really do have a problem. The problem is that "encode" sounds like "make this Unicode" to people who aren't familiar with Unicode.
- andrewstuart 11y agoI don't think it is obvious and intuitive and unambiguous what the difference is between ENcoding and DEcoding.
- Veedrac 11y agoA string is an abstract bit of text. You need to encode this into a particular memory representation of the text. Bytes hold a bunch of data in some encoding. It could be an image, UTF-8 or LZMA compressed ASCII. Once you know the encoding, to reconstruct the data you decode into a semantically meaningful form. To put it another way, imagine the terms were "serialize" and "deserialize". Of course one serializes to and deserializes from binary data. Just replace "{,de}serialize" with "{en,de}code" and you're done.
- lawpoop 11y agoYou mean the prefixes 'en-' and 'de-'?
- odonnellryan 11y agoEncode and Decode are slightly subjective... why not something like to_bytes and from_bytes? Maybe not the best names, but definitely clearer on the meaning.
- takeda 11y agoNot really. Veedrac had a good analogy, think of text as something abstract, for example imagine text is an image or sound, if you want to store it in bytes you need to encode it, and to read back you decode it. As to_bytes/from_bytes, actually python provides it too: to_bytes -> bytes(<text>) from_bytes -> str(<bytes>)
- odonnellryan 11y agoIt makes sense for sure, just isn't super intuitive - if it was people wouldn't be so confused.
- daveguy 11y agoMy way of remembering str -> decode; (s->d); sad. sad that I have to remember this and that it is still the "future of the language" despite significant and ongoing resistance. EDIT: This last line of the article pretty much sums it up: "We structured the transition thinking the community would come along with us in leaving Python 2 behind, but that turned out not to be the case and instead we have taken some more time and are using a Python 2/3 compatible subset of the language to manage the transition." I wonder if python 3.6 or 3.7 will acquiesce to this and give an option for something like "from __past__ import str" like we have for __future__ in 2.7 to make these backward incompatibilities easier to deal with.
- s73v3r 11y agoWhy? What possible reason would there be to use the old behavior, except for laziness?
- daveguy 11y agoNice. Thanks for attributing a negative personality trait as the only reason someone would prefer any other way.
- chc 11y agoI don't know if it's been edited since you posted this, but at the moment, the comment is a question asking why somebody would want this other than laziness.
- daveguy 11y agoTrue. I guess some would consider this a positive personality trait in a programmer. I think that's only for system design though (lazy for minimizing future effort required).
- squeaky-clean 11y agoBecause I'd like to be able to use the googleads Python client library within Python 3 without needing to fix 62,000 lines of code? I guess you could call that laziness if you want.
- dietrichepp 11y agoHm, I never saw this as ambiguous at all, except for a few weird encodings that Python has as "convenience" method. Here's how you remember it: "Unicode" is not an encoding. It never was, it never will be. Of course, the data must be encoded in memory somehow, but in Python 3, you cannot be sure what encoding that is because it's not really exposed to the user. From what I understand, there are different encodings that string objects will use, transparently, in order to save memory! You always "encode" something into bytes, and "decode" bytes back into something. There should be exactly two functions, because the functions have different types: "encode" is str -> bytes, "decode" is bytes -> str. Explicit is better than implicit. output = input.decode(self.coding) With Python 3, I instantly know that "input" is bytes and "output" is str.
- hahainternet 11y agoIn [1]: len("नि") Out[1]: 2 So close, but it's not really a string or bytes, but a codepoint array. You can even iterate it: In [2]: for c in "नि": ...: print("Character {}".format(c)) ...: Character न Character ि
- Veedrac 11y agoAlthough true, the aim of the transition was to enforce encoding-correctness. Operations on strs should never break the encoding itself[1], but preserving the semantic meaning of the text itself is basically impossible for arbitrary data. [1] Sadly surrogate escapes break this. Rust's division into String:OsString:Vec<u8> makes that aspect a lot cleaner.
- dietrichepp 11y agoI'm getting tired of this argument, we had it in the Rust community as well and I was tired of it there, too. I've seen the same argument over and over again, and every argument had the same hole that I see in your post: What, exactly, is a "character"? Please answer me that, and then you can write a proposal for how the API should work, exactly, and why this works with Indic and German and Korean and everything else. Strings have to be arrays. It is inevitable. We are on Von Neumann machines with L1 cache lines that are something like 64 bytes wide, so making strings into something other than arrays is a complete non-starter. Because strings are arrays, we use array indexes to slice strings. The indexes are integers, because that just makes sense on a Von Neumann machine. So your complaint is that you can slice a string in "bad" ways. What makes it bad? You're trying to prevent people from slicing up grapheme clusters, which is a noble goal. But in practice, we get the indexes from methods like str.find() or re.match(), and we treat them as more or less opaque, so we don't end up slicing grapheme clusters very often in practice. Grapheme cluster slicing requires a big Unicode table in memory anyway, so in the rare case that you need it, you can pay the high performance cost for using it. In the meantime, formats like JSON and XML are defined in terms of code points, so using code point arrays eliminates a class of bugs where you could accidentally make malformed XML or JSON, which would then get completely REJECTED by the receiving side, causing your Jabber client to quit or your web browser to show a bunch of mojibake. And let me ask you this: what do I get when I write: x = "a" y = "\u0301" Is the resulting len(x + y) == 1? But len(x) == 1 and len(y) == 1? Can you tell me what the correct behavior is? Might you end up with bugs in programs because len(x + y) != len(x) + len(y)? Or do you introduce extra code points into the string when you concatenate them? Please, tell me what you actually think is correct behavior. It is far, far more useful than pointing out that something is "wrong" on purely semantic arguments.
- the_mitsuhiko 11y ago> .encode() and .decode() are so ambiguous and unintuitive They are not. What is unintuitive is the default encoding in Python where an encode can trigger an implicit encode and the other way round. The `encode()`/`decode()` availability of strings was never a problem has you have many bytes -> bytes and str -> str codecs;
- bpicolo 11y agoDisagree. The original commenter is correct in saying that the naming scheme is not obvious. Something like to_bytes(encoding) would be a lot more clear.
- the_mitsuhiko 11y ago> Disagree. The original commenter is correct in saying that the naming scheme is not obvious. Something like to_bytes(encoding) would be a lot more clear. But then function would do something completely different. "\x01\x02".encode('zlib') for instance is a bytes to bytes operation. The problem is that "foo".encode('utf-8') does not give you an exception. If the coercion would not be enabled you would get an error: >>> reload(sys).setdefaultencoding('undefined') >>> 'foo'.encode('utf-8') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/Cellar/python/.../encodings/undefined.py", line 22, in decode raise UnicodeError("undefined encoding") UnicodeError: undefined encoding That's not any worse than what Python 3 does: >>> b'foo'.encode('utf-8') Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'bytes' object has no attribute 'encode'
- kenko 11y ago> "\x01\x02".encode('zlib') for instance is a bytes to bytes operation. I think the inclusion of things like zlib (or rot13 or whatever) was a conceptual error that just fosters confusion.
- the_mitsuhiko 11y ago
- marklgr 11y agoEncoded things are not meant to be human readable, like some sort of "secret code". The underlying bytes of some Unicode string probably won't be human-readable if you read it byte per byte--it will be like the platform's "secret code" for your string. Conversely, you can decode the byte message into something readable, ie. your Unicode string.
- jlarocco 11y agoIt is easy to confuse the names, but it takes 2 seconds in the REPL or in Idle to figure it out, and the concept behind them is pretty obvious, IMO. I'll take an occasional 2 seconds in the REPL over the headache of debugging codec issues any day of the week.
- zeeboo 11y agoNobody has a problem with encode/decode when talking about, say, json. You have some semantic object that you want to encode to some bytes representation, and some bytes representation that you want to decode into some semantic object. The confusions is around people viewing byte strings implicitly as ascii codepoints, and not fully understanding what the things they're looking at actually are.
- wiz21 11y agoI concur 100%. Most of the problems with unicode are like most problem with floats : tons of people don't get how it works (and the good reasons that make all of that tricky)
- est 11y ago99% encoding problems in Python 2 is because str(), the fundamental type converting function, is not safe. output = ''.join(str(x) for x in some_list) Can you spot the danger here? Can you imagine how many hidden str() calls in every lib/modules?
- coldtea 11y ago>Nobody has a problem with encode/decode when talking about, say, json. That's maybe because JSON is by the standard supposed to be utf-8, always.
- aidos 11y agoI think you've misunderstood. json encoding is the process of converting native types to utf8 encoded data. The point of the comment is that everyone knows that you json encode native entities to the utf8 representation, and json decode from utf8 to native entities. The same is true of unicode. It's not ambiguous at all.
- coldtea 11y ago>The same is true of unicode. It's not ambiguous at all. Only utf-8 is a very specific encoding, so converting to it is straightforward, whereas unicode is an umbrella term and the most ambiguous thing you can imagine. Even saying "the same is true of unicode" shows confusion, as utf-8 is itself unicode. As are numerous different encodings, and then you have normalization and other subtleties.
- rcthompson 11y agoI've really tried to figure out how to use these methods properly, but in the end, the only successful strategy I've found essentially boils down to randomly sprinkling encodes and decodes throughout my code until errors stop happening. Half the time I end up using an encode where my interpretation of the documentation suggested a decode would be required, or vice versa. (This is in Python 2. I haven't tried Python 3 yet since I don't do much Python coding these days.)
- andreasvc 11y agoThat's not the proper way. You should rarely need encode/decode if you make sure everything within Python is unicode. You ensure this by being specifying the encoding when getting data from external sources such as files, the network, or external libraries.
- rcthompson 11y agoOf course I'm well aware this isn't the proper way to do it. I'm just saying it's the only way I've found that is successful with any degree of regularity. I've tried following the documentation as best I can, but like I said, half the time an encode ends up working where I thought I needed a decode or vice versa.
- auxym 11y agoPython 3 really does help with this. You rarely need to use encode and decode, it's done automatically when opening a file in text mode. You do have to specify the encoding you want though.
- takeda 11y agoThis is how to look at it. First you need to think of text and binary (bytes) as two distinct types. The text most of the times is how you interact with user, so things that you will display to the user, or what user enters to you. Now, the text is stored in unicode (how, it's not our concern, Python abstracts it from us), the bytes is representation how it's stored in files, send over a network etc. Now if you need to store text in a file, or send over network you need to encode it (most of the time as UTF-8), if you receive data which supposed to be text you decode it. It might be helpful to think of a text as a sound and bytes as mp3. You encode sound as mp3 to store it in a file, and when you want to play it back you decode it. There are some things can cause a confusion. For example if you write text to a file, python will automatically apply the conversion. Things like that are there so you don't have to do the conversion every single time.
- aidos 11y agoThe confusion you're talking about is a Python 2 problem, not one of ambiguity. Encoding and decoding are pretty well defined (though, I've never thought about the formal definition before). When you have an entity in its native form, it needs to be _encoded_ for the purposes of communication (in a broad sense). The encoded message can then be to be _decoded_ back to the natural form. There is no ambiguity. Really, the reason people get them mixed up in Python is because Python 2 totally stacked it by adding str.encode and unicode.decode. In Python 2, you can _decode_ unicode to unicode – which it does by silently _encoding_ as ascii first. This operation is total madness. >>> u = u'\xe9' >>> u.decode('utf8') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 0: ordinal not in range(128) The error here is in the step where you _encode_ the unicode to acsii, something you didn't ask it to do at all. And similarly, you can encode str to str (where str is really bytes in Python 2, another issue that adds to the confusion). Don't get me wrong. I feel your pain. When I was using Python 2 I also got confused about what form things were in and where I needed to encode / decode. Honestly, once I switched to Python 3, that cognitive overhead just totally vanished. str is the natural form of text, and if I need to store / communicate it I _encode_ it to bytes (utf8, generally). When I'm loading a stored/transmitted message, I _decode_ it to its natural text form. There are edge cases that make certain situations more complex, but in terms of general usage, I feel Python 3 really got this stuff right.
- mark-r 11y agoI agree that encode() and decode() are ambiguous, I find myself pausing to make sure I'm using the right one. You can use bytes(string,encoding) to replace encode(). Unfortunately it doesn't have a default encoding, which makes it a pain to use. And str(bstring) isn't symmetric, it can't replace decode().
- est 11y agoThe problem is that both u''.encode() and ''.encode() (at least in Python 2) exists. Why?
- Too 11y agoFor me this confusion arises because of ducktyping together with ambiguous interfaces. Take open() as example, it can return a file that reads either in binary more or automatically decodes into unicode strings, just based on if you passed a string argument mode='rb' vs mode='r', also if you enable universal newlines it will implicitly assume some encoding. This in itself all kind of makes sense once you know it, but the danger lies in that it's code that worked in py2, and even now it will still run because most functions that work on bytes also work on unicode. It's not until you try to combine these variables with data from other sources(like string literals) which were/weren't unicode you notice this and then you don't know which side wad correct so you randomly just shotgun either decode on one side or encode on the other side. This could be 100 lines away from where the problem should actually have been solved and people being people solves the symptoms instead of the causes.