4 ms·

> you would use byte strings and you would never have to call encode or touch UTF-8 before passing the byte string to a TCP socket. I'll start by adding that i

by keyme 13y ago

> you would use byte strings and you would never have to call encode or touch UTF-8 before passing the byte string to a TCP socket. I'll start by adding that it's also incredibly annoying to declare each string to be a byte string, if this wasn't clear from my original rant. Additionally, your advice is broken. Take a look at this example, with the shelve module (from standard library). s=shelve.open('/tmp/a') s[b'key']=1 Results in: AttributeError: 'bytes' object has no attribute 'encode' So in this case, my byte string can't be used as a key here, apparently. Of course, a string is expected, and this isn't really a string. My use case was trying to use a binary representation of a hash as the key here. What's more natural than that. Could easily do that in Python 2. Not so easy now. I can find endless examples for this, so your advice about "just using byte strings" is invalid. Conversions are inevitable. And this annoys me. > What you're saying about Unicode scares me. Yeah, I know full well what you're scared of. If I'm designing everything from scratch, using Unicode properly is easy. This, however, is not the case when implementing existing protocols, or reading file formats that don't use Unicode. That's where things begin being annoying when your strings are no longer strings.

deathanatos 13y ago
> Additionally, your advice is broken. Take a look at this example, with the shelve module (from standard library). His advice was sound, and referred to your example of TCP stream data (which is binary). Your example regards the shelve library. > So in this case, my byte string can't be used as a key here, apparently. shelve requires strings as a keys. This is documented, though not particularly clearly. > so your advice about "just using byte strings" is invalid. Let me attempt to rephrase his advice. Use bytestrings where your data is a string of bytes. Use strings where your data is human-readable text. Covert to and from bytestrings when serializing to something like network or storage. > Conversions are inevitable. Absolutely, because bytestrings and (text)strings are two different types. > And this annoys me. There is no real alternative though, because there is no way to automatically convert between the two. Python 2 made many assumptions, and these were often invalid and led to bugs. Python 3 does not; in places where it does not have the required information, you must provide it. > when implementing existing protocols, or reading file formats that don't use Unicode. I'm afraid it's still the same. A protocol that uses unicode requires you to code something like "decode('utf-8')" (if UTF-8 is what it uses), one that does not requires "decode('whatever-it-uses-instead')". If it isn't clear what encoding the file format or protocol stores textual data in, then that's a bug with the file format or protocol, not Python. Regardless though, Python doesn't know (and can't know) what encoding the file or protocol uses.

by keyme 13y ago

deathanatos 13y ago

> Additionally, your advice is broken. Take a look at this example, with the shelve module (from standard library). His advice was sound, and referred to your example of TCP stream data (which is binary). Your example regards the shelve library. > So in this case, my byte string can't be used as a key here, apparently. shelve requires strings as a keys. This is documented, though not particularly clearly. > so your advice about "just using byte strings" is invalid. Let me attempt to rephrase his advice. Use bytestrings where your data is a string of bytes. Use strings where your data is human-readable text. Covert to and from bytestrings when serializing to something like network or storage. > Conversions are inevitable. Absolutely, because bytestrings and (text)strings are two different types. > And this annoys me. There is no real alternative though, because there is no way to automatically convert between the two. Python 2 made many assumptions, and these were often invalid and led to bugs. Python 3 does not; in places where it does not have the required information, you must provide it. > when implementing existing protocols, or reading file formats that don't use Unicode. I'm afraid it's still the same. A protocol that uses unicode requires you to code something like "decode('utf-8')" (if UTF-8 is what it uses), one that does not requires "decode('whatever-it-uses-instead')". If it isn't clear what encoding the file format or protocol stores textual data in, then that's a bug with the file format or protocol, not Python. Regardless though, Python doesn't know (and can't know) what encoding the file or protocol uses.