2 ms·

It craps me off that I still need to look up which direction to encode/decode strings and other things like bytes and ascii even after programming Python 3 for

It craps me off that I still need to look up which direction to encode/decode strings and other things like bytes and ascii even after programming Python 3 for more than 10 years. It's a huge usability fail on the part of Python that it's not obvious and easily memorable how to do this.

maxbond 4y ago
You encode strings to bytes, you decode bytes to strings. A string is an abstract representation of text, which can be encoded into whatever supported format you'd like; bytes are a concrete representation of binary data, such as text encoded in UTF8, which, if it represents text, can be decoded to obtain a string. If you think about it, that's the only way that makes sense; if you wanted to go from, say, UTF8 to UTF16, you'd need to have an intermediate step that was abstract and formatless. Similarly, when you're operating on text, you don't want it to have a particular format; the format should only exist at the edges, where you take text in and output it. The format is a way to communicate with other programs, not components within your program. My advice would be, if that doesn't stick in your head (totally legitimate), write down a code example and stick it in a personal wiki or something. I find it really helps me to understand something once and more or less copy paste it forever.
- andrewstuart 4y ago
  I understand the reasons, it’s Python's fault that it’s not easy to remember and program. This is a key reason why people hated Python 3 so much IMO. For context I’m a huge advocate of Python 3, except for this (and lambda functions, but that’s another whine session).
saila 4y ago
It's not that complicated nor is it a Python specific issue. This is a good article on the topic: https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim... A simplistic summary is that you always decode to unicode and always encode to bytes. Another way to think about it is that Unicode is an idealized character model that is reified into a byte encoding, usually UTF-8, for practical usage. It seems to me that this is something a lot of developers don't think too much about until they run into some obscure encoding/decoding issue, but it's kind of a fundamental aspect of programming that's worth spending some time grokking.
- andrewstuart 4y ago
  It’s not that I don’t understand it. The problem is that encode and decode are directional constructs that change depending on context. That’s a poor UI in Python.
  - maxbond 4y ago
    Genuinely curious, what would you like it to look like?
    - aliqot 4y ago
      golang has a wonderful solution
      - maxbond 4y ago
        Would you care to elaborate on what it is and why it's better for the benefit of those of us who don't know Go? It's tricky to look up a different solution you aren't familiar since the terminology is different. What I'm seeing currently doesn't really look that different to Python, but with different names for the types and such. Taking a stab at it; This looks simple, and runes are great (similar to Python): https://golangdocs.com/strings-in-golang https://golangdocs.com/strings-in-golang But something that bothers me is that converting an array of bytes to a string looks to implicitly be ASCII, or possibly UTF8. This document doesn't say. This document acknowledges encodings and clarifies that literals are UTF8, but makes no mention of byte arrays: https://go.dev/blog/strings https://go.dev/blog/strings I've poked through as many SO questions as I care to and am not clear on how to use different encodings. Looks to me like this is a difference in philosophy; Python would say, explicit is better then implicit, and I'm inclined to agree. If I'm not understanding properly I'm happy to be corrected.
    - magicalhippo 4y ago
      I don't have skin in the game, but at least the .Net way is quite explicit, which I like: byte[] unicodeBytes = Encoding.UTF8.GetBytes(inputString); // Perform the conversion from one encoding to the other. byte[] asciiBytes = Encoding.Convert(Encoding.UTF8, Encoding.ASCII, unicodeBytes); string outputString = Encoding.ASCII.GetString(asciiBytes); Adapted from https://learn.microsoft.com/en-us/dotnet/api/system.text.encoding?view=net-7.0 https://learn.microsoft.com/en-us/dotnet/api/system.text.enc...

maxbond 4y ago

You encode strings to bytes, you decode bytes to strings. A string is an abstract representation of text, which can be encoded into whatever supported format you'd like; bytes are a concrete representation of binary data, such as text encoded in UTF8, which, if it represents text, can be decoded to obtain a string. If you think about it, that's the only way that makes sense; if you wanted to go from, say, UTF8 to UTF16, you'd need to have an intermediate step that was abstract and formatless. Similarly, when you're operating on text, you don't want it to have a particular format; the format should only exist at the edges, where you take text in and output it. The format is a way to communicate with other programs, not components within your program. My advice would be, if that doesn't stick in your head (totally legitimate), write down a code example and stick it in a personal wiki or something. I find it really helps me to understand something once and more or less copy paste it forever.

andrewstuart 4y ago
I understand the reasons, it’s Python's fault that it’s not easy to remember and program. This is a key reason why people hated Python 3 so much IMO. For context I’m a huge advocate of Python 3, except for this (and lambda functions, but that’s another whine session).

saila 4y ago

It's not that complicated nor is it a Python specific issue. This is a good article on the topic: https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim... A simplistic summary is that you always decode to unicode and always encode to bytes. Another way to think about it is that Unicode is an idealized character model that is reified into a byte encoding, usually UTF-8, for practical usage. It seems to me that this is something a lot of developers don't think too much about until they run into some obscure encoding/decoding issue, but it's kind of a fundamental aspect of programming that's worth spending some time grokking.

andrewstuart 4y ago
It’s not that I don’t understand it. The problem is that encode and decode are directional constructs that change depending on context. That’s a poor UI in Python.
- maxbond 4y ago
  Genuinely curious, what would you like it to look like?
  - aliqot 4y ago
    golang has a wonderful solution
    - maxbond 4y ago
      Would you care to elaborate on what it is and why it's better for the benefit of those of us who don't know Go? It's tricky to look up a different solution you aren't familiar since the terminology is different. What I'm seeing currently doesn't really look that different to Python, but with different names for the types and such. Taking a stab at it; This looks simple, and runes are great (similar to Python): https://golangdocs.com/strings-in-golang https://golangdocs.com/strings-in-golang But something that bothers me is that converting an array of bytes to a string looks to implicitly be ASCII, or possibly UTF8. This document doesn't say. This document acknowledges encodings and clarifies that literals are UTF8, but makes no mention of byte arrays: https://go.dev/blog/strings https://go.dev/blog/strings I've poked through as many SO questions as I care to and am not clear on how to use different encodings. Looks to me like this is a difference in philosophy; Python would say, explicit is better then implicit, and I'm inclined to agree. If I'm not understanding properly I'm happy to be corrected.
  - magicalhippo 4y ago
    I don't have skin in the game, but at least the .Net way is quite explicit, which I like: byte[] unicodeBytes = Encoding.UTF8.GetBytes(inputString); // Perform the conversion from one encoding to the other. byte[] asciiBytes = Encoding.Convert(Encoding.UTF8, Encoding.ASCII, unicodeBytes); string outputString = Encoding.ASCII.GetString(asciiBytes); Adapted from https://learn.microsoft.com/en-us/dotnet/api/system.text.encoding?view=net-7.0 https://learn.microsoft.com/en-us/dotnet/api/system.text.enc...