6 ms·

Nobody has a problem with encode/decode when talking about, say, json. You have some semantic object that you want to encode to some bytes representation, and s

by zeeboo 11y ago

Nobody has a problem with encode/decode when talking about, say, json. You have some semantic object that you want to encode to some bytes representation, and some bytes representation that you want to decode into some semantic object. The confusions is around people viewing byte strings implicitly as ascii codepoints, and not fully understanding what the things they're looking at actually are.

wiz21 11y ago
I concur 100%. Most of the problems with unicode are like most problem with floats : tons of people don't get how it works (and the good reasons that make all of that tricky)
- est 11y ago
  99% encoding problems in Python 2 is because str(), the fundamental type converting function, is not safe. output = ''.join(str(x) for x in some_list) Can you spot the danger here? Can you imagine how many hidden str() calls in every lib/modules?
coldtea 11y ago
>Nobody has a problem with encode/decode when talking about, say, json. That's maybe because JSON is by the standard supposed to be utf-8, always.
- aidos 11y ago
  I think you've misunderstood. json encoding is the process of converting native types to utf8 encoded data. The point of the comment is that everyone knows that you json encode native entities to the utf8 representation, and json decode from utf8 to native entities. The same is true of unicode. It's not ambiguous at all.
  - coldtea 11y ago
    >The same is true of unicode. It's not ambiguous at all. Only utf-8 is a very specific encoding, so converting to it is straightforward, whereas unicode is an umbrella term and the most ambiguous thing you can imagine. Even saying "the same is true of unicode" shows confusion, as utf-8 is itself unicode. As are numerous different encodings, and then you have normalization and other subtleties.
    - aidos 11y ago
      This is not my area of expertise, so my understanding may be incomplete. If some is "in Unicode" it's a list of codepoints - that is the "thing" to be encoded. I don't see that ambiguity. It's a list of numbers, it's not fluffy kittens in the network or something. UTF-8 is a way of transcribing that list of numbers as 1s and 0s. That is the encoding process. Once it's encoded to 1s and 0s, it's not anything. It's just 1s and 0s. One person's utf8 encoded 1s and 0s may be another person's audio signal. It doesn't really make sense to talk about the encoded data as being utf8. It's the encoded / decoding process that's utf8. Sure, it might be utf8 decodable. In terms of json, it's the same deal. You have the "thing", simple nested objects/lists/numbers in memory. It only makes sense to talk about json when you're talking about the process of transcribing those. In both cases you have a "thing" (I think of it as "real", but in a way it's an abstract concept - how does Python represent a dict, I have no idea), you can encode those things into a tangible list of 1s and 0s. That's why they're the same. And people aren't generally confused about the json encoding decoding process.

by zeeboo 11y ago

wiz21 11y ago

I concur 100%. Most of the problems with unicode are like most problem with floats : tons of people don't get how it works (and the good reasons that make all of that tricky)

est 11y ago
99% encoding problems in Python 2 is because str(), the fundamental type converting function, is not safe. output = ''.join(str(x) for x in some_list) Can you spot the danger here? Can you imagine how many hidden str() calls in every lib/modules?

coldtea 11y ago

>Nobody has a problem with encode/decode when talking about, say, json. That's maybe because JSON is by the standard supposed to be utf-8, always.

aidos 11y ago
I think you've misunderstood. json encoding is the process of converting native types to utf8 encoded data. The point of the comment is that everyone knows that you json encode native entities to the utf8 representation, and json decode from utf8 to native entities. The same is true of unicode. It's not ambiguous at all.
- coldtea 11y ago
  >The same is true of unicode. It's not ambiguous at all. Only utf-8 is a very specific encoding, so converting to it is straightforward, whereas unicode is an umbrella term and the most ambiguous thing you can imagine. Even saying "the same is true of unicode" shows confusion, as utf-8 is itself unicode. As are numerous different encodings, and then you have normalization and other subtleties.
  - aidos 11y ago
    This is not my area of expertise, so my understanding may be incomplete. If some is "in Unicode" it's a list of codepoints - that is the "thing" to be encoded. I don't see that ambiguity. It's a list of numbers, it's not fluffy kittens in the network or something. UTF-8 is a way of transcribing that list of numbers as 1s and 0s. That is the encoding process. Once it's encoded to 1s and 0s, it's not anything. It's just 1s and 0s. One person's utf8 encoded 1s and 0s may be another person's audio signal. It doesn't really make sense to talk about the encoded data as being utf8. It's the encoded / decoding process that's utf8. Sure, it might be utf8 decodable. In terms of json, it's the same deal. You have the "thing", simple nested objects/lists/numbers in memory. It only makes sense to talk about json when you're talking about the process of transcribing those. In both cases you have a "thing" (I think of it as "real", but in a way it's an abstract concept - how does Python represent a dict, I have no idea), you can encode those things into a tangible list of 1s and 0s. That's why they're the same. And people aren't generally confused about the json encoding decoding process.