4 ms·

The only thing you are forced to think about is whether you're working with strings or with bytes. Unicode is handled transparently (but you do need to know the

by 1st1 11y ago

The only thing you are forced to think about is whether you're working with strings or with bytes. Unicode is handled transparently (but you do need to know the encoding when you're decoding bytes). How it's stored internally is none of your concern.

Avernar 11y ago
Yes it is. Every conversion to and from the unicode type and every expansion of the unicode type from 1 to 2 to 4 bytes eats up processor time, cache/memory bandwidth and puts more pressure on the garbage collector. And I don't want to be forced to distinguish between strings and bytes. To me they are one and the same as I use UTF-8 for everything. Not once in my code do I need to encode/decode anything.
- mark-r 11y ago
  Most of us aren't so lucky. I really do wish we lived in a 100% UTF-8 world. But even if we did, I don't see how e.g. regular expressions would work.
  - Avernar 11y ago
    The world seems to be headed to UTF-8 for everything. That's why having Python3 force us away from it doesn't make sense to me. As for regular expressions, they work perfectly if your delimiters are in the ASCII range. If you're trying to match a single character then they don't work for Unicode even when using Python3's unicode type. The reason is that code points are not characters.
    - 1st1 11y ago
      UTF-8 is just an encoding. Unicode is a more general term. Python 3 allows you to convert bytes to unicode from any encoding, including utf-8 (which is default). Using utf-8 as the internal unicode representation is flawed too - index and length operations are slow. For regexes, \w matches any unicode word character (unless ASCII mode is on), just use it on strings. If you use it on bytes, then yes, you can only match bytes.
      - Avernar 11y ago
        > Using utf-8 as the internal unicode representation is flawed too - index and length operations are slow. And this perfectly illustrates what I posted in another comment elsewhere. Thinking that doing operations on code points (indexing and length) is the proper thing to do. You need to use a unicode library to work on graphemes and take into account grapheme clusters. > For regexes, \w matches any unicode word character From what I've read, \w just matches code points. I did find that the https://pypi.python.org/pypi/regex https://pypi.python.org/pypi/regex module supports \X which matches graphemes as per the unicode spec http://www.unicode.org/reports/tr29/ http://www.unicode.org/reports/tr29/.
- Avernar 11y ago
  Nice, down voted for someone not agreeing with me. I'd rather know what you though was wrong with what I wrote. I've always wanted to know how things are done internally. Makes me a better programmer. What I said about all this extra conversion does to performance is correct, especially when scaling up. And my choice to use UTF-8 internally is a valid way to do things. I make sure that what comes in from the outside world is UTF-8 (either by rejecting or converting) and then don't have to worry about it from that point on. The bulk of my code deals with outbound data so no need for me to do any encoding as it's already UTF-8.

by 1st1 11y ago

Avernar 11y ago

Yes it is. Every conversion to and from the unicode type and every expansion of the unicode type from 1 to 2 to 4 bytes eats up processor time, cache/memory bandwidth and puts more pressure on the garbage collector. And I don't want to be forced to distinguish between strings and bytes. To me they are one and the same as I use UTF-8 for everything. Not once in my code do I need to encode/decode anything.

mark-r 11y ago
Most of us aren't so lucky. I really do wish we lived in a 100% UTF-8 world. But even if we did, I don't see how e.g. regular expressions would work.
- Avernar 11y ago
  The world seems to be headed to UTF-8 for everything. That's why having Python3 force us away from it doesn't make sense to me. As for regular expressions, they work perfectly if your delimiters are in the ASCII range. If you're trying to match a single character then they don't work for Unicode even when using Python3's unicode type. The reason is that code points are not characters.
  - 1st1 11y ago
    UTF-8 is just an encoding. Unicode is a more general term. Python 3 allows you to convert bytes to unicode from any encoding, including utf-8 (which is default). Using utf-8 as the internal unicode representation is flawed too - index and length operations are slow. For regexes, \w matches any unicode word character (unless ASCII mode is on), just use it on strings. If you use it on bytes, then yes, you can only match bytes.
    - Avernar 11y ago
      > Using utf-8 as the internal unicode representation is flawed too - index and length operations are slow. And this perfectly illustrates what I posted in another comment elsewhere. Thinking that doing operations on code points (indexing and length) is the proper thing to do. You need to use a unicode library to work on graphemes and take into account grapheme clusters. > For regexes, \w matches any unicode word character From what I've read, \w just matches code points. I did find that the https://pypi.python.org/pypi/regex https://pypi.python.org/pypi/regex module supports \X which matches graphemes as per the unicode spec http://www.unicode.org/reports/tr29/ http://www.unicode.org/reports/tr29/.
Avernar 11y ago
Nice, down voted for someone not agreeing with me. I'd rather know what you though was wrong with what I wrote. I've always wanted to know how things are done internally. Makes me a better programmer. What I said about all this extra conversion does to performance is correct, especially when scaling up. And my choice to use UTF-8 internally is a valid way to do things. I make sure that what comes in from the outside world is UTF-8 (either by rejecting or converting) and then don't have to worry about it from that point on. The bulk of my code deals with outbound data so no need for me to do any encoding as it's already UTF-8.