3 ms·

> it uses code point semantics, which is almost worse than UTF-16 semantics What's so bad about storing a unicode string as a series of codepoints? UTF-8 is al

> it uses code point semantics, which is almost worse than UTF-16 semantics What's so bad about storing a unicode string as a series of codepoints? UTF-8 is also a series of codepoints in essence. > When you look at newer languages, they almost always choose something more sensible, and UTF-8 is almost always at the heart of it. I still think that UTF-8 for memory is a bad idea. For example, it wastes a lot of bits to make sure that a string can be decoded starting from an arbitrary byte offset. This is generally a desirable property for files and some networking applications, but it's an absolute waste of space when used in memory.

chrismorgan 3y ago
Code point semantics, not scalar value semantics which is what Unicode strings are supposed to be. That is, a Python string can contain surrogate code points, which cannot be encoded in any UTF-*: >>> '\udead'.encode('utf-8') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeEncodeError: 'utf-8' codec can't encode character '\udead' in position 0: surrogates not allowed >>> '\udead'.encode('utf-16') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeEncodeError: 'utf-16' codec can't encode character '\udead' in position 0: surrogates not allowed (UTF-8 is a sequence of 8-bit code units, representing a sequence of Unicode scalar values.) —⁂— > For example, it wastes a lot of bits to make sure that a string can be decoded starting from an arbitrary byte offset. If you ditch self-synchronisability and ASCII purity of extension by ditching the first two bits (“1-0”) on continuation bytes, you can encode some more briefly: roughly, U+0800–U+1FFF go from three-byte to two-byte, and U+10000–U+FFFFF go from four-byte to three-byte, leaving just U+100000–10FFFF as four-byte. But honestly the difference isn’t often that much (unlike UTF-16 → UTF-8, which roughly halves memory usage most of the time, commonly even saving when dealing with text that makes heavy use of other scripts that are more compact in UTF-16 than in UTF-8 since you commonly mix things in with ASCII markup), and if you’re really trying to shave memory, other techniques like compression are far more effective. I love microoptimisation (and spent a large number of hours on my Casio GFX-9850GB PLUS, where having only 30KB of memory encouraged learning to shave single bytes here and there!), but there’s also a lot to be said for consistency. Yes, some purposes could have an encoding that is more efficient than UTF-8. But doing this complicates matters quite a bit, and I think we’re better off with just UTF-8. Also I will note that the self-synchronising nature of UTF-8 is useful for some operations: you can scan through strings much more quickly that way. So it’s not pure waste as a memory representation.
- deleted 3y ago
  [deleted]

chrismorgan 3y ago

Code point semantics, not scalar value semantics which is what Unicode strings are supposed to be. That is, a Python string can contain surrogate code points, which cannot be encoded in any UTF-*: >>> '\udead'.encode('utf-8') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeEncodeError: 'utf-8' codec can't encode character '\udead' in position 0: surrogates not allowed >>> '\udead'.encode('utf-16') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeEncodeError: 'utf-16' codec can't encode character '\udead' in position 0: surrogates not allowed (UTF-8 is a sequence of 8-bit code units, representing a sequence of Unicode scalar values.) —⁂— > For example, it wastes a lot of bits to make sure that a string can be decoded starting from an arbitrary byte offset. If you ditch self-synchronisability and ASCII purity of extension by ditching the first two bits (“1-0”) on continuation bytes, you can encode some more briefly: roughly, U+0800–U+1FFF go from three-byte to two-byte, and U+10000–U+FFFFF go from four-byte to three-byte, leaving just U+100000–10FFFF as four-byte. But honestly the difference isn’t often that much (unlike UTF-16 → UTF-8, which roughly halves memory usage most of the time, commonly even saving when dealing with text that makes heavy use of other scripts that are more compact in UTF-16 than in UTF-8 since you commonly mix things in with ASCII markup), and if you’re really trying to shave memory, other techniques like compression are far more effective. I love microoptimisation (and spent a large number of hours on my Casio GFX-9850GB PLUS, where having only 30KB of memory encouraged learning to shave single bytes here and there!), but there’s also a lot to be said for consistency. Yes, some purposes could have an encoding that is more efficient than UTF-8. But doing this complicates matters quite a bit, and I think we’re better off with just UTF-8. Also I will note that the self-synchronising nature of UTF-8 is useful for some operations: you can scan through strings much more quickly that way. So it’s not pure waste as a memory representation.

deleted 3y ago
[deleted]