4 ms·

Internally Python holds a string as an array of uint32. A utf-8 representation is created on demand from it (and cached). So pansa2 is basically correct [^1].

Internally Python holds a string as an array of uint32. A utf-8 representation is created on demand from it (and cached). So pansa2 is basically correct [^1]. IMO, while this may not be optimal, it's far better than the more arcane choice made by other systems. For example, due to reasons only Microsoft can understand, Windows is stuck with UTF-16. [1] Actually it's more intelligent. For example, Python automatically uses uint8 instead of uint32 for ASCII strings.

deleted 7mo ago
[deleted]
nslsm 7mo ago
Read first paragraph here https://devblogs.microsoft.com/oldnewthing/20190830-00/?p=102823 https://devblogs.microsoft.com/oldnewthing/20190830-00/?p=10...
zahlman 7mo ago
There is no caching of a "utf-8 representation". You may check for example: >>> x = '日本語'*100000000 >>> import time >>> t = time.time(); y = x.encode(); time.time() - t # takes nontrivial time >>> t = time.time(); y = x.encode(); time.time() - t # not cached; not any faster Generally, the only reason this would happen implicitly is for I/O; actual operations on the string operate directly on the internal representation. Python uses either 8, 16 or 32 bits per character according to the maximum code point found in the string; uint8 is thus used for all strings representable in Latin-1, not just "ASCII". (It does have other optimizations for ASCII strings.) The reason for Windows being stuck with UTF-16 is quite easy to understand: backwards compatibility. Those APIs were introduced before there supplementary Unicode planes, such that "UTF-16" could be equated with UCS-2; then the surrogate-pair logic was bolted on top of that. Basically the same thing that happened in Java.
- cloudbonsai 7mo ago
  > There is no caching of a "utf-8 representation". No there certainly is. This is documented in the official API documentation: UTF-8 representation is created on demand and cached in the Unicode object. https://docs.python.org/3/c-api/unicode.html#unicode-objects In particular, Python's Unicode object (PyUnicodeObject) contains a field named utf8. This field is populated when PyUnicode_AsUTF8AndSize() is first called and reused thereafter. You can check the exact code I'm talking about here: https://github.com/python/cpython/blob/main/Objects/unicodeobject.c#L4081-L4117 https://github.com/python/cpython/blob/main/Objects/unicodeo... Is it clear enough?
  - zahlman 7mo ago
    The C API may provide for it, but I'm not seeing a way to access that from Python. This sort of thing is provided for people writing C extensions who need to interface to other C code. (And the code search seems to be broken; it can't find me the definition of `unicode_fill_utf8` although I'm sure it's obvious enough.)

deleted 7mo ago

[deleted]

nslsm 7mo ago

Read first paragraph here https://devblogs.microsoft.com/oldnewthing/20190830-00/?p=102823 https://devblogs.microsoft.com/oldnewthing/20190830-00/?p=10...

zahlman 7mo ago

There is no caching of a "utf-8 representation". You may check for example: >>> x = '日本語'*100000000 >>> import time >>> t = time.time(); y = x.encode(); time.time() - t # takes nontrivial time >>> t = time.time(); y = x.encode(); time.time() - t # not cached; not any faster Generally, the only reason this would happen implicitly is for I/O; actual operations on the string operate directly on the internal representation. Python uses either 8, 16 or 32 bits per character according to the maximum code point found in the string; uint8 is thus used for all strings representable in Latin-1, not just "ASCII". (It does have other optimizations for ASCII strings.) The reason for Windows being stuck with UTF-16 is quite easy to understand: backwards compatibility. Those APIs were introduced before there supplementary Unicode planes, such that "UTF-16" could be equated with UCS-2; then the surrogate-pair logic was bolted on top of that. Basically the same thing that happened in Java.

cloudbonsai 7mo ago
> There is no caching of a "utf-8 representation". No there certainly is. This is documented in the official API documentation: UTF-8 representation is created on demand and cached in the Unicode object. https://docs.python.org/3/c-api/unicode.html#unicode-objects In particular, Python's Unicode object (PyUnicodeObject) contains a field named utf8. This field is populated when PyUnicode_AsUTF8AndSize() is first called and reused thereafter. You can check the exact code I'm talking about here: https://github.com/python/cpython/blob/main/Objects/unicodeobject.c#L4081-L4117 https://github.com/python/cpython/blob/main/Objects/unicodeo... Is it clear enough?
- zahlman 7mo ago
  The C API may provide for it, but I'm not seeing a way to access that from Python. This sort of thing is provided for people writing C extensions who need to interface to other C code. (And the code search seems to be broken; it can't find me the definition of `unicode_fill_utf8` although I'm sure it's obvious enough.)