4 ms·
It makes sense if you remove "where a character"... Or change it to "where a character is a Unicode codepoint". This is basically saying Python uses a fixed-le
by xiaq 11y ago
It makes sense if you remove "where a character"... Or change it to "where a character is a Unicode codepoint".
This is basically saying Python uses a fixed-length encoding (UTF-32) where each codepoint takes constant storage, as opposed to UTF-8 where the storage of a character depends on where it is in the Unicode plane. A useful analog, if you know C, is that Python always uses wchar_t[] where each element is a character, while rust uses char[] and then the elements are not characters but UTF-8 bytes.
There are obvious up- and downsides to both approaches.
EDIT: As suggested by two children, with Python 3.3+, the fixed-length encoding Python uses depends on the highest codepoint in the string and is often not UTF-32. With earlier versions there are "narrow" and "wide" builds which use UTF-16 and UTF-32 respectively.
- lucian1900 11y agoPython (at least 3) actually uses the narrowest encoding that can fit the string, so it'll rarely actually use UTF-32.
- xiaq 11y agoThanks, that is good to know.
- ubernostrum 11y agoOn Python 2.2 through 3.2, CPython could be compiled with "narrow" or "wide" Unicode strings. On a "narrow" build, Unicode strings internally were represented using two-byte characters and surrogate pairs. On a "wide" build, Unicode strings were represented internally using four-byte characters. On Python 3.3+, the distinction is gone, and Python uses either latin-1, UCS-2 or UCS-4 depending on the highest codepoint in the string (i.e., a string containing only codepoints in latin-1 will be stored internally as latin-1; a string containing at least one codepoint outside the BMP will be stored internally as UCS-4, etc.).
- masklinn 11y agoIt's still fixed-length codepoint encoding, even if the specific encoding depends on content.