3 ms·
> Python 3's string types are internally ASCII (or Latin-1, I'm not sure), UTF-16, or UTF-32 depending on the contents of the string, but the API effectively as
by the_mitsuhiko 10y ago
> Python 3's string types are internally ASCII (or Latin-1, I'm not sure), UTF-16, or UTF-32 depending on the contents of the string, but the API effectively assumes that the encoding is UTF-32.
That is incorrect. The API assumes the string is O(1) indexable. It makes no assumptions or guarantees about the internal encoding.
> The main point is that you don't want to be passing around binary data internally and expecting people to decode charsets as needed--the knowledge of charsets will usually be dropped.
This is an incorrect blanket statement and I can give you plenty of examples where this is exactly what you will have to do because the knowledge of the encoding is not available until later. This is in fact a common problem that Python 3 has because it decodes to unicode so early that it had to come up with surrogate-escapes to roundtrip bytes in case the encoding it guessed/inferred was wrong.
- mattashii 10y agoHe's talking about CPython, the default interpreter/compiler for Python in many systems. CPython internally uses 1-byte representation if the maximum Unicode code point of its chars is 127 or less (therefore ASCII), 2-byte representation for values less than 32767 (thus utf-16, but only the first half-page), and for the rest it uses UTF-32, which means all lookups can be done in O(1). Assuming UTF-8 encoding for source files is not that strange, as it is getting more and more the default or preferred encoding for string-based I/O.
- the_mitsuhiko 10y agoI understand how CPython works however the external API makes no guarantees about any internal encoding. The implementation of the storage also changed many times.