4 ms·
> Recent versions only convert to UTF-32 when a wide character is present It's non-BMP characters that trigger the conversion to 4 bytes per codepoint. Unfort
by Avernar 11y ago
> Recent versions only convert to UTF-32 when a wide character is present
It's non-BMP characters that trigger the conversion to 4 bytes per codepoint. Unfortunately Emoji are non-BMP and are super common.
- rspeer 11y agoIf you're dealing with emoji -- and yes, they are incredibly common -- you'd better not use any Python before 3.3. Non-BMP characters in 2.x-3.2 give you implementation-specific behavior. It can vary from one system to the next. Some string methods fail on "narrow builds" of Python (the ones that store their characters in UCS-2), while others return different results than they would on a "wide build".
- Avernar 11y agoNot an issue because I don't use the Unicode type. All my data comes in as UTF-8 on which I do a quick validation check. Then it's just handled as bytes all the way to and from my database.
- rspeer 11y agoSure, that is an option that's always available. You don't have to worry if Python's string methods are wrong if you're opting not to use them.