3 ms·
if all text is UTF-8, then most operations can be performed on bytes without worrying about any transformations whatsoever This is incorrect for anything other
by ubernostrum 10y ago
if all text is UTF-8, then most operations can be performed on bytes without worrying about any transformations whatsoever
This is incorrect for anything other than an ASCII-only world. In general, "just use UTF-8" is a shorthand for "just let me pretend non-ASCII doesn't exist", because it leads to people writing code that assumes one byte == one character.
By rubbing your nose in the difference between str and bytes, Python 3 complicates your life.
Having had a lot of experience with it, I would instead say Python 3 makes you think about things and find bugs at the appropriate time -- up-front during the development process -- rather than at an inappropriate time, like 3AM on a Sunday when your pager goes off because you trusted the "helpful" but wrong way of doing things.
- bubuga 10y ago> In general, "just use UTF-8" is a shorthand for "just let me pretend non-ASCII doesn't exist", because it leads to people writing code that assumes one byte == one character. Only if people writing code are entirely oblivious to what UTF-8 is. The most basic feature of UTF-8 is that it's a variable-length encoding scheme. If the main feature is it's variable-length, why would any programmer assume the length doesn't vary?
- ubernostrum 10y agoIf the main feature is it's variable-length, why would any programmer assume the length doesn't vary? Because UTF-8 is the Unicode encoding that looks just like ASCII, so long as the characters stay within the set supported by ASCII, so it lets them say they're doing "Unicode" while really just writing code to handle ASCII.
- tomjen3 10y agoI am a Dane - we have three letters in our alphabet that are not represented in ascii (æ, ø and å) so let me assure you I don't except text to be in ascii, but I do expect it to be in UTF-8 and will consider it a bug if it isn't.
- Too 10y agoBu åäöæø are still in the < 255 range so they only occupy one byte, as any other "normal" ascii character.