2 ms·
The problem isn't access to individual codepoints, it's mixing strings with different encodings without any way of tracking the encoding or knowing that you're
by nostrademons 13y ago
The problem isn't access to individual codepoints, it's mixing strings with different encodings without any way of tracking the encoding or knowing that you're doing something wrong.
UTF-8 everywhere works great when you can enforce it. On the level of an individual project, you can enforce it. On the level of a language ecosystem, you can't, and you need to. Otherwises you end up with some libraries who assume their internals are UTF-8 encoded strings, some libraries that assume they are ASCII strings, some libraries that assume the caller is handling encoding issues and will take care of ensuring that it's all UTF-8, some that make no assumptions at all and carry around the encoding everywhere, and some that just haven't thought about the problem and silently break when you use them with data that came from other libraries.
It's interesting that Go and Java - both languages with mature Unicode handling - still have a distinction between uninterpreted bytes and UTF-8 or UCS-2 text. In Go, you have separate []bytes and string types, even though Go strings are just UTF-8 byte sequences. In Java, you have byte[] and String. The problem is not a technical one of how to represent strings, it's a social one of how to get all the library authors on a language to agree on a convention of how to handle encoding.
- geocar 13y ago> it's a social one of how to get all the library authors on a language to agree on a convention of how to handle encoding. Ok, but the solution isn't to make everyone rewrite their libraries by holding the future ransom if they do not. It could have been real simple: let python track encoding when it's known and throw an exception if they mix oldstr(unknown) with newstr(py3). In py3 modules newstr is str, and in py2 modules oldstr is str. Now you only have to fix the bits where they mix and the programmer can always choose to fix it in the new code.