5 ms·

> One of the big ironies of all this is how well C has stood the test of time. C strings interoperate beautifully with UTF8 I wouldn't say that. I'd rather cl

> One of the big ironies of all this is how well C has stood the test of time. C strings interoperate beautifully with UTF8 I wouldn't say that. I'd rather claim that UTF-8 has been engineered to be ASCII pass-through compatible because of the lacking Unicode support in most C-codebases out there. I mean... If you compare Unicode support in platforms and OSes where this was clearly a thought (Windows, C#, Java) to platforms with the more naive approach (Linux, C, PHP, etc), you will see a very clear picture of which side has the most unicode bugs and encoding errors. And I say that as a Linux-guy. C is terrible for text-processing. UTF-8 was designed as it was because of that. Trying to paint C as the best choice here because someone found a working solution they could also apply to terrible C-code is clearly seeing things backwards.

yoklov 8y ago
While I agree that it's not really a measure of C standing the test of time, I kind of doubt that an alternate universe where 'C interop' / 'ASCII compatibility' weren't a design consideration, we'd have ended up with something much better (or TBH much different) than UTF-8.
the_mitsuhiko 8y ago
> If you compare Unicode support in platforms and OSes where this was clearly a thought (Windows, C#, Java) Their insistence on UCS2 however gave us the worst outcome of all. We have forever lost all north of 23 bits for characters due to UTF16. The result of this misdesign creates bigger problem than C’s char type.
- mikeash 8y ago
  UCS-2 was all there was at the time, though. Blame Unicode for thinking 65,536 code points would be enough.
  - gpderetta 8y ago
    That's not a very good excuse though: UTF-8 was invented, by Thompson and Pike, specifically because existing solutions were not deemed adequate for their needs. "Reasonable people adapt themselves to the world. Unreasonable people attempt to adapt the world to themselves. All progress, therefore, depends on unreasonable people"
    - mikeash 8y ago
      When Java and friends were being designed, it looked like 16 bits would be adequate. They didn’t insist on UCS-2 over something better, they chose what looked like the best option at the time. It later turned out not to be. UTF-16 was created to fix that problem without having to completely redo their strings from scratch. Likewise, C was created in a time when it looked like 7 or 8 bits would suffice. That turned out to be wrong. UTF-8 was invented to solve that problem without redoing C strings from scratch.
slavik81 8y ago
Windows is the platform where I see most of my Unicode problems. Here's an example of one of them: hello.py: print("こ") Ubuntu: $ python3 --version Python 3.5.2 $ python3 hello.py こ $ python3 hello.py > out.txt $ cat out.txt こ Windows: C:\Users\u\tmp>python --version Python 3.7.1 C:\Users\u\tmp>python hello.py ? C:\Users\u\tmp>python hello.py > out.txt Traceback (most recent call last): File "hello.py", line 1, in <module> print("?") File "C:\Users\u\AppData\Local\Programs\Python\Python37-32\lib\encodings\ cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\u3053' in position 0: character maps to <undefined>
- sametmax 8y ago
  The cmd console is terrible. It has a limited set of unicode code point, and worse, an even more limited set of fonts support. The python unidecode pypi package is your best friend if you have to deal with it.
  - josteink 8y ago
    > The cmd console is terrible. Agreed. And it's not really as much a native Windows app, as it is an emulator for the "good old" times of MS-DOS, which has received the necessary adjustments to keep chugging along, but little else. Given that you probably have what is the single weakest link unicode-wise you can find anywhere in the entirety of the Windows universe. It's really a completely obsoleted, abandoned mess. The fact that we had to wait until Windows 10 before you could actually resize the window freely should tell you at what level you should put your expectations. If you want to live your life in the command-line on Windows (for python or whatever else), do yourself a favour and get any other terminal. There's really quite a few to choose from. CMDer, ConEmu and MinTTY springs to mind.
    - jacobush 8y ago
      What about this then? https://stackoverflow.com/a/47843552 https://stackoverflow.com/a/47843552 Claims unicode support is decent in Windows 8 and up.
jcelerier 8y ago
> I mean... If you compare Unicode support in platforms and OSes where this was clearly a thought (Windows, C#, Java) to platforms with the more naive approach (Linux, C, PHP, etc), you will see a very clear picture of which side has the most unicode bugs and encoding errors. so... windows ? unicode has always worked fine for me on linux, and has always been a complete pain in windows - as a software user and even worse as a developer. The only problem until ~ 6-7 years ago was libre unicode fonts with complete coverage but they are now pretty good.
TorKlingberg 8y ago
You are right that UTF-8 was created to pass safely through C code that isn't Unicode aware. But that turned out to be the right design everywhere. Every system that went with UTF-16 or UCS-2 is either carefully switching to UTF-8, or suffering from various encoding bugs and inefficiencies.
- quietbritishjim 8y ago
  It is not true that UTF8 is the right design "everywhere". Perhaps it is the right design when you have strings that are stored and later read by various programs, but it's not necessarily right for in-memory strings. Python 3 internally uses ASCII, UCS2 or UCS4 for its strings depending on which is most space efficient but still capable of representing the string. It can do that because Python strings are immutable* and because it is impossible for user code to see what the encoding is (to see the bytes of a string you must explicitly convert it, at which point you specify the destination encoding and Python ensures the translation is correct). If you join a UCS2 string with a UCS4 string then the result is automatically UCS4 and there's no way to tell from user code (except by memory usage)! There's a good reason for this: indexing into a UTF8 string takes O(n) time because you must parse all the bytes before that point in order to count the number of characters. Iterating over a string by index would either take O(n^2) time or you would have to use some sort of awkward string iterator. If you do not provide any way to index by character (rather than byte position) then I would argue that your data structure is not really a "string". I suppose an alternative implementation would be to use UTF-8 but have an auxiliary data structure that maps between character indices and byte positions. * Immutability matters because if you had a mutable string in ASCII and you changed one character to be a code point >127 then you'd need to copy the string before making the modification, which if possible at all would be O(n) for what you would expect to be O(1).
  - bradleyjg 8y ago
    I agree with much of what you said here, but 'character', and therefore indexing by character, isn't a well behaved concept in the unicode world. Normalization helps somewhat, though it's another expense at string creation time. However, even after normalization there are plenty of graphemes that don't have a precomposed characters. Our fundamental ideas about how a string should work and what kinds of things we might want to do with one are deeply influenced by the languages spoken by the pioneers of the computer age. Short of someone from a very different language tradition not being exposed to these implicit assumptions in their formative programming years I don't think we are ever going to know what a different path would look like.

yoklov 8y ago

While I agree that it's not really a measure of C standing the test of time, I kind of doubt that an alternate universe where 'C interop' / 'ASCII compatibility' weren't a design consideration, we'd have ended up with something much better (or TBH much different) than UTF-8.

the_mitsuhiko 8y ago

> If you compare Unicode support in platforms and OSes where this was clearly a thought (Windows, C#, Java) Their insistence on UCS2 however gave us the worst outcome of all. We have forever lost all north of 23 bits for characters due to UTF16. The result of this misdesign creates bigger problem than C’s char type.

mikeash 8y ago
UCS-2 was all there was at the time, though. Blame Unicode for thinking 65,536 code points would be enough.
- gpderetta 8y ago
  That's not a very good excuse though: UTF-8 was invented, by Thompson and Pike, specifically because existing solutions were not deemed adequate for their needs. "Reasonable people adapt themselves to the world. Unreasonable people attempt to adapt the world to themselves. All progress, therefore, depends on unreasonable people"
  - mikeash 8y ago
    When Java and friends were being designed, it looked like 16 bits would be adequate. They didn’t insist on UCS-2 over something better, they chose what looked like the best option at the time. It later turned out not to be. UTF-16 was created to fix that problem without having to completely redo their strings from scratch. Likewise, C was created in a time when it looked like 7 or 8 bits would suffice. That turned out to be wrong. UTF-8 was invented to solve that problem without redoing C strings from scratch.

slavik81 8y ago

Windows is the platform where I see most of my Unicode problems. Here's an example of one of them: hello.py: print("こ") Ubuntu: $ python3 --version Python 3.5.2 $ python3 hello.py こ $ python3 hello.py > out.txt $ cat out.txt こ Windows: C:\Users\u\tmp>python --version Python 3.7.1 C:\Users\u\tmp>python hello.py ? C:\Users\u\tmp>python hello.py > out.txt Traceback (most recent call last): File "hello.py", line 1, in <module> print("?") File "C:\Users\u\AppData\Local\Programs\Python\Python37-32\lib\encodings\ cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\u3053' in position 0: character maps to <undefined>

sametmax 8y ago
The cmd console is terrible. It has a limited set of unicode code point, and worse, an even more limited set of fonts support. The python unidecode pypi package is your best friend if you have to deal with it.
- josteink 8y ago
  > The cmd console is terrible. Agreed. And it's not really as much a native Windows app, as it is an emulator for the "good old" times of MS-DOS, which has received the necessary adjustments to keep chugging along, but little else. Given that you probably have what is the single weakest link unicode-wise you can find anywhere in the entirety of the Windows universe. It's really a completely obsoleted, abandoned mess. The fact that we had to wait until Windows 10 before you could actually resize the window freely should tell you at what level you should put your expectations. If you want to live your life in the command-line on Windows (for python or whatever else), do yourself a favour and get any other terminal. There's really quite a few to choose from. CMDer, ConEmu and MinTTY springs to mind.
  - jacobush 8y ago
    What about this then? https://stackoverflow.com/a/47843552 https://stackoverflow.com/a/47843552 Claims unicode support is decent in Windows 8 and up.

jcelerier 8y ago

> I mean... If you compare Unicode support in platforms and OSes where this was clearly a thought (Windows, C#, Java) to platforms with the more naive approach (Linux, C, PHP, etc), you will see a very clear picture of which side has the most unicode bugs and encoding errors. so... windows ? unicode has always worked fine for me on linux, and has always been a complete pain in windows - as a software user and even worse as a developer. The only problem until ~ 6-7 years ago was libre unicode fonts with complete coverage but they are now pretty good.

TorKlingberg 8y ago

You are right that UTF-8 was created to pass safely through C code that isn't Unicode aware. But that turned out to be the right design everywhere. Every system that went with UTF-16 or UCS-2 is either carefully switching to UTF-8, or suffering from various encoding bugs and inefficiencies.

quietbritishjim 8y ago
It is not true that UTF8 is the right design "everywhere". Perhaps it is the right design when you have strings that are stored and later read by various programs, but it's not necessarily right for in-memory strings. Python 3 internally uses ASCII, UCS2 or UCS4 for its strings depending on which is most space efficient but still capable of representing the string. It can do that because Python strings are immutable* and because it is impossible for user code to see what the encoding is (to see the bytes of a string you must explicitly convert it, at which point you specify the destination encoding and Python ensures the translation is correct). If you join a UCS2 string with a UCS4 string then the result is automatically UCS4 and there's no way to tell from user code (except by memory usage)! There's a good reason for this: indexing into a UTF8 string takes O(n) time because you must parse all the bytes before that point in order to count the number of characters. Iterating over a string by index would either take O(n^2) time or you would have to use some sort of awkward string iterator. If you do not provide any way to index by character (rather than byte position) then I would argue that your data structure is not really a "string". I suppose an alternative implementation would be to use UTF-8 but have an auxiliary data structure that maps between character indices and byte positions. * Immutability matters because if you had a mutable string in ASCII and you changed one character to be a code point >127 then you'd need to copy the string before making the modification, which if possible at all would be O(n) for what you would expect to be O(1).
- bradleyjg 8y ago
  I agree with much of what you said here, but 'character', and therefore indexing by character, isn't a well behaved concept in the unicode world. Normalization helps somewhat, though it's another expense at string creation time. However, even after normalization there are plenty of graphemes that don't have a precomposed characters. Our fundamental ideas about how a string should work and what kinds of things we might want to do with one are deeply influenced by the languages spoken by the pioneers of the computer age. Short of someone from a very different language tradition not being exposed to these implicit assumptions in their formative programming years I don't think we are ever going to know what a different path would look like.