5 ms·

Maybe the errors in the article are more a statement of how complicated and improperly communicated Unicode was... and mostly still is! While I think I understa

by jordigh 10y ago

Maybe the errors in the article are more a statement of how complicated and improperly communicated Unicode was... and mostly still is! While I think I understand most of how UTF-8 works, I still have to read and re-read how codepoints and planes and encodings and decodings work together. It's a pretty complicated beast that could very easily be misunderstood when it was less popular than it is now. It's still widely misunderstood today.

TazeTSchnitzel 10y ago
I'm not sure that's fair, Unicode's encodings are pretty straightforward, particularly compared to some other character sets. Most of the complexity comes above the encoding level.
- 0x0 10y ago
  It also doesn't help that the classic LAMP stack has very confusing defaults and badly named functions: * PHP has functions named "utf8_encode()" and "utf8_decode()", when they should have been called "latin1_to_utf8_transcode()" and "utf8_to_latin1_transcode()" * MySQL for the longest time used latin1 as a default character set, then introduced an insufficient character set called "utf8" which only allows up to 3 bytes, not enough for all possible utf8 encoded codepoints, then introduced a proper implementation called "utf8mb4". * mysql connectors and client libraries often default their "client character set" setting to latin1, causing "silent" transcodes against the "server character set" and table column character sets. Also, because their "latin1" charset is more or less a binary-safe encoding, it is very easy to get double latin1-to-utf8 transcoded data in the database, something that often goes by unnoticed as long as data is merely received-inserted-selected-output to a browser, until you start to work on substrings or case insensitive searches etc. * In Java, there are tons of methods that work on the boundary between bytes and characters that allows not specifying an encoding, which then silenty falls back to an almost randomly set system encoding * Many languages such as Java, JavaScript and the unicode variants of win32 were unfortunately designed at a time where unicode characters could fit into 16bits, with the devastating result that the data type "char" is too small to store a single unicode character. It also plays hell on substring indexing. In short, the APIs are stacked against the beginning programmer and doesn't make it obvious that when you go from working with abstract "characters" to byte streams, there is ALWAYS an encoding involved.
  - jordigh 10y ago
    Does any programming language get Unicode right all the way? I thought Python did it mostly correctly, but for example with the composing characters, I would argue that it gets it wrong if you try to reverse a Unicode string.
    - 0x0 10y ago
      I've had the least trouble when using Apple's Objective-C (NSString), and Microsoft's C# - these two at least make you take conscious decisions when transcoding to bytes.
    - deathanatos 10y ago
      My basic litmus test for "does this language support Unicode" is, "does iterating over a string get me code points?"¹ Rust, and recent versions of Python 3 (but not early versions of Python 3, and definitely not 2…) pass this test. I believe that all of JavaScript, Java, C#, C, C++ … all fail. (Frankly, I'm not sure anything in that list even has built-in functionality in the standard library for doing code-point iteration. You have to more or less write it yourself. I think C# comes the closest, by having some Unicode utility functions that make the job easier, but still doesn't directly let you do it.) ¹Code units are almost always, in my experience, the wrong layer to work at. One might argue that code points are still too low level, but this is a basic litmus test (I don't disagree that code points are often wrong, it's mostly a matter of what can I actually get from a language). > try to reverse a Unicode string. A good example of where even code points don't suffice.
      - pjmlp 10y ago
        C# is built on top of what Windows offers, so I imagine it matches a mix of DBCS and Unicode APIs.
        int_19h 10y ago
        All strings in C# are UTF-16. However, it exposes the encoding directly as a sequence of 16-bit ints. In other words, if you iterate over a string or index it, you're getting those, and not codepoints (i.e. it doesn't account for surrogate pairs). Note that this only applies to iteration and indexing. All string functions do understand surrogates properly. On the other hand, C# (or rather .NET) has a way to iterate over text elements in a string. This is one level higher than code points, in that it folds combining characters: https://msdn.microsoft.com/en-us/library/system.globalization.stringinfo(v=vs.110).aspx https://msdn.microsoft.com/en-us/library/system.globalizatio...
- Dylan16807 10y ago
  Right. The encoding is just figuring out how many bits you need for a number and putting those bits in a template. With UTF-8 having four templates and UTF-16 having two.
syncsynchalt 10y ago
Hello, sorry for the incoming information dump! You are probably familiar with all of this, but I've still typed it out for any passing interested reader. I've listed my responses in decreasing importance, feel free to stop after the next paragraph: Unicode doesn't have to be complex; it is simply a single map of characters (such as "a" or "ﷺ") to a codepoint (such as "97" or "65018"). "Encodings": Encodings predate Unicode, and we had to deal with encoding before Unicode was ever invented (eg. "is this text in iso-8859-1, or is it koi8-r?"). Unicode has actually simplified the problem of encodings, for two reasons: 1) it's increasingly likely that any text sent over the internet is in the UTF-8 transformation of Unicode. 2) there is a mapping for any encoding into a Unicode transformation such as UTF-16 or UTF-8, therefore you only need three routines for text work: transform_encoding_to_unicode, process_text_as_unicode, transform_unicode_to_encoding. "Planes": Planes are an aspect of Unicode that you may not need to concern yourself with, if you work in UTF-8 (as most Unix software does). However if you're in the world of Windows, Java, or Javascript (which use UTF-16 instead), then you may need to know that characters in planes other than 0 are encoded as 4 bytes instead of 2 bytes. Hopefully the tools available in your language of choice hide this implementation detail from you! Misc: Beyond that it's true that there's an infinitely complex system of rabbit holes (think normalization, hyphenation, character ordering, right-to-left, ligature decomposition, and so on) that you can delve into when learning i18n, however Unicode wasn't the cause of them; they all existed before Unicode was invented! They're simply the messy but rewarding aspect of dealing with the written word.

by jordigh 10y ago

TazeTSchnitzel 10y ago

I'm not sure that's fair, Unicode's encodings are pretty straightforward, particularly compared to some other character sets. Most of the complexity comes above the encoding level.

0x0 10y ago
It also doesn't help that the classic LAMP stack has very confusing defaults and badly named functions: * PHP has functions named "utf8_encode()" and "utf8_decode()", when they should have been called "latin1_to_utf8_transcode()" and "utf8_to_latin1_transcode()" * MySQL for the longest time used latin1 as a default character set, then introduced an insufficient character set called "utf8" which only allows up to 3 bytes, not enough for all possible utf8 encoded codepoints, then introduced a proper implementation called "utf8mb4". * mysql connectors and client libraries often default their "client character set" setting to latin1, causing "silent" transcodes against the "server character set" and table column character sets. Also, because their "latin1" charset is more or less a binary-safe encoding, it is very easy to get double latin1-to-utf8 transcoded data in the database, something that often goes by unnoticed as long as data is merely received-inserted-selected-output to a browser, until you start to work on substrings or case insensitive searches etc. * In Java, there are tons of methods that work on the boundary between bytes and characters that allows not specifying an encoding, which then silenty falls back to an almost randomly set system encoding * Many languages such as Java, JavaScript and the unicode variants of win32 were unfortunately designed at a time where unicode characters could fit into 16bits, with the devastating result that the data type "char" is too small to store a single unicode character. It also plays hell on substring indexing. In short, the APIs are stacked against the beginning programmer and doesn't make it obvious that when you go from working with abstract "characters" to byte streams, there is ALWAYS an encoding involved.
- jordigh 10y ago
  Does any programming language get Unicode right all the way? I thought Python did it mostly correctly, but for example with the composing characters, I would argue that it gets it wrong if you try to reverse a Unicode string.
  - 0x0 10y ago
    I've had the least trouble when using Apple's Objective-C (NSString), and Microsoft's C# - these two at least make you take conscious decisions when transcoding to bytes.
  - deathanatos 10y ago
    My basic litmus test for "does this language support Unicode" is, "does iterating over a string get me code points?"¹ Rust, and recent versions of Python 3 (but not early versions of Python 3, and definitely not 2…) pass this test. I believe that all of JavaScript, Java, C#, C, C++ … all fail. (Frankly, I'm not sure anything in that list even has built-in functionality in the standard library for doing code-point iteration. You have to more or less write it yourself. I think C# comes the closest, by having some Unicode utility functions that make the job easier, but still doesn't directly let you do it.) ¹Code units are almost always, in my experience, the wrong layer to work at. One might argue that code points are still too low level, but this is a basic litmus test (I don't disagree that code points are often wrong, it's mostly a matter of what can I actually get from a language). > try to reverse a Unicode string. A good example of where even code points don't suffice.
    - pjmlp 10y ago
      C# is built on top of what Windows offers, so I imagine it matches a mix of DBCS and Unicode APIs.
      - int_19h 10y ago
        All strings in C# are UTF-16. However, it exposes the encoding directly as a sequence of 16-bit ints. In other words, if you iterate over a string or index it, you're getting those, and not codepoints (i.e. it doesn't account for surrogate pairs). Note that this only applies to iteration and indexing. All string functions do understand surrogates properly. On the other hand, C# (or rather .NET) has a way to iterate over text elements in a string. This is one level higher than code points, in that it folds combining characters: https://msdn.microsoft.com/en-us/library/system.globalization.stringinfo(v=vs.110).aspx https://msdn.microsoft.com/en-us/library/system.globalizatio...
Dylan16807 10y ago
Right. The encoding is just figuring out how many bits you need for a number and putting those bits in a template. With UTF-8 having four templates and UTF-16 having two.

syncsynchalt 10y ago

Hello, sorry for the incoming information dump! You are probably familiar with all of this, but I've still typed it out for any passing interested reader. I've listed my responses in decreasing importance, feel free to stop after the next paragraph: Unicode doesn't have to be complex; it is simply a single map of characters (such as "a" or "ﷺ") to a codepoint (such as "97" or "65018"). "Encodings": Encodings predate Unicode, and we had to deal with encoding before Unicode was ever invented (eg. "is this text in iso-8859-1, or is it koi8-r?"). Unicode has actually simplified the problem of encodings, for two reasons: 1) it's increasingly likely that any text sent over the internet is in the UTF-8 transformation of Unicode. 2) there is a mapping for any encoding into a Unicode transformation such as UTF-16 or UTF-8, therefore you only need three routines for text work: transform_encoding_to_unicode, process_text_as_unicode, transform_unicode_to_encoding. "Planes": Planes are an aspect of Unicode that you may not need to concern yourself with, if you work in UTF-8 (as most Unix software does). However if you're in the world of Windows, Java, or Javascript (which use UTF-16 instead), then you may need to know that characters in planes other than 0 are encoded as 4 bytes instead of 2 bytes. Hopefully the tools available in your language of choice hide this implementation detail from you! Misc: Beyond that it's true that there's an infinitely complex system of rabbit holes (think normalization, hyphenation, character ordering, right-to-left, ligature decomposition, and so on) that you can delve into when learning i18n, however Unicode wasn't the cause of them; they all existed before Unicode was invented! They're simply the messy but rewarding aspect of dealing with the written word.