4 ms·

I am not sure whether these are dark corners of Unicode or just complex language support. When dealing with text, you need to learn and educate yourself about t

I am not sure whether these are dark corners of Unicode or just complex language support. When dealing with text, you need to learn and educate yourself about text. International text is much more complex than localized text, but if you intend to support that, learn it. Sure, frameworks exist that hide a lot of the complexity, but then you hit a specific bug/corner case where the framework fails you, and since you are not familiar with the intricacies of the language-specific problem, you are clueless how to proceed and fix it, whereas if you invested the time to learn about it, it would have been much easier. A similar (albeit rather simpler and more limited) problem is calendrical calculus, where people who have little to no grasp of how to perform correct date operations do complex calendar applications and fail spectacularly in some edge cases. Call me crazy, but if you are dealing with text, have some time set for research before you start your development.

masklinn 11y ago
> I am not sure whether these are dark corners of Unicode or just complex language support. Either, neither, both. Some are intrinsic to Unicode's purpose of encoding human text, others are accidents of Unicode history, yet others are design decisions which could have gone other ways (which may or may not have been better) > Call me crazy, but if you are dealing with text, have some time set for research before you start your development. The issues being most people will have a hard time justifying a year of linguistic and calligraphic study before the project gets to start (whether employed or independent) and most languages have "string"-manipulation facilities which are easy, obvious and wrong.
- LeoNatan25 11y ago
  I think a year is quite the exaggeration, but of course, it depends on the project. If you are developing a complex word processor, page layout or publishing software, you bet your ass you should devote a year and even more to get it right. In other projects, even take two-three weeks of research before development will do wonders to complement the existing frameworks which also assist you in development.
chipsy 11y ago
As you point out, many of these features are dependent on how well they're handled by the application developer. If Unicode were "just" about rendering pre-made glyphs consistently it would not be nearly so hard, but it aims to do more - sorting, capitalization, string length - and that's also why it tends to fall over in production systems. Nobody is ever going to get Unicode completely right, just a subset of it for the languages they've successfully localized. The simplest character encoding you could hope to work with is something like a single-line calculator or vending machine display - fixed-width, no line breaks, just the Arabic numerals and maybe a decimal point, some mathematical symbols, or a limited English alphabet to display "INSERT CASH". Any featureset above that produces issues. Just line breaks alone are responsible for all sorts of strange behaviors. I think it's a bit magical that we've managed to do so much with text given the starting situation. At each step - from early telegram encodings through the proliferation of emoji - the implementations had to codify a thing that was previously left open, develop rules around its use, etc. We've made language more systematic than it ever was in history, for the benefit of machines to parse and process it.
- ygra 11y ago
  Unicode is complex because language is complex. Before we had that complexity in Unicode there were a lot of languages and scripts that couldn't be represented accurately (or at all) in computers. Mixing scripts in one document was nigh impossible. I'm not sure that's a world we want back. It's funny though, how people are not even aware of the fact that different languages are different until they see something break with Unicode. Casing and collation rules have been language-specific before. It's just that before lots of software didn't even attempt to do it right. Again, a world I'd rather not have back again.
- frobozz 11y ago
  The decimal mark requires some localisation (see https://en.wikipedia.org/wiki/Decimal_mark#Countries_using_Arabic_numerals_with_decimal_comma https://en.wikipedia.org/wiki/Decimal_mark#Countries_using_A...). Let's restrict the calculator to whole numbers only.

masklinn 11y ago

> I am not sure whether these are dark corners of Unicode or just complex language support. Either, neither, both. Some are intrinsic to Unicode's purpose of encoding human text, others are accidents of Unicode history, yet others are design decisions which could have gone other ways (which may or may not have been better) > Call me crazy, but if you are dealing with text, have some time set for research before you start your development. The issues being most people will have a hard time justifying a year of linguistic and calligraphic study before the project gets to start (whether employed or independent) and most languages have "string"-manipulation facilities which are easy, obvious and wrong.

LeoNatan25 11y ago
I think a year is quite the exaggeration, but of course, it depends on the project. If you are developing a complex word processor, page layout or publishing software, you bet your ass you should devote a year and even more to get it right. In other projects, even take two-three weeks of research before development will do wonders to complement the existing frameworks which also assist you in development.

chipsy 11y ago

As you point out, many of these features are dependent on how well they're handled by the application developer. If Unicode were "just" about rendering pre-made glyphs consistently it would not be nearly so hard, but it aims to do more - sorting, capitalization, string length - and that's also why it tends to fall over in production systems. Nobody is ever going to get Unicode completely right, just a subset of it for the languages they've successfully localized. The simplest character encoding you could hope to work with is something like a single-line calculator or vending machine display - fixed-width, no line breaks, just the Arabic numerals and maybe a decimal point, some mathematical symbols, or a limited English alphabet to display "INSERT CASH". Any featureset above that produces issues. Just line breaks alone are responsible for all sorts of strange behaviors. I think it's a bit magical that we've managed to do so much with text given the starting situation. At each step - from early telegram encodings through the proliferation of emoji - the implementations had to codify a thing that was previously left open, develop rules around its use, etc. We've made language more systematic than it ever was in history, for the benefit of machines to parse and process it.

ygra 11y ago
Unicode is complex because language is complex. Before we had that complexity in Unicode there were a lot of languages and scripts that couldn't be represented accurately (or at all) in computers. Mixing scripts in one document was nigh impossible. I'm not sure that's a world we want back. It's funny though, how people are not even aware of the fact that different languages are different until they see something break with Unicode. Casing and collation rules have been language-specific before. It's just that before lots of software didn't even attempt to do it right. Again, a world I'd rather not have back again.
frobozz 11y ago
The decimal mark requires some localisation (see https://en.wikipedia.org/wiki/Decimal_mark#Countries_using_Arabic_numerals_with_decimal_comma https://en.wikipedia.org/wiki/Decimal_mark#Countries_using_A...). Let's restrict the calculator to whole numbers only.