36 ms·
What does Unicode support actually mean?
- NullPrefix 6y ago>characters that display as white space but actually are not because they represent a nordic rune carved on the side of the stone not visible from the front Can't tell if serious or not
- BiteCode_dev 6y agoSame. At first I though "this must be a joke", but then I though, "well, it can make sense in some contexts". And now I don't know if I'm fool or not.
- GlitchMr 6y agoI don't recognize that one, but there are in fact characters that render as whitespace but aren't, for example U+2800 (BRAILLE PATTERN BLANK).
- a1369209993 6y agoWhich is particularly egregious, since in actual braille, that is a whitespace character (specifically, a space with no dots in it represents the character... space! 0x20).
- throwaway_pdp09 6y agoThere is considered to be semantics in individual glyphs. So IIRC there is a unicode minus sign and a unicode dash and even if they are visibly indistinguishable you're not supposed to mix them. I'm no unicode expert though. I doubt the unicode consortium is comprised of fools and nincompoops. Best assume they know what they're doing.
- a1369209993 6y agoI'm not claiming that BRAILLE PATTERN BLANK is the same character as SPACE (I don't particularly disagree, but that's not my point); I'm claiming that BRAILLE PATTERN BLANK is a whitespace character, just like SPACE or NEWLINE or U+2003 EM SPACE (pretty much any of the U+200xs, really), regardless of whether it's the same character.
- throwaway_pdp09 6y agoI exactly get your point. I don't know why either, but I'm assuming there's a reason.
- a1369209993 6y agoFair enough, but I'm assuming the reason, whatever it is, is at least as stupid as the reason why U+01F1 exists and is not encoded as 44 5A.
- throwaway_pdp09 6y agoWhy on earth would unicode U+01F1, with it's obvious hex encoding of 01F1, be encoded as hex value 44 5A (a Han character https://www.fileformat.info/info/unicode/char/445a/index.htm https://www.fileformat.info/info/unicode/char/445a/index.htm)?
- wnoise 6y agoHe meant U+44 U+5A. The ASCII letters D and Z.
- throwaway_pdp09 6y agoThen in that case because some apparent pairs of letters are in fact a single letter in some languages. Per the DZ wiki page someone else has given, DZ is a distinct single letter in hungarian and slovak at least. Double letters being a single letter isn't rare, such as ll and dd in welsh. In a strange sense it arguably occurs in english in th and th, which I could argue are double-letter representations of single letters, which were originally eth and thorn https://en.wikipedia.org/wiki/Eth https://en.wikipedia.org/wiki/Eth https://en.wikipedia.org/wiki/Thorn_%28letter%29 https://en.wikipedia.org/wiki/Thorn_%28letter%29
- LgWoodenBadger 6y agoThis is particularly annoying when trying to sanitize input in a language like Java, where String.trim() only removes space (maybe tab?) and the pre-defined Regex classes in Pattern define some of the whitespace characters, but not all. Horizontal whitespace characters [ \t\xA0\u1680\u180e\u2000-\u200a\u202f\u205f\u3000] Vertical whitespace characters [\n\x0B\f\r\x85\u2028\u2029] Whitespace characters [ \t\n\x0B\f\r]
- dane-pgp 6y agoIt sounds like it's inspired by the (very real) example of the Ogham Space Mark: https://www.youtube.com/watch?v=2yWWFLI5kFU https://www.youtube.com/watch?v=2yWWFLI5kFU
- zeveb 6y agoVery cool video, but it sounds like that space-that-isn't-a-space would also apply to a runic interpunct as well as to the Latin interpunct he mentions.
- sdflhasjd 6y ago> Unicode support in my experience seems to be one of those hand wavy things where most people respond to the question of “Do you support unicode?” with > > Yeah we support emojis, so yes we support unicode! And sometimes it's not even that simple, because things that say they support UTF-8 can't encode the full range. Looking at you, MySQL and "utf8mb3"
- doubleunplussed 6y agoEmoji have been a boon for unicode support. In order to support emoji, a lot of software ends up supporting unicode in general. I think I read that this was intentional by the unicode consortium - it is arguable that emoji don't belong in unicode, but is included to enhance adoption.
- sdflhasjd 6y agoUnfortunately, I think control of the Emoji code blocks by the Unicode Consortium stifles its potential.
- tom_mellior 6y agoCould you elaborate?
- sdflhasjd 6y agoIt has more limited usefulness outiside of North America and Japan when it comes to cultural or geographically limited characters, especially food, drink and clothing; The Consortium does recognise this, but with a measly 50+100 new emoji per year, it will be forever before we see this improved significantly.
- microtherion 6y agoWithout that control, you would be free to call custom emoji from the vasty deep, but without vendor support, that would not do you much good. I find it preferable to have a limited pipeline of additions, but with considerable vendor support.
- hibbelig 6y agoAnother thing OP does not mention is normalization: The German character ä can be represented as a single character in Unicode, but it can also be represented as the letter a followed by a combining diacritical mark. Both will be displayed the same way. I guess most Germans are not aware of these differences and would expect that Ctrl+F treats them the same.
- kergonath 6y agoNever mind digraphs or ligatures that can be one or two characters depending on the language.
- EForEndeavour 6y agoAny self-respecting Ctrl+F should match "ä" when the user searches for "a." Chrome and Firefox do.
- karatinversion 6y agoOn my firefox (68.9.0esr), a Ctrl+F for "The German character a" finds no results. Also, as a Finnish user, I would not be impressed by a search for "talli" (stables) matching "tälli" (blow), or "länteen" (westwards) matching "lanteen" ('of the hip').
- tom_mellior 6y ago> On my firefox (68.9.0esr), a Ctrl+F for "The German character a" finds no results. On mine (77.0.1) it finds the OP's "The German character ä" as well as yours. Conversely, searching for "The German character ä" finds the OP's original and your "a" version. There is a "match diacritics" button for enabling/disabling this behavior.
- earthboundkid 6y ago"Should match" above was too strong. "Should match for users with locale set to English" is right though. In English, theoretically you should write "café" and "résumé" but practically speaking, it doesn't matter and no one cares if you omit the accents. In the US, there's starting to be more push from people with ñ in their name to get support for the correct letter, but most government systems have zero support for anything but ASCII all capitals (sorry McWhoever).
- hibbelig 6y agoI understand that Han unification in Unicode has resulted in the same character encoding distinct glyphs, depending on the language in use. So there would be a Japanese Kanji that looks similar to, but distinct from, a Chinese character. And both would be encoded as the same character in Unicode. And that character would look one way when included in a Japanese document and another way when included in a Chinese document. If the document contains both Japanese and Chinese, and the character appears in both parts, would a Japanese user expect to find both occurrences when entering one of them?
- TorKlingberg 6y agoIdeally, the document format should specify which parts are in what language. Both MS Word and HTML can do that.
- lucio 6y agoThe idea of UNICODE was to avoid that.
- numpad0 6y agoWas. Whoever thought it a good idea to merge letters with similar shapes to same code points must’ve been drinking way too much ancient Chinese civilization kool-aid.
- jerf 6y agoI think they were trying to keep Unicode fitting into 16 bits. We later gave that up, but now it's kinda too late to go back and de-unify.
- dhosek 6y agoIIRC, ISO-10646 initially preserved each of the 16-bit national codings for Han characters (including three separate Chinese encodings). I don't know if that has been retained at all in later versions of the standard and a cursory reading of the relevant wikipedia pages is not informative.
- skissane 6y agoUnicode supports 150+ scripts which can be used to write thousands of languages. No matter how good your Unicode support, there is probably some obscure script for which you do the wrong thing. But, in the real world, most software (whether proprietary or open source) supports a limited set of languages, and you test it works correctly for all of them, and every now and again you might add support for a new language (due to business requirements, or, in the case of open source, because someone volunteers to do it)-at which point any issues specific to that language and its script are found and fixed.
- gumby 6y agoThe unicode consortium maintains open source software to handle a lot of hard cases (not just case folding but canonicalization, formatting, line breaking...). Doesn't magically make all your problems go away but takes away a lot of hard work that you're most likely to get wrong. http://site.icu-project.org/home http://site.icu-project.org/home
- hibbelig 6y agoMy take on this is that it's difficult to communicate the exact details of “Unicode support”, so perhaps it's alright to say that one supports Unicode, generally speaking, but Unicode is complicated and there are corners that aren't supported fully.
- cryptonector 6y agoMy take is that it is important to understand the basics: - UTFs - [canonical] equivalence (form, normalization) - [canonical] decomposition - [canonical] precomposition - case mapping - case-folding - localization - the reason for all this complexity (hint: it's not the UC's fault) You don't have to really grok the details, much less memorize them. It does not much more than TFA's ~1,500 words to explain the concepts.
- a1369209993 6y ago> - the reason for all this complexity > (hint: it's not the UC's fault) This is a common fallacy, that just because some of the complexity is unavoidable and irreducible, means that the UC hasn't made things much worse than they needed to be with things like code points (which are neither full characters (see eg U+308 combining umlaut) nor single characters (see eg U+1F1 the two letters D and Z)) or emoji. But, yes, with the possible exception of normalization, all of those are things you'd need to deal with in some fashion regardless of Unicode.
- cryptonector 6y agoEven combining codepoints are a thing that predates Unicode and simplifies a number of things, so they were a very useful thing to have in Unicode. First off, diacritical marks have generally been just that: marks that get added to other characters. Notionally that is a great deal like combining codepoints. For example, typewriter fonts, and later ASCII, were designed to have useful overstrike combinations. So you could type á as a<BS>', both in typewriters and in ASCII! Yes, ASCII was a variable-length, multi-byte internationalized encoding for Latin scripts. For example, this is why Spanish used to drop diacritical marks when up-casing: because there was no room in those typewriter fonts for accents on upper-case letters, though nowadays we have much more luxurious fonts, and so you see marks kept on up-cased Spanish a lot more than one used to. Second, diacritical marks as combining marks adds a great deal of flexibility and a fair degree of future-proofing to the system. This is a restatement of the first point, I know. Read on before telling me that flexibility is nice but expensive. Third, it is much simpler to close canonical composition to new compositions by having canonical decomposition than it is to keep adding pre-compositions for every sensible and new character. Part of this is that we have a fairly limited codepoint codespace while pre-compositions are essentially a cartesian explosion, and cartesian explosion means consuming codepoint codespace very quickly. Cartesian explosions complicate software as well (more on this below). This really gets back to the point about flexibility. Fourth, scripts and languages tend to have decomposition strongly built into their rules. It would complicate things to only let them have pre-composed characters! For example, Hangul is really a phonetic script, but is arranged to look syllabic. Now, clearly Hangul is a lot more like Latin in terms of semantics than like Hiragana, say, so you should want to process Hangul text as phonetic (letters) rather than syllabic, but the syllabic aspect of it can't be ignored completely. So Hangul gets precompositions (lots, thanks to cartesian explosion) but even in NFC Hangul is decomposed because that is much more useful for handling in software. Emoji are mostly like ideographic CJK gluphs, except with color. The color aspect is what truly makes emojis a new thing in human scripts, but even color can be a combining mark. Emoji clearly illustrates the fact that our scripts are still evolving. This is a critical thing to understand: our languages and scripts are not set in stone! You could say that there was no need to add emoji all you want, but they were getting added in much software, and users were using them -- eventually the UC was going to have to acknowledge their existence. Aside from emoji, which are truly an innovation (though, again, not the UC's), conceptual decomposition of characters had long been a part of scripts. It was there because it was useful. It's entirely natural, and a very good thing too, that it's there in Unicode as well. Decomposition, and therefore combining codepoints, was probably unavoidable. Regarding digraphs and ligatures, these too are part of our scripts, and we can't very well ignore them completely. This is not to say that the UC didn't make mistakes. Han unification was a mistake that resulted from not inventing UTF-8 sooner. Indeed, the UC did not even invent UTF-8. Some mistakes are going to be unavoidable, but decomposition and combining codepoints are absolutely not a mistake. Finally, we're not likely to ever replace Unicode -- not within the next few decades anyways. However much you might think Unicode is broken, however widespread, right, or wrong that perception might be, Unicode is an unavoidble fact of developers' lives. One might as well accept it as it is, understand it as it is, and move on.
- cryptonector 6y ago> Case folding can be done in one or two ways, simple and full Oh, it's worse than that. There can be locale specific tailorings of case folding rules. By the way, do not confuse case-folding mappings with case mappings for up- and down-casing. These are different in Unicode. Also, don't think this is Unicode's fault. This is the fault of humans for creating such rich scripts and languages and rules.
- kergonath 6y ago> Because cote, coté, côte and côté are all different words with the same upper-case representation COTE. The last bit is wrong, at least in French. The upper case representation of “côté” is “CÔTÉ”. It infuriates me to no end that Microsoft does not accept that letters in caps keep their accents. I can believe that it might be language-dependent, though. Also, a search engine that does not find “œuf” when you type “oe” is broken. Very interesting article, though. I wish more developer were aware of this.
- AlanYx 6y ago>It infuriates me to no end that Microsoft does not accept that letters in caps keep their accents. That behaviour is actually locale-dependant in Microsoft apps. Setting Microsoft Word to "French (Canada)" uses accented capitals, while "French (France)" does not. I imagine there's a lively internal debate about the latter. France's Académie Française actually says the behaviour you'd prefer is the proper one, but the local typographic culture is heavily influenced by a history of typewriter use where such accents weren't available.
- an_d_rew 6y agoHooray! Somebody who knows about the different capitalization rules between Québec and France! There are dozens of us! Dozens! ;-)
- VeejayRampay 6y agothere are no differences in this case though, you're supposed to accentuate when you capitalize in France as well
- ta17711771 6y agoHa. Maybe you guys should use the French Canadian search engine!
- kergonath 6y agoI must admit I’ve never tried to set it to Canadian French. I’m not surprised that the standards might be higher. Also, there’s a box to tick somewhere to set it to a sane behaviour. It’s just the default, which needs to be changed every time, which nobody bothers to do. There isn’t any debate really, in the sense that virtually nobody remotely interested in typesetting advocates for unaccented caps. All the french books on typography and spelling emphasise that accents change the meaning of words and that they should be kept, provided that you have a device that supports them. It’s just an urban legend amplified by stupid defaults in commonly-used applications. Right, I’m going to stop ranting now :)
- rini17 6y agoIs it turing complete yet? (as in, whether the primitive of comparing/collating two unicode strings can be used for arbitrary computation)
- js2 6y agoJust last week I noticed the description for the movie "Us" in the HBO Max app on iOS and AppleTV displayed all of the left-leaning and right-leaning quotes and double-quotes as question marks, but displayed ® correctly. My guess is something in their pipeline is using latin1 encoding[1]. Which is to say, just getting everything onto UTF-8 (or UTF-16 if you must) would be a huge win. [1] https://en.wikipedia.org/wiki/ISO/IEC_8859-1#Quotation_marks https://en.wikipedia.org/wiki/ISO/IEC_8859-1#Quotation_marks
- SAI_Peregrinus 6y agoThe idea that "case insensitivity" is a generally meaningful concept is a falsehood. It applies to a small subset of writing systems, and horribly breaks elsewhere.
- g-b-r 6y agoI don't think it's such a small subset...
- mcswell 6y agoIf you count writing systems, a small subset has an upper/lower distinction: Greek, Latin, and Cyrillic, and maybe a handful of others. Not Arabic, Chinese, Japanese, Devanagari, Tamil, Thai, Thaana, Bengali, Hebrew, or a bunch of other scripts. If on the other hand you count the languages that use writing systems with an upper/lower case distinction, then most use Latin, a much smaller set use Cyrillic; and the rest of the writing systems are used by one or two languages each, with the notable exception of the Arabic script. So most scripts lack an upper/lower case distinction, but probably most languages use a script that does have such a distinction.
- SAI_Peregrinus 6y agoExactly, and among those languages that have a case distinction quite a few have at least one edge case (like German ß vs ss vs ẞ) where the transformation isn't bijective. Or it's position-dependent. If you want case insensitivity, and you also want internationalization, you'd better hope you have a good library that handles the edge cases, or you'll get a bunch of errors. Case-sensitive is far easier to program.
- g-b-r 6y agoWhen dealing with case you rarely (if ever) need bijective transformations... Case-insensitive search/comparison requires a temporary mapping, not a permanent transformation; the German case is easily handled in Unicode's CaseFolding.txt by just mapping all three of them either to ss or to ß (depending on the implementer, if he prefers a length-invariant mapping or not). Position-dependent? I read about it, but the Unicode case folding algorithm doesn't support it, so I imagine it's not considered useful for case-insensitive comparison You will probably do want a good library when dealing with Unicode, but for this case thing it actually doesn't seem to be that complex to implement by hand... I don't doubt case-sensitive is far easier to program , but you usually program for some user's sake, not for your own pleasure :) And in most cases a normal user expects case-insensitive search
- amelius 6y agoIs anyone working on a decentralized version of Unicode? Currently, one organization decides what symbols/emoji we can use, and I think that's stifling our expression.
- justwalt 6y agoHow would you like to see things change?
- toast0 6y agoThere's a ton of recorded history of decentralized character encodings. Some people long for the days when a character was a byte and a byte was a character, but that model doesn't fit the world's languages. The unicode consortium has been historically inclusive enough to avoid alienating enough people to cause a revolt. They have defined ranges where they will not assign code points as private use areas, and you can use those for whatever symbols you need, but of course there's no mechanism to resolve disputes over use, and you would need your own method to determine which font was appropriate, and its likely to be challenging to use outside of applications you control.
- zzo38computer 6y agoWell, my own software tends not to use Unicode. Sometimes, it only uses ASCII. Sometimes, it doesn't care about character coding, as long as it is compatible with principle of extended ASCII. Sometimes, it is a combination of the two (such as ASCII only for commands, but comments can include any characters that are compatible with principle of extended ASCII). In one case (VGMCK), it does parse UTF-8 in some contexts (such as #TITLE commands), but the only thing it does with the decoded data is to convert it to UTF-16, since the output format (VGM) contains UTF-16 text. A UTF-8 byte order mark at the beginning of the file is not acceptable, though. But I have thought of some mechanisms to declare character sets and character mapping, to determine which font is appropriate, etc. Unicode is a valid choice, but even if you select Unicode as your character set of use, you must specify the language code and the Unicode version. You can also specify more than one mapping, and custom mappings, for example if you are using CSUR to write in Klingon and English, then you can declare both Unicode and CSUR, together with the relevant version numbers. I thought of a similar mechanism for declaring character sets and character mapping for TAVERN32 (which doesn't exist yet, but it is meant to be an improved text adventure game VM having the advantages of both TAVERN and Glulx). One lump declares the character sets and character mapping, and can include fallbacks if wanted, so that if ASCII mappings are declared, then it can work with any implementation even if they do not know about that character set. The story file might also (optionally) include other lumps with bitmap fonts, so that also allows it to work even if the character set is unknown (as long as it can display graphics, which some implementations may be incapable of). However, in this case, the character mapping is also relevant for compression too, and not only for determining character sets. There is then the possibility that multiple codes will be mapped to the same ASCII character (or sequence of characters in ASCII or any other character set), avoiding the problem of compatibility.
- z3t4 6y agoAnd the article didnt even get started on emojis, there are for example two different ways you can define skin color of an emoji. Most programming language dont have unicode support so its up to the developer how to handle them.
- rurban 6y agoHe forgot about stricter identifier rules, almost no product got right. Normalization. And the complete lack of unicode support in the most basic libraries or programs, like the libc, stdlibc++ or coreutils. As long as you cannot compare strings, you cannot search for them. As long as identifiers are treated as binary (e.g filesystem paths, names, ...), there are tons of security holes in the most basic things, like kernel drivers. Adding unicode support to any product does not make much sense then. Rather restrict it to Latin-1 or the current locale to be on the safe side. Nobody knows the security recommendation for identifiers.
- HelloNurse 6y agoTLDR: OMG text is difficult! In this article, unusually, text is difficult because of naive expectations, not because of incompetence. For example, the treatment of case-insensitive comparisons is quite good, but the author thinks that counting "characters" should be simple and well-defined, that conversion between uppercase and lowercase characters should be reversible, and that that one can "obviously use well supported libraries that solve this problem for you".