49 ms·

I Can’t Write My Name in Unicode

• angersock 12y ago
  I came in expecting to read an article bemoaning some niche language and playing the diversity card. I was not disappointed, but as I kept reading, the author made some very good points. I don't really care that the organization is run by white men who speak English, because frankly the entire computing industry and telecommunications industry is based on that. I'm not going to argue about the original sin there, because the facts speak for themselves: they got shit done. What I do find really disturbing, though, is that somehow both Linear A and Linear B are included, in their (as known) entirety, while Bengali is not--two languages spoken and written by nobody for over a millenia are included whereas a language spoken and written by millions of people is not. That's bad. The discussion of somehow creating a CJK superset and then deviations to support each of those languages is worse, and the author's remark about trying to "unify" Greek, Roman, Cyrillic, and Swedish struck home. Personally, I think that it's nice to have an ever-decreasing number of languages to support, preferably English. The first part of that is because it's annoying enough to unambiguously parse one human language much less dozens, and the second part is pure convenience and small-mindedness on my part. That said, we don't have to buy into diversity or identity or anything else to note that something here is amiss. This is bad allocation of engineering resources. EDIT: And yes, one could make the argument that the computer/teleco industry is being produced in Shenzhen, to which I would respond that the intellectual capital was handled by the West. Nintendo hired SGI, Sony IBM. There is certainly a wonderful and burgeoning domestic talent, but the paths were set elsewhere, to the best of my knowledge.
  • wavefunction 12y ago
    I really would like to downvote you for your harping about "diversity" whatever you mean by that. I can guess what you mean, as I live in the United States and am exposed to other people who harp similarly thus. I'm not going to down vote you though, just point out your weird usage of the term and mention that this is about proper information transmission, the foundation of computing and the Internet. So maybe get off your agenda regarding whatever you mean by "diversity" and start to see the people of all types as fellow humans and not actors in your own private psycho-drama.
  • com2kid 12y ago
    Han unification makes things really hard for programmers. You end up with code that tries to guess what language a string is in to pick out which character set should be used! It is an absolute nightmare and a horrid idea.
    • 1ris 12y ago
      Well, if you find Hiragana/Katakana it's Japanese, if you find Chữ Nôm it's Vietnamese. Otherwise it's Chinese (Well, given the definition of "language" is very hard in the context of Chinese). From a purly theoritical perspecitve the Han unification looks like a great idea. Image the horror of normalisations if it didn't happen. ; and greek questionmark would have been a joke in comparison.
    • theon144 12y ago
      Isn't the idea technically that the code shouldn't even have to guess? Why isn't this the case?
      • Someone 12y ago
        Imagine a world where the British always write the lowercase letter g as a single-story glyph (http://en.m.wikipedia.org/wiki/G#Typographic_variants http://en.m.wikipedia.org/wiki/G#Typographic_variants). The colonies start writing it identically, but after a while, they start writing it as a double-story g. After a century or so, nobody in he colonies writes the single-story variant, and all Brits always do. The unicode consortium studies the case and concludes that there is only a single g with variations in the way it is written. Because of that, it creates a single code point for the lowercase 'g' character. Now, suppose a web page stores the text 'goto' in Unicode as the code points for 'g', 'o', 't', and 'o'. To write the code that renders that string, you will find you need to know whether the text is written in British English or in colonial English.
        • Anderkent 12y ago
          Isn't this the question of a font? In which case the client chooses if they want to use a font with a double-story or a single-story g?
  • beat 12y ago
    Bengali is the seventh most widely spoken language in the world, with more native speakers than Russian. That's hardly a "niche language". And, as pg has pointed out, making the computer industry English-centric and America-centric is greatly limiting us. Intelligence and will to power are equally distributed across the globe. But institutional barriers to success are not. This is a classic example of an institutional barrier, and it makes all of us poorer.
    • exelius 12y ago
      Right, but some languages are insanely complex to implement. It might be a better idea to teach English to people around the globe rather than cater to every individual need (which will still leave people unable to communicate across languages). I'm not saying other languages should go away -- but the world would also benefit from having a "universal" language, which is more or less English at this point (Mandarin is spoken by more people but is rarely spoken outside of Asia). If we want to maximize intelligence and will to power, it would be best if like-minded individuals could communicate regardless of where they're from or what their native language is.
      • mbillie1 12y ago
        > some languages are insanely complex to implement I don't understand this, is there more to implementing a language than creating glyphs for its character set? I wouldn't think the linguistic complexity would matter at all, only the number of glyphs in the 'alphabet' or similar?
        • Crito 12y ago
          http://en.wikipedia.org/wiki/Complex_text_layout http://en.wikipedia.org/wiki/Complex_text_layout Hypothetically you could construct a language where the glyphs are easy to generate procedurally on the fly by people who are fluent in that language, but who's full space of possible glyphs is staggeringly massive. Suppose a language with tens or hundreds of thousands of "base" glyphs, but with a unique variant on each glyph depending on what is to the left and right of it. With that alone, for N base glyphs, you could have N^2 variants of each glyph. I don't know if that sort of language exists. I don't see any reason why it couldn't though.
  • makomk 12y ago
    Yeah, there's only one minor problem with the article: those emoji symbols it bemoans were added for the Japanese market. Western and English-language support for them was very much an afterthought. What's more, the reason they were added was because Japanese mobile providers were already offering emoji support through various non-standard and not entirely compatible encodings.
  • frivoal 12y ago
    Han unification has been overly aggressive about merging some characters, but the basic principle is not as flawed as it is some times (as in this article) made to sound. The vast majority of Japanese and Chinese characters are not only similar, they are identical. Not all are. Some are clearly different characters deriving from a common historical root, and should not be unified. Sometimes, when characters are a bit different, it is a matter of font. Out of the tends of thousands of characters used by these languages, it is not surprising that mistakes have been made, and yes, han unification has been a bit too far reaching. But not unifying at all would have been much worse. You don't want a in German and a in English to be different letters just because Helvetica and Baskerville look different. Also, since CJK languages are comfortable with having an unbounded set of letters to work from, they tend to be comfortable with declaring every odd thing a separate letter. Witness the mess that half-width katakana is. Or notice that the origin of most emoji is not in the disregard of white old men for "lesser" languages, but comes from importing existing characters from Japanese encodings. Which also hints at one more factor. Historically, and still very much to this day, Japan has been much more active in standardization bodies than most other non western countries. Unsurprisingly, Japanese is much better supported by modern software than many other non western languages.
    • com2kid 12y ago
      > The vast majority of Japanese and Chinese characters are not only similar, they are identical. Not all are. Some are clearly different characters deriving from a common historical root, and should not be unified. And what about traditional versus simplified? Which glyph set do I use? Oh wait, thanks to Han unification, I now need to rely on bloody environment variables to decide! For Chinese text, rendering a string of text involves not just the string, but also the local runtime. Joy of joys. > You don't want a in German and a in English to be different letters just because Helvetica and Baskerville look different. If following the spec means my users become partially illiterate then there are problems with the spec.
      • Tomte 12y ago
        Antiqua, Fractura, Schwabacher, Textura and all those other charming variants of writing European languages don't have seperate code points for all those presentational variations. Do we white Western men happen to discriminate against ourselves?
  • konstruktor 12y ago
    I can't think of any telecommunication or computing device that I own or have touched in the last few years that was built by white, English speaking men. Regarding software, I googled a bit to find numbers, and one of the first articles actually giving numbers says 19 percent of software developers are in the largest English speaking country, the US: http://www.techrepublic.com/blog/european-technology/there-are-185-million-software-developers-in-the-world-but-which-country-has-the-most/ http://www.techrepublic.com/blog/european-technology/there-a...
  • Dewie 12y ago
    > Personally, I think that it's nice to have an ever-decreasing number of languages to support, preferably English. The first part of that is because it's annoying enough to unambiguously parse one human language much less dozens, and the second part is pure convenience and small-mindedness on my part. I'll go further and claim that it's all about convenience and small-mindedness. It's so typical of a certain group of programmers to complain like this. Ugh, why do we have different languages and ways of doing things? Let's just converge on the one I grew up with, or use[1]. Hopefully you won't be forced to work on this kind of thing, or any kind of work that you don't care about/think is worth it. But until you do, maybe you could keep your self-centred navel gazing to yourself. [1] In the case of English, most programmers who express this opinion write perfectly good English, though they might not be native speakers. They wouldn't be inconvenienced at all if suddenly everyone forgot about their own language and started speaking English, especially since they don't seem to care about languages beyond having some way of communicating with other people.
• pavlov 12y ago
  By the title, I assumed this would be an article about how 2-year-olds can learn to do some interesting stuff on a smartphone. [Edit -- Sorry about this lame joke. I've suffered the consequences in downvotes.]
  • spiritplumber 12y ago
    I thought it was going to be about illiteracy.
• nemo 12y ago
  I wonder if the author has submitted a proposal to get the missing glyph for their name added. You don't need to be a member of the consortium to propose adding a missing glyph/updating the standard. The point of the committee as I understand it isn't to be an expert in all forms of writing, but to take the recommendations from scholars/experts and get a working implementation, though more diverse representation of language groups would definitely be a positive change. http://unicode.org/pending/proposals.html http://unicode.org/pending/proposals.html Also, the CJK unification project sounds horrible.
  • vilhelm_s 12y ago
    It sounds like the glyph is in unicode already, but expressed using combining characters?
    • breadbox 12y ago
      Imagine if the letter Q had been left out of Unicode's Latin alphabet. The argument against it is that it can be written with a capital O combined with a comma. (That's going to play hell with naive sorting algorithms, of course, but oh well.) Oh, and also imagine your name is Quentin.
      • vilhelm_s 12y ago
        Combining characters are not just used for Bengali though. E.g. umlauted letters in European languages can also be expressed using combining characters, and implementations need to deal with those when sorting.
      • icebraining 12y ago
        But the letter wasn't left out of Unicode; it's actually typed in the article. It's just internally represented as multiple codepoints, much like one of parts of my name (é) may be. Frankly, this is irrelevant to the actual problem, which is the input system, and which has nothing to do with Unicode. Nothing prevents a single key from typing multiple codepoints at once.
• mlmonkey 12y ago
  I get the author's point, but assigning blame to the Unicode Consortium is incorrect. The Indian government dropped the ball here. They went their separate way with "ISCII" and other such misguided efforts, instead of cooperating with UC. To me, the UC is just a platform. The government is the de facto safeguarder of the people's interests; if it drops the ball, it should be taken to task, not the provider of the platform.
  • shkkmo 12y ago
    So blame the indian government for any problems with ISCII. The problems with Unicode support for global languages are indeed to be blamed on the Unicord Consortium. Nothing the indian governemnt did prevents the UC from availing themselves of the global knowledge needed to create a good global standard.
    • mlmonkey 12y ago
      do you have any evidence that the UC has actively ignored requests from Bengali speakers? Has any Bengali speaker made proposals to the UC for fixing these issues? If yes, and the UC chose to ignore them, then there is some blame to be assigned with the UC. Otherwise, this is a non-issue. Take, for example, Tibetan. The number of Tibetan speakers is minuscule compared to, say, Bengali. But still Tibetan has good support, because enough people took an interest in getting it supported and worked with the UC to get it done.
      • deleted 12y ago
        [deleted]
        • frivoal 12y ago
          Not true. Anybody with an email address can send feedback. And individuals who want to be more active and become an actual member have a $75 membership fee.
      • yuriks 12y ago
        I think it needs to be stressed that anyone can submit character proposals to the consortium and work with them to get them included in the next versions of the standard. You don't need to be a member (By paying the hefty fee. Someone needs to pay for the operational costs of the consortium.) to have your suggestions taken into account.
• theon144 12y ago
  I don't understand; I don't feel like character combination using the zero width joiner is on the same level as 13375p34k. It looks like the character just doesn't have a separate code-point, but is instead a composite, but still technically "reachable" from within Unicode, no?
  • cosarara97 12y ago
    It's like typing ` + o to get "ò", isn't it? You can argue that ò is actually an o with that tilde, while that character is not ত + ্ + an invisible joining character, but that's an input method thing, and there is a ৎ character after all.
    • jamie_ca 12y ago
      They're on the same key on my keyboard, but ` is a grave, ~ is a tilde.
    • zhemao 12y ago
      But ò has it's own codepoint, whereas the ending character in the Aditya's name does not.
  • gopalv 12y ago
    That's effectively the only way to write several indian language with unicode - ZWJ + NZWJ and a decent font which supports all the ligatures. The recent Bullshit Sans font is a clear example to describe how ligatures works. And for Malayalam there are consonant patterns which do not match itself to the Sanskrit model which makes it rather odd to write half-consonants which are full syllables (വ്യഞ്ജനം +് zwj). And my name is written with one of those (ഗോപാൽ) & I can't say I'm mad about it because I found Unicode to be elegant in another way. Somewhere in the early 2000s, I was amazed to find out that the Unicode layouts for Malayalam as bytes in UTF-8 were sortable as-is. As a programmer, I found that detail of encoding to sort order to be very fascinating as it meant that I had to do nothing to handle Malayalam text in my programs - the collation order is implicit, provided everyone reads the ZWJ and NZWJ in their sorting orders.
  • Manishearth 12y ago
• qntm 12y ago
  > He proudly announces that there are ‘no fewer than 147 Indian dialects’ – a pathetically inaccurate count. (Today, India has 57 non-endangered and 172 endangered languages, each with multiple dialects – not even counting the many more that have died out in the century since My Fair Lady took place) So, how many were there really? At the time, I mean.
  • rpedroso 12y ago
    I think the numbers are somewhat disputed. The People's Linguistic Survey of India says there are at least 780, with ~220 having died out in the last half century.[1] The Anthropological Survey of India reported 325 languages.[2] The discrepancies are made particularly tricky because of the somewhat ambiguous distinction between languages and dialects. [1] http://blogs.reuters.com/india/2013/09/07/india-speaks-780-languages-220-lost-in-last-50-years-survey/ http://blogs.reuters.com/india/2013/09/07/india-speaks-780-l... [2] http://books.google.com/books?id=VjGdDo75UssC&pg=PA145 http://books.google.com/books?id=VjGdDo75UssC&pg=PA145
    • qntm 12y ago
      So, this person thinks there are 229 languages in India, but actually there's a minimum of 325, possibly twice that?
      • rpedroso 12y ago
        I'm not an anthropological linguist, so I can't say with any authority. I only listed two figures there, but there are many more disputed figures.[1] There are a lot of difficulties classifying languages in India because the line between language and dialect can be tricky to pin down. [1] http://en.wikipedia.org/wiki/Languages_of_India#Inventories http://en.wikipedia.org/wiki/Languages_of_India#Inventories
  • pc2g4d 12y ago
    I believe the number of "dialects" named in My Fair Lady can be largely explained by the lack of clear distinction between language and dialect over the years. From [1]: "There is no universally accepted criterion for distinguishing a language from a dialect. A number of rough measures exist, sometimes leading to contradictory results. The distinction is therefore subjective and depends on the user's frame of reference." Getting upset about Henry Higgins's estimation of the number of Indian "dialects" in a play from many decades ago doesn't make sense to me. His character was deliberately portrayed as a regressive lout, and terminology has surely changed in the intervening years. [1]: http://en.wikipedia.org/wiki/Dialect#Dialect_or_language http://en.wikipedia.org/wiki/Dialect#Dialect_or_language
• Epenthesis 12y ago
  Wait, "ত + ্ + ‍ = ‍ৎ" is nothing like "\ + / + \ + / = W". The Bengali script is (mostly) an abugida. Ie, consonants have an inherent vowel (/ɔ/ in the case of Bengali), which can be overriden with a diacritic representing a different vowel. To write /t/ in Bengali, you combine the character for /tɔ/, "ত", with the "vowel silencing diacritic" to remove the /ɔ/, " ্". As it happens, for "ত", the addition of the diacritic changes the shape considerably more than it usually does, but it's a perfectly legitimate to suggest the resulting character is still a composition of the other two (for a more typical composition, see "ঢ "+" ্" = "ঢ্"). As it happens, the same character ("ৎ") is also used for /tjɔ/ as in the "tya" of "aditya". Which suggests having a dedicated code point for the character could make sense. But Unicode isn't being completely nutso here.
  • elFarto 12y ago
    I don't understand Bengali at all, but the character ৎ does have it's own Unicode codepoint (U+09CE, BENGALI LETTER KHANDA TA). It was introduced in Unicode 4.1 in 2005.
    • chimeracoder 12y ago
      Yes, this is what the article says: > Until 2005, Unicode did not have one of the characters in the Bengali word for “suddenly”. The codepoint does exist now, but it took ten years before it was included.
      • Manishearth 12y ago
        ... and emoji was added to the Unicode standard in 2010. The title of your article is fallacious :/ (though I agree with most of it)
  • ubasu 12y ago
    Native Bengali here. The ligature used for the last letter of "haTaat" (suddenly) is not the same as the last ligature in "aditya" - the latter doesn't have the circle at the top. More generally, using the vowel silencing diacritic (hasanta) along with a separate ligature for the vowel ending - while theoretically correct - does not work because no one writes that way! Not using the proper ligatures makes the test essentially unreadable.
• microcolonel 12y ago
  “Whatever path we take, it’s imperative that the writing system of the 21st century be driven by the needs of the people using it. In the end, a non-native speaker – even one who is fluent in the language – cannot truly speak on behalf the monolingual, native speaker.” Not sure how the author can simultaneously say this, while criticizing the CJK unification, which makes total sense, and has never been a point of contention in the communities concerned with it. I regularly work with both Chinese and Japanese texts, and have studied Japanese language for more than five years; more than not opposing CJK unification, I benefit from it greatly. It means that I can search for the same character(yes, the same, I said it) and get results in Japanese, Chinese, and sometimes Korean context without having to consult some sort of unicode etymology resource. There should not be three codepoints for 中, 日, 力, and most other characters. The simplified and specific post-han forms have their own codepoints anyway, so there is(to my eyes at least) no value in fragmenting instances of the same character.
  • mehta 12y ago
    > [...] CJK unification[...] has never been a point of contention in the communities concerned with it. I am not very familar with the CJK unification project so take my points with a grain of salt. > more than not opposing CJK unification, I benefit from it greatly. I think think that is a different point of view. Isn't it? You are seeing your benefit whereas the author is seeing his. Here's an alternative solution: What if the search engine understood what you were searching for and returned results in all the languages? Unification can result in a lot of information loss the same way a photo can be compressed but it comes at the cost of loss in quality. > so there is(to my eyes at least) no value in fragmenting instances of the same character. But no-one is fragmenting instances of the same character. They _are_ different characters from different languages. To take an example from the article, I am not sure how I feel about combining B and β. You are either ignoring the whole of English speaker population or the greek speaking one. Given that you have complete flexibility to assign a code for both of them, why not do it(responsibly)?
    • Navarr 12y ago
      The example characters expressed: 日、中、力 however are written the same way in both Chinese and Japanese from my understanding. (Albeit, I studied Japanese). There are admittedly variations which should be done separately, however unification of visually identical glyphs is a "good thing" imho
• yarper 12y ago
  I Can Text You A Pile of Poo, But I Can’t Write My Name ... - by Aditya Mukerjee on March 17th, 2015 What is the glyph missing from this? I know its not ideal but some uncommon glyphs have always been omitted from charsets, for example ASCII never included http://en.wikipedia.org/wiki/%C3%86 http://en.wikipedia.org/wiki/%C3%86, and it was replaced by "ae" in common language. http://en.wikipedia.org/wiki/Hanlon%27s_razor http://en.wikipedia.org/wiki/Hanlon%27s_razor
  • fixermark 12y ago
    This is also why we've (almost) all taken to writing résumé as resume. We've even managed to build text-based search engines that do a pretty decent job of guessing which one we mean.
• dominotw 12y ago
  I am an Indian and it shocks me that Indians are still blaming the British after 70 yrs of independence. Is 70 years of Independence not enough to make your language "first class citizen" ? Ofcourse Bengali is second class language because Bengalis didn't invent the standard. Can we stop blaming white people for everything. Seriously WTF.
  • yosefk 12y ago
    Hebrew (and I'd guess Arabic and other right-to-left languages) work rather badly in Unicode when it comes to bidirectional rendering; however, to the extent that it's a result of Israeli/Egyptian/Saudi/etc. companies and/or governments failing to pay $18K (the figure from TFA) to pay for the consortium membership, I kinda blame them and not the consortium. I mean, it's not a whole lot of money, even for a smallish company, not to mention a biggish state. Also this bit was just amazing: "It took half a century to replace the English-only ASCII with Unicode, and even that was only made possible with an encoding that explicitly maintains compatibility with ASCII, allowing English speakers to continue ignoring other languages." Seriously? Nobody forces anyone to use UTF-8, and even if it were the only encoding available - by how much the cost of these extra bytes reduces the standard of living in non-English-speaking countries, exactly? It is in fact unfortunate that the pile of poo is standardized while characters from languages spoken by hundreds of millions are not, however. A pity that a good and surprising (to some) point was mixed with all that "white men" business.
    • cplease 12y ago
      > Hebrew (and I'd guess Arabic and other right-to-left languages) work rather badly in Unicode when it comes to bidirectional rendering; however, to the extent that it's a result of Israeli/Egyptian/Saudi/etc. companies and/or governments failing to pay $18K (the figure from TFA) to pay for the consortium membership I'm sorry? How does this have anything to do with encoding, versus software toolkits? Unicode and encoding generally has little to nothing to do with input methods or rendering, and if you can explain why Unicode in particular works better in one direction I am genuinely curious. > It is in fact unfortunate that the pile of poo is standardized while characters from languages spoken by hundreds of millions are not, however I don't think in 2015 Unicode lacks codepoints for any glyph in regular usage by hundreds of millions. OP has not cited any; the one he explicitly complains of was in fact added in 2005. Before someone says Chinese, that's a bit more like citing some archaic English word missing from a spellchecker. I am skeptical that there is a Han character missing from Unicode that >> 1 million people recognize much less use on a regular basis.
• hawkice 12y ago
  Getting rid of CJK unification would better model actual language change in the future (France, for instance, has a group that keeps a rigorous definition of the French language up to date -- I would enjoy giving them a subset of the codes to define how to write French). But the general principle sounds odd. Should 家, the simplified Chinese character and 家, the traditional Chinese character have different codepoints? Should no French be written using characters with lower, English code points because of their need for a couple standard characters? Should latin be written using a whole new set of code points even though it needs no code points not contained in ascii?
  • tjradcliffe 12y ago
    There was an academic proposal in the '90's for something called "multicode" (IIRC) that did exactly this: every character had a language associated with it, so there were as many encodings for "a" as there were languages that used "a", and all of them were different, or at least every character was somehow tagged so the language it "came from" was somehow identifiable. Fortunately, it never caught on. The notion that some particular squiggle "belongs" to one culture or language is kind of quaint in a globalized world. We should all be able to use the same "a", and not insist that we have our own national or cultural "a". The position becomes more absurd when you consider how many versions of some languages there are. Do Australians, South Africans and Scots all get their own "a" for their various versions of English? What about historical documents? Do Elizabethan poets need their own character set? Medieval chroniclers? Building identity politics into character sets is a bad idea. Unifying as much as practically possible is a good idea. Every solution is going to have some downsides, some of them extremely ugly, but surely solutions that tend toward homogenization and denationalization are to be preferred over ones that enable nationalists and cultural isolationists.
    • RodericDay 12y ago
      > Building identity politics into character sets is a bad idea. Unifying as much as practically possible is a good idea. Every solution is going to have some downsides, some of them extremely ugly, but surely solutions that tend toward homogenization and denationalization are to be preferred over ones that enable nationalists and cultural isolationists. glib white supremacists are the best kind of white supremacists. "it's progress!"
• sdg1 12y ago
  Not sure if the l33tspeak analogy is fully justified. In case of the "missing" letter (called khanda-ta in Bengali) for the Bengali equivalent of "suddenly", historically, it has been a derivative of the ta-halant form (ত + ্ + ‍ ). As the language evolved, khanda-ta became a grapheme of its own, and Unicode 4.1 did encode it as a distinct grapheme. A nicely written review of the discussions around the addition can be found here: http://www.unicode.org/L2/L2004/04252-khanda-ta-review.pdf http://www.unicode.org/L2/L2004/04252-khanda-ta-review.pdf I could write the author's name fine: আদিত্য. A search with the string in the Bengali version of Wikipedia pulls up quite a few results as well, so other people are writing it too. The final "letter" in that string is a compound character, and there's no clear evidence that it needs to be treated as an independent one. Even while in primary school, we were taught the final "letter" in the author's name as a conjunct. In contrast, for the khanda-ta case, it could be shown that modern Bengali dictionaries explicitly referred to khanda-ta as an independent character. For me, many of these problems are more of an input issue, than an encoding issue. Non latin languages have had to shoe-horn their script onto keyboard layouts designed for latin-scripts, and that has been always suboptimal. With touch devices we have newer ways to think about this problem, and people are starting to try out things. [Disclosure: I was involved in the Unicode discussions about khanda-ta (I was not affiliated with a consortium member) and I have been involved with Indic localization projects for the past 15 years]
  • lilyball 12y ago
    > For me, many of these problems are more of an input issue, than an encoding issue. I think you've hit the nail on the head here. I'm a native English speaker, so I may in fact be making bad assumptions here, but I think the biggest issue here is that people conflate text input systems with text encoding systems. Unicode is all about representing written text in a way that computers can understand. But the way that a given piece of text is represented in Unicode bears only a loose relation to the way the text is entered by the user, and the way it's rendered to screen. A failure at the text input level (can't input text the way you expect), or a failure at the text rendering level (text as written doesn't render the way you expect), are both quite distinct from a failure of Unicode to accurately represent text.
• josephschmoe 12y ago
  Am I the only person who thought unifying the Greco-Roman language characters actually sounds like a good idea?
  • mmastrac 12y ago
    Yeah, I thought that "No native English speaker would ever think to try “Greco Unification”" was a poor argument. In seems like a reasonable idea.
    • Someone1234 12y ago
      I think their argument was: The characters look the same (e.g. Russian's first character and the English A) but have different meanings. So in this example if you searched for the English word "Eat" that is also a completely legal Russian word (E, A, and T, exist in English and Russian), however it means nothing remotely similar. I don't know if they're right or wrong. I am just saying that might be the point they were trying to make. You could make a Greco Unified unicode set and it would work fairly well, but you might wind up with some confusing edge cases where it isn't clear what language you're reading (literally). This could be particularly problematic for automation (e.g. language detection). Since in some situations any Greco-like language could look similar to any other (in particular as the text gets shorter).
      • mrgriscom 12y ago
        I think the real reason was to preserve round-trip compatibility (legacy char -> unicode char -> legacy char) with the existing encodings for those alphabets.
      • josephschmoe 12y ago
        English, French, German, Italian, Spanish and several other European languages have mostly identical character sets and even large numbers of similar or identical words. Computers detect these languages just fine. I think we'll be okay.
      • gcv 12y ago
• Torgo 12y ago
  Minoan script and Mormon Deseret are in there because somebody stepped up.
• jhpriestley 12y ago
  For the first few thousand years of writing, you could write whatever you wanted. For example, you could make up a new alphabet if you wanted to. Or come up with symbols, like = or $ or @, which represented some concept previously expressed with words. Moveable type, and then typewriters, meant that we had to decide upfront which characters could be used. That is, you could use any symbol you wanted, but you might have to cut it yourself, cast it in metal, etc. Now, with the infinitely more flexible digital technology, we have achieved the ultimate level of rigidity: using a new character would mean petitioning a committee somewhere.
• agumonkey 12y ago
  I'm not that surprised Unicode includes very shallow code points (pile of poo), because nobody really cares. Bengali on the other hand, requires no mess up in order to satisfies the Billion+ users.
• a2tech 12y ago
  Ok, we all understand it sucks. Whats the fix? You think Unicode is racist and terrible. Its stupidly difficult to work with for sure, but racist is a stretch. What do you propose? More complexity layered on a system that people don't understand isn't really a fix.
  • VLM 12y ago
    Also they organizational mystery of how to open the floodgates for unpopular languages while still keeping Klingon and Tolkien Elvish out is something of a mystery.
• outworlder 12y ago
  > He proudly announces that there are ‘no fewer than 147 Indian dialects’ – a pathetically inaccurate count. Wow. How can a country function like this? Is everyone proficient in their native language plus a 'common' one, or are all interactions supposed to be translated inside the same country? Regardless of historical and cultural value, if that's the case, it seems... inefficient. I do realize that there are more countries like this, but the number of languages seems way too high. I am really curious how that works.
  • masklinn 12y ago
    > How can a country function like this? Is everyone proficient in their native language plus a 'common' one Yes, there may be multiple "common" languages if the country is large/populated enough or for historical reasons (Switzerland has 4 official languages — though official acts only have to be provided in 3 of them, Swiss-German, French and Italian — and they only have 8 million people) > Regardless of historical and cultural value, if that's the case, it seems... inefficient. Well telling people to fuck off with their generations-old gobbledygook and imposing a brand new language on them tends to make them kind-of restless, so unless you're willing to assert your declarations in blood (or at least in the specific suppression of non-primary languages)… The latter has happened quite bit, e.g. post-revolutionary France tried very hard to stamp out both dialects and non-french languages until very recently (the EU shows a distinct lack of appreciation for the finer points of linguicide and linguistic discrimination), the UK did the same throughout the Empire. > I do realize that there are more countries like this Almost all of them aside from former british colonies (where native languages were by and large eradicated), at least to an extent (many countries carried out linguistic unification as part of their rise as nation-states, to various levels of success).
  • Dewie 12y ago
    > I do realize that there are more countries like this, "More"? I wouldn't be surprised if it was most.
  • zhemao 12y ago
    India has two "official languages" used across the country in business, government, and education: Hindi and English. There are also more widely used regional languages recognized by state governments, such as Tamil, Gujarati, Punjabi, and the author's native Bengali. Educated Indians are usually multilingual, speaking Hindi, which they learn at school, and one or more regional languages. This is not a unique situation. Every country with a large enough area and a long enough history has a vast array of minority languages and dialects. North America is just different because it was settled relatively recently and the indigenous population was almost entirely wiped out.
• fredo699 12y ago
  How fucking melodramatic do you have to get before you can invoke the invasion and colonization of India by the British Empire in a conversation about Unicode?
• Htsthbjig 12y ago
  I think the author misses the point completely. Things like this: "No native English speaker would ever think to try “Greco Unification” and consolidate the English, Russian, German, Swedish, Greek, and other European languages’ alphabets into a single alphabet." The author probably ignores that different European languages used different alphabet scripts until very recently. For example, Gothic and other different script were used in lots of books. I have old books that take something like a week to be able to read fast, and they are in German!. But it was chaos and it unified into a single script. Today you open any book, Greek, Russian, English or German and they all use the same standard script, although they include different glyphs. There is a convention for every symbol. In cyrilic you see an "A" and a "a". In fact, any scientific or technical book includes Greek letters as something normal. It should also be pointed out that latin characters are not latin, but modified latin. E.g Lower case letters did not exist on Roman's empire. It were included by other languages and "unified". Abut CJK, I am not an expert but I had lived on China, Japan and Korea, and in my opinion it has been pushed by the Governments of those countries because it has lots of practical value for those countries. Learning Chinese characters is difficult enough. If they don't simplify it people just won't use it, when they can use Hangul or kana. With smartphones and tablets people there are not hand writing Chinese anymore. It makes no sense to name yourself with characters nobody can remember.
  • Dylan16807 12y ago
    Right, I was going to argue against that too. Changing fonts every letter will always look weird. There's a lot of different ways to shape these letters that are all counted as the same.
  • gurkendoktor 12y ago
    > Learning Chinese characters is difficult enough. If they don't simplify it people just won't use it CJK unification doesn't make learning Chinese characters any easier, it only means that characters that look exactly the same will use the same code point. This is not related to simplifying characters. Also, I regularly see people hand-writing Chinese characters on tablets, so it does happen. And for those who use Pinyin to enter characters, it doesn't matter if the characters are simplified or traditional or unified or whatever, because they only have to pick the right one in their IME.
• ramviswanadha 12y ago
  The argument that author makes is "every letter in English alphabet is represented, why not every letter/grapheme in Bengali/Tamil/Telugu/Name your language" argument is specious at best
  • zhemao 12y ago
    Except that the whole purpose of Unicode is to create a character encoding that "enables people around the world to use computers in any language" - taken from the Unicode Consortium website. Bengali is also not an obscure language. It is the 10th most spoken language in the world and the national language of Bangladesh.
    • zhemao 12y ago
      And besides, the argument isn't "Why aren't all Bengali characters represented when all English characters are?" The argument is "Why aren't all Bengali characters represented when a pile of poo is?"
      • webitorial 12y ago
        No, the argument is "Why didn't the Unicode authors make the same technical choice I would have based on my limited knowledge of this topic -- including not knowing that the pile of poo was created for use in the Japanese market; as well as not knowing that I could participate myself for just $75, not the $18,000 lie in my story; and not understanding the nuances of international standardization; or reviewing the list of international liaisons to the Unicode organization where much of the language-specific work is done?"
    • TheCoelacanth 12y ago
      Bangladesh is a member of the Unicode Consortium. Why haven't they ensured that Bengali is well supported? The Unicode Consortium's job isn't to support every language in the world; it's to coordinate between the multitude of different parties who do the actual work to ensure that they produce can coexist nicely.
• gurkendoktor 12y ago
  The poo emoji is not part of Unicode because rich white old cis-gendered mono-lingual oppressive hetero men thought it'd be a fun idea (outrage now!!). Emoji were adopted from existing cellphone "characters" in Japan, and Japan is famously lagging in Unicode adoption because some Japanese names cannot (could not?) be written in Unicode. It all just seems to be very normal (=inefficient, quirky) design by committee.
• bane 12y ago
  Out of curiosity, before Unicode came along what was the state of the art for encoding/writing/displaying Bengali? Sometimes I think issues with Unicode might be because it's trying to solve issues for languages that haven't yet arrived at a good solution for themselves yet. Latin-using languages ended up going through a very long orthographic simplification and unification process after centuries of divergent orthographic development. These changes all occurred to simplify or improve some aspect of the language: improve readability, increase typesetting speed, reduce typesetting pieces. Early personal computers even did away with lowercase letters and some punctuation marks completely before they were grudgingly reintroduced. I'm more familiar with Hangul (Korean), which has sometimes complex rules for composing syllables but has undergone fairly rapid orthographic changes once it was accepted into widespread and official use: dropping antiquated or dialectal letters, dropping antiquated tone markers, spelling revisions, etc. In addition, Chinese characters are now completely phased out in North Korea and are rapidly disappearing in the South. It's my personal theory that the acceleration of this orthographic change has had to do with the increased need to mechanize (and later computerize) writing. Koreans independently came up with workable solutions to encode, write and display their system, sometimes by changing the system, sometimes by figuring out how to mechanically and then algorithmically encode the complex rules for their script. It appears that a happy medium has been found and Korean users happily pound away at keyboards for billions of keystrokes every year. I'm digressing here, but pre-unicode, how had Bengali users solved the issue of mechanically and algorithmically encoding, inputting and displaying their own language? Is it just a case that the unicode team is ignorant of these solutions and hasn't transferred this hard-earned knowledge over? (note, I came across this page as I was writing this, how suitable was this solution? http://www.dsource.in/resource/devanagari-letterforms-history/devanagari_letterforms/letterforms_for_typewriter.html http://www.dsource.in/resource/devanagari-letterforms-histor...) I'm asking these questions out of ignorance of course, I don't know the story on either side. On a second point, I'm deeply concerned about Unicode getting turned into a font-repository for slightly different (or even the same) character, that just happens to end up in another language. For example, Cherokee uses quite a few Latin letters (and numbers) (and quite a few original inventions). Is it really necessary to store all the Latin letters it uses again? Would a reader of Tsalagi really care too much if I used W or Ꮃ? When does a character go from being a letter to being a specific depiction of that letter?
• ckoerner 12y ago
  The comparison to the play "My Fair Lady" is not very convincing. I would suggest the author remove it as it weakens the argument. First, the fictional character states 'no fewer than' as an admittance that there are more. Second, even if we consider this as a valid complaint, the author himself points out in his example that this 'common sentiment' is from a play written a century ago. Using a fictional character from a time when the world's knowledge of language was incredibly smaller than it is today does not help support your goals.
• vacri 12y ago
  It sounds like the author is looking to be offended. Talking about Bengali being the seventh largest native language, then saying that US$18k is too expensive for a stake in solving this problem? Emojis with skin tones, something every human can use, that have arrived before a character that only Bengalis use is taken as an "outright insult"?
  • zhemao 12y ago
    > Talking about Bengali being the seventh largest native language, then saying that US$18k is too expensive for a stake in solving this problem? West Bengal and Bangladesh aren't exactly the richest places in the world. > Emojis with skin tones, something every human can use, that have arrived before a character that only Bengalis use is taken as an "outright insult"? Nobody is being inconvenienced by the inability to send emojis with a darker skin tone. People definitely are being inconvenienced by not being able to write a common character of their native language.
    • wbkang 12y ago
      $18k is really a small amount of money for any governmental organization including those places. Even North Korea participates in the process.
      • webitorial 12y ago
        Any Bengali scholar can join the work for $75.
    • vacri 12y ago
      I work with a Bangladeshi, and he says that there are more millionaires in Bangladesh than here in Australia. Yes, the people are poor on average, but there's still a lot of money sloshing around. As to your second point, the top-voted comment in the thread talks about writing exactly that character and gives an example. From the ensuing discussion with the article author, it seems that rather than an inconvenience per se, it's more of a subtle issue. sdg1 also mentions that there's little in the way of local standards, which may be hindering a formal uptake by unicode. Ultimately the answer is to push the issue with the standards body. If you aren't going to front the money for voting power, then pester the people who do have the power, and make it easier for them by helping them to standardise the codepoints for the language. But whichever path is taken, they must be engaged professionally - saying things like "I take it as an outright insult that you haven't accommodated me yet" is not going to help.
• marincounty 12y ago
  My English professor biggest pet peeve. He told us day one. It was: 1. Don't use cliches in your writing. 2. Don't ever use emoji in any communication. p.s. Emoji wasen't even around when I went to school. I bet Mr. Taylor woukd have had a field day though? Never been the fan of any smiley face, even rotated 90 degrees.
• drawkbox 12y ago
  It is much easier to criticize than to fix it. While it is good to bring awareness to this, we are still growing in this area. In fact we should applaud the efforts so far that we even have a standard that somewhat works for most of the digital world. Does it need to evolve further, yes. I am sure the engineers and multi-lingual people that stepped up to do Unicode and organize it aren't trying to exclude anyone. Largely it comes down to who has been investing in the progress and time. It may even be easier to fund and move this along further in this time, it was hard to fund anything like this before the internet really hit and largely relied on corporations to fund software evolution and standards. In no way should the engineers or group getting us this far (UC) be chided or lambasted for progressing us to this step, this is truly a case of no good deed goes unpunished.
  • webitorial 12y ago
    Generally true, but the problem here is not an issue of bandwidth or racism. Unicode can represent this character, but does so with two codepoints, a technical decision the author doesn't feel is useful. He blames this on the dominance of white people in the work (a questionable assumption, given he didn't link to the extensive list of international liaisons and international standards bodies). The participants in Unicode, including a native Bengali speaker who responded above, considered the argument presented but chose a different path to be consistent with how other characters are treated. The author needs to more carefully distinguish the codepoint, input, and rendering issues raised in his argument.
• lxe 12y ago
  > No native English speaker would ever think to try “Greco Unification” and consolidate the English, Russian, German, Swedish, Greek, and other European languages’ alphabets into a single alphabet. This actually is a pretty good idea. Cyrillic, Latin, and other "Greco" scripts share quite a lot of characters. There's no need for both А (http://www.fileformat.info/info/unicode/char/0410/index.htm http://www.fileformat.info/info/unicode/char/0410/index.htm) and A (http://www.fileformat.info/info/unicode/char/0041/index.htm http://www.fileformat.info/info/unicode/char/0041/index.htm) beyond ASCII and other legacy compatibility.
  • TazeTSchnitzel 12y ago
    Well, it depends. Do you unify those with identical glyphs but different origins? Do you unify only those with identical glyphs and the same origins?
• volune 12y ago
  get your hundreds of millions of native speakers off their asses and solve the problem! the current internet is in roman glyphs because that's what all the people doing the work speak!
  • zhemao 12y ago
    Given the number of Indians working in the software field, I think it's safe to say that there are quite a few Bengali speakers "doing the work" to keep the internet running. Actually, I'd posit that the majority of programmers are not native English speakers.
• PSeitz 12y ago
  The problem here is not that these people are white, it's the languages they are not speaking.
  • webitorial 12y ago
    And a native Bengali speaker has discussed his input into the Unicode discussions and why he disagreed with the author. What do you know about the linguistic input into the work or the participants? Did you check the huge international lists of liaisons that the author ignored? It costs $75 to join Unicode.
• jkot 12y ago
  I dont speak Bengali, but some european languages have similar problem. In Czech language letter "ch" is a single character with its own place in alphabet.
• LeoPanthera 12y ago
  This article is imbued with its own form of curious racism. In particular, I became suspicious of its motives at the line: > "It took half a century to replace the English-only ASCII with Unicode, and even that was only made possible with an encoding that explicitly maintains compatibility with ASCII, allowing English speakers to continue ignoring other languages." ASCII-compatibilty was essential to ensure the adoption of Unicode. It's not because English speakers wanted to ignore anyone or anything, it's because without it, it would never have been adopted, and we'd be in a much worse position today. In other words, the explanation is technical, not racial.
  • chris_wot 12y ago
    It's sort of hilarious that he said that, given that English speakers were able to encode every character in 7-bit ASCII. The issues around standardising characters was because non-English characters were being squabbled about between the French, Russians and a whole bunch of other non-English countries. In essence, he's not understood that really ASCII was used as the base for Unicode because it was widely used. In fact. It's actually ISO8859-1 that has been used because of its wide coverage of a variety of languages, far more than any of the other 8859-x character sets. I cannot speak to anything else he's said, aside from saying that trying to encode all the world's languages is bloody hard. Even when a limited number of countries try to nut out a standard for 128 characters, it's a nightmare. And don't forget that they were competing with EBCDIC. I wrote about it here: http://randomtechnicalstuff.blogspot.com.au/2009/05/unicode-and-oracle.html http://randomtechnicalstuff.blogspot.com.au/2009/05/unicode-...
• kenko 12y ago
  Why was the title changed?
  • elchief 12y ago
    Because poo
• 6t6t6 12y ago
  What is amazing here is that everybody is talking about the character "ৎ" instead of talking about the real issue. I guess that the main point of the article is that the Unicode Standard is being decided by a bunch of North American and European people, instead of being decided by the speakers of the languages Unicode is intended to. In my opinion, for instance, Han Unification is a botch and no Japanese would consider that it makes sense.
• robmccoll 12y ago
  "in My Fair Lady (based on George Bernard Shaw’s Pygmalion)" which of course draws on Ovid's Metamorphoses poem about Pygmalion from ancient Greek tradition.
• deleted 12y ago
  [deleted]
• wbkang 12y ago
  I see many comments about Han unification being a bad idea but I am not seeing any reason why it was such a bad idea. I am from a CJK country and I find it makes a lot of sense. Most commonly used characters should be considered identical regardless of whether it is used in Chinese Japanese Korean or Vietnamese. Sure there are some characters that are rendered improperly depending on your font but I don't think that makes Han unification a fundamentally bad idea.
  • iso8859-1 12y ago
    Is it really depending on the font, or is it depending on some language metadata? Having it depend on the font seems stupid, since a font ideally be able to represent a languages which use a script encodable using Unicode.
    • chris_wot 12y ago
      You have the world's greatest username and I was wondering when you would turn up!
  • deleted 12y ago
    [deleted]
• kaptain 12y ago
  Most of the discussion here centers on how linguistic commitments ought to drive decision making in determining the Unicode spec. It's already been covered how Unicode provides a number to symbol (i.e. codepoint) mapping; composition, rendering, input, etc. is left to the system implementation to determine. I actually worked with Lee Collins (he was my manager) and a whole bunch of the ITG crew at Apple in the early 00's. The critique that this was just a bunch of white dudes that only loved their own mother tongues but studied other languages dispassionately as a superficial proof of their right to determine an international encoding spec is, to me, misinformed, and DEEPLY offensive. I had just gotten out of college and it was AMAZING to see how much these people loved language! Like it was almost weird. Our department had a very diverse linguistic background and, oftentimes, when you had a question about a specific language property you could just go down the hall and just ask "Is this the way it should be?" All the discussion here happened on the unicode mailing lists as well as in passing at lunch and at team meetings. Lots of people felt VERY passionately about particular things; I just liked watching the drama. But to write an article like this that somehow intimates that people didn't care is wrong. People cared A LOT. It's been touched in a couple of comments, but a big factor in Unicode was also the commercial viability of certain decisions. You have Apple, Adobe, Microsoft, etc with all of their previous attempts at internationalization/localization. If you wanted these large companies to adopt a new standard you had to give them a relative easy pathway to conversion and adoption. I think the article, in general, lacks a perspective that dismisses the work of the Unicode team as well as the different stakeholders. Historically, accessible computing was birthed in the United States. The tools and processes are naturally Ameri-centric. I'm not saying it's right, I'm just saying it is. The fact that the original contributors to Unicode were (mostly?) American shouldn't be a surprise; they had the most at stake in creating interoperable systems that spanned linguistic boundaries. http://www.unicode.org/history/earlyyears.html http://www.unicode.org/history/earlyyears.html A purely academic solution may have been better. That's not certain. But it's pretty clear that any solution that didn't address the needs of pre-existing commercial systems would never be adopted. I'm surprised this hasn't been emphasized more. I have more thoughts on how to solve the problem that the author complains about but I'll leave that for another day.
• ticking 12y ago
  Even though I'm not a native english speaker and couldn't write my name in ASCI, I really despise Unicode. Its broken technically and a setback socially. Unicode itself is such a unfathomably huge project that it's impossible to do it right, too many languages, too many weird writing systems, and too many ways to do mathematical notation on paper that can't be expressed. Just look at the code pages, they are an utter mess. Computers and ASCI were a chance to start anew, to establish english as a universal language, spoken by everybody. The pressure on governments who would wanted to partake in the digital revolution would have forced them to introduce it as an official secondary language. Granted english is not the nicest language, but is the best candidate we have in terms of adoption, and relative simplicity (Mandarin is another contester, but several thousand logograms are really impractical to encode.). Take a look at the open source world, where everybody speaks english and collaborates regardless of nationality. One of the main factors why this is possible, is that we found a common language, forced on us by the tools and programming languages we use. If humanity wants get rid of wars, poverty and nationalism, we have to find a common language first. A simple encoding and universal communication is a feature, fragmented communication is the bug. Besides. UTF-8 is broken because it doesn't allow for constant time random character access and length counting.
  • krdln 12y ago
    Why do you think English is the best candidate for the universal language, how do you define simplicity? First of all, pronunciation and spelling are almost unrelated and you have to learn them separately. That results in really different accents throughout the world. Even if you look at AmE and BrE, they differ much at the word level. Which one you want to choose? Besides, personally I find English really ambiguous and density of idioms in average text repelling, although that's only a subjective opinion. Usage of Latin alphabet in English seems like it's on plus, but there's at least one language that uses that simple alphabet better. > Besides. UTF-8 is broken because it doesn't allow for constant time random character and length counting. And why you'd want that? And how do you define length? Are you a troll?
    • ticking 12y ago
      English is the best candidate because it has the second largest user base (1.2 Billion vs 1.3 Billion for Mandarin), http://en.wikipedia.org/wiki/List_of_languages_by_total_number_of_speakers http://en.wikipedia.org/wiki/List_of_languages_by_total_numb... and is twice as spoken as the third most popular language Spanish. (0.55 Billion) If I got to pick the universal language, it would be Lojban (a few hundred speakers), but that is not a realistic goal, teaching the other 6 Billion people a language that is already spoken by 1/7th of the population is at least plausible. > Why would you want that... Why would, you not want that?! Many popular programming languages are based on array indexing through pointer arithmetic, having a variable width encoding there is a horrible idea, because you have to iterate through the text to get to an index. Length is the number of characters, which is just the number of bytes in ASCI, but has to be calculated by looking at every character in UTF-8.
• kalleboo 12y ago
  Re: the Han Unification debate that's going on in parallel here, I think CJK unification makes sense from the Unicode point of view (although if they had to choose again after the expansion beyond 16-bit I doubt they'd bother with the political headache). The problem stems from the fact that only a few high-level text storage formats (HTML, MS Word, etc) have a way to mark what language some text is in. There's no way to include both Japanese and Chinese in a git commit log, or a comment on Hacker News. Sure you can say "that's just the problem of the software developer!" but that's what we said about supporting different character sets before Unicode, and we abandoned that attitude. Hacker News is never going to add that feature on their own. What's needed is either a "language switch" glyph in Unicode (like the right-to-left/left-to-right ones) or a layer ontop of Unicode which does implement one that gets universally implemented in OSes and rendering systems.
• olgeni 12y ago
  I searched for "white" and of course I found a few, as expected.
• enupten 12y ago
  I couldn't agree with you more on this! But there is more to it when it comes to India, it is mostly that "nobody really cares". It is sad that while India is develops, it is rapidly leaving its many languages behind when it comes to the computer; indeed, while Chinese/Arabic keyboards are extremely common, one would be hard pressed to find Hindi, "Nagari", (let alone Bengali) keyboards in India. This despite the nauseous linguistic jingoism in the country's political history. Indeed there has been, sadly very little to show, and it has infact, gotten worse past independence, as with many things; see, http://www.columbia.edu/cu/mesaas/faculty/directory/pollock_pub/classics.pdf http://www.columbia.edu/cu/mesaas/faculty/directory/pollock_... I would not be surprised, at this rate, if India turns into a monoculture, two centuries hence. P.S: Speaking of representation, the script the OP talks of, belongs to the set of phonetically accurate scripts; one which while being populated mostly by Indic scripts, is given - quite disgracefully - the name "abugida", after a lone Semitic script - "Geez" - from Ethiopia - which ironically is probably derived from one of the Indic ones. Systematic biases are far too apparent in Indology.
• sirseal 12y ago
  Let's use UTF-16: http://en.wikipedia.org/wiki/UTF-16 http://en.wikipedia.org/wiki/UTF-16. Problem solved.
• 4bpp 12y ago
  I'd imagine that from the point of view of a Unicode consortium member, the question as to whether to include a particular Bengali glyph some argued to be obsolete looks more like "should we lower the threshold for what characters in a language are considered deserving of a separate codepoint, potentially exposing ourselves to a deluge of O(#codepoints/#characters per language) requests for obscure variant characters until we actually run out of them", whereas the question as to whether to include a sign for poop in the end boils down to "can we spare O(1) codepoints to prove to the world that we are not humourless fascists". The particular decision in this case might well be ill-informed, but I think any judgement that the Unicode Consortium is engaging in cultural supremacism (as opposed to doing the usual thing of wanting anglo-american capitalist money) is somewhat far-fetched. The right solution, I think, would be to replace Unicode with a truly intrinsically variable-length standard such as an unbounded UTF-8 - many of the arguments that were fielded in favour of having the option of fixed-width encodings seem to have melted away now that almost everything that interfaces with users has a layer of high-level glue code, naive implementations of strings have been deemed harmful to security and even ostensibly "embedded" platforms can get away with Java. Rather than having an overwhelmed committee of American industrialists decide over the faith of every single codepoint, then, they could simply allocate a prefix to every catalogued writing system and defer the rest to non-technical authorities whose suggested lists would only require basic rubber-stamp sanity-checking. > "You can't write your name in your native language, but at least you can tweet your frustration with an emoji face that's the same shade of brown as yours!" This seems fairly characteristic of the apparent belief of social justice activists - and I can't imagine that Unicode's inclusion of skin colours would be a result of anything other than pressure by the same - that they can improve the whole world with remedies conceived against the background of US American race/identity politics.
  • webitorial 12y ago
    Had nothing to do with proving they were not humorless fascists. There was a legitimate need for a universal codepoint among Japanese cellphone operators. Keep in mind, Japanese is a language where frequently entire concepts are represented in a single character, so this isn't perhaps as odd as you think. "Poo" had a specific semantic in "cellphone Japanese" that the market demanded. To be used interoperably with other Unicode characters, various 'emoji' were added to Unicode. I, for one, retain the right to be called a humorless fascist.
• anon4 12y ago
  Or maybe you could start writing using a normal alphabet.
• deleted 12y ago
  [deleted]
• coolguy43 12y ago
  It's really hard. Try to learn english with native speaker first like at http://preply.com/en/skype/english-native-speakers http://preply.com/en/skype/english-native-speakers.
• azinman2 12y ago
  This is one of the most interesting HN post+comments I've read yet, in part because it mixes technology with culture and history. It also takes advantage of the diverse HN community and their own native languages. As an American (English speaker) who has studied French, Hebrew and Japanese, I can appreciate the complexity of input balanced against standards and the needs of programmers. It's a fucking hard problem, but I don't think that blaming the unicode consortium is the right place to do so. They seem to be doing a reasonably good job in trying to get everything in, and really they need input from outsiders to do this well. It requires people with linguistic & technical backgrounds which is probably why random governments may have a harder time providing input. Further from all the points people are making about uppercase/lowercase/hyphenation across languages, it sounds to me like there really needs to be a super-standarized open source implementation of the things you want to do with text, not just purely encode it. I don't think that exists, and it might be a good place for the unicode people to branch to.
• ramgorur 12y ago
  I can type anything in bengali without any issue -- হঠাৎ, কিংকর্তব্যবিমূঢ়, সংশপ্তক, বর্ষাকাল... Not sure about other "second class" languages, this open source phonetic (unicode) keyboard is extremely popular in bangladesh, and everyone does all sorts of bengali stuffs with it -- https://www.omicronlab.com/avro-keyboard.html https://www.omicronlab.com/avro-keyboard.html