7 ms·

I came in expecting to read an article bemoaning some niche language and playing the diversity card. I was not disappointed, but as I kept reading, the author m

I came in expecting to read an article bemoaning some niche language and playing the diversity card. I was not disappointed, but as I kept reading, the author made some very good points. I don't really care that the organization is run by white men who speak English, because frankly the entire computing industry and telecommunications industry is based on that. I'm not going to argue about the original sin there, because the facts speak for themselves: they got shit done. What I do find really disturbing, though, is that somehow both Linear A and Linear B are included, in their (as known) entirety, while Bengali is not--two languages spoken and written by nobody for over a millenia are included whereas a language spoken and written by millions of people is not. That's bad. The discussion of somehow creating a CJK superset and then deviations to support each of those languages is worse, and the author's remark about trying to "unify" Greek, Roman, Cyrillic, and Swedish struck home. Personally, I think that it's nice to have an ever-decreasing number of languages to support, preferably English. The first part of that is because it's annoying enough to unambiguously parse one human language much less dozens, and the second part is pure convenience and small-mindedness on my part. That said, we don't have to buy into diversity or identity or anything else to note that something here is amiss. This is bad allocation of engineering resources. EDIT: And yes, one could make the argument that the computer/teleco industry is being produced in Shenzhen, to which I would respond that the intellectual capital was handled by the West. Nintendo hired SGI, Sony IBM. There is certainly a wonderful and burgeoning domestic talent, but the paths were set elsewhere, to the best of my knowledge.

wavefunction 12y ago
I really would like to downvote you for your harping about "diversity" whatever you mean by that. I can guess what you mean, as I live in the United States and am exposed to other people who harp similarly thus. I'm not going to down vote you though, just point out your weird usage of the term and mention that this is about proper information transmission, the foundation of computing and the Internet. So maybe get off your agenda regarding whatever you mean by "diversity" and start to see the people of all types as fellow humans and not actors in your own private psycho-drama.
com2kid 12y ago
Han unification makes things really hard for programmers. You end up with code that tries to guess what language a string is in to pick out which character set should be used! It is an absolute nightmare and a horrid idea.
- 1ris 12y ago
  Well, if you find Hiragana/Katakana it's Japanese, if you find Chữ Nôm it's Vietnamese. Otherwise it's Chinese (Well, given the definition of "language" is very hard in the context of Chinese). From a purly theoritical perspecitve the Han unification looks like a great idea. Image the horror of normalisations if it didn't happen. ; and greek questionmark would have been a joke in comparison.
- theon144 12y ago
  Isn't the idea technically that the code shouldn't even have to guess? Why isn't this the case?
  - Someone 12y ago
    Imagine a world where the British always write the lowercase letter g as a single-story glyph (http://en.m.wikipedia.org/wiki/G#Typographic_variants http://en.m.wikipedia.org/wiki/G#Typographic_variants). The colonies start writing it identically, but after a while, they start writing it as a double-story g. After a century or so, nobody in he colonies writes the single-story variant, and all Brits always do. The unicode consortium studies the case and concludes that there is only a single g with variations in the way it is written. Because of that, it creates a single code point for the lowercase 'g' character. Now, suppose a web page stores the text 'goto' in Unicode as the code points for 'g', 'o', 't', and 'o'. To write the code that renders that string, you will find you need to know whether the text is written in British English or in colonial English.
    - Anderkent 12y ago
      Isn't this the question of a font? In which case the client chooses if they want to use a font with a double-story or a single-story g?
beat 12y ago
Bengali is the seventh most widely spoken language in the world, with more native speakers than Russian. That's hardly a "niche language". And, as pg has pointed out, making the computer industry English-centric and America-centric is greatly limiting us. Intelligence and will to power are equally distributed across the globe. But institutional barriers to success are not. This is a classic example of an institutional barrier, and it makes all of us poorer.
- exelius 12y ago
  Right, but some languages are insanely complex to implement. It might be a better idea to teach English to people around the globe rather than cater to every individual need (which will still leave people unable to communicate across languages). I'm not saying other languages should go away -- but the world would also benefit from having a "universal" language, which is more or less English at this point (Mandarin is spoken by more people but is rarely spoken outside of Asia). If we want to maximize intelligence and will to power, it would be best if like-minded individuals could communicate regardless of where they're from or what their native language is.
  - mbillie1 12y ago
    > some languages are insanely complex to implement I don't understand this, is there more to implementing a language than creating glyphs for its character set? I wouldn't think the linguistic complexity would matter at all, only the number of glyphs in the 'alphabet' or similar?
    - Crito 12y ago
      http://en.wikipedia.org/wiki/Complex_text_layout http://en.wikipedia.org/wiki/Complex_text_layout Hypothetically you could construct a language where the glyphs are easy to generate procedurally on the fly by people who are fluent in that language, but who's full space of possible glyphs is staggeringly massive. Suppose a language with tens or hundreds of thousands of "base" glyphs, but with a unique variant on each glyph depending on what is to the left and right of it. With that alone, for N base glyphs, you could have N^2 variants of each glyph. I don't know if that sort of language exists. I don't see any reason why it couldn't though.
makomk 12y ago
Yeah, there's only one minor problem with the article: those emoji symbols it bemoans were added for the Japanese market. Western and English-language support for them was very much an afterthought. What's more, the reason they were added was because Japanese mobile providers were already offering emoji support through various non-standard and not entirely compatible encodings.
frivoal 12y ago
Han unification has been overly aggressive about merging some characters, but the basic principle is not as flawed as it is some times (as in this article) made to sound. The vast majority of Japanese and Chinese characters are not only similar, they are identical. Not all are. Some are clearly different characters deriving from a common historical root, and should not be unified. Sometimes, when characters are a bit different, it is a matter of font. Out of the tends of thousands of characters used by these languages, it is not surprising that mistakes have been made, and yes, han unification has been a bit too far reaching. But not unifying at all would have been much worse. You don't want a in German and a in English to be different letters just because Helvetica and Baskerville look different. Also, since CJK languages are comfortable with having an unbounded set of letters to work from, they tend to be comfortable with declaring every odd thing a separate letter. Witness the mess that half-width katakana is. Or notice that the origin of most emoji is not in the disregard of white old men for "lesser" languages, but comes from importing existing characters from Japanese encodings. Which also hints at one more factor. Historically, and still very much to this day, Japan has been much more active in standardization bodies than most other non western countries. Unsurprisingly, Japanese is much better supported by modern software than many other non western languages.
- com2kid 12y ago
  > The vast majority of Japanese and Chinese characters are not only similar, they are identical. Not all are. Some are clearly different characters deriving from a common historical root, and should not be unified. And what about traditional versus simplified? Which glyph set do I use? Oh wait, thanks to Han unification, I now need to rely on bloody environment variables to decide! For Chinese text, rendering a string of text involves not just the string, but also the local runtime. Joy of joys. > You don't want a in German and a in English to be different letters just because Helvetica and Baskerville look different. If following the spec means my users become partially illiterate then there are problems with the spec.
  - Tomte 12y ago
    Antiqua, Fractura, Schwabacher, Textura and all those other charming variants of writing European languages don't have seperate code points for all those presentational variations. Do we white Western men happen to discriminate against ourselves?
konstruktor 12y ago
I can't think of any telecommunication or computing device that I own or have touched in the last few years that was built by white, English speaking men. Regarding software, I googled a bit to find numbers, and one of the first articles actually giving numbers says 19 percent of software developers are in the largest English speaking country, the US: http://www.techrepublic.com/blog/european-technology/there-are-185-million-software-developers-in-the-world-but-which-country-has-the-most/ http://www.techrepublic.com/blog/european-technology/there-a...
Dewie 12y ago
> Personally, I think that it's nice to have an ever-decreasing number of languages to support, preferably English. The first part of that is because it's annoying enough to unambiguously parse one human language much less dozens, and the second part is pure convenience and small-mindedness on my part. I'll go further and claim that it's all about convenience and small-mindedness. It's so typical of a certain group of programmers to complain like this. Ugh, why do we have different languages and ways of doing things? Let's just converge on the one I grew up with, or use[1]. Hopefully you won't be forced to work on this kind of thing, or any kind of work that you don't care about/think is worth it. But until you do, maybe you could keep your self-centred navel gazing to yourself. [1] In the case of English, most programmers who express this opinion write perfectly good English, though they might not be native speakers. They wouldn't be inconvenienced at all if suddenly everyone forgot about their own language and started speaking English, especially since they don't seem to care about languages beyond having some way of communicating with other people.

wavefunction 12y ago

I really would like to downvote you for your harping about "diversity" whatever you mean by that. I can guess what you mean, as I live in the United States and am exposed to other people who harp similarly thus. I'm not going to down vote you though, just point out your weird usage of the term and mention that this is about proper information transmission, the foundation of computing and the Internet. So maybe get off your agenda regarding whatever you mean by "diversity" and start to see the people of all types as fellow humans and not actors in your own private psycho-drama.

com2kid 12y ago

Han unification makes things really hard for programmers. You end up with code that tries to guess what language a string is in to pick out which character set should be used! It is an absolute nightmare and a horrid idea.

1ris 12y ago
Well, if you find Hiragana/Katakana it's Japanese, if you find Chữ Nôm it's Vietnamese. Otherwise it's Chinese (Well, given the definition of "language" is very hard in the context of Chinese). From a purly theoritical perspecitve the Han unification looks like a great idea. Image the horror of normalisations if it didn't happen. ; and greek questionmark would have been a joke in comparison.
theon144 12y ago
Isn't the idea technically that the code shouldn't even have to guess? Why isn't this the case?
- Someone 12y ago
  Imagine a world where the British always write the lowercase letter g as a single-story glyph (http://en.m.wikipedia.org/wiki/G#Typographic_variants http://en.m.wikipedia.org/wiki/G#Typographic_variants). The colonies start writing it identically, but after a while, they start writing it as a double-story g. After a century or so, nobody in he colonies writes the single-story variant, and all Brits always do. The unicode consortium studies the case and concludes that there is only a single g with variations in the way it is written. Because of that, it creates a single code point for the lowercase 'g' character. Now, suppose a web page stores the text 'goto' in Unicode as the code points for 'g', 'o', 't', and 'o'. To write the code that renders that string, you will find you need to know whether the text is written in British English or in colonial English.
  - Anderkent 12y ago
    Isn't this the question of a font? In which case the client chooses if they want to use a font with a double-story or a single-story g?

beat 12y ago

Bengali is the seventh most widely spoken language in the world, with more native speakers than Russian. That's hardly a "niche language". And, as pg has pointed out, making the computer industry English-centric and America-centric is greatly limiting us. Intelligence and will to power are equally distributed across the globe. But institutional barriers to success are not. This is a classic example of an institutional barrier, and it makes all of us poorer.

exelius 12y ago
Right, but some languages are insanely complex to implement. It might be a better idea to teach English to people around the globe rather than cater to every individual need (which will still leave people unable to communicate across languages). I'm not saying other languages should go away -- but the world would also benefit from having a "universal" language, which is more or less English at this point (Mandarin is spoken by more people but is rarely spoken outside of Asia). If we want to maximize intelligence and will to power, it would be best if like-minded individuals could communicate regardless of where they're from or what their native language is.
- mbillie1 12y ago
  > some languages are insanely complex to implement I don't understand this, is there more to implementing a language than creating glyphs for its character set? I wouldn't think the linguistic complexity would matter at all, only the number of glyphs in the 'alphabet' or similar?
  - Crito 12y ago
    http://en.wikipedia.org/wiki/Complex_text_layout http://en.wikipedia.org/wiki/Complex_text_layout Hypothetically you could construct a language where the glyphs are easy to generate procedurally on the fly by people who are fluent in that language, but who's full space of possible glyphs is staggeringly massive. Suppose a language with tens or hundreds of thousands of "base" glyphs, but with a unique variant on each glyph depending on what is to the left and right of it. With that alone, for N base glyphs, you could have N^2 variants of each glyph. I don't know if that sort of language exists. I don't see any reason why it couldn't though.

makomk 12y ago

Yeah, there's only one minor problem with the article: those emoji symbols it bemoans were added for the Japanese market. Western and English-language support for them was very much an afterthought. What's more, the reason they were added was because Japanese mobile providers were already offering emoji support through various non-standard and not entirely compatible encodings.

frivoal 12y ago

Han unification has been overly aggressive about merging some characters, but the basic principle is not as flawed as it is some times (as in this article) made to sound. The vast majority of Japanese and Chinese characters are not only similar, they are identical. Not all are. Some are clearly different characters deriving from a common historical root, and should not be unified. Sometimes, when characters are a bit different, it is a matter of font. Out of the tends of thousands of characters used by these languages, it is not surprising that mistakes have been made, and yes, han unification has been a bit too far reaching. But not unifying at all would have been much worse. You don't want a in German and a in English to be different letters just because Helvetica and Baskerville look different. Also, since CJK languages are comfortable with having an unbounded set of letters to work from, they tend to be comfortable with declaring every odd thing a separate letter. Witness the mess that half-width katakana is. Or notice that the origin of most emoji is not in the disregard of white old men for "lesser" languages, but comes from importing existing characters from Japanese encodings. Which also hints at one more factor. Historically, and still very much to this day, Japan has been much more active in standardization bodies than most other non western countries. Unsurprisingly, Japanese is much better supported by modern software than many other non western languages.

com2kid 12y ago
> The vast majority of Japanese and Chinese characters are not only similar, they are identical. Not all are. Some are clearly different characters deriving from a common historical root, and should not be unified. And what about traditional versus simplified? Which glyph set do I use? Oh wait, thanks to Han unification, I now need to rely on bloody environment variables to decide! For Chinese text, rendering a string of text involves not just the string, but also the local runtime. Joy of joys. > You don't want a in German and a in English to be different letters just because Helvetica and Baskerville look different. If following the spec means my users become partially illiterate then there are problems with the spec.
- Tomte 12y ago
  Antiqua, Fractura, Schwabacher, Textura and all those other charming variants of writing European languages don't have seperate code points for all those presentational variations. Do we white Western men happen to discriminate against ourselves?

konstruktor 12y ago

I can't think of any telecommunication or computing device that I own or have touched in the last few years that was built by white, English speaking men. Regarding software, I googled a bit to find numbers, and one of the first articles actually giving numbers says 19 percent of software developers are in the largest English speaking country, the US: http://www.techrepublic.com/blog/european-technology/there-are-185-million-software-developers-in-the-world-but-which-country-has-the-most/ http://www.techrepublic.com/blog/european-technology/there-a...

Dewie 12y ago

> Personally, I think that it's nice to have an ever-decreasing number of languages to support, preferably English. The first part of that is because it's annoying enough to unambiguously parse one human language much less dozens, and the second part is pure convenience and small-mindedness on my part. I'll go further and claim that it's all about convenience and small-mindedness. It's so typical of a certain group of programmers to complain like this. Ugh, why do we have different languages and ways of doing things? Let's just converge on the one I grew up with, or use[1]. Hopefully you won't be forced to work on this kind of thing, or any kind of work that you don't care about/think is worth it. But until you do, maybe you could keep your self-centred navel gazing to yourself. [1] In the case of English, most programmers who express this opinion write perfectly good English, though they might not be native speakers. They wouldn't be inconvenienced at all if suddenly everyone forgot about their own language and started speaking English, especially since they don't seem to care about languages beyond having some way of communicating with other people.