8 ms·

> I could write the author's name fine: আদিত্য Author here. Well, yes and no. The jophola at the end is not actually given its own codepoint[0]. The best anal

> I could write the author's name fine: আদিত্য Author here. Well, yes and no. The jophola at the end is not actually given its own codepoint[0]. The best analogy I can give is to a ligature in English[1]. The Bengali fonts that you have installed happen to render it as a jophola, the way some fonts happen to render "ff" as "ﬀ" but that's not the same thing as saying that it actually is a jophola (according to the Unicode standard). The difference between the jophola and an English ligature, though, is English ligatures are purely aesthetic. Typing two "f" characters in a row has the same obvious semantic meaning as the ﬀ ligature, whereas the characters that are required to type a jophola have no obvious semantic, phonetic, or orthographic connection to the jophola. [0] http://unicode.org/charts/PDF/U0980.pdf http://unicode.org/charts/PDF/U0980.pdf [1] Some fonts will render (e.g.) two "f"s in a row as if they were a ligature, even though it's not a true ﬀ(U+FB00).

yuriks 12y ago
Unicode makes extensive use of combining characters for european languages, for example to produce diacritics: ìǒ or even for flag emoji. A correct rendering system will properly combine those, and if it doesn't then that's a flaw in the implementation, not the standard. It seems like you're trying to single out combining pairs as "less legitimate" when they're extensively used in the standard.
- chimeracoder 12y ago
  > Unicode makes extensive use of combining characters for european languages, for example to produce diacritics: ìǒ or even for flag emoji. But it doesn't, for example say that a lowercase "b" is simply "a lowercase 'l' followed by an 'o' followed by an invisible joiner", because no native English speaker thinks of the character "b" as even remotely related to "lo" when reading and writing. > It seems like you're trying to single out combining pairs as "less legitimate" when they're extensively used in the standard. I'm saying that Unicode only does it in English where it makes semantic sense to a native English speaker. It does it in Bengali even where it makes little or no semantic sense to a native Bengali speaker.
  - maxlybbert 12y ago
    > > It seems like you're trying to single out combining pairs as "less legitimate" when they're extensively used in the standard. > I'm saying that Unicode only does it in English where it makes semantic sense to a native English speaker. Well, combining characters almost never come up in English. The best I can think of would be the use of cedillas, diaereses, and acute accents in words like façade, coördinate and renownèd (I've been reading Tolkien's translation of Beowulf, and he used renownèd a lot). Thinking about the Spanish I learned in high school, ch, ll, ñ, and rr are all considered separate letters (i.e., the Spanish alphabet has 30 letters; ch is between c and d, ll is between l and m, ñ is between n and o, and rr is between r and s; interestingly, accented vowels aren't separate letters). Unicode does not provide code points for ch, ll, or rr; and ñ has a code point more from historical accident than anything (the decision to start with Latin1). Then again, I don't think Spanish keyboards have separate keys for ch, ll, or rr. Portuguese, on the other hand, doesn't officially include k or y in the alphabet. But it uses far more accents than Spanish. So, a, ã and á are all the same letter. In a perfect world, how would Unicode handle this? Either they accept the Spanish view of the world, or the Portuguese view. Or, perhaps, they make a big deal about not worrying about languages and instead worrying about alphabets ( http://www.unicode.org/faq/basic_q.html#4 http://www.unicode.org/faq/basic_q.html#4 ). They haven't been perfect. And they've certainly changed their approach over time. And I suspect they're including emoji to appear more welcoming to Japanese teenagers than they were in the past. But (1) combining characters aren't second-class citizens, and (2) the standard is still open to revisions ( http://www.unicode.org/alloc/Pipeline.html http://www.unicode.org/alloc/Pipeline.html ).
    - darklajid 12y ago
      I'm coming from a German background and I sympathize with the author. German has 4 (7 if you consider cases) non-ASCII characters: äüöß(and upper-case umlauts). All of these are unique, well-defined codepoints. That's not related to composing on a keyboard. In fact, although I'm German I'm using the US keyboard layout and HAD to compose these characters now. But I wouldn't need to and the result is a single codepoint again..
- hetman 12y ago
  Except unicode also has invididual codepoints for "ǒ" and "ì".
sdg1 12y ago
> The Bengali fonts that you have installed happen to render it as a jophola It's not only the Bengali font - the text rendering framework of my operating system also needs to have a bunch of complex rules to figure out that a jophola needs to be rendered. It also needs to know that the visual ordering of i-kar is before the preceding consonant cluster (দ in আদিত্য). > the characters that are required to type a jophola have no semantic, phonetic, or orthographic connection to the jophola. Not so sure about that. The fact that it's called a "jo"-phola points to a relationship. The relationship may have become less apparent as the script has evolved (though there are words such as সহ্য which makes the relationship more visible), but the distinction is still not as pronounced as between "ta" and "khanda-ta". For the khanda-ta case, it was explicit from the the then-current editions of the dictionaries produced by the language bodies of both Bangladesh and West Bengal that the character had become distinct (স্বতন্ত্র বর্ণ was the phrase that was used). As far as I know, there hasn't been any such claim about jophola from the language bodies. Also, if you look at the collation in Bengali dictionaries, jo-phola is treated as (্+য) for collation.
- chimeracoder 12y ago
  Is Bengali your first language? While one can make the case that ত্য is simply "'to' - 'o' + 'ya' = 'to'"[0][1], it's rather confusing mental acrobatics, and it doesn't reflect either how the writing system is taught, or how native speakers use it and think of it on a day-to-day basis. If anything, your comment makes a stronger argument for consolidating ই and ি (they are literally the same letter and phoneme, but written differently in different contexts) than for combining the viram and য into the jophola. [0] To non-Bengali speakers reading this, yes, this is how that construction would work, and yes, I am aware that the arithmetic doesn't appear to add up (which I guess is part of the point). [1] Also, now that I think about it, the য is a consonant, not a vowel, so using it in place of a vowel is doubly awkward. This is particularly an issue in Bengali, where sounds that might be consonants in English (like "r" and "l") can be either consonants or vowels in Bengali, depending on the word.
  - sdg1 12y ago
    > Is Bengali your first language? Yes. > [...] it's rather confusing mental acrobatics, and it doesn't reflect either how the writing system is taught, or how native speakers use it and think of it on a day-to-day basis. Mental acrobatics are part-and-parcel of the language, either in digital or non-digital form. If I were to spell out your name aloud, I would end with "ত-এ য-ফলা", which doesn't really say anything about how ত্য is pronounced. While writing on paper, we say "ক-এ ইকার", and then we reorder what we just said to write the ইকার in front of the ক. Even more complicated mental acrobatics - we say ক-এ ওকার, and then proceed to write half of the ওকার in front of the ক and then the other half, after the ক. We don't necessarily think about these when we carry out these acrobatics in our head, but they exist, and we have made the layer on top of the encoding system (rendering, and to some extent, input) deal with these acrobatics as well. My point in the original comment (and to some extent in the preceding one) was to emphasize that a lot of these issues are at the input method level - we should not have to think about encoding as long as it accurately and unambiguously represent whatever we want it to represent. Just out of curiosity - I would be interested to know more about your learning experience that you feel is not well aligned with the representation of jophola as it is currently.
ramviswanadha 12y ago
> whereas the characters that are required to type a jophola have no obvious semantic, phonetic, or orthographic connection to the jophola. Then it is an input method issue not an encoding issue.
deleted 12y ago
[deleted]
png_hero 12y ago
Doesn't this reply invalidate your whole point of the article? Seems like there was a lot of hard work put in on making the "khanda-ta" work properly?
Manishearth 12y ago
(I can't read Bengali, so I'm not entirely sure what the johphola is, but I'm trying to relate this to Devanagari -- if my analogy is mistaken or if you don't know Devanagari, let me know) I don't see anything discriminatory about not giving glyphs their own codepoints. Devanagari has tons of glyphs which are logically broken up into consonants, modifying diacritics, and bare vowels. Do we really need separate codepoints for these[1] when they can be made by combinations of these[2]? I mentioned this as a reply to another comment, but it's only a Unicode problem if: - There is no way to write the glyph as a combination of code points - There is a way to write the glyph as a combination of code points, but the same combination could mean something else (not counting any rendering mistakes, the question is about if Unicode defines it to mean something else) If it's hard to input, that's the fault of the input method. If it doesn't combine right, it's the fault of the font. [1]: http://en.wikipedia.org/wiki/Devanagari#Biconsonantal_conjuncts http://en.wikipedia.org/wiki/Devanagari#Biconsonantal_conjun... [2]: http://en.wikipedia.org/wiki/Devanagari_%28Unicode_block%29 http://en.wikipedia.org/wiki/Devanagari_%28Unicode_block%29

yuriks 12y ago

Unicode makes extensive use of combining characters for european languages, for example to produce diacritics: ìǒ or even for flag emoji. A correct rendering system will properly combine those, and if it doesn't then that's a flaw in the implementation, not the standard. It seems like you're trying to single out combining pairs as "less legitimate" when they're extensively used in the standard.

chimeracoder 12y ago
> Unicode makes extensive use of combining characters for european languages, for example to produce diacritics: ìǒ or even for flag emoji. But it doesn't, for example say that a lowercase "b" is simply "a lowercase 'l' followed by an 'o' followed by an invisible joiner", because no native English speaker thinks of the character "b" as even remotely related to "lo" when reading and writing. > It seems like you're trying to single out combining pairs as "less legitimate" when they're extensively used in the standard. I'm saying that Unicode only does it in English where it makes semantic sense to a native English speaker. It does it in Bengali even where it makes little or no semantic sense to a native Bengali speaker.
- maxlybbert 12y ago
  > > It seems like you're trying to single out combining pairs as "less legitimate" when they're extensively used in the standard. > I'm saying that Unicode only does it in English where it makes semantic sense to a native English speaker. Well, combining characters almost never come up in English. The best I can think of would be the use of cedillas, diaereses, and acute accents in words like façade, coördinate and renownèd (I've been reading Tolkien's translation of Beowulf, and he used renownèd a lot). Thinking about the Spanish I learned in high school, ch, ll, ñ, and rr are all considered separate letters (i.e., the Spanish alphabet has 30 letters; ch is between c and d, ll is between l and m, ñ is between n and o, and rr is between r and s; interestingly, accented vowels aren't separate letters). Unicode does not provide code points for ch, ll, or rr; and ñ has a code point more from historical accident than anything (the decision to start with Latin1). Then again, I don't think Spanish keyboards have separate keys for ch, ll, or rr. Portuguese, on the other hand, doesn't officially include k or y in the alphabet. But it uses far more accents than Spanish. So, a, ã and á are all the same letter. In a perfect world, how would Unicode handle this? Either they accept the Spanish view of the world, or the Portuguese view. Or, perhaps, they make a big deal about not worrying about languages and instead worrying about alphabets ( http://www.unicode.org/faq/basic_q.html#4 http://www.unicode.org/faq/basic_q.html#4 ). They haven't been perfect. And they've certainly changed their approach over time. And I suspect they're including emoji to appear more welcoming to Japanese teenagers than they were in the past. But (1) combining characters aren't second-class citizens, and (2) the standard is still open to revisions ( http://www.unicode.org/alloc/Pipeline.html http://www.unicode.org/alloc/Pipeline.html ).
  - darklajid 12y ago
    I'm coming from a German background and I sympathize with the author. German has 4 (7 if you consider cases) non-ASCII characters: äüöß(and upper-case umlauts). All of these are unique, well-defined codepoints. That's not related to composing on a keyboard. In fact, although I'm German I'm using the US keyboard layout and HAD to compose these characters now. But I wouldn't need to and the result is a single codepoint again..
hetman 12y ago
Except unicode also has invididual codepoints for "ǒ" and "ì".

sdg1 12y ago

> The Bengali fonts that you have installed happen to render it as a jophola It's not only the Bengali font - the text rendering framework of my operating system also needs to have a bunch of complex rules to figure out that a jophola needs to be rendered. It also needs to know that the visual ordering of i-kar is before the preceding consonant cluster (দ in আদিত্য). > the characters that are required to type a jophola have no semantic, phonetic, or orthographic connection to the jophola. Not so sure about that. The fact that it's called a "jo"-phola points to a relationship. The relationship may have become less apparent as the script has evolved (though there are words such as সহ্য which makes the relationship more visible), but the distinction is still not as pronounced as between "ta" and "khanda-ta". For the khanda-ta case, it was explicit from the the then-current editions of the dictionaries produced by the language bodies of both Bangladesh and West Bengal that the character had become distinct (স্বতন্ত্র বর্ণ was the phrase that was used). As far as I know, there hasn't been any such claim about jophola from the language bodies. Also, if you look at the collation in Bengali dictionaries, jo-phola is treated as (্+য) for collation.

chimeracoder 12y ago
Is Bengali your first language? While one can make the case that ত্য is simply "'to' - 'o' + 'ya' = 'to'"[0][1], it's rather confusing mental acrobatics, and it doesn't reflect either how the writing system is taught, or how native speakers use it and think of it on a day-to-day basis. If anything, your comment makes a stronger argument for consolidating ই and ি (they are literally the same letter and phoneme, but written differently in different contexts) than for combining the viram and য into the jophola. [0] To non-Bengali speakers reading this, yes, this is how that construction would work, and yes, I am aware that the arithmetic doesn't appear to add up (which I guess is part of the point). [1] Also, now that I think about it, the য is a consonant, not a vowel, so using it in place of a vowel is doubly awkward. This is particularly an issue in Bengali, where sounds that might be consonants in English (like "r" and "l") can be either consonants or vowels in Bengali, depending on the word.
- sdg1 12y ago
  > Is Bengali your first language? Yes. > [...] it's rather confusing mental acrobatics, and it doesn't reflect either how the writing system is taught, or how native speakers use it and think of it on a day-to-day basis. Mental acrobatics are part-and-parcel of the language, either in digital or non-digital form. If I were to spell out your name aloud, I would end with "ত-এ য-ফলা", which doesn't really say anything about how ত্য is pronounced. While writing on paper, we say "ক-এ ইকার", and then we reorder what we just said to write the ইকার in front of the ক. Even more complicated mental acrobatics - we say ক-এ ওকার, and then proceed to write half of the ওকার in front of the ক and then the other half, after the ক. We don't necessarily think about these when we carry out these acrobatics in our head, but they exist, and we have made the layer on top of the encoding system (rendering, and to some extent, input) deal with these acrobatics as well. My point in the original comment (and to some extent in the preceding one) was to emphasize that a lot of these issues are at the input method level - we should not have to think about encoding as long as it accurately and unambiguously represent whatever we want it to represent. Just out of curiosity - I would be interested to know more about your learning experience that you feel is not well aligned with the representation of jophola as it is currently.

ramviswanadha 12y ago

> whereas the characters that are required to type a jophola have no obvious semantic, phonetic, or orthographic connection to the jophola. Then it is an input method issue not an encoding issue.

deleted 12y ago

[deleted]

png_hero 12y ago

Doesn't this reply invalidate your whole point of the article? Seems like there was a lot of hard work put in on making the "khanda-ta" work properly?

Manishearth 12y ago

(I can't read Bengali, so I'm not entirely sure what the johphola is, but I'm trying to relate this to Devanagari -- if my analogy is mistaken or if you don't know Devanagari, let me know) I don't see anything discriminatory about not giving glyphs their own codepoints. Devanagari has tons of glyphs which are logically broken up into consonants, modifying diacritics, and bare vowels. Do we really need separate codepoints for these[1] when they can be made by combinations of these[2]? I mentioned this as a reply to another comment, but it's only a Unicode problem if: - There is no way to write the glyph as a combination of code points - There is a way to write the glyph as a combination of code points, but the same combination could mean something else (not counting any rendering mistakes, the question is about if Unicode defines it to mean something else) If it's hard to input, that's the fault of the input method. If it doesn't combine right, it's the fault of the font. [1]: http://en.wikipedia.org/wiki/Devanagari#Biconsonantal_conjuncts http://en.wikipedia.org/wiki/Devanagari#Biconsonantal_conjun... [2]: http://en.wikipedia.org/wiki/Devanagari_%28Unicode_block%29 http://en.wikipedia.org/wiki/Devanagari_%28Unicode_block%29