12 ms·

As a person living with the CJK languages (I’m specifically a Korean), I find that some of these problems are prominent, even in the 21th century. There are an

As a person living with the CJK languages (I’m specifically a Korean), I find that some of these problems are prominent, even in the 21th century. There are an excessive amount of programs that conflate key presses & text input, and ones that don’t consider input methods. I use macOS & Linux, and while the default text handling system called Cocoa Text System in macOS handles input methods well, almost all applications that implement it’s own, like big apps like Eclipse and Firefox, don’t get this right. On Linux, it’s terrifying; I’ve never seen any app that allows input systems to work naturally, and after a week of use you get used to pressing space & backspace after finishing every Hangul word. The Unix-style composability they want (apps should work whether or not input methods are used - and looks like Linux users that use Latin characters don’t use any input methods (opposed to macOS where Latin characters are input by a Latin input system), so looks like this state will persist. About the emoticons, I’m not that concerned with that since most (if not all) users won’t really input the color modifier separately (or even encounter files that have a separate one), so you can just select a sensible behavior like the #2 or 3 or 4. Users who understand the color modifiers, and other Unicode fiasco will understand what is happening under the hood, and ones that don’t will just think the file is broken and none of the behaviors will make sense, whatever you do.

hsivonen 7y ago
> I use macOS & Linux, and while the default text handling system called Cocoa Text System in macOS handles input methods well, almost all applications that implement it’s own, like big apps like Eclipse and Firefox, don’t get this right. What specific IME problems do you have with Firefox on Mac? > On Linux, it’s terrifying; I’ve never seen any app that allows input systems to work naturally, and after a week of use you get used to pressing space & backspace after finishing every Hangul word. Do you mean you have to press space twice and erase the second space? With IBus?
- dfcowell 7y ago
  There are no spaces between words in Chinese or Japanese. Pressing space confirms the current selection in the Japanese IME, which is expected behavior. Where some Linux implementations get it wrong is they also insert a space after the word, meaning the user has to select the desired word in the IME with the space bar and then remove the erroneous inserted space. Edit: Correction based on feedback below. Previously stated that Hangul does not have spaces.
  - tasogare 7y ago
    > There are no spaces between words in Hangul Wrong, there are spaces between words in Korean. It’s in Japanese and Chinese that there isn’t. And in Vietnamese there are spaces between everyone syllables, even in words.
    - bicolao 7y ago
      Vietnamese does not really belong in CJK group because it's written with Latin alphabet.
      - dfcowell 7y ago
        Traditional Vietnamese (before French colonialism) used Chinese characters.
      - tasogare 7y ago
        That was not my point. I mentioned Vietnamese because the spacing it uses is interesting. Also like dfcowell said, Vietnamese used to be written with han & chu nôm characters (respectively Chinese characters and Chinese-like characters created by Vietnamese), a lot of which are encoded in Unicode. Hence the existence of the CJKV acronym.
      - kevin_thibedeau 7y ago
        It is actually CJKV to deal with historical Vietnamese. From Unicode spec: > Although the term “CJK”—Chinese, Japanese, and Korean is used throughout this text to describe the languages that currently use Han ideographic characters, it should be noted that earlier Vietnamese writing systems were based on Han ideographs. Consequently, the term “CJKV” would be more accurate in a historical sense. Han ideographs are still used for historical, religious, and pedagogical purposes in Vietnam.
    - dfcowell 7y ago
      Thanks for the fact check. Not sure your comment on Vietnamese is accurate though. I work in a company with ~35% native Vietnamese speakers and I’ve seen plenty of multi-syllable words. Are you talking about traditional Vietnamese (when it still used Chinese characters) or modern Vietnamese (post-French-colonialism) which uses the Latin alphabet with accents?
SiVal 7y ago
For decades I've been saying that all text input should go through OS-level IMEs (input method editors). For more complex scripts, the need is obvious, but even writing in English, we can get great benefits from a system that expands abbreviations, replaces easy-to-type sequences with proper Unicode chars, runs little scripts and inserts the output, gives you quick dictionary/thesaurus lookups, gives you emmet-style powers, etc., whenever you're writing and in any app. Of course, everyone's preferences will be different, so you'll get default IMEs with default configs, but with the idea that you can reconfigure or replace them entirely with systems that work the way you want to work everywhere you input text. There have been utilities that do this sort of things for decades, but they've always been treated as clever hacks rather than standard text input. In other words, instead of powerful text input methods being the exception, they would be the rule, and apps that didn't use them would be the exception.
- saurik 7y ago
  So that sort of makes sense, but then I realize I am looking for all of those behaviors to be different in different contexts (such as writing code vs. writing an essay).
  - TheSmiddy 7y ago
    The OS knows the destination app (and with the correct apis the context within that app) so this would be trivial to implement once the base exists.
- Pxtl 7y ago
  > For decades I've been saying that all text input should go through OS-level IMEs (input method editors). Which is why it's infuriating when the built-in widgets of a platform don't include simple operations for obvious workflows like filtering characters, masked input, etc. That's what leads developers to have to roll their own implementation using keydown/keyup. I can't tell you how many GUI frameworks have forced me to roll my own numeric input widget.
  - ygra 7y ago
weeb_throwaway 7y ago
Another issue with CJK is vertical right-to-left writing: https://www.w3.org/International/articles/vertical-text/ https://www.w3.org/International/articles/vertical-text/ It's pretty popular with japanese novels/manga and traditional chinese but really hard to do without bugs on the web.
- L_Rahman 7y ago
  Hadn't fully registered till this comment, the degree to which the modern web is anchored to horizontal (usually left-to-right) writing and the design patterns of vertical scrolling that come with that assumption.
  - rcthompson 7y ago
    It's not just the web, it's all of computing, even down to the hardware design. A vertical scroll wheel is standard on all mice. A horizontal scrolling method of some sort is not, and even on mice that include one, it's usually not as good as the vertical one (e.g. leaning the wheel left and right).
    - deleted 7y ago
      [deleted]
zbraniecki 7y ago
Hi! I work on Firefox! Would you be open to share your experience and file bugs on issues you encountered? I'd love to help with the CJK support in Firefox!
- vezycash 7y ago
  Hi. Sent in feedback yesterday concerning addon privacy. Specifically, the option to grant or restrict addon access on per site basis. I'd want to be able to right click an addon icon in the toolbar and click, "Don't run on this site." And have even more options in the extension detail page.
  - zapzupnz 7y ago
    This isn't relevant to the thread, nor to what the Firefox dev was asking for.
- fouc 7y ago
  OR better yet, use the base OS editor support.
- ken 7y ago
  I think the point is to get native behavior, you need to use the native functionality. When you try to emulate it, you'll always be missing something. We see this in every corner of Firefox: the text editing, the toolbar, the context menus, the form controls, etc. Yet Firefox seems to be all about writing everything from scratch. There are many bugs filed for using native functionality, like [1], that have been open for decades with no activity. 20 years ago, it wasn't done because "it's a time thing", and since then it's racked up other bugs as dependencies because stuff's just broken. This isn't a situation where you can tweak a couple little problems and call it done. This is a fundamental change in the Firefox architecture. Asking for more bug reports is not going to help. [1]: https://bugzilla.mozilla.org/show_bug.cgi?id=34572 https://bugzilla.mozilla.org/show_bug.cgi?id=34572
  - hsivonen 7y ago
    Firefox uses native IMEs. Most of the time they aren't broken in obvious ways. To the extent they are broken in non-obvious ways, it's unhelpful not to say how on the level that would allow the problem to be reproduced. Firefox can't just use native text edit controls. First, Firefox needs to support contenteditable, which doesn't map to an OS-supplied text box. Second, the multiprocess architecture leads to a situation where the UI process talks to the native IME API but the Web content process hosts the text being edited, so native text boxes don't work even for things that look similar to OS-supplied text boxes.
ptero 7y ago
This is interesting. Not doubting your story, but my personal experience, as a Cyrillic user, is opposite: at least in early versions of Windows apps I constantly struggled with a random mix of hardcoded assumptions for encodings, key presses, characters and data stored which often produced gibberish on screen. When I switched to Linux everything just works out of the box: I can copy and paste text between gvim, xterm, etc. with no issues. I admit that this is likely due to app writers, not underlying OS. And my experience is only with single-byte characters. Just my 2c.
usr1106 7y ago
> and looks like Linux users that use Latin characters don’t use any input methods Right, I prefer slim systems and I typically uninstall everything input method related that my distro has chosen to preinstall. I cannot read or memorize a single CJK character, so why would I need that. Aditionally programming and computers for me means English, although that is not my mother tongue. I would never install anything in my mother tongue. Basically I use my mother tongue (and a couple of other European languages I speak) only in Email, chat or maybe some web form. I can feel your pain though, because 10+ years ago we had the same problem with the couple of non-ASCII characters you need in most European languages. In order to have the situation in Linux improve there just need to be enough CJK contributors to fix existing bugs. And reviewers / unit test cases to make sure we Westeners don't break it again with our next commit.
- pcr910303 7y ago
  > Right, I prefer slim systems and I typically uninstall everything input method related that my distro has chosen to preinstall. I cannot read or memorize a single CJK character, so why would I need that. Yes, I'm exactly talking about this mindset. This is basically why Linux has such poor input method support. Because English has a special privilege of not needing input methods to be input in, combined with the fact that the majority of Linux application programmers use English only, that means basically all apps that don't consider i18n seriously are by default 'wrong', opposed to apps running on Windows/macOS which are by default 'right'. > Basically I use my mother tongue (and a couple of other European languages I speak) only in Email, chat or maybe some web form. Does that mean European languages are able to being input without special input methods? > I can feel your pain though, because 10+ years ago we had the same problem with the couple of non-ASCII characters you need in most European languages. The non-ASCII characters fit in the character array model that most western people think in, and as a plus they are fittable in the upper half of ASCII. Asian CJK languages require a different model from the western ones. > In order to have the situation in Linux improve there just need to be enough CJK contributors to fix existing bugs. It's a failing fight. That only works on an ideal world where every program has enough contributors. Thats not true.
  - 7y ago
microcolonel 7y ago
> On Linux, it’s terrifying; I’ve never seen any app that allows input systems to work naturally Kind of interested to hear what sort of input method you're using. Even my terminal emulator supports Japanese input methods well, through IBus. Maybe it's just that as a non-native I'm more often going to convert chunk by chunk anyway, I have noticed that some input methods do not do bulk conversion well; I must say I never thought hangeul would even warrant conversion other than composing the syllables, is it because you mix in hanja conversion? I think that your problems will mostly a matter of the quality of the input method, IBus and the input method system in GTK+ at least seem to be not preventing anyone from writing better input methods. I feel it with Firefox though, then again Firefox is very poor quality software in my experience, almost everything is at least subtly wrong, and there seems to be more interest in niche feature work than basic work on product quality. I could in some ways say the same for Eclipse, every time an SDK I want to use is only documented in terms of their special Eclipse frontend, I get a bit depressed.
innocenat 7y ago
> On Linux, it’s terrifying; I’ve never seen any app that allows input systems to work naturally, and after a week of use you get used to pressing space & backspace after finishing every Hangul word. The Unix-style composability they want (apps should work whether or not input methods are used - and looks like Linux users that use Latin characters don’t use any input methods (opposed to macOS where Latin characters are input by a Latin input system), so looks like this state will persist. Living in Japan and using Linux almost all the time, I never remember having any problem with typing in Japanese whatsoever.
- cyborgx7 7y ago
  Learning japanese and full-time Linux user. Getting Japanese writing to work was a pain in the ass and still doesn't work everywhere.
  - innocenat 7y ago
    I use ibus and mozc (i.e. what come with Ubuntu) and never have any problem.
- kbumsik 7y ago
  Korean case is totally different from Japanese. I guess Korean is very unique in this area, because Korean Hangul characters consists of multiple small characters. Input systems needs to have a state system to allow type multiple small characters to complete a character. The problems usually come from improper state systems.
  - Izkata 7y ago
    How is that any different from Japanese Kanji?
    - Qwertystop 7y ago
      Kanji is logographic: each symbol is a complete word, phrase, or idea. Hangul is alphabetic-syllabic: each symbol is a segment (vowel-or-consonant), except that they're written in two- or three-letter blocks each representing a syllable.
munmaek 7y ago
> On Linux, it’s terrifying; Are you using ibus? I'm on debian and once a character is finished, it automatically moves onto the next character (unless you press space, enter, etc). All I have to do is ctrl+space to switch input methods. I've had issues with terminals like the character building not working in alacritty. The only major annoyance I've found is having to install and configure ibus/ibus-daemon, and CJK fonts.

hsivonen 7y ago

> I use macOS & Linux, and while the default text handling system called Cocoa Text System in macOS handles input methods well, almost all applications that implement it’s own, like big apps like Eclipse and Firefox, don’t get this right. What specific IME problems do you have with Firefox on Mac? > On Linux, it’s terrifying; I’ve never seen any app that allows input systems to work naturally, and after a week of use you get used to pressing space & backspace after finishing every Hangul word. Do you mean you have to press space twice and erase the second space? With IBus?

dfcowell 7y ago
There are no spaces between words in Chinese or Japanese. Pressing space confirms the current selection in the Japanese IME, which is expected behavior. Where some Linux implementations get it wrong is they also insert a space after the word, meaning the user has to select the desired word in the IME with the space bar and then remove the erroneous inserted space. Edit: Correction based on feedback below. Previously stated that Hangul does not have spaces.
- tasogare 7y ago
  > There are no spaces between words in Hangul Wrong, there are spaces between words in Korean. It’s in Japanese and Chinese that there isn’t. And in Vietnamese there are spaces between everyone syllables, even in words.
  - bicolao 7y ago
    Vietnamese does not really belong in CJK group because it's written with Latin alphabet.
    - dfcowell 7y ago
      Traditional Vietnamese (before French colonialism) used Chinese characters.
    - tasogare 7y ago
      That was not my point. I mentioned Vietnamese because the spacing it uses is interesting. Also like dfcowell said, Vietnamese used to be written with han & chu nôm characters (respectively Chinese characters and Chinese-like characters created by Vietnamese), a lot of which are encoded in Unicode. Hence the existence of the CJKV acronym.
    - kevin_thibedeau 7y ago
      It is actually CJKV to deal with historical Vietnamese. From Unicode spec: > Although the term “CJK”—Chinese, Japanese, and Korean is used throughout this text to describe the languages that currently use Han ideographic characters, it should be noted that earlier Vietnamese writing systems were based on Han ideographs. Consequently, the term “CJKV” would be more accurate in a historical sense. Han ideographs are still used for historical, religious, and pedagogical purposes in Vietnam.
  - dfcowell 7y ago
    Thanks for the fact check. Not sure your comment on Vietnamese is accurate though. I work in a company with ~35% native Vietnamese speakers and I’ve seen plenty of multi-syllable words. Are you talking about traditional Vietnamese (when it still used Chinese characters) or modern Vietnamese (post-French-colonialism) which uses the Latin alphabet with accents?

SiVal 7y ago

For decades I've been saying that all text input should go through OS-level IMEs (input method editors). For more complex scripts, the need is obvious, but even writing in English, we can get great benefits from a system that expands abbreviations, replaces easy-to-type sequences with proper Unicode chars, runs little scripts and inserts the output, gives you quick dictionary/thesaurus lookups, gives you emmet-style powers, etc., whenever you're writing and in any app. Of course, everyone's preferences will be different, so you'll get default IMEs with default configs, but with the idea that you can reconfigure or replace them entirely with systems that work the way you want to work everywhere you input text. There have been utilities that do this sort of things for decades, but they've always been treated as clever hacks rather than standard text input. In other words, instead of powerful text input methods being the exception, they would be the rule, and apps that didn't use them would be the exception.

saurik 7y ago
So that sort of makes sense, but then I realize I am looking for all of those behaviors to be different in different contexts (such as writing code vs. writing an essay).
- TheSmiddy 7y ago
  The OS knows the destination app (and with the correct apis the context within that app) so this would be trivial to implement once the base exists.
Pxtl 7y ago
> For decades I've been saying that all text input should go through OS-level IMEs (input method editors). Which is why it's infuriating when the built-in widgets of a platform don't include simple operations for obvious workflows like filtering characters, masked input, etc. That's what leads developers to have to roll their own implementation using keydown/keyup. I can't tell you how many GUI frameworks have forced me to roll my own numeric input widget.
- ygra 7y ago

weeb_throwaway 7y ago

Another issue with CJK is vertical right-to-left writing: https://www.w3.org/International/articles/vertical-text/ https://www.w3.org/International/articles/vertical-text/ It's pretty popular with japanese novels/manga and traditional chinese but really hard to do without bugs on the web.

L_Rahman 7y ago
Hadn't fully registered till this comment, the degree to which the modern web is anchored to horizontal (usually left-to-right) writing and the design patterns of vertical scrolling that come with that assumption.
- rcthompson 7y ago
  It's not just the web, it's all of computing, even down to the hardware design. A vertical scroll wheel is standard on all mice. A horizontal scrolling method of some sort is not, and even on mice that include one, it's usually not as good as the vertical one (e.g. leaning the wheel left and right).
  - deleted 7y ago
    [deleted]

zbraniecki 7y ago

Hi! I work on Firefox! Would you be open to share your experience and file bugs on issues you encountered? I'd love to help with the CJK support in Firefox!

vezycash 7y ago
Hi. Sent in feedback yesterday concerning addon privacy. Specifically, the option to grant or restrict addon access on per site basis. I'd want to be able to right click an addon icon in the toolbar and click, "Don't run on this site." And have even more options in the extension detail page.
- zapzupnz 7y ago
  This isn't relevant to the thread, nor to what the Firefox dev was asking for.
fouc 7y ago
OR better yet, use the base OS editor support.
ken 7y ago
I think the point is to get native behavior, you need to use the native functionality. When you try to emulate it, you'll always be missing something. We see this in every corner of Firefox: the text editing, the toolbar, the context menus, the form controls, etc. Yet Firefox seems to be all about writing everything from scratch. There are many bugs filed for using native functionality, like [1], that have been open for decades with no activity. 20 years ago, it wasn't done because "it's a time thing", and since then it's racked up other bugs as dependencies because stuff's just broken. This isn't a situation where you can tweak a couple little problems and call it done. This is a fundamental change in the Firefox architecture. Asking for more bug reports is not going to help. [1]: https://bugzilla.mozilla.org/show_bug.cgi?id=34572 https://bugzilla.mozilla.org/show_bug.cgi?id=34572
- hsivonen 7y ago
  Firefox uses native IMEs. Most of the time they aren't broken in obvious ways. To the extent they are broken in non-obvious ways, it's unhelpful not to say how on the level that would allow the problem to be reproduced. Firefox can't just use native text edit controls. First, Firefox needs to support contenteditable, which doesn't map to an OS-supplied text box. Second, the multiprocess architecture leads to a situation where the UI process talks to the native IME API but the Web content process hosts the text being edited, so native text boxes don't work even for things that look similar to OS-supplied text boxes.

ptero 7y ago

This is interesting. Not doubting your story, but my personal experience, as a Cyrillic user, is opposite: at least in early versions of Windows apps I constantly struggled with a random mix of hardcoded assumptions for encodings, key presses, characters and data stored which often produced gibberish on screen. When I switched to Linux everything just works out of the box: I can copy and paste text between gvim, xterm, etc. with no issues. I admit that this is likely due to app writers, not underlying OS. And my experience is only with single-byte characters. Just my 2c.

usr1106 7y ago

> and looks like Linux users that use Latin characters don’t use any input methods Right, I prefer slim systems and I typically uninstall everything input method related that my distro has chosen to preinstall. I cannot read or memorize a single CJK character, so why would I need that. Aditionally programming and computers for me means English, although that is not my mother tongue. I would never install anything in my mother tongue. Basically I use my mother tongue (and a couple of other European languages I speak) only in Email, chat or maybe some web form. I can feel your pain though, because 10+ years ago we had the same problem with the couple of non-ASCII characters you need in most European languages. In order to have the situation in Linux improve there just need to be enough CJK contributors to fix existing bugs. And reviewers / unit test cases to make sure we Westeners don't break it again with our next commit.

pcr910303 7y ago
> Right, I prefer slim systems and I typically uninstall everything input method related that my distro has chosen to preinstall. I cannot read or memorize a single CJK character, so why would I need that. Yes, I'm exactly talking about this mindset. This is basically why Linux has such poor input method support. Because English has a special privilege of not needing input methods to be input in, combined with the fact that the majority of Linux application programmers use English only, that means basically all apps that don't consider i18n seriously are by default 'wrong', opposed to apps running on Windows/macOS which are by default 'right'. > Basically I use my mother tongue (and a couple of other European languages I speak) only in Email, chat or maybe some web form. Does that mean European languages are able to being input without special input methods? > I can feel your pain though, because 10+ years ago we had the same problem with the couple of non-ASCII characters you need in most European languages. The non-ASCII characters fit in the character array model that most western people think in, and as a plus they are fittable in the upper half of ASCII. Asian CJK languages require a different model from the western ones. > In order to have the situation in Linux improve there just need to be enough CJK contributors to fix existing bugs. It's a failing fight. That only works on an ideal world where every program has enough contributors. Thats not true.
- 7y ago

microcolonel 7y ago

> On Linux, it’s terrifying; I’ve never seen any app that allows input systems to work naturally Kind of interested to hear what sort of input method you're using. Even my terminal emulator supports Japanese input methods well, through IBus. Maybe it's just that as a non-native I'm more often going to convert chunk by chunk anyway, I have noticed that some input methods do not do bulk conversion well; I must say I never thought hangeul would even warrant conversion other than composing the syllables, is it because you mix in hanja conversion? I think that your problems will mostly a matter of the quality of the input method, IBus and the input method system in GTK+ at least seem to be not preventing anyone from writing better input methods. I feel it with Firefox though, then again Firefox is very poor quality software in my experience, almost everything is at least subtly wrong, and there seems to be more interest in niche feature work than basic work on product quality. I could in some ways say the same for Eclipse, every time an SDK I want to use is only documented in terms of their special Eclipse frontend, I get a bit depressed.

innocenat 7y ago

> On Linux, it’s terrifying; I’ve never seen any app that allows input systems to work naturally, and after a week of use you get used to pressing space & backspace after finishing every Hangul word. The Unix-style composability they want (apps should work whether or not input methods are used - and looks like Linux users that use Latin characters don’t use any input methods (opposed to macOS where Latin characters are input by a Latin input system), so looks like this state will persist. Living in Japan and using Linux almost all the time, I never remember having any problem with typing in Japanese whatsoever.

cyborgx7 7y ago
Learning japanese and full-time Linux user. Getting Japanese writing to work was a pain in the ass and still doesn't work everywhere.
- innocenat 7y ago
  I use ibus and mozc (i.e. what come with Ubuntu) and never have any problem.
kbumsik 7y ago
Korean case is totally different from Japanese. I guess Korean is very unique in this area, because Korean Hangul characters consists of multiple small characters. Input systems needs to have a state system to allow type multiple small characters to complete a character. The problems usually come from improper state systems.
- Izkata 7y ago
  How is that any different from Japanese Kanji?
  - Qwertystop 7y ago
    Kanji is logographic: each symbol is a complete word, phrase, or idea. Hangul is alphabetic-syllabic: each symbol is a segment (vowel-or-consonant), except that they're written in two- or three-letter blocks each representing a syllable.

munmaek 7y ago

> On Linux, it’s terrifying; Are you using ibus? I'm on debian and once a character is finished, it automatically moves onto the next character (unless you press space, enter, etc). All I have to do is ctrl+space to switch input methods. I've had issues with terminals like the character building not working in alacritty. The only major annoyance I've found is having to install and configure ibus/ibus-daemon, and CJK fonts.