3 ms·

How many bytes should be removed from your gap buffer when the user presses backspace? Does it matter if it's Hangul?

by torstenvl 3y ago

jameshart 3y ago
It's possible that pressing backspace needs to ADD bytes.
- gumby 3y ago
  What is that case? Hmm, I could imagine that you have a precomposed character with two diacritical marks; if there were no precomposed character with the remaining diacritical you'd need to replace the precomposed character with the base character and diacritical. However I would imagine the better UX would be for backspace to remove the entire character (whether diacriticals were composed or not) because that's how people think of their characters. Is there a real world case? I'd be excited to learn it.
  - torstenvl 3y ago
    > What is that case? ... Hangul. What happens when the user presses backspace when the carat is positioned just past the graphical representation of U+AC01? It should by default result in U+AC01 being replaced by U+1100 and U+1161. (In some IME implementations the user might have to configure it, e.g., "Delete by jaso unit" in Windows 7.)
    - gumby 3y ago
      Thanks!
    - qalmakka 3y ago
      I don't understand why the felt the need to add precomposed characters for Hangul. Why? Why couldn't they just let the system compose them instead?
      - gumby 3y ago
        There was probably an existing character set that had precomposed characters and Unicode always includes existing character sets for round trip idempotency. Imagine if hanzi had been encoded as radicals…
        qalmakka 3y ago
        Yeah, Chinese characters radicals are not an alphabet per se, while Hangul has separate, standalone letters. It's literally a matter of composing them in a fixed way, compared to Chinese characters which tend to go bananas on how many thousands of ways they can be composed.
        gumby 3y ago
        Yeah, in case my message wasn't clear: my comment about the radicals was in support of encoding precomposed characters.
        dhosek 3y ago
        Exactly this. When Unicode 1.0 came out, there was no standardized way to indicate that f i should compose into ﬁ, let alone that ᄋ ㅡ ᄀ should compose into 윽. IIRC, composing of East Asian scripts into han or hangul happened at the firmware level, not at the app or OS level. If you didn’t have the hardware built into your keyboard, you simply could not type anything in East Asian scripts. A lot of the inconsistencies in Unicode (like the difference in how vowels are handled in Thai vs Indic scripts) come down to how the original 8-bit encodings worked pre-Unicode.
        qalmakka 3y ago
        Very interesting, I guess that legacy really is the root cause of everything. Do you have any links or such about the history of Unicode?
        dhosek 3y ago
        Not especially. I was around for a lot of it (I shared an office with one of the contributors to Unicode 1.0 back in the late 80s and used to have a massive binder with every ECMA code page in it (which included a lot of non-European codes). The other thing about Han unification that people don’t realize is that there were five different East Asian 16-bit encodings (the original draft of ISO-10646 had a 32-bit encoding which included all of them). I worked on a project in the early 90s where we needed multiple-script capabilities (Roman, Cyrillic and Japanese) and ended up using JIS because it was a well-defined encoding that we could buy fonts that employed it (although we still had to contract with Bitstream to get a version of their font that was split into 8-bit segments for our software). There was JIS (Japanese), the Korean code (which included both Jamo and pre-composed Hangul syllables) and not one, not two, but three, count ’em, three different Chinese encodings: Mainland China, Hong Kong and Taiwan. Since the original goal of Unicode was to try to be a 16-bit fixed-width encoding (as opposed to the competing 32-bit fixed-width encoding of ISO-10646), there was really no choice but to do Han Unification. Even so, the five 16-bit encodings still linger.
theamk 3y ago
Your app, unless it's a text editor or UI toolkit, should not care. The easiest way to support Unicode is to avoid supportimg it. Leave text editing to UI widgets. Leave truncation to web browsers. Avoid fancy marquee effect on the terminal. Your program may look less fancy but it will automatically support unicode, even future extensions.
- torstenvl 3y ago
  This is not good advice. Most software is not an "app," and even if it is, you at least need to be able to tell the difference between, e.g., ISO 8859-1, ISO 8859-15, and UTF-8. Otherwise you'll get text talking about cutting doorways 8œ feet tall or submitting a rÃ©sumÃ©. Any software that needs to process information effectively needs to understand character sets, including Unicode.
  - theamk 3y ago
    Unless you deal with very old data, just use UTF-8 [0]. Mac terminal uses UTF-8 by default, most Linuxes use UTF-8 by default, and if you are stuck with Windows, there are libraries which let you use UTF-8 everywhere. And if you do deal with old data, still prefer to use UTF-8, and make sure you only translate character sets if user asks / they are explicitly specified. And for the love of god, _please_ don't just hardcode two common encodings (like 8859-1 and 8859-15)... I used to deal with encodings a lot, and the only time I actually had data loss from encoding problems was from misguided apps which assume 8859-1 (and tried to convert to ASCII by stripping accents). Please either assume UTF-8 or assume nothing and keep data as-is. And that's it, _most_ software does not need to know more. Let's look at some examples: - For file archiver, assume utf-8 everywhere and don't worry about encoding at all (unless you planning to deal with very old archives, in this case you'd need encoding support. But for something made from scratch, don't bother.) _Especially_ don't add any encoding support for "extract to console" function -- if I need a different encoding I can pipe into iconv myself, thank you very much. - For API client, you don't need to worry about encoding either.. A very old HTTP server might return data in something other than utf-8 but (1) your HTTP library likely handles it already (2) how many such servers are left, anyway? - For XML analyzer, use proper XML library, they will handle encodings for you. - For web dashboard or a database, keep everything in utf-8 and things will just work. - For ETL-like applications, many modern data sources are already in UTF-8.. and if they are not, convert everything to utf-8 ASAP. In other words: while it's useful to know that encodings exist, unless you are working with very old data or legacy systems, utf-8 is the only thing you need. [0] http://utf8everywhere.org/ http://utf8everywhere.org/

How many bytes should be removed from your gap buffer when the user presses backspace? Does it matter if it's Hangul?

jameshart 3y ago

It's possible that pressing backspace needs to ADD bytes.

gumby 3y ago
What is that case? Hmm, I could imagine that you have a precomposed character with two diacritical marks; if there were no precomposed character with the remaining diacritical you'd need to replace the precomposed character with the base character and diacritical. However I would imagine the better UX would be for backspace to remove the entire character (whether diacriticals were composed or not) because that's how people think of their characters. Is there a real world case? I'd be excited to learn it.
- torstenvl 3y ago
  > What is that case? ... Hangul. What happens when the user presses backspace when the carat is positioned just past the graphical representation of U+AC01? It should by default result in U+AC01 being replaced by U+1100 and U+1161. (In some IME implementations the user might have to configure it, e.g., "Delete by jaso unit" in Windows 7.)
  - gumby 3y ago
    Thanks!
  - qalmakka 3y ago
    I don't understand why the felt the need to add precomposed characters for Hangul. Why? Why couldn't they just let the system compose them instead?
    - gumby 3y ago
      There was probably an existing character set that had precomposed characters and Unicode always includes existing character sets for round trip idempotency. Imagine if hanzi had been encoded as radicals…
      - qalmakka 3y ago
        Yeah, Chinese characters radicals are not an alphabet per se, while Hangul has separate, standalone letters. It's literally a matter of composing them in a fixed way, compared to Chinese characters which tend to go bananas on how many thousands of ways they can be composed.
        gumby 3y ago
        Yeah, in case my message wasn't clear: my comment about the radicals was in support of encoding precomposed characters.
      - dhosek 3y ago
        Exactly this. When Unicode 1.0 came out, there was no standardized way to indicate that f i should compose into ﬁ, let alone that ᄋ ㅡ ᄀ should compose into 윽. IIRC, composing of East Asian scripts into han or hangul happened at the firmware level, not at the app or OS level. If you didn’t have the hardware built into your keyboard, you simply could not type anything in East Asian scripts. A lot of the inconsistencies in Unicode (like the difference in how vowels are handled in Thai vs Indic scripts) come down to how the original 8-bit encodings worked pre-Unicode.
        qalmakka 3y ago
        Very interesting, I guess that legacy really is the root cause of everything. Do you have any links or such about the history of Unicode?
        dhosek 3y ago
        Not especially. I was around for a lot of it (I shared an office with one of the contributors to Unicode 1.0 back in the late 80s and used to have a massive binder with every ECMA code page in it (which included a lot of non-European codes). The other thing about Han unification that people don’t realize is that there were five different East Asian 16-bit encodings (the original draft of ISO-10646 had a 32-bit encoding which included all of them). I worked on a project in the early 90s where we needed multiple-script capabilities (Roman, Cyrillic and Japanese) and ended up using JIS because it was a well-defined encoding that we could buy fonts that employed it (although we still had to contract with Bitstream to get a version of their font that was split into 8-bit segments for our software). There was JIS (Japanese), the Korean code (which included both Jamo and pre-composed Hangul syllables) and not one, not two, but three, count ’em, three different Chinese encodings: Mainland China, Hong Kong and Taiwan. Since the original goal of Unicode was to try to be a 16-bit fixed-width encoding (as opposed to the competing 32-bit fixed-width encoding of ISO-10646), there was really no choice but to do Han Unification. Even so, the five 16-bit encodings still linger.

theamk 3y ago

Your app, unless it's a text editor or UI toolkit, should not care. The easiest way to support Unicode is to avoid supportimg it. Leave text editing to UI widgets. Leave truncation to web browsers. Avoid fancy marquee effect on the terminal. Your program may look less fancy but it will automatically support unicode, even future extensions.

torstenvl 3y ago
This is not good advice. Most software is not an "app," and even if it is, you at least need to be able to tell the difference between, e.g., ISO 8859-1, ISO 8859-15, and UTF-8. Otherwise you'll get text talking about cutting doorways 8œ feet tall or submitting a rÃ©sumÃ©. Any software that needs to process information effectively needs to understand character sets, including Unicode.
- theamk 3y ago
  Unless you deal with very old data, just use UTF-8 [0]. Mac terminal uses UTF-8 by default, most Linuxes use UTF-8 by default, and if you are stuck with Windows, there are libraries which let you use UTF-8 everywhere. And if you do deal with old data, still prefer to use UTF-8, and make sure you only translate character sets if user asks / they are explicitly specified. And for the love of god, _please_ don't just hardcode two common encodings (like 8859-1 and 8859-15)... I used to deal with encodings a lot, and the only time I actually had data loss from encoding problems was from misguided apps which assume 8859-1 (and tried to convert to ASCII by stripping accents). Please either assume UTF-8 or assume nothing and keep data as-is. And that's it, _most_ software does not need to know more. Let's look at some examples: - For file archiver, assume utf-8 everywhere and don't worry about encoding at all (unless you planning to deal with very old archives, in this case you'd need encoding support. But for something made from scratch, don't bother.) _Especially_ don't add any encoding support for "extract to console" function -- if I need a different encoding I can pipe into iconv myself, thank you very much. - For API client, you don't need to worry about encoding either.. A very old HTTP server might return data in something other than utf-8 but (1) your HTTP library likely handles it already (2) how many such servers are left, anyway? - For XML analyzer, use proper XML library, they will handle encodings for you. - For web dashboard or a database, keep everything in utf-8 and things will just work. - For ETL-like applications, many modern data sources are already in UTF-8.. and if they are not, convert everything to utf-8 ASAP. In other words: while it's useful to know that encodings exist, unless you are working with very old data or legacy systems, utf-8 is the only thing you need. [0] http://utf8everywhere.org/ http://utf8everywhere.org/