3 ms·

Is there any standard system where each byte/word maps to one character/grapheme? I feel there is a general sentiment that not being able to jump to the Nth cha

by geokon 5y ago

Is there any standard system where each byte/word maps to one character/grapheme? I feel there is a general sentiment that not being able to jump to the Nth character is programmatically.. irritating and disappointing. I'm sure such a system wouldn't support some languages - but in the words of Lord Farquaad "That's a sacrifice I'm willing to make". Most of the world's languages would do just fine and it'd make sense to exclude right to left arabic ligatured text in, for instance, your monospaced computer code. I'm guessing you could extract a subset of UTF-8 - but has anyone done anything like that?

deleted 5y ago
[deleted]
eyelidlessness 5y ago
You’re basically describing the various character encodings which preceded adoption of Unicode. No, UTF-8 doesn’t support that. That’s the sacrifice it was willing to make, to eliminate the need to distinguish between character encodings: a string isn’t a sequence of bytes. Even if you don’t appreciate that convenience (and the highly standardized, easily abstracted rules of combined bytes), maybe you’ll appreciate that the complexity of supporting multilingual users would mean multipart data, arbitrarily fractured. As in a single text message using multiple character sets could be dozens of payload boundaries. Does that really sound easier or more painless than UTF-8’s variable-length code points?
- geokon 5y ago
  From what I understood of the link there is the "grapheme" which is an intermediary representation between being UTF-8 and being drawn on the screen. Like all possible glyphs have been prerendered. It's not got some simple graphics engine to draw those on the fly. Couldn't you directly provide an index in the grapheme "cache" ?
  - eyelidlessness 5y ago
    The index is larger than you think.
dahfizz 5y ago
> Is there any standard system where each byte/word maps to one character/grapheme? > I'm sure such a system wouldn't support some languages > I'm guessing you could extract a subset of UTF-8 I can't tell if you're joking or not. You're perfectly describing ASCII.
- geokon 5y ago
  Okay but ascii only covers English. Doesn't even cover all European languages. There is probably something between ascii and utf-8 where you could cover prolly like 90% of people's needs with English Chinese Japanese Korean Greek Russian. Weird languages that need ligatures and combining symbols are the exception (and in some cases like Korean there is a very finite amount of combinations)

by geokon 5y ago

deleted 5y ago

[deleted]

eyelidlessness 5y ago

You’re basically describing the various character encodings which preceded adoption of Unicode. No, UTF-8 doesn’t support that. That’s the sacrifice it was willing to make, to eliminate the need to distinguish between character encodings: a string isn’t a sequence of bytes. Even if you don’t appreciate that convenience (and the highly standardized, easily abstracted rules of combined bytes), maybe you’ll appreciate that the complexity of supporting multilingual users would mean multipart data, arbitrarily fractured. As in a single text message using multiple character sets could be dozens of payload boundaries. Does that really sound easier or more painless than UTF-8’s variable-length code points?

geokon 5y ago
From what I understood of the link there is the "grapheme" which is an intermediary representation between being UTF-8 and being drawn on the screen. Like all possible glyphs have been prerendered. It's not got some simple graphics engine to draw those on the fly. Couldn't you directly provide an index in the grapheme "cache" ?
- eyelidlessness 5y ago
  The index is larger than you think.

dahfizz 5y ago

> Is there any standard system where each byte/word maps to one character/grapheme? > I'm sure such a system wouldn't support some languages > I'm guessing you could extract a subset of UTF-8 I can't tell if you're joking or not. You're perfectly describing ASCII.

geokon 5y ago
Okay but ascii only covers English. Doesn't even cover all European languages. There is probably something between ascii and utf-8 where you could cover prolly like 90% of people's needs with English Chinese Japanese Korean Greek Russian. Weird languages that need ligatures and combining symbols are the exception (and in some cases like Korean there is a very finite amount of combinations)