3 ms·

I've been toying with the idea for a text editor where instead of working with Unicode, convert Unicode into and out of a simpler format to use internally. All

I've been toying with the idea for a text editor where instead of working with Unicode, convert Unicode into and out of a simpler format to use internally. All text would be one code point=one character and laid out left to right. When loading a file, any Unicode not supported would be encoded as "illegal characters" that would just convert back to the original code points when saved. Also, all right-to-left text would be reordered when loaded, and back when saved. (as long as there are no embeddings) A weird drawback however is that Backspace and Delete would not act on logical text order, but as "Delete Left" and "Delete Right". However, there should be a hotkey for swapping default text direction/justificiation globally that I think could make up for some of that. I think that, if you are honest about the limitations of your program and convey those to your user as a consistent model, the user would be more willing to accept that model. Just don't claim to support Unicode according to spec if it is likely that only a subset will be supported correctly.

nikeee 2y ago
How are arbitrary grapheme clusters represented in this model?
crazygringo 2y ago
> convert Unicode into and out of a simpler format to use internally But what simpler format? If it supports all of Unicode it's not simpler. If it doesn't, then it's missing necessary features entire languages rely on. And missing emoji. I don't understand what benefit you're trying to provide.
- Nevermark 2y ago
  Unicode includes characters built from multiple characters. This means questions (functions) like “how many characters do I have”, and indexing in general, are not as straight forward as they could be. So storing each visual character’s character group together, as a single element, results in a simpler interface from a programming standpoint. Not necessarily as efficient, since either indirection or larger character footprints (or both) are required, but simpler. This is definitely the route I would go, while implementing other complexity. After that, possibly undoing this “simplicity optimization”, would be a small price to pay for the development convenience.
- Findecanor 2y ago
  Simpler for the text editor logic, not necessarily simpler for the encoder/decoder. > If it supports all of Unicode it's not simpler. The point is that it doesn't. The Unicode that it does not handle should get marked as "illegal character", rendered as boxes and not get mangled unless the user messes with it. Support for more of Unicode could be added incrementally, and in a single subsystem instead of everywhere in the text editor.
  - crazygringo 2y ago
    I still don't understand what you're arguing for. >> If it supports all of Unicode it's not simpler. > The point is that it doesn't. But then it doesn't work. Why would you not want to support all of Unicode? > The Unicode that it does not handle should get marked as "illegal character", rendered as boxes and not get mangled unless the user messes with it. That's what already happens. We already show boxes when the codepoint is unknown or there's no glyph in the font. Their codepoints are preserved. > Support for more of Unicode could be added incrementally, and in a single subsystem instead of everywhere in the text editor. It already is. New OS's get upgraded to new Unicode versions. Text editors don't usually have to change a thing. I don't know what problem you're trying to solve here.
globalnode 2y ago
I was brutal with mine... force all unicode to ascii and replace unicode with question marks. Like you said, I notify the user (which is only me) and it gets saved as ascii. I have no illusions that it doesnt support unicode but its still a useful editor for me. And PS: Text editors are really really hard.

nikeee 2y ago

How are arbitrary grapheme clusters represented in this model?

crazygringo 2y ago

> convert Unicode into and out of a simpler format to use internally But what simpler format? If it supports all of Unicode it's not simpler. If it doesn't, then it's missing necessary features entire languages rely on. And missing emoji. I don't understand what benefit you're trying to provide.

Nevermark 2y ago
Unicode includes characters built from multiple characters. This means questions (functions) like “how many characters do I have”, and indexing in general, are not as straight forward as they could be. So storing each visual character’s character group together, as a single element, results in a simpler interface from a programming standpoint. Not necessarily as efficient, since either indirection or larger character footprints (or both) are required, but simpler. This is definitely the route I would go, while implementing other complexity. After that, possibly undoing this “simplicity optimization”, would be a small price to pay for the development convenience.
Findecanor 2y ago
Simpler for the text editor logic, not necessarily simpler for the encoder/decoder. > If it supports all of Unicode it's not simpler. The point is that it doesn't. The Unicode that it does not handle should get marked as "illegal character", rendered as boxes and not get mangled unless the user messes with it. Support for more of Unicode could be added incrementally, and in a single subsystem instead of everywhere in the text editor.
- crazygringo 2y ago
  I still don't understand what you're arguing for. >> If it supports all of Unicode it's not simpler. > The point is that it doesn't. But then it doesn't work. Why would you not want to support all of Unicode? > The Unicode that it does not handle should get marked as "illegal character", rendered as boxes and not get mangled unless the user messes with it. That's what already happens. We already show boxes when the codepoint is unknown or there's no glyph in the font. Their codepoints are preserved. > Support for more of Unicode could be added incrementally, and in a single subsystem instead of everywhere in the text editor. It already is. New OS's get upgraded to new Unicode versions. Text editors don't usually have to change a thing. I don't know what problem you're trying to solve here.

globalnode 2y ago

I was brutal with mine... force all unicode to ascii and replace unicode with question marks. Like you said, I notify the user (which is only me) and it gets saved as ascii. I have no illusions that it doesnt support unicode but its still a useful editor for me. And PS: Text editors are really really hard.