3 ms·
Out of curiosity, is there a more compact alternative to Unicode that excludes all precomposed chars like ñ, but also for all the Asian languages, so you'd basi
by eviks 3y ago
Out of curiosity, is there a more compact alternative to Unicode that excludes all precomposed chars like ñ, but also for all the Asian languages, so you'd basically have some set of "strokes" that can compose anything else? And if you need to add semantic meaning to identically looking results, that would be another non-visual composable indicator
- Dylan16807 3y agoWhat's your definition of "compact" here? And there's a lot of ways to put together a character, so I worry you'd end up with something closer to a vector format than a normal character encoding.
- eviks 3y ago> What's your definition of "compact" here? Number of elements in an encoding 5: ˜ n o ñ õ 3: ˜ n o > so I worry you'd end up with something closer to a vector format than a normal character encoding. then stop 1 step before that :)
- lifthrasiir 3y agoMuch easier said than done. Try to actually do that. A good start would be IDS.TXT I've linked above; this gives about 500 approximately atomic characters to start with [1]. Now extend this to the entirety of Ideographic Variation Database [2], which tries to solve the most cited problems with Han unification. And then add some kind of semantic annotation as you've suggested (I have no idea, maybe you have a better idea). [1] 123 components that are already encoded, 121 components ({01}, ...) that are partial characters, ~123 "unpresentable" components (?), and ~115 "minor" variations (〾) that may require additional components or two. [2] https://unicode.org/ivd/ https://unicode.org/ivd/
- eviks 3y ago> Much easier said than done. Try to actually do that Ha-ha, thanks for the offer, but I'll pass - all that complexity is precisely why I'm not even going to try, it requires a lot of dedicated team effort put into it, a simple "try" is doomed to fail.
- Dylan16807 3y agoI worry that "1 step before that" requires a database of information about how to actually put the pieces together for a huge fraction of characters. And that without the database it doesn't work right, and with the database it's like the current system with extra steps and extra layers of complicated abstraction. You'd be able to encode some novel characters but that's a high cost.
- jeroenhd 3y agoIf you know exactly what text you're going to render, you can create your own font, relocate all the characters, and fit everything very compactly into UTF-8. You'll need to convert external input to your encoding scheme, but it can save a lot of bytes for some corpuses. If you try to standardise against "what things look like", you'll inevitable run into trouble like the famous Turkic I (I (Iı) vs İ (İi) vs I (Ii) where "İ".toLower().toUpper() != "İ"). Sure, this means characters like é can theoretically be stored as either 0xE9 or 0xC3A9, but if you can pick what encoding you use, you can also optimise for the smallest length.
- eviks 3y ago> you'll inevitable run into trouble like the famous Turkic nope > if you need to add semantic meaning to identically looking results, that would be another non-visual composable indicator so in this case you'd have an extra indicator that would be appended to ˙ to make it İi and not Ii
- Findecanor 3y agoI'd think that such an encoding would waste encoding space on stroke combinations that are not valid, and therefore not be more compact than Unicode in practice. I think that the TRON encoding supports vector graphics in addition to all Chinese, Japanese and Korean Han/Hanja/Kanji characters as separate code points. The vector encoding is for spelling of names (family names and places) that you wouldn't find in dictionaries. https://en.wikipedia.org/wiki/TRON_(encoding) https://en.wikipedia.org/wiki/TRON_(encoding)
- lifthrasiir 3y agoI should first point out that you probably meant Han characters (so called "CJK(V) ideographs", an unfortunate misnomer), and many Asian languages do not use them at all if we are talking about the number of languages. But yeah, let's talk about strokes first. Almost all Han characters do consist of a small finite number of strokes and the stroke order itself is reasonably canonical... only for a single point of time in one area. There are lots of examples where the stroke order or even the stroke composition itself differ from time to time and among countries. And you won't be able to correctly encode an archaic character which canonical stroke order is not known---you have to guess, and you will be frequently wrong but wouldn't be able to fix that. So the next possibility is something like character description languages [1], and people tried a lot of them, so much that we even have a standardized description sublanguage in Unicode [2]. And again, there are so many possibilities that we can't have a single canonical description. Andrew West has maintained a comprehensive IDS database for all Unicode Han characters [3], and while sometimes the first level of decomposition looks clear (字 is clearly 宀 + 子) subsequent decompositions are much more ambiguous (is 子 atomic or 了 + 一?). Ultimately this approach takes too much work to be worthy, so most standards including Unicode have treated them as atomic characters with a room for glyphic variations. [1] https://en.wikipedia.org/wiki/Chinese_character_description_languages https://en.wikipedia.org/wiki/Chinese_character_description_... [2] But those ideographic description sequences (IDS) are meant to describe characters (like, ⿱丶王 means 丶 over 王) and not equivalent to characters themselves (⿱丶王 doesn't equal to 主). [3] https://babelstone.co.uk/CJK/IDS.TXT https://babelstone.co.uk/CJK/IDS.TXT
- numpad0 3y agoThe original Droid Sans Fallback is exactly that. Caveat is it still has Han Unification problem(all Asian characters with "similar" shapes are each comingled into single code points across languages, making a "unitary Asian" font impossible).