3 ms·

This post got me wondering: what would a character set not requiring backward compatibility with ASCII look like? I tried to put the question to the StackOverfl

This post got me wondering: what would a character set not requiring backward compatibility with ASCII look like? I tried to put the question to the StackOverflow folks, but it was apparently off topic: http://stackoverflow.com/questions/7128748/if-ascii-compatibility-were-not-an-issue-what-would-an-ideal-character-encoding Thoughts on a non-ASCII encumbered character set? What can be improved over UTF-8?

thristian 15y ago
The obvious 'native' format for Unicode data is UTF-32, which forgoes ASCII-compatibility and gets you regular, consistent sizing. Unfortunately, Unicode itself is an ASCII-encumbered character set (or at least Latin-1-encumbered, and Latin-1 is ASCII-encumbered). Pretty much all the contents of Unicode are there because somebody needed them¹, so I imagine a non-ASCII-encumbered Universal Character Set would still contain all the ASCII characters... somewhere... but they'd likely be shuffled around a bit from the Unicode we know and love. ¹: I believe Unicode encodes some meaningless Kanji/Hanzi glyphs that were created by accidentally confusing two other, genuine glyphs; I'm pretty sure it only does so because it inherited them from legacy pre-Unicode encodings.
- guard-of-terra 15y ago
  Is it UTF-32LE or UTF-32BE? The good thing about UTF-8 is that you don't have to choose.
dustingetz 15y ago
what would the web look like if we didn't care about backwards compat? ...who cares?
- troutwine 15y ago
  How terribly unconstructive of you. 1. Sarcasm is not wit. 2. Dismissal by analogy is no dismissal at all. 3. The polite individual addresses well intentioned questions in good faith or not at all. Now, I've asked my question because, in a community of peers of various stripe, it's entirely possible that there will be members more versed in character encodings than myself and that there are, indeed, more optimal ways to encode characters which are not taken for need to maintain backwards compatibility with the ASCII character set. One poster mentioned UTF-32 as a non-backwards compatible example which, while space inefficient, is a good-faith answer to my question. As to your clumsy dismisal by analogy question, here are some things off the top of my head: * Substitute byte-code standards for browser javascript. * Specify parsing, rendering semantics from the start for all markup/presentation languages. * Automated browser compliance testing from the start for all standards. * Effective client-side user storage. * Include stateful communication channels from the word go. Innovation in a field of endeavor only occurs by examining the assumptions of the field and invalidating them as the context of their being changes. Those that don't care to do so are absolute hacks, stirring the sewer's mirk in the hopes that a shiny bobble will ocassional bubble up from the depths.
  - dustingetz 15y ago
    sorry, should have been more careful with my tone. i only meant that improving UTF-8 is interesting CS research but not terribly useful in real life.
    - troutwine 15y ago
      Your premise does not necessarily grant your conclusion without more elaboration. How it does research necessarily not have a benefit for 'real life'? Or, to put it another way, why is it basic research is, by default, divorced from 'real life'? Presumably if there were a more efficient character encoding with sufficient advantages there would indeed be some 'useful' aspect to it. However, and I'd like to drive this point home, you have no business deriding the well-intentioned questions of others if you have nothing to contribute. Thus far you've outright dismissed even the validity of raising the questions--'who cares'--and, after being confronted, backpedaled somewhat but still dismissed the question's utility--'not terribly useful in real life'--without being so kind as to explain why, as if it were self-evident. Perhaps to you, but this somewhat the point of my last harangue: innovation starts with questions, even those which are seemingly naive. You have been, to this point, not uncommonly, sadly, but rude indeed. If you have thoughts to share on the subject I would love to hear them. To contribute only blase dismissal tends to make individuals of less than adamantian character cease or hide their well-natured questioning of the world about them. I do not mean to suggest, of course, that every question posed should be met with twee praise: no, indeed. Rather, explorations should be met with good faith, disagreements elaborated such that rational observers might find in the conversation new ideas, or strengthening of their own. Our individual actions steer the culture in which we find ourselves; I hope we can both agree that it is a better world in which basic exploration is the norm; neither dogged travel in a well-worn rut if indeed a better road might be found.

thristian 15y ago

The obvious 'native' format for Unicode data is UTF-32, which forgoes ASCII-compatibility and gets you regular, consistent sizing. Unfortunately, Unicode itself is an ASCII-encumbered character set (or at least Latin-1-encumbered, and Latin-1 is ASCII-encumbered). Pretty much all the contents of Unicode are there because somebody needed them¹, so I imagine a non-ASCII-encumbered Universal Character Set would still contain all the ASCII characters... somewhere... but they'd likely be shuffled around a bit from the Unicode we know and love. ¹: I believe Unicode encodes some meaningless Kanji/Hanzi glyphs that were created by accidentally confusing two other, genuine glyphs; I'm pretty sure it only does so because it inherited them from legacy pre-Unicode encodings.

guard-of-terra 15y ago
Is it UTF-32LE or UTF-32BE? The good thing about UTF-8 is that you don't have to choose.

dustingetz 15y ago

what would the web look like if we didn't care about backwards compat? ...who cares?

troutwine 15y ago
How terribly unconstructive of you. 1. Sarcasm is not wit. 2. Dismissal by analogy is no dismissal at all. 3. The polite individual addresses well intentioned questions in good faith or not at all. Now, I've asked my question because, in a community of peers of various stripe, it's entirely possible that there will be members more versed in character encodings than myself and that there are, indeed, more optimal ways to encode characters which are not taken for need to maintain backwards compatibility with the ASCII character set. One poster mentioned UTF-32 as a non-backwards compatible example which, while space inefficient, is a good-faith answer to my question. As to your clumsy dismisal by analogy question, here are some things off the top of my head: * Substitute byte-code standards for browser javascript. * Specify parsing, rendering semantics from the start for all markup/presentation languages. * Automated browser compliance testing from the start for all standards. * Effective client-side user storage. * Include stateful communication channels from the word go. Innovation in a field of endeavor only occurs by examining the assumptions of the field and invalidating them as the context of their being changes. Those that don't care to do so are absolute hacks, stirring the sewer's mirk in the hopes that a shiny bobble will ocassional bubble up from the depths.
- dustingetz 15y ago
  sorry, should have been more careful with my tone. i only meant that improving UTF-8 is interesting CS research but not terribly useful in real life.
  - troutwine 15y ago
    Your premise does not necessarily grant your conclusion without more elaboration. How it does research necessarily not have a benefit for 'real life'? Or, to put it another way, why is it basic research is, by default, divorced from 'real life'? Presumably if there were a more efficient character encoding with sufficient advantages there would indeed be some 'useful' aspect to it. However, and I'd like to drive this point home, you have no business deriding the well-intentioned questions of others if you have nothing to contribute. Thus far you've outright dismissed even the validity of raising the questions--'who cares'--and, after being confronted, backpedaled somewhat but still dismissed the question's utility--'not terribly useful in real life'--without being so kind as to explain why, as if it were self-evident. Perhaps to you, but this somewhat the point of my last harangue: innovation starts with questions, even those which are seemingly naive. You have been, to this point, not uncommonly, sadly, but rude indeed. If you have thoughts to share on the subject I would love to hear them. To contribute only blase dismissal tends to make individuals of less than adamantian character cease or hide their well-natured questioning of the world about them. I do not mean to suggest, of course, that every question posed should be met with twee praise: no, indeed. Rather, explorations should be met with good faith, disagreements elaborated such that rational observers might find in the conversation new ideas, or strengthening of their own. Our individual actions steer the culture in which we find ourselves; I hope we can both agree that it is a better world in which basic exploration is the norm; neither dogged travel in a well-worn rut if indeed a better road might be found.