7 ms·

I already ranted about unicode earlier today, my main argument is, that unicode is what happens if everybody qualified thinks: "That's a great idea, of course y

by yk 10y ago

I already ranted about unicode earlier today, my main argument is, that unicode is what happens if everybody qualified thinks: "That's a great idea, of course you have to handle X and Y and Z and I just remember that I forgot to fill out several warranty cards." This blog post is a nice example, I have absolutely no idea how these new code points are supposed to look like, since I only spend an afternoon to implement the unicode best practices from the Arch wiki, instead of subscribing to some unicode standard mailing list. (Except the one symbol which was redefined to a symbol that does not carry the semantic meaning of "standby symbol" anywhere outside of the unicode standard.) In my opinion there are two ways forward, one burn the entire thing. Or alternatively, force the unicode committee to produce an authoritative and complete font, in triplicate, and in their own blood.

detaro 10y ago
Well, if you want to know what they can look like, the blogpost has images, an embedded webfont and links to the reference font for the new symbols. And AFAIK providing reference images that are freely usable is required for all new symbol proposals.
- yk 10y ago
  It does not work on Firefox with noScript. (And as a matter of fact, prohibiting random blog posts from rendering unicode, executing complex numerical calculations on my graphics card or delivering exploits is kind of the purpose of noScript...)
  - valleyer 10y ago
    You prohibit pages from "rendering Unicode"? Why?
  - cbd1984 10y ago
    NoScript does not prevent pages from "rendering Unicode", whatever that's supposed to mean. I use it myself, and the only thing it does is selectively block Javascript.
pron 10y ago
I think that the BMP -- Basic Multilingual Plane or the first 16-bit of Unicode characters -- is pretty reasonable, and covers fairly well everything we may consider as text (all alphabets in current use plus mathematical symbols). Anything beyond that, from emojis and pictograms to ancient Greek musical notation is pretty... weird. I think it would have made much more sense to have something like image tags: a special codepoint would introduce a link to a URL containing a sequence of glyphs, followed by an index into that sequence. Those glyphs would be guaranteed not to change (in any meaningful way), and devices would be free to cache them. This way, anything that isn't real text, would standardize representation, too, instead of just a vague "meaning". Another standard could relate those glyphs to one another in some way, giving them standard semantics and means of translation (i.e. "Egyptian hieroglyphics"). This would also allow each of those (emojis or hieroglyphics) to evolve their standards independent of a single universal standard that means little.
- yk 10y ago
  I mostly agree. In my opinion, separating code points from encodings (without providing a replacement for txt files) was the original sin of unicode. That just adds a lot of complexity with very little gain. (Suppose we would have standardized from ASCII to a 64 bit per character standard encoding, then text files would grow by a factor of 8 and nothing important would have happened.^1) ^1 I am pretty sure that some use cases actually profit a lot, but neither text file formats (since most text does not contain 2^64 different characters zip would work nicely) nor networking (since most data on the internet is either video or torrents) seem to be among them. So probably they would not be huge fields.
  - skybrian 10y ago
    Text files growing by 8x would be a non-starter. It would be nice to standardize on something but what do you have against UTF8?
    - pron 10y ago
      It's not bad, but it's complicated, as it requires an O(n) algorithm to jump to a specific character. Unicode should have been capped at 16 bits, and doubling text files in size is fine. An alternate representation of simplified UTF-8 would have kept compatibility with old ASCII files.
johncolanduoni 10y ago
The Unicode tables include examples for all graphical code points: http://unicode.org/charts/ http://unicode.org/charts/. If you really wanted you can make them into a font (most of them seem to be vectorized), but since I'm guessing you see most of the added code points as useless why do you care if they show up as boxes? What harm is this stuff causing or going to cause to the standard? We have hundreds of thousands of unassigned code points. Meanwhile, a lot of the "Ys and Zs" added to Unicode have proved to be extremely useful. Unicode's math operator and letter-styling support is what made MathJax (and more generally MathML) possible. They've also helped big time when it comes to accessibility (e.g. screen readers) for mathematics on the internet. Should we have shunted that off to another standard and made the creators of screen readers completely restructure their offerings so they can deal with Unicode characters and "Mathicode" characters? Assuming anyone bothered to implement it, how would that be better than just adding a Unicode category and spending a meager amount of space?
- yk 10y ago
  Your answer illustrates my point perfectly, first the lack of a reasonable fall back mechanism. Of course I can start a hex editor, get the utf-8 encoding and then look up the code point ( and theoretically add that character to a open font). A default font would just ship with every OS out there, and suddenly there would be a working fall back. Second mathematical symbols, consider the case were I get a text file considering mostly of ASCII 7 and some mathematical symbols which may render as mathematical symbols or as Chinese characters, since there is no way to specify the encoding in a text file and so I have to guess the encoding. (That is not helped by the roughly 17 standardized encodings that mostly agree with utf-8.)
  - mungoman2 10y ago
    > Of course I can start a hex editor, get the utf-8 encoding and then look up the code point Tip: a quicker way is to copy the unrendered box and Google it.
  - leni536 10y ago
    > The Unicode tables include examples for all graphical code points: http://unicode.org/charts/ http://unicode.org/charts/. If you really wanted you can make them into a font No, last time I checked you are not legally allowed to do that.

by yk 10y ago

detaro 10y ago

Well, if you want to know what they can look like, the blogpost has images, an embedded webfont and links to the reference font for the new symbols. And AFAIK providing reference images that are freely usable is required for all new symbol proposals.

yk 10y ago
It does not work on Firefox with noScript. (And as a matter of fact, prohibiting random blog posts from rendering unicode, executing complex numerical calculations on my graphics card or delivering exploits is kind of the purpose of noScript...)
- valleyer 10y ago
  You prohibit pages from "rendering Unicode"? Why?
- cbd1984 10y ago
  NoScript does not prevent pages from "rendering Unicode", whatever that's supposed to mean. I use it myself, and the only thing it does is selectively block Javascript.

pron 10y ago

I think that the BMP -- Basic Multilingual Plane or the first 16-bit of Unicode characters -- is pretty reasonable, and covers fairly well everything we may consider as text (all alphabets in current use plus mathematical symbols). Anything beyond that, from emojis and pictograms to ancient Greek musical notation is pretty... weird. I think it would have made much more sense to have something like image tags: a special codepoint would introduce a link to a URL containing a sequence of glyphs, followed by an index into that sequence. Those glyphs would be guaranteed not to change (in any meaningful way), and devices would be free to cache them. This way, anything that isn't real text, would standardize representation, too, instead of just a vague "meaning". Another standard could relate those glyphs to one another in some way, giving them standard semantics and means of translation (i.e. "Egyptian hieroglyphics"). This would also allow each of those (emojis or hieroglyphics) to evolve their standards independent of a single universal standard that means little.

yk 10y ago
I mostly agree. In my opinion, separating code points from encodings (without providing a replacement for txt files) was the original sin of unicode. That just adds a lot of complexity with very little gain. (Suppose we would have standardized from ASCII to a 64 bit per character standard encoding, then text files would grow by a factor of 8 and nothing important would have happened.^1) ^1 I am pretty sure that some use cases actually profit a lot, but neither text file formats (since most text does not contain 2^64 different characters zip would work nicely) nor networking (since most data on the internet is either video or torrents) seem to be among them. So probably they would not be huge fields.
- skybrian 10y ago
  Text files growing by 8x would be a non-starter. It would be nice to standardize on something but what do you have against UTF8?
  - pron 10y ago
    It's not bad, but it's complicated, as it requires an O(n) algorithm to jump to a specific character. Unicode should have been capped at 16 bits, and doubling text files in size is fine. An alternate representation of simplified UTF-8 would have kept compatibility with old ASCII files.

johncolanduoni 10y ago

The Unicode tables include examples for all graphical code points: http://unicode.org/charts/ http://unicode.org/charts/. If you really wanted you can make them into a font (most of them seem to be vectorized), but since I'm guessing you see most of the added code points as useless why do you care if they show up as boxes? What harm is this stuff causing or going to cause to the standard? We have hundreds of thousands of unassigned code points. Meanwhile, a lot of the "Ys and Zs" added to Unicode have proved to be extremely useful. Unicode's math operator and letter-styling support is what made MathJax (and more generally MathML) possible. They've also helped big time when it comes to accessibility (e.g. screen readers) for mathematics on the internet. Should we have shunted that off to another standard and made the creators of screen readers completely restructure their offerings so they can deal with Unicode characters and "Mathicode" characters? Assuming anyone bothered to implement it, how would that be better than just adding a Unicode category and spending a meager amount of space?

yk 10y ago
Your answer illustrates my point perfectly, first the lack of a reasonable fall back mechanism. Of course I can start a hex editor, get the utf-8 encoding and then look up the code point ( and theoretically add that character to a open font). A default font would just ship with every OS out there, and suddenly there would be a working fall back. Second mathematical symbols, consider the case were I get a text file considering mostly of ASCII 7 and some mathematical symbols which may render as mathematical symbols or as Chinese characters, since there is no way to specify the encoding in a text file and so I have to guess the encoding. (That is not helped by the roughly 17 standardized encodings that mostly agree with utf-8.)
- mungoman2 10y ago
  > Of course I can start a hex editor, get the utf-8 encoding and then look up the code point Tip: a quicker way is to copy the unrendered box and Google it.
- leni536 10y ago
  > The Unicode tables include examples for all graphical code points: http://unicode.org/charts/ http://unicode.org/charts/. If you really wanted you can make them into a font No, last time I checked you are not legally allowed to do that.