3 ms·

There are two big lessons from this discussion, IMHO. Firstly, broad character sets are all very well, but they have limited value if you can’t rely on everyon

There are two big lessons from this discussion, IMHO. Firstly, broad character sets are all very well, but they have limited value if you can’t rely on everyone using your text being able to see them. Something like the ö (o-umlaut) is a reasonable thing to expect in modern Western fonts, but what about the emoticons I’m increasingly seeing in e-mails, or more advanced mathematical operators, or the cat in the title of this very article that many people commenting here can’t see in some contexts? We need much better standardisation and prioritisation of sets of related glyphs that are, for example, permitted by the coding standards for a software project or supported by a font file. ASCII is too small, all of Unicode is too big, and picking glyphs for inclusion in fonts one at a time is too fine-grained for this purpose. Secondly, it is crazy that with literally a million code points available in Unicode world, we don’t seem to have new control characters for “begin literal” and “end literal” to mark a range of text that should be interpreted verbatim regardless of context. Instead, we’re still using horrible hacks like quoting and escaping in environments like command lines and source code, and in text file formats like comma- or tab-separated values. These kinds of techniques are, invariably, horribly error-prone and terrible for usability; after all, in the case we’re discussing, it seems to be the apostrophe that is causing more problems than the ö! I think the computing world would be a much easier place for many, many people if there were one universal standard way of saying, “This is plain text”.

Jabbles 14y ago
Is it too much to expect that every device should be capable of rendering every unicode point (latest version at release) in at least one font? With the option to fall back to that font if it tries to render something in a font that doesn't have that character? Obviously there are issues with unicode phishing of domain names and other cases where you might want to signal that a character is "strange", but surely the memory and processing requirements for this are low enough now. It doesn't have to be a good font!
- Chris_Newton 14y ago
  Is it too much to expect that every device should be capable of rendering every unicode point (latest version at release) in at least one font? Yes, I think it probably is. Unicode is vast, and the 100,000+ characters specified in the latest standard include numerous obscure, specialised, or downright gimmicky ones. The effort required to create just one font that supports even a crude version of each and every character is probably measured in human lifetimes. Imposing that kind of burden as a barrier to entry for any new platform seems unrealistic.
  - ygra 14y ago
    For one, all those glyphs don't need to be drawn by a single person or even coexist in the same font (they cannot, anyway, given current font formats). Of the 110182 characters Unicode 6.2 defines Windows 8 (the only thing I can test at the moment) includes glyphs for 102082 of them out of the box. The missing ones fall mostly into either »rarely-used CJK ideograph« or »historical script« (like Hieroglyphs). Unicode serves many different needs and not all characters are necessary to support in a general-purpose OS. There are fonts to cover the missing pieces and professionals working in fields requiring those usually have them installed. There is also little benefit to provide a single font that encompasses all of Unicode. Designers pick fonts for aesthetic reasons and every script has different styles (although Latin, Greek and Cyrillic are fairly similar which is why they usually are all included in every font). E.g. you have the main distinctions into serif and sans-serif (for non-decorative body text). This distinction never existed for scripts like Han, Hebrew, Arabic, various Indic scripts, etc. So if you were to create only one font, what are your choices to include for every script? Pan-Unicode fonts are mostly useful as fallback fonts to ensure that you can see some rarely-used glyphs but for nearly all practical purposes they cannot be used for anything else. It's also an enormous effort beyond creating the glyphs because you'll have to include kerning tables, define positions where combining characters appear, etc. Those are often issues that make such pan-Unicode fonts unusable because yes, they may contain plenty of glyphs but cannot be used reliably to render text that goes beyond simple scripts (and diacritic placement can even be wrong with just Latin.
    - Chris_Newton 14y ago
- Edootjuh 14y ago
  You might be interested in GNU Unifont: http://unifoundry.com/unifont.html http://unifoundry.com/unifont.html
  - iso8859-1 14y ago
    Or some of these: http://en.wikipedia.org/wiki/Free_software_Unicode_typefaces http://en.wikipedia.org/wiki/Free_software_Unicode_typefaces
Evbn 14y ago
Um, how could you ever showll someone the string-termination character without escaping it?
ygra 14y ago
> we don’t seem to have new control characters for “begin literal” and “end literal” to mark a range of text that should be interpreted verbatim regardless of context That is not of Unicode's concern. Markup is reserved for higher-level things and is not done within Unicode. There are historical oddities like language tags and variation selectors but the former are deprecated and the latter exist to serve a particular need. You will never see semantics like those you describe applied to code points in Unicode. That's not what Unicode is for.
- Chris_Newton 14y ago
  That is not of Unicode's concern. Markup is reserved for higher-level things and is not done within Unicode. Well, I’m not sure what authority you’d cite for your statements about what is or is not Unicode’s concern, but right on page 3 in chapter 1 we have: “The Unicode Standard began with a simple goal: to unify the many hundreds of conflicting ways to encode characters, replacing them with a single, universal standard. [...] Whenever textual data was converted between different programs or platforms, there was a substantial risk of corruption.” That sounds familiar, given the various contexts I mentioned before. I wonder whether the cost of all the software bugs ever caused by getting quoting/escaping wrong is greater than other repeat offenders like dereferencing nulls. Then on page 8 in chapter 2 there are several explicit text processes mentioned, including user interaction and treating text as bulk data. I do understand that a basic principle of Unicode is to represent plain text rather than mark-up, but it seems to me that avoiding problems with representing widely used text strings in different ways using different quoting and escaping conventions is consistent with that principle and exactly the kind of lack of standardisation that Unicode should help with. There are already more than 60 control characters in Unicode. There are also many characters that aren’t described as control characters but which convey things like layout information or rendering hints, such as the non-breaking space, soft hyphen, and direction controls. I know that language tags are deprecated, but those are rather complicated and context-specific and, as the standard itself observes, supporting them is a significant burden on implementers, so I don’t think it is reasonable to equate that situation with what I suggested. On balance, as a practical matter, if you're working in a text-only medium, your character set is all you have. If anything else must be done using some form of mark-up built with the same character set, then a simple, unambiguous, standardised way to switch from one to the other seems entirely consistent with the general goals of the Unicode project. Links to relevant sources: http://www.unicode.org/versions/Unicode6.2.0/ch01.pdf http://www.unicode.org/versions/Unicode6.2.0/ch01.pdf http://www.unicode.org/versions/Unicode6.2.0/ch02.pdf http://www.unicode.org/versions/Unicode6.2.0/ch02.pdf

Jabbles 14y ago

Is it too much to expect that every device should be capable of rendering every unicode point (latest version at release) in at least one font? With the option to fall back to that font if it tries to render something in a font that doesn't have that character? Obviously there are issues with unicode phishing of domain names and other cases where you might want to signal that a character is "strange", but surely the memory and processing requirements for this are low enough now. It doesn't have to be a good font!

Chris_Newton 14y ago
Is it too much to expect that every device should be capable of rendering every unicode point (latest version at release) in at least one font? Yes, I think it probably is. Unicode is vast, and the 100,000+ characters specified in the latest standard include numerous obscure, specialised, or downright gimmicky ones. The effort required to create just one font that supports even a crude version of each and every character is probably measured in human lifetimes. Imposing that kind of burden as a barrier to entry for any new platform seems unrealistic.
- ygra 14y ago
  For one, all those glyphs don't need to be drawn by a single person or even coexist in the same font (they cannot, anyway, given current font formats). Of the 110182 characters Unicode 6.2 defines Windows 8 (the only thing I can test at the moment) includes glyphs for 102082 of them out of the box. The missing ones fall mostly into either »rarely-used CJK ideograph« or »historical script« (like Hieroglyphs). Unicode serves many different needs and not all characters are necessary to support in a general-purpose OS. There are fonts to cover the missing pieces and professionals working in fields requiring those usually have them installed. There is also little benefit to provide a single font that encompasses all of Unicode. Designers pick fonts for aesthetic reasons and every script has different styles (although Latin, Greek and Cyrillic are fairly similar which is why they usually are all included in every font). E.g. you have the main distinctions into serif and sans-serif (for non-decorative body text). This distinction never existed for scripts like Han, Hebrew, Arabic, various Indic scripts, etc. So if you were to create only one font, what are your choices to include for every script? Pan-Unicode fonts are mostly useful as fallback fonts to ensure that you can see some rarely-used glyphs but for nearly all practical purposes they cannot be used for anything else. It's also an enormous effort beyond creating the glyphs because you'll have to include kerning tables, define positions where combining characters appear, etc. Those are often issues that make such pan-Unicode fonts unusable because yes, they may contain plenty of glyphs but cannot be used reliably to render text that goes beyond simple scripts (and diacritic placement can even be wrong with just Latin.
  - Chris_Newton 14y ago
Edootjuh 14y ago
You might be interested in GNU Unifont: http://unifoundry.com/unifont.html http://unifoundry.com/unifont.html
- iso8859-1 14y ago
  Or some of these: http://en.wikipedia.org/wiki/Free_software_Unicode_typefaces http://en.wikipedia.org/wiki/Free_software_Unicode_typefaces

Evbn 14y ago

Um, how could you ever showll someone the string-termination character without escaping it?

ygra 14y ago

> we don’t seem to have new control characters for “begin literal” and “end literal” to mark a range of text that should be interpreted verbatim regardless of context That is not of Unicode's concern. Markup is reserved for higher-level things and is not done within Unicode. There are historical oddities like language tags and variation selectors but the former are deprecated and the latter exist to serve a particular need. You will never see semantics like those you describe applied to code points in Unicode. That's not what Unicode is for.

Chris_Newton 14y ago
That is not of Unicode's concern. Markup is reserved for higher-level things and is not done within Unicode. Well, I’m not sure what authority you’d cite for your statements about what is or is not Unicode’s concern, but right on page 3 in chapter 1 we have: “The Unicode Standard began with a simple goal: to unify the many hundreds of conflicting ways to encode characters, replacing them with a single, universal standard. [...] Whenever textual data was converted between different programs or platforms, there was a substantial risk of corruption.” That sounds familiar, given the various contexts I mentioned before. I wonder whether the cost of all the software bugs ever caused by getting quoting/escaping wrong is greater than other repeat offenders like dereferencing nulls. Then on page 8 in chapter 2 there are several explicit text processes mentioned, including user interaction and treating text as bulk data. I do understand that a basic principle of Unicode is to represent plain text rather than mark-up, but it seems to me that avoiding problems with representing widely used text strings in different ways using different quoting and escaping conventions is consistent with that principle and exactly the kind of lack of standardisation that Unicode should help with. There are already more than 60 control characters in Unicode. There are also many characters that aren’t described as control characters but which convey things like layout information or rendering hints, such as the non-breaking space, soft hyphen, and direction controls. I know that language tags are deprecated, but those are rather complicated and context-specific and, as the standard itself observes, supporting them is a significant burden on implementers, so I don’t think it is reasonable to equate that situation with what I suggested. On balance, as a practical matter, if you're working in a text-only medium, your character set is all you have. If anything else must be done using some form of mark-up built with the same character set, then a simple, unambiguous, standardised way to switch from one to the other seems entirely consistent with the general goals of the Unicode project. Links to relevant sources: http://www.unicode.org/versions/Unicode6.2.0/ch01.pdf http://www.unicode.org/versions/Unicode6.2.0/ch01.pdf http://www.unicode.org/versions/Unicode6.2.0/ch02.pdf http://www.unicode.org/versions/Unicode6.2.0/ch02.pdf