3 ms·

> Q: What is Unicode? > A: Unicode is the universal character encoding, maintained by the Unicode Consortium. This encoding standard provides the basis for pro

> Q: What is Unicode? > A: Unicode is the universal character encoding, maintained by the Unicode Consortium. This encoding standard provides the basis for processing, storage and interchange of text data in any language in all modern software and information technology protocols. https://home.unicode.org/basic-info/faq/ https://home.unicode.org/basic-info/faq/

layer8 4y ago
Unicode uses the term “character encoding form” or “character encoding scheme” for what is normally referred to or abbreviated as “character encoding” or “charset” (see e.g. RFC 8187), and uses “character encoding” or “coded character set” for the abstract assignment of natural numbers to the abstract characters in a character repertoire, which is more usually referred to as just “[coded] character set” (cf. also UCS = Unicode Character Set). This different use of terminology can cause confusion. The GP is correct that Unicode as a whole is not what is colloquially meant by “encoding”.
brewmarche 4y ago
In your quote encoding refers to assigning numbers (code points in Unicode parlance) to characters (I am simplifying here, I know the definition of character in Unicode is not that easy). It’s like a catalogue of scripts. We have to extend it when we encounter new scripts that are not catalogued yet (or when we create new emojis) Converting a byte sequence to a Unicode code point sequence and vice-versa is called transformation format (or more generally an encoding form, but then might not be deterministic) by Unicode (see <https://www.unicode.org/faq/utf_bom.html#gen2 https://www.unicode.org/faq/utf_bom.html#gen2>). Unicode specifies UTF-8, -16 and -32. We do not have to change these formats unless the catalogue hit the limits of 32 bits (not a big problem for UTF-8 but for the other two formats). These formats are already able to encode code points that are not assigned yet. And the confusion now is that a lot of people call what Unicode calls transformation format (i.e. the byte to code point mapping) encoding as well. The term charset is also used sometimes. PS: Note that a goal of Unicode is to be able to accommodate legacy encoding/charsets by having a broad enough catalogue. This is so that these legacy encoding which may come with their own catalogue can be mapped to the Unicode catalogue. So we have control codes (even though not part of any “proper” human script), precomposed letters (there is a code point for à although it could be represented by a + combining `), things like the Greek terminal form of sigma separately encoded, although that could be done in font-rendering (like generally done for Arabic), and a lot more to aid with mapping and roundtrips.
- tzot 4y ago
  Just a note about the Greek terminal form of sigma: when dealing with Greek numerals, ςʹ (the final form of sigma) is 6 while σʹ (the non-final form of sigma) is 200; they need to be differentiated whatever the font engine decides to render.
  - brewmarche 4y ago
    Thanks, good to know. I’ve also realised that determining which form to use in mathematical formulas is maybe not that straightforward. Edit: by which I mean that I’ve only seen the non terminal form so far in maths but it’s hard to write an algorithm that distinguishes between a word ending in sigma and some juxtaposition of Greek variables.
jll29 4y ago
Regardless of official terminology, there are two levels: 1. Map a character to a unique number in a character set (in Unicode: called codepoint) 2. Map a number that represents a character in a character set to a bit pattern for storage (transiently or persistently, internally or externally). Unicode code points can be bit-encoded in various ways: UTF8, UCS2 and UCS4/UTF32. The original code points permit the same character to be represented in various ways, which makes equality checks non-trivial: for instance a character like "ä" can be represented as a single character or alternatively as a composition of "a" + umlaut accent (2 characters). So far, this is all about plain text, so we are not talking about font families or character properties (bold, italics, underlined) or orientation (super-script, sup-script). Ken Lunde's opus magnum is the standard book on representing text in various languages other than English, with a focus on Asian languages: https://www.oreilly.com/library/view/cjkv-information-processing/9780596156114/ https://www.oreilly.com/library/view/cjkv-information-proces...

layer8 4y ago

Unicode uses the term “character encoding form” or “character encoding scheme” for what is normally referred to or abbreviated as “character encoding” or “charset” (see e.g. RFC 8187), and uses “character encoding” or “coded character set” for the abstract assignment of natural numbers to the abstract characters in a character repertoire, which is more usually referred to as just “[coded] character set” (cf. also UCS = Unicode Character Set). This different use of terminology can cause confusion. The GP is correct that Unicode as a whole is not what is colloquially meant by “encoding”.

brewmarche 4y ago

In your quote encoding refers to assigning numbers (code points in Unicode parlance) to characters (I am simplifying here, I know the definition of character in Unicode is not that easy). It’s like a catalogue of scripts. We have to extend it when we encounter new scripts that are not catalogued yet (or when we create new emojis) Converting a byte sequence to a Unicode code point sequence and vice-versa is called transformation format (or more generally an encoding form, but then might not be deterministic) by Unicode (see <https://www.unicode.org/faq/utf_bom.html#gen2 https://www.unicode.org/faq/utf_bom.html#gen2>). Unicode specifies UTF-8, -16 and -32. We do not have to change these formats unless the catalogue hit the limits of 32 bits (not a big problem for UTF-8 but for the other two formats). These formats are already able to encode code points that are not assigned yet. And the confusion now is that a lot of people call what Unicode calls transformation format (i.e. the byte to code point mapping) encoding as well. The term charset is also used sometimes. PS: Note that a goal of Unicode is to be able to accommodate legacy encoding/charsets by having a broad enough catalogue. This is so that these legacy encoding which may come with their own catalogue can be mapped to the Unicode catalogue. So we have control codes (even though not part of any “proper” human script), precomposed letters (there is a code point for à although it could be represented by a + combining `), things like the Greek terminal form of sigma separately encoded, although that could be done in font-rendering (like generally done for Arabic), and a lot more to aid with mapping and roundtrips.

tzot 4y ago
Just a note about the Greek terminal form of sigma: when dealing with Greek numerals, ςʹ (the final form of sigma) is 6 while σʹ (the non-final form of sigma) is 200; they need to be differentiated whatever the font engine decides to render.
- brewmarche 4y ago
  Thanks, good to know. I’ve also realised that determining which form to use in mathematical formulas is maybe not that straightforward. Edit: by which I mean that I’ve only seen the non terminal form so far in maths but it’s hard to write an algorithm that distinguishes between a word ending in sigma and some juxtaposition of Greek variables.

jll29 4y ago

Regardless of official terminology, there are two levels: 1. Map a character to a unique number in a character set (in Unicode: called codepoint) 2. Map a number that represents a character in a character set to a bit pattern for storage (transiently or persistently, internally or externally). Unicode code points can be bit-encoded in various ways: UTF8, UCS2 and UCS4/UTF32. The original code points permit the same character to be represented in various ways, which makes equality checks non-trivial: for instance a character like "ä" can be represented as a single character or alternatively as a composition of "a" + umlaut accent (2 characters). So far, this is all about plain text, so we are not talking about font families or character properties (bold, italics, underlined) or orientation (super-script, sup-script). Ken Lunde's opus magnum is the standard book on representing text in various languages other than English, with a focus on Asian languages: https://www.oreilly.com/library/view/cjkv-information-processing/9780596156114/ https://www.oreilly.com/library/view/cjkv-information-proces...