4 ms·

If you take a peek at an extended ASCII table (like the one at https://www.ascii-code.com/ https://www.ascii-code.com/), you'll notice that 0xC5 specifies a pre

If you take a peek at an extended ASCII table (like the one at https://www.ascii-code.com/ https://www.ascii-code.com/), you'll notice that 0xC5 specifies a precomposed capital A with ring above. It predates Unicode. Accepting that that's the case, and acknowledging that forward compatibility from ASCII to Unicode is a good thing (so we don't have any more encodings, we're just extending the most popular one), and understanding that you're going to have the ring-above diacritic in Unicode anyway... you kind of just end up with both representations.

arp242 3y ago
Everything can just be pre-composed; Unicode doesn't need composing characters. There's history here, with Unicode originally having just 65k characters, and hindsight is always 20/20, but I do wish there was a move towards deprecating all of this in favour of always using pre-composed. Also: what you linked isn't "ASCII" and "extended ASCII" doesn't really mean anything. ASCII is a 7-bit character set with 128 characters, and there are dozens, if not hundreds, of 8-bit character sets with 256 characters. Both CP-1252 and ISO-8859-1 saw wide use for Latin alphabet text, but others saw wide use for text in other scripts. So if you give me a document and tell me "this is extended ASCII" then I still don't know how to read it and will have to trail-and-error it. I don't think Unicode after U+007F is compatible with any specific character set? To be honest I never checked, and I don't see in what case that would be convenient. UTF-8 is only compatible with ASCII, not any specific "extended ASCII".
- zokier 3y ago
  For roundtripping e.g. https://en.wikipedia.org/wiki/VSCII https://en.wikipedia.org/wiki/VSCII you do need both composing characters and precomposed characters.
- bandrami 3y ago
  > Unicode doesn't need composing characters But it does, IIRC, for both Bengali and Telugu.
  - arp242 3y ago
    Only because they chose to do it like that. It doesn't need to.
    - kbolino 3y ago
      Considering that Unicode did not invent combining diacritics, it follows that simple compatibility with existing encodings demanded it. Now that Unicode's goals have expanded beyond simply representing what already exists, precomposed characters would be too limiting.
- adrian_b 3y ago
  In my opinion, only the reverse could be true, i.e. that Unicode does not need pre-composed characters because everything can be written with composing characters. The pre-composed characters are necessary only for backwards compatibility. It is completely unrealistic to expect that Unicode will ever provide all the pre-composed characters that have ever been used in the past or which will ever be desired in the future. There are pre-composed characters that do not exist in Unicode because they have been very seldom used. Some of them may even be unused in any language right now, but they have been used in some languages in the past, e.g. in the 19th century, but then they have been replaced by orthographic reforms. Nevertheless, when you digitize and OCR some old book, you may want to keep its text as it was written originally, so you want the missing composed characters. Another case that I have encountered where I needed composed characters not existing in Unicode was when choosing a more consistent transliteration for languages that do not use the Latin alphabet. Many such languages use quite bad transliteration systems, precisely because whoever designed them has attempted to use only whatever restricted character set was available at that time. By choosing appropriate composing characters it is possible to design improved transliterations.
  - arp242 3y ago
    > It is completely unrealistic to expect that Unicode will ever provide all the pre-composed characters that have ever been used in the past or which will ever be desired in the future. I agree it's unlikely this will ever happen, but as far as I know there aren't really any serious technical barriers, and from purely a technical point of view it could be done if there was a desire to do so. There are plenty of rarely used codepoints in Unicode already, and while adding more is certainly an inconvenience, the status quo is also inconvenient, which is why we have one of those "wow, I just discovered Unicode normalisation!" (and variants thereof) posts on the front-page here every few months. Your last paragraph can be summarize as "it makes it easier to innovate with new diacritics". This is actually an interesting point – in the past anyone could "just" write a new character and it may or may not get any uptake, just as anyone can "just" coin a new word. I've bemoaned this inability to innovate before. That is not inherent to Unicode but computerized alphabets in general, and I that composing characters alleviates at least some of that is probably the best reason I've heard for favouring compose characters. I'm actually also okay with just using composing characters and deprecating the pre-composed forms. Overall I feel that pre-composed is probably better, partly because that's what most text currently uses and partly because it's simpler, but that's the lesser issue – the more important one that it would be nice to move towards "one obviously canonical" form that everything uses.
- kps 3y ago
  > I don't think Unicode after U+007F is compatible with any specific character set? The ‘early’ Unicode alphabetic code blocks came from ISO 8859 encodings¹, e.g. the Unicode Cyrillic block follows ISO 8859-5, the Greek and Coptic block follows ISO 8859-7, etc. ¹ https://en.wikipedia.org/wiki/ISO/IEC_8859 https://en.wikipedia.org/wiki/ISO/IEC_8859

arp242 3y ago

Everything can just be pre-composed; Unicode doesn't need composing characters. There's history here, with Unicode originally having just 65k characters, and hindsight is always 20/20, but I do wish there was a move towards deprecating all of this in favour of always using pre-composed. Also: what you linked isn't "ASCII" and "extended ASCII" doesn't really mean anything. ASCII is a 7-bit character set with 128 characters, and there are dozens, if not hundreds, of 8-bit character sets with 256 characters. Both CP-1252 and ISO-8859-1 saw wide use for Latin alphabet text, but others saw wide use for text in other scripts. So if you give me a document and tell me "this is extended ASCII" then I still don't know how to read it and will have to trail-and-error it. I don't think Unicode after U+007F is compatible with any specific character set? To be honest I never checked, and I don't see in what case that would be convenient. UTF-8 is only compatible with ASCII, not any specific "extended ASCII".

zokier 3y ago
For roundtripping e.g. https://en.wikipedia.org/wiki/VSCII https://en.wikipedia.org/wiki/VSCII you do need both composing characters and precomposed characters.
bandrami 3y ago
> Unicode doesn't need composing characters But it does, IIRC, for both Bengali and Telugu.
- arp242 3y ago
  Only because they chose to do it like that. It doesn't need to.
  - kbolino 3y ago
    Considering that Unicode did not invent combining diacritics, it follows that simple compatibility with existing encodings demanded it. Now that Unicode's goals have expanded beyond simply representing what already exists, precomposed characters would be too limiting.
adrian_b 3y ago
In my opinion, only the reverse could be true, i.e. that Unicode does not need pre-composed characters because everything can be written with composing characters. The pre-composed characters are necessary only for backwards compatibility. It is completely unrealistic to expect that Unicode will ever provide all the pre-composed characters that have ever been used in the past or which will ever be desired in the future. There are pre-composed characters that do not exist in Unicode because they have been very seldom used. Some of them may even be unused in any language right now, but they have been used in some languages in the past, e.g. in the 19th century, but then they have been replaced by orthographic reforms. Nevertheless, when you digitize and OCR some old book, you may want to keep its text as it was written originally, so you want the missing composed characters. Another case that I have encountered where I needed composed characters not existing in Unicode was when choosing a more consistent transliteration for languages that do not use the Latin alphabet. Many such languages use quite bad transliteration systems, precisely because whoever designed them has attempted to use only whatever restricted character set was available at that time. By choosing appropriate composing characters it is possible to design improved transliterations.
- arp242 3y ago
  > It is completely unrealistic to expect that Unicode will ever provide all the pre-composed characters that have ever been used in the past or which will ever be desired in the future. I agree it's unlikely this will ever happen, but as far as I know there aren't really any serious technical barriers, and from purely a technical point of view it could be done if there was a desire to do so. There are plenty of rarely used codepoints in Unicode already, and while adding more is certainly an inconvenience, the status quo is also inconvenient, which is why we have one of those "wow, I just discovered Unicode normalisation!" (and variants thereof) posts on the front-page here every few months. Your last paragraph can be summarize as "it makes it easier to innovate with new diacritics". This is actually an interesting point – in the past anyone could "just" write a new character and it may or may not get any uptake, just as anyone can "just" coin a new word. I've bemoaned this inability to innovate before. That is not inherent to Unicode but computerized alphabets in general, and I that composing characters alleviates at least some of that is probably the best reason I've heard for favouring compose characters. I'm actually also okay with just using composing characters and deprecating the pre-composed forms. Overall I feel that pre-composed is probably better, partly because that's what most text currently uses and partly because it's simpler, but that's the lesser issue – the more important one that it would be nice to move towards "one obviously canonical" form that everything uses.
kps 3y ago
> I don't think Unicode after U+007F is compatible with any specific character set? The ‘early’ Unicode alphabetic code blocks came from ISO 8859 encodings¹, e.g. the Unicode Cyrillic block follows ISO 8859-5, the Greek and Coptic block follows ISO 8859-7, etc. ¹ https://en.wikipedia.org/wiki/ISO/IEC_8859 https://en.wikipedia.org/wiki/ISO/IEC_8859