6 ms·

Conversion between encodings doesn’t require a database or knowledge of combining characters.

poorlyknit 4y ago
This. But it strenghtens the arguments that programming environments should just come with some sort of support for the most common encoding forms.
- nine_k 4y ago
  It does: say, Latin-1 has one-byte code points for characters like ê, but a UTF-8 sequence may represent it as an e followed by a combining circumflex.
  - lultimouomo 4y ago
    Latin-1 is a different character set; you do not need knowledge of code points to convert between different UTF encodings.
  - int_19h 4y ago
    That does not require you to know which Unicode characters are classified as combining marks. It only requires that a Unicode sequence U+0045 U+0302 is translated to Latin-1 character 0xEA (or vice versa).
kevin_thibedeau 4y ago
A library that only extracts code points will do more damage than not having one at all. If you have to decode Unicode you presumably want to parse it some of the time. Not supporting the needs for string processing with multi-point graphemes leads to broken Unicode "support" that doesn't actually work with all valid Unicode.
- Ferrotin 4y ago
  Plenty of applications just need to take strings in one format and pass them along in another format without doing feats of string processing to them.
  - int_19h 4y ago
    To elaborate: UTF-8 is the most common I/O encoding, but UTF-16 is often what you need to work with the OS APIs (Win32, macOS) or popular frameworks (Qt).
    - TillE 4y ago
      The funny thing is that C++ sort of has a flexible string class which uses system-dependent conversions, std::filesystem::path It's super convenient for working with Windows file APIs, but everything else is still a pain.
    - Ferrotin 4y ago
      Yep. There are still some annoyances like invalid UTF-16, so maybe there is more is needed than just the basic transformations — some precise handling or parameters specifying how to handle these edge cases could be necessary. Handling UTF-8 versus WTF-8, and properly round-tripping these representations, is probably a bigger issue than graphemes and canonical forms for most users.
    - josephg 4y ago
      Also unfortunately the web browser. Javascript strings are UTF-16 (Well, UCS2).
      - Yoric 4y ago
        This used to be the case when JS was created. However, I don't think that this is true anymore. It's definitely not under the hood. Nor, if I recall correctly, if you attempt to convert them from/to typed arrays. Maybe if you use `fromCharCode`?
        4y ago

This. But it strenghtens the arguments that programming environments should just come with some sort of support for the most common encoding forms.

nine_k 4y ago
It does: say, Latin-1 has one-byte code points for characters like ê, but a UTF-8 sequence may represent it as an e followed by a combining circumflex.
- lultimouomo 4y ago
  Latin-1 is a different character set; you do not need knowledge of code points to convert between different UTF encodings.
- int_19h 4y ago
  That does not require you to know which Unicode characters are classified as combining marks. It only requires that a Unicode sequence U+0045 U+0302 is translated to Latin-1 character 0xEA (or vice versa).

kevin_thibedeau 4y ago

A library that only extracts code points will do more damage than not having one at all. If you have to decode Unicode you presumably want to parse it some of the time. Not supporting the needs for string processing with multi-point graphemes leads to broken Unicode "support" that doesn't actually work with all valid Unicode.

Ferrotin 4y ago
Plenty of applications just need to take strings in one format and pass them along in another format without doing feats of string processing to them.
- int_19h 4y ago
  To elaborate: UTF-8 is the most common I/O encoding, but UTF-16 is often what you need to work with the OS APIs (Win32, macOS) or popular frameworks (Qt).
  - TillE 4y ago
    The funny thing is that C++ sort of has a flexible string class which uses system-dependent conversions, std::filesystem::path It's super convenient for working with Windows file APIs, but everything else is still a pain.
  - Ferrotin 4y ago
    Yep. There are still some annoyances like invalid UTF-16, so maybe there is more is needed than just the basic transformations — some precise handling or parameters specifying how to handle these edge cases could be necessary. Handling UTF-8 versus WTF-8, and properly round-tripping these representations, is probably a bigger issue than graphemes and canonical forms for most users.
  - josephg 4y ago
    Also unfortunately the web browser. Javascript strings are UTF-16 (Well, UCS2).
    - Yoric 4y ago
      This used to be the case when JS was created. However, I don't think that this is true anymore. It's definitely not under the hood. Nor, if I recall correctly, if you attempt to convert them from/to typed arrays. Maybe if you use `fromCharCode`?
      - 4y ago