5 ms·

Unicode support requires incorporating their database into a library. At a minimum you need to know which code points are combining chars. For a language with f

Unicode support requires incorporating their database into a library. At a minimum you need to know which code points are combining chars. For a language with five to ten year update cycles should everyone be stuck with outdated data if the Unicode standard is revised in the interim?

Ferrotin 4y ago
Conversion between encodings doesn’t require a database or knowledge of combining characters.
- poorlyknit 4y ago
  This. But it strenghtens the arguments that programming environments should just come with some sort of support for the most common encoding forms.
  - nine_k 4y ago
    It does: say, Latin-1 has one-byte code points for characters like ê, but a UTF-8 sequence may represent it as an e followed by a combining circumflex.
    - lultimouomo 4y ago
      Latin-1 is a different character set; you do not need knowledge of code points to convert between different UTF encodings.
    - int_19h 4y ago
      That does not require you to know which Unicode characters are classified as combining marks. It only requires that a Unicode sequence U+0045 U+0302 is translated to Latin-1 character 0xEA (or vice versa).
- kevin_thibedeau 4y ago
  A library that only extracts code points will do more damage than not having one at all. If you have to decode Unicode you presumably want to parse it some of the time. Not supporting the needs for string processing with multi-point graphemes leads to broken Unicode "support" that doesn't actually work with all valid Unicode.
  - Ferrotin 4y ago
    Plenty of applications just need to take strings in one format and pass them along in another format without doing feats of string processing to them.
    - int_19h 4y ago
      To elaborate: UTF-8 is the most common I/O encoding, but UTF-16 is often what you need to work with the OS APIs (Win32, macOS) or popular frameworks (Qt).
      - TillE 4y ago
        The funny thing is that C++ sort of has a flexible string class which uses system-dependent conversions, std::filesystem::path It's super convenient for working with Windows file APIs, but everything else is still a pain.
      - Ferrotin 4y ago
        Yep. There are still some annoyances like invalid UTF-16, so maybe there is more is needed than just the basic transformations — some precise handling or parameters specifying how to handle these edge cases could be necessary. Handling UTF-8 versus WTF-8, and properly round-tripping these representations, is probably a bigger issue than graphemes and canonical forms for most users.
      - josephg 4y ago
        Also unfortunately the web browser. Javascript strings are UTF-16 (Well, UCS2).
arka2147483647 4y ago
All operating systems have the unicode database saved somewhere. There should just be a standard way of accessing it. Just like filesystem. Edit; that is; it does not have to be linked in the standard lib. Can be a data file somewhere, or a a shared lib.
- duskwuff 4y ago
  There's precedent for this, too: time zones! Time zone data can change over time, and as such it's typically stored in system files and loaded at runtime, rather than being embedded in executables. Locales have some similar behavior as well.
- thwarted 4y ago
  Like the timezone database, many things of which ship(ped) with their own copy.
  - apaprocki 4y ago
    It's funny how much time I (and probably a number of others) spend undoing this decision and making things all refer to one copy of the data. There's usually also one per programming language/runtime, so the list only grows as new languages/runtime keep coming into the picture. Everyone's so good at patching vulnerabilities, but did you plan for your app breaking in production for all users in particular timezones because a random service in a chain of a 6, say, re-implemented in Go last quarter, or maybe... changed their Node app to port from using Moment to the standard Intl object 10 months ago, thus depending on the ICU C++ library, and a separate copy of the data that is no longer correct? Try to assess that risk and your head will hurt. These are the sharp edges in a polyglot environment that usually aren't considered when deciding to bring in new languages. Unicode CLDR has the same issue, but the pain usually isn't acute because it takes much, much longer for new glyphs or other data to appear in common use or start appearing in data feeds or from OS APIs/input. As long as you're at least on some update schedule, it's usually fine, and there's usually fewer things to update.
    - duskwuff 4y ago

Ferrotin 4y ago

Conversion between encodings doesn’t require a database or knowledge of combining characters.

poorlyknit 4y ago
This. But it strenghtens the arguments that programming environments should just come with some sort of support for the most common encoding forms.
- nine_k 4y ago
  It does: say, Latin-1 has one-byte code points for characters like ê, but a UTF-8 sequence may represent it as an e followed by a combining circumflex.
  - lultimouomo 4y ago
    Latin-1 is a different character set; you do not need knowledge of code points to convert between different UTF encodings.
  - int_19h 4y ago
    That does not require you to know which Unicode characters are classified as combining marks. It only requires that a Unicode sequence U+0045 U+0302 is translated to Latin-1 character 0xEA (or vice versa).
kevin_thibedeau 4y ago
A library that only extracts code points will do more damage than not having one at all. If you have to decode Unicode you presumably want to parse it some of the time. Not supporting the needs for string processing with multi-point graphemes leads to broken Unicode "support" that doesn't actually work with all valid Unicode.
- Ferrotin 4y ago
  Plenty of applications just need to take strings in one format and pass them along in another format without doing feats of string processing to them.
  - int_19h 4y ago
    To elaborate: UTF-8 is the most common I/O encoding, but UTF-16 is often what you need to work with the OS APIs (Win32, macOS) or popular frameworks (Qt).
    - TillE 4y ago
      The funny thing is that C++ sort of has a flexible string class which uses system-dependent conversions, std::filesystem::path It's super convenient for working with Windows file APIs, but everything else is still a pain.
    - Ferrotin 4y ago
      Yep. There are still some annoyances like invalid UTF-16, so maybe there is more is needed than just the basic transformations — some precise handling or parameters specifying how to handle these edge cases could be necessary. Handling UTF-8 versus WTF-8, and properly round-tripping these representations, is probably a bigger issue than graphemes and canonical forms for most users.
    - josephg 4y ago
      Also unfortunately the web browser. Javascript strings are UTF-16 (Well, UCS2).

arka2147483647 4y ago

All operating systems have the unicode database saved somewhere. There should just be a standard way of accessing it. Just like filesystem. Edit; that is; it does not have to be linked in the standard lib. Can be a data file somewhere, or a a shared lib.

duskwuff 4y ago
There's precedent for this, too: time zones! Time zone data can change over time, and as such it's typically stored in system files and loaded at runtime, rather than being embedded in executables. Locales have some similar behavior as well.
thwarted 4y ago
Like the timezone database, many things of which ship(ped) with their own copy.
- apaprocki 4y ago
  It's funny how much time I (and probably a number of others) spend undoing this decision and making things all refer to one copy of the data. There's usually also one per programming language/runtime, so the list only grows as new languages/runtime keep coming into the picture. Everyone's so good at patching vulnerabilities, but did you plan for your app breaking in production for all users in particular timezones because a random service in a chain of a 6, say, re-implemented in Go last quarter, or maybe... changed their Node app to port from using Moment to the standard Intl object 10 months ago, thus depending on the ICU C++ library, and a separate copy of the data that is no longer correct? Try to assess that risk and your head will hurt. These are the sharp edges in a polyglot environment that usually aren't considered when deciding to bring in new languages. Unicode CLDR has the same issue, but the pain usually isn't acute because it takes much, much longer for new glyphs or other data to appear in common use or start appearing in data feeds or from OS APIs/input. As long as you're at least on some update schedule, it's usually fine, and there's usually fewer things to update.
  - duskwuff 4y ago