4 ms·

> It's kind of amazing that something as basic as character encodings - at least the basics like UTF-16 ↔ UTF-8 ↔ stdio-encoding! - is something that's still no

> It's kind of amazing that something as basic as character encodings - at least the basics like UTF-16 ↔ UTF-8 ↔ stdio-encoding! - is something that's still not in the C++ standard library. how would that work if you're on a microcontroller ? if $GOV_AGENCY says "product XXX is made according to international standards, including ISO/IEC 14882:2026" and international standard ISO/IEC 14882:2026 now says that a complete Unicode implementation has to be provided otherwise you're not compliant, any device with less than the 20-30-ish megabytes of memory needed for the unicode database won't be able to pass certification even if they don't handle text at any point, e.g. it's some dsp filter somewhere in a camera lense

klodolph 4y ago
Various parts of the standard are marked as OPTIONAL, so you do not need to support them to have a compliant implementation. This includes, at the moment, any kind of Unicode. You may be aware that C and C++ are still used in legacy environments... "legacy" in the sense that these environments have been in continuous use for a long time. > __STDC_ISO_10646__ > An integer literal of the form yyyymmL (for example, 199712L). If this symbol is defined, then every character in the Unicode required set, when stored in an object of type wchar_t, has the same value as the code point of that character. The Unicode required set consists of all the characters that are defined by ISO/IEC 10646, along with all amendments and technical corrigenda as of the specified year and month. You can see the wording, "if this symbol is defined". Note that even if your implementation supports Unicode, and has all sorts of character conversion tables and character property tables, you can easily arrange for those to be statically linked, and only included if actually used. This is normally how standard libraries work in programming environments for resource-constrained systems like microcontrollers. Another interesting macro is the __STDC_HOSTED__ macro--basically, if __STDC_HOSTED__ is missing, then large chunks of the library will not be available. Still completely standards-compliant.
Hello71 4y ago
you only need 30 MB if you use the ridiculously bloated icu library. musl implements POSIX locale APIs including iconv and wcwidth plus the entire rest of the libc in around half a megabyte. if you need case folding, character classes, or other special functionality, utf8proc is about 0.3 MB, and libunistring is about 1.5 MB. additionally, if your device has less than a few megabytes of storage, you'll almost certainly be using static linking, which automatically prunes unused code from properly designed libraries.
rcxdude 4y ago
You already can't fit the entire standard library on many microcontrollers people use C++ on (hell, you can't even fit the entire C standard library on most of the micros I use, and they aren't exactly the smallest ones on the market). It still works because if you use the right compiler and linker options then you will only include the parts of the standard library that you actually use.
flohofwoe 4y ago
UTF-8 vs UTF-16 vs UTF-32 encoding/decoding are just simple bit-twiddling transformations [1], you don't need a 'complete UNICODE implemenentation' for this (and ISO/IEC be damned, those conversion functions would still be useful to have in the C and C++ stdlibs without full 'UNICODE support' - which is hardly needed anyway, unless you're doing 'actual' text processing or need to write your own text renderer) [1] https://github.com/llvm-mirror/llvm/blob/master/lib/Support/ConvertUTF.cpp https://github.com/llvm-mirror/llvm/blob/master/lib/Support/...

klodolph 4y ago

Various parts of the standard are marked as OPTIONAL, so you do not need to support them to have a compliant implementation. This includes, at the moment, any kind of Unicode. You may be aware that C and C++ are still used in legacy environments... "legacy" in the sense that these environments have been in continuous use for a long time. > __STDC_ISO_10646__ > An integer literal of the form yyyymmL (for example, 199712L). If this symbol is defined, then every character in the Unicode required set, when stored in an object of type wchar_t, has the same value as the code point of that character. The Unicode required set consists of all the characters that are defined by ISO/IEC 10646, along with all amendments and technical corrigenda as of the specified year and month. You can see the wording, "if this symbol is defined". Note that even if your implementation supports Unicode, and has all sorts of character conversion tables and character property tables, you can easily arrange for those to be statically linked, and only included if actually used. This is normally how standard libraries work in programming environments for resource-constrained systems like microcontrollers. Another interesting macro is the __STDC_HOSTED__ macro--basically, if __STDC_HOSTED__ is missing, then large chunks of the library will not be available. Still completely standards-compliant.

Hello71 4y ago

you only need 30 MB if you use the ridiculously bloated icu library. musl implements POSIX locale APIs including iconv and wcwidth plus the entire rest of the libc in around half a megabyte. if you need case folding, character classes, or other special functionality, utf8proc is about 0.3 MB, and libunistring is about 1.5 MB. additionally, if your device has less than a few megabytes of storage, you'll almost certainly be using static linking, which automatically prunes unused code from properly designed libraries.

rcxdude 4y ago

You already can't fit the entire standard library on many microcontrollers people use C++ on (hell, you can't even fit the entire C standard library on most of the micros I use, and they aren't exactly the smallest ones on the market). It still works because if you use the right compiler and linker options then you will only include the parts of the standard library that you actually use.

flohofwoe 4y ago

UTF-8 vs UTF-16 vs UTF-32 encoding/decoding are just simple bit-twiddling transformations [1], you don't need a 'complete UNICODE implemenentation' for this (and ISO/IEC be damned, those conversion functions would still be useful to have in the C and C++ stdlibs without full 'UNICODE support' - which is hardly needed anyway, unless you're doing 'actual' text processing or need to write your own text renderer) [1] https://github.com/llvm-mirror/llvm/blob/master/lib/Support/ConvertUTF.cpp https://github.com/llvm-mirror/llvm/blob/master/lib/Support/...